Tuesday, May 17, 2016

Spark window function, failure: ``union'' expected but `(' found

this a very weird error when I try to run a simple window ranking, all looks good from the syntax perspective.
team=[("Lakers","WEST",29 ),("Golden State","WEST",89 ),
      ("MIA HEAT","EAST",79 ),("SAS","WEST",9 ),
      ("RAPTORS","EAST",29 ) ]

sql.createDataFrame(
    sc.parallelize(
        team).map(lambda x: Row(Team=x[0],Division=x[1], Score=x[2])))\
    .registerAsTable("team")

print sql.sql("SELECT team, division, score, rank() OVER (PARTITION BY division ORDER BY score desc)"              "  as rank FROM team").take(10)

And I got this errors complaining the syntax
4j.protocol.Py4JJavaError: An error occurred while calling o36.sql.
: java.lang.RuntimeException: [1.43] failure: ``union'' expected but `(' found

SELECT team, division, score, rank() OVER (PARTITION BY division ORDER BY score desc)  as rank FROM team
                                          ^
 at scala.sys.package$.error(package.scala:27)
 at org.apache.spark.sql.catalyst.AbstractSparkSQLParser.parse(AbstractSparkSQLParser.scala:36)
 at org.apache.spark.sql.catalyst.DefaultParserDialect.parse(ParserDialect.scala:67)
 at org.apache.spark.sql.SQLContext$$anonfun$2.apply(SQLContext.scala:211)
 at org.apache.spark.sql.SQLContext$$anonfun$2.apply(SQLContext.scala:211)

To Fix this, please make sure you are using HiveContext instead of SqlContext

6 comments:

Sarah said...

A control ping-pong table is nine legs in length by 5 legs wide.watch Roland Garros live

Sarah said...

It rests 30 creeps wide off the ground and is composed from Masonite and secured with an extraordinary region that prevents the ball from diminishing down because of rubbing.Roland Garros Direct

Sarah said...

The covering is generally green or turquoise in shading, however other proper shades are accessible, for example, dark. plunging the work area in 50 percent is a white net that seems like a golf net. french open live

Unknown said...

Thanks. This post saved me hours of over thinking

Rachel Bilson said...

It’s always my passion to write creative articles. Have worked on a lot of innovative and interesting topics. Read it and it’s informative. Recommend you to provide your feedback & suggestions to work on more titles. Contact me or reach me if you would like to know more about my profile.
My work:Pureflix on Roku

marvinericksen said...

Hey, This is me Marvin Ericksen writing from Florida. So how’s life going-on, Feeling Enthusiastic or feeling dull. I felt bored with my life. So here I am and going to write my blogs. Actually I am really interested in reading and writing. But my all-time hobbies are movies and TV Shows and Series.
My blog: cbs.com/roku

 
Locations of visitors to this page