Window Functions in Spark SQL using HiveQL
SparkでHiveContextを用いてwindow functionを使用するサンプル。
build.sbt
...
libraryDependencies ++= Seq(
"org.apache.spark" %% "spark-core" % "1.5.2" % "provided",
"org.apache.spark" %% "spark-sql" % "1.5.2" % "provided",
"org.apache.spark" %% "spark-hive" % "1.5.2" % "provided"
)
...
scalaサンプル
import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.sql.SQLContext
import org.apache.spark.sql.hive.HiveContext
...
val conf = new SparkConf()
val sc = new SparkContext(conf)
val sqlContext = new HiveContext(sc)
import sqlContext.implicits._
val src = sqlContext.read.json(test_data)
src.registerTempTable("test")
val r = sqlContext.sql("select id, time, row_number() over(partition by id order by time) num from test")
...
Author And Source
この問題について(Window Functions in Spark SQL using HiveQL), 我々は、より多くの情報をここで見つけました https://qiita.com/ike_s/items/c0bfbf85029118a1cbdf著者帰属：元の著者の情報は、元のURLに含まれています。著作権は原作者に属する。
Content is automatically searched and collected through network algorithms . If there is a violation . Please contact us . We will adjust (correct author information ,or delete content ) as soon as possible .