Spark2 DataSet 创建新行之flatMap


val dfList = List(("Hadoop", "Java,SQL,Hive,HBase,MySQL"), ("Spark", "Scala,SQL,DataSet,MLlib,GraphX"))
dfList: List[(String, String)] = List((Hadoop,Java,SQL,Hive,HBase,MySQL), (Spark,Scala,SQL,DataSet,MLlib,GraphX))

case class Book(title: String, words: String)

val df=dfList.map{p=>Book(p._1,p._2)}.toDS()
df: org.apache.spark.sql.Dataset[Book] = [title: string, words: string]

df.show
+------+--------------------+
| title|               words|
+------+--------------------+
|Hadoop|Java,SQL,Hive,HBa...|
| Spark|Scala,SQL,DataSet...|
+------+--------------------+

df.flatMap(_.words.split(",")).show
+-------+
|  value|
+-------+
|   Java|
|    SQL|
|   Hive|
|  HBase|
|  MySQL|
|  Scala|
|    SQL|
|DataSet|
|  MLlib|
| GraphX|
+-------+

 


免责声明!

本站转载的文章为个人学习借鉴使用,本站对版权不负任何法律责任。如果侵犯了您的隐私权益,请联系本站邮箱yoyou2525@163.com删除。



 
粤ICP备18138465号  © 2018-2025 CODEPRJ.COM