scala集合转化为DS/DF
1 case class TestPerson(name: String, age: Long, salary: Double) 2 val tom = TestPerson("Tom Hanks",37,35.5) 3 val sam = TestPerson("Sam Smith",40,40.5) 4 val PersonList = mutable.MutableList[TestPerson]() 5 PersonList += tom 6 PersonList += sam 7 8 val personDS = PersonList.toDS() 9 println(personDS.getClass) 10 personDS.show() 11 12 val personDF = PersonList.toDF() 13 println(personDF.getClass) 14 personDF.show() 15 personDF.select("name", "age").show()
输出结果：
class org.apache.spark.sql.Dataset +---------+---+------+ | name|age|salary| +---------+---+------+ |Tom Hanks| 37| 35.5| |Sam Smith| 40| 40.5| +---------+---+------+ class org.apache.spark.sql.DataFrame +---------+---+------+ | name|age|salary| +---------+---+------+ |Tom Hanks| 37| 35.5| |Sam Smith| 40| 40.5| +---------+---+------+ +---------+---+ | name|age| +---------+---+ |Tom Hanks| 37| |Sam Smith| 40| +---------+---+
参考资料：
https://stackoverflow.com/questions/39397652/convert-scala-list-to-dataframe-or-dataset