spark中常用轉換操作keys 、values和mapValues

1.keys

功能：

　　返回所有鍵值對的key

示例

val list = List("hadoop","spark","hive","spark")
val rdd = sc.parallelize(list)
val pairRdd = rdd.map(x => (x,1))
pairRdd.keys.collect.foreach(println)

結果

hadoop
spark
hive
spark
list: List[String] = List(hadoop, spark, hive, spark)
rdd: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[142] at parallelize at command-3434610298353610:2
pairRdd: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[143] at map at command-3434610298353610:3

2.values

功能：

　　返回所有鍵值對的value

示例

val list = List("hadoop","spark","hive","spark")
val rdd = sc.parallelize(list)
val pairRdd = rdd.map(x => (x,1))
pairRdd.values.collect.foreach(println)

結果

1
1
1
1
list: List[String] = List(hadoop, spark, hive, spark)
rdd: org.apache.spark.rdd.RDD[String] = ParallelCollectionRDD[145] at parallelize at command-3434610298353610:2
pairRdd: org.apache.spark.rdd.RDD[(String, Int)] = MapPartitionsRDD[146] at map at command-3434610298353610:3

3.mapValues(func)

功能：

　　對鍵值對每個value都應用一個函數，但是，key不會發生變化。

示例　

val list = List("hadoop","spark","hive","spark")
val rdd = sc.parallelize(list)
val pairRdd = rdd.map(x => (x,1))
pairRdd.mapValues(_+1).collect.foreach(println)//對每個value進行+1

結果

(hadoop,2)
(spark,2)
(hive,2)
(spark,2)