spark combineByKey用法

 

本例子是根据某个字段作为key，然后将记录合并为list集合。

 

        type myType = (String,Double)
        val goodsTradeRankRdd = goodsTradeRankDF.rdd.map(item => {
            val tenantId = item.getAs[String]("tenantId")
            val spuId = item.getAs[String]("spuId")
            val totalQty = item.getAs[Double]("totalQty")
            (tenantId,score(spuId,totalQty))
        }).combineByKey(
           List(_), (c:List[score],v) => v::c, (c1:List[score],c2:List[score]) => c1:::c2
        ).map(x => goodsTradeRank(x._1,x._2))