一、參數說明
啟動Spark任務時，在沒有配置spark.yarn.archive或者spark.yarn.jars時， 會看到不停地上傳jar，非常耗時；使用spark.yarn.archive可以大大地減少任務的啟動時間，整個處理過程如下。
二、spark.yarn.archive使用
1.在本地創建zip文件
silent@bd01:~/env/spark$ cd jars/ silent@bd01:~/env/spark$ zip spark2.0.0.zip ./*
注：zip包為全量包
2.上傳至HDFS並更改權
silent@bd01:~/env/spark$ /usr/ndp/current/hdfs_client/bin/hdfs dfs -mkdir /tmp/spark-archive silent@bd01:~/env/spark$ /usr/ndp/current/hdfs_client/bin/hdfs dfs -put ./spark2.0.0.zip /tmp/spark-archive silent@bd01:~/env/spark$ /usr/ndp/current/hdfs_client/bin/hdfs dfs -chmod 775 /tmp/spark-archive/spark2.0.0.zip.zip
3.配置spark-defaut.conf
spark.yarn.archive hdfs:///tmp/spark-archive/spark2.0.0.zip
三、spark.yarn.jars使用說明
1. 上傳依賴jar包
silent@bd01:~/env/spark$ /usr/ndp/current/hdfs_client/bin/hdfs dfs -mkdir hdfs://bd01/user/asiainfo/jars/
silent@bd01:~/env/spark$ /usr/ndp/current/hdfs_client/bin/hdfs dfs -put ./spark2.0.0.zip hdfs://bd01/user/asiainfo/jars/
silent@bd01:~/env/spark$ /usr/ndp/current/hdfs_client/bin/hdfs dfs -chmod 775 hdfs://bd01/user/asiainfo/jars/spark2.0.0.zip.zip
2.配置spark-defaut.conf
spark.yarn.jars=local:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/spark/jars/*,local:/opt/cloudera/parcels/CDH-6.1.0-1.cdh6.1.0.p0.770702/lib/spark/hive/*,hdfs://bd01/user/asiainfo/jars/*.jar
注：本地配置local,hdfs標記為hdfs目錄即可