Spark- 使用第三方依賴解析IP地址

使用 github上已有的開源項目
1)git clone https://github.com/wzhe06/ipdatabase.git

2)編譯下載的項目: mvn clean package- DskipTests

3)安裝jar包到自己的 maven倉庫

　　mvn install: install-file -Dfile=${編譯的jar包路徑}/target/ipdatabase-1.0-SNAPSHOT jar  -DgroupId=com.ggstar  -DartifactId=ipdatabase   -Dversion=1.0  -Dpackaging=jar

4)添加依賴到pom

<dependency>
　　<groupId>com.ggstar<groupId>
　　<artifactId>ipdatabase</artifactId>
　　<version>1.0</version>
</dependency>

 

<dependency>
　　<groupId>org.apache.poi</groupId>
　　<cartifactId>poi-ooxml</artifactId>
　　<version>3.14</version>
</dependency>
<dependency>
　　<groupId>org.apache.poi</groupId> 
　　<cartifactId>poi</artifactId>
 　<version>3.14</version> 
</dependency>

5)將源碼main/resource下的ipDatabase.csv和ipRegion.xlxs拷貝到當前項目的resource目錄下

6)ip解析工具類

/**
  * IP解析工具類
  */
object IpUtils {

  def getCity(ip:String): Unit ={
    IpHepler.findRegionByIp(ip)
  }
}

7）打包到yarn運行

在pom文件排除spark打包，因為環境上有。

<!--scala 依賴-->
    <dependency>
      <groupId>org.scala-lang</groupId>
      <artifactId>scala-library</artifactId>
      <version>${scala.version}</version>
      <scope>provided</scope>
    </dependency>
    <!--SparkSQL-->
    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-sql_2.11</artifactId>
      <version>${spark.version}</version>
      <scope>provided</scope>
    </dependency>

    <dependency>
      <groupId>org.apache.spark</groupId>
      <artifactId>spark-hive_2.11</artifactId>
      <version>${spark.version}</version>
      <scope>provided</scope>
    </dependency>

 

打包時注意，pom.xml中需要添加如下plugin

<plugin>
        <artifactId>maven-assembly-plugin</artifactId>
        <configuration>
          <archive>
            <mainfest>
              <mainClass></mainClass>
            </mainfest>
          </archive>
          <descriptRefs>
            <descriptRef>
              jar-with-dependencies
            </descriptRef>
          </descriptRefs>
        </configuration>
      </plugin>
      <plugin>

 

提交運行

/bin/spark-submit
class com.rz.log.SparkstatcleanJobYARN
--name SparkstatcleanJobYARN
--master yarn
--executor-memory 1G
--num-executors 1\
--files /home/hadoop/Lib/ipDatabase. CSV, /home/hadoop/lib/ipRegion XlSx \
/home/hadoop/lib/sql-1.0-jar-with-dependencies.jar \
hdfs://hadoop001:8020/imooc/input/* hdfs://hadoop001: 8020/imooc/clean