hive 學習系列四（用戶自定義函數）

如果入參是簡單的數據類型，直接繼承UDF，實現一個或者多個evaluate 方法。

具體流程如下：

1，實現大寫字符轉換成小寫字符的UDF

package com.example.hive.udf;

import org.apache.hadoop.hive.ql.exec.UDF;
import org.apache.hadoop.io.Text;

public class Lower extends UDF {
    public Text evaluate(final Text s) {
        if (s == null) {
            return null;
        }
        return new Text(s.toString().toLowerCase());
    }
}

2，打包成jar 包。

建立maven 項目，使用maven 打包。
這里打包成的jar 包是，hiveudf-1.0.0.jar

3，上傳到hdfs 路徑上。

[root@master /opt]# hadoop fs -mkdir -p /user/hive/udf
18/06/07 09:41:09 WARN util.NativeCodeLoader: Unable 
to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@master /opt]# hadoop fs -put hiveudf-1.0.0.jar  /user/hive/udf
18/06/07 09:41:24 WARN util.NativeCodeLoader: Unable to 
load native-hadoop library for your platform... using builtin-java classes where applicable
[root@master /opt]# hadoop fs -ls /user/hive/udf 
18/06/07 09:41:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library
 for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r--   3 root supergroup       8020 2018-06-07 09:41 /user/hive/udf/hiveudf-1.0.0.jar
[root@master /opt]#

4, 在Hive 命令行里面創建函數。

add jar hdfs:////udf/hiveudf-1.0.0.jar;
create temporary function lower as 'com.example.hive.udf.Lower';

hive> delete jar  hiveudf-1.0.0.jar;
hive> list jars
    > ;
hive> add jar hdfs:///user/hive/udf/hiveudf-1.0.0.jar
    > ;
Added [/tmp/416cfcca-9ea0-4eaf-9e54-8154b440f3a9_resources/hiveudf-1.0.0.jar] to class path
Added resources: [hdfs:///user/hive/udf/hiveudf-1.0.0.jar]
hive> list jars;
/tmp/416cfcca-9ea0-4eaf-9e54-8154b440f3a9_resources/hiveudf-1.0.0.jar
hive> create temporary function lower as 'com.example.hive.udf.Lower';
OK
Time taken: 0.594 seconds
hive> 

5，然后就可以用這個注冊的函數了。

hive> select lower('AbcDEfg')
    > ;
OK
abcdefg
Time taken: 1.718 seconds, Fetched: 1 row(s)
hive> 

至於入參是復雜數據類型，比如Array 等， 可以繼承GenericUDF

1，同樣的，先寫一個類，繼承GenericUDF，

此自定義函數實現的是，把一個點，根據經緯度，轉換成一個字符串。

package com.zbra.udf;


import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDF;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.DoubleObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;

/**
 * 針對復雜數據
 */
public class GeoUdf extends GenericUDF {

    private DoubleObjectInspector doubleObjectInspector01;
    private DoubleObjectInspector doubleObjectInspector02;

    public ObjectInspector initialize(ObjectInspector[] objectInspectors) throws UDFArgumentException {
        if (objectInspectors.length != 2) {
            throw new UDFArgumentLengthException("arrayContainsExample only takes 2 arguments: String,  String");
        }
        // 1. 檢查是否接收到正確的參數類型
        ObjectInspector a = objectInspectors[0];
        ObjectInspector b = objectInspectors[1];
        if (!(a instanceof DoubleObjectInspector) || !(b instanceof DoubleObjectInspector)) {
            throw new UDFArgumentException("first argument must be a double, second argument must be a double");
        }

        this.doubleObjectInspector01 = (DoubleObjectInspector) a;
        this.doubleObjectInspector02 = (DoubleObjectInspector) b;

        return PrimitiveObjectInspectorFactory.javaStringObjectInspector;
    }

    public Object evaluate(DeferredObject[] deferredObjects) throws HiveException {

        Double lat = this.doubleObjectInspector01.get(deferredObjects[0].get());
        Double lng = this.doubleObjectInspector02.get(deferredObjects[1].get());

        if (lat == null || lng == null) {
            return new String("");
        }

        return new GeoHash(lat, lng).getGeoHashBase32();
    }

    public String getDisplayString(String[] strings) {
        if (strings.length == 2) {
            return "geo_hash(" + strings[0] + ", " + strings[1] + ")";
        } else {
            return "傳入的參數不對...";
        }
    }
}

2，打包成jar 包

本文中打包成hiveudf-1.0.0.jar

3，同樣的上傳到hdfs 路徑中

[root@master /opt]# hadoop fs -mkdir -p /user/hive/udf
18/06/07 09:41:09 WARN util.NativeCodeLoader: Unable 
to load native-hadoop library for your platform... using builtin-java classes where applicable
[root@master /opt]# hadoop fs -put hiveudf-1.0.0.jar  /user/hive/udf
18/06/07 09:41:24 WARN util.NativeCodeLoader: Unable to 
load native-hadoop library for your platform... using builtin-java classes where applicable
[root@master /opt]# hadoop fs -ls /user/hive/udf 
18/06/07 09:41:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library
 for your platform... using builtin-java classes where applicable
Found 1 items
-rw-r--r--   3 root supergroup       8020 2018-06-07 09:41 /user/hive/udf/hiveudf-1.0.0.jar
[root@master /opt]#

4, 創建自定義函數。

hive> list jars;
/tmp/3794df3a-687a-45dd-93d3-d6a712c43e85_resources/hiveudf-1.0.0.jar
hive> delete jar /tmp/3794df3a-687a-45dd-93d3-d6a712c43e85_resources/hiveudf-1.0.0.jar
    > ;
Deleted [/tmp/3794df3a-687a-45dd-93d3-d6a712c43e85_resources/hiveudf-1.0.0.jar] from class path
hive> add jar hdfs:///user/hive/udf/hiveudf-1.0.0.jar;
Added [/tmp/3794df3a-687a-45dd-93d3-d6a712c43e85_resources/hiveudf-1.0.0.jar] to class path
Added resources: [hdfs:///user/hive/udf/hiveudf-1.0.0.jar]
hive> create temporary function geohash as 'com.zbra.udf.GeoUdf';
OK
Time taken: 0.145 seconds

5, 使用如下:

hive> select geohash(12.0d, 123.0d);
OK
wdpkqbtc
Time taken: 0.8 seconds, Fetched: 1 row(s)
hive> select geohash(cast('12' as Double), cast('123' as Double));
OK
wdpkqbtc
Time taken: 0.733 seconds, Fetched: 1 row(s)
hive>