Hadoop 3クラスタ構築——hive追加カスタム関数UDTF(1行入力、複数行出力)

前編:
Hadoop 3クラスタによる仮想マシンのインストール
Hadoop 3クラスタ構築——hadoopをインストールし、環境を配置する
Hadoop 3クラスタ構築-ntpサービスの構成
Hadoop 3クラスタ構築-hiveインストール
Hadoop 3クラスタ構築-hbaseインストールと簡単な操作
Hadoop 3クラスタ構築-hiveカスタム関数UDF追加
Hadoop 3クラスタ構築——hive追加カスタム関数UDTF
 
前編ではudtf関数は,1行入力,1行出力のみである.udtfは1行の入力,複数行の出力が可能である.
以下の要件を簡単に説明します.

      ，    ，         

直接コード:

package com.venn.udtf;

import org.apache.hadoop.hive.ql.exec.UDFArgumentException;
import org.apache.hadoop.hive.ql.exec.UDFArgumentLengthException;
import org.apache.hadoop.hive.ql.metadata.HiveException;
import org.apache.hadoop.hive.ql.udf.generic.GenericUDTF;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
import org.apache.hadoop.hive.serde2.objectinspector.StructObjectInspector;
import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;

import java.util.ArrayList;

/**
 * Created by venn on 5/20/2018.
 * SplitHour : split hour
 */
public class SplitHour extends GenericUDTF {

    /**
     * add the column name
     * @param args
     * @return
     * @throws UDFArgumentException
     */
    @Override
    public StructObjectInspector initialize(ObjectInspector[] args) throws UDFArgumentException {
        if (args.length != 1) {
            throw new UDFArgumentLengthException("ExplodeMap takes only one argument");
        }
        if (args[0].getCategory() != ObjectInspector.Category.PRIMITIVE) {
            throw new UDFArgumentException("ExplodeMap takes string as a parameter");
        }

        ArrayList fieldNames = new ArrayList();
        ArrayList fieldOIs = new ArrayList();
        fieldNames.add("begintime");
        fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
        fieldNames.add("endtime");
        fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
        fieldNames.add("hour");
        fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);
        fieldNames.add("seconds");
        fieldOIs.add(PrimitiveObjectInspectorFactory.javaStringObjectInspector);


        return ObjectInspectorFactory.getStandardStructObjectInspector(fieldNames, fieldOIs);
    }

    /**
     * process the column
     * @param objects
     * @throws HiveException
     */
    public void process(Object[] objects) throws HiveException {


        String [] input = objects[0].toString().split(",");
        // 2018-06-06 10:25:35
        String beginTime = input[0];
        String endTime = input[1];

        String[] result = new String[4];
        result[0] = beginTime;
        result[1] = endTime;

        // begintime
        int bhour = Integer.parseInt(beginTime.substring(11, 13));
        int bmin = Integer.parseInt(beginTime.substring(14, 16));
        int bsecond = Integer.parseInt(beginTime.substring(17, 19));
        // endtime
        int ehour = Integer.parseInt(endTime.substring(11, 13));
        int emin = Integer.parseInt(endTime.substring(14, 16));
        int esecond = Integer.parseInt(endTime.substring(17, 19));

        // 1.if begin hour equal end hour, second is : (emin - bmin) * 60 + (esecond - bsecond)
        if (bhour == ehour) {
            result[2] = String.valueOf(bhour);
            result[3] = String.valueOf((emin - bmin) * 60 + (esecond - bsecond));
            forward(result);
            return;
        }

        boolean flag = true;
        //TODO    ，           ，           
        while (bhour != ehour) {
            result[2] = String.valueOf(bhour);

            if(flag){
                flag = false;
            // 2. if begintime hour != endtime, the first hour, second is : 3600 - bmin * 60 - bsecond
                result[3] = String.valueOf(3600 - bmin * 60 - bsecond);
            }else {
                // 3. next hour is 3600
                result[3] = String.valueOf(3600);
            }
            bhour += 1;
            //    hive
            forward(result);
        }

        result[2] = String.valueOf(bhour);
        // 4. the end hour is : emin  * 60 + esecond
        result[3] = String.valueOf( emin  * 60 + esecond);
        forward(result);

    }

    public void close() throws HiveException {

    }

}

 
udtf関数紹介参加前編
使い方は前編を参照
Hadoop 3クラスタ構築——hive追加カスタム関数UDTF
 
サンプル:

hive> select split_hour( concat(begintime,',',endtime)) from viewlog where log_date=20180401 limit 10;
OK
begintime    endtime    hour    seconds
2018-04-01 10:26:14    2018-04-01 10:26:21    10    7
2018-04-01 07:21:47    2018-04-01 07:22:23    7    36
2018-04-01 15:18:08    2018-04-01 15:18:11    15    3
2018-04-01 18:05:13    2018-04-01 18:05:28    18    15
2018-04-01 07:18:34    2018-04-01 07:18:52    7    18
2018-04-01 23:28:32    2018-04-01 23:29:44    23    72
2018-04-01 06:34:11    2018-04-01 06:34:17    6    6
2018-04-01 14:02:40    2018-04-01 14:03:33    14    53
2018-04-01 17:30:23    2018-04-01 17:30:26    17    3
2018-04-01 12:15:07    2018-04-01 12:15:11    12    4
2018-04-01 06:53:40    2018-04-01 07:02:09    6    380
2018-04-01 06:53:40    2018-04-01 07:02:09    7    129
Time taken: 2.238 seconds, Fetched: 12 row(s)

かたづける
 
転載先:https://www.cnblogs.com/Springmoon-venn/p/9286670.html