hive 中两张表做关联查询时，查询某些字段会出现异常，底层文件存储采用 parquet。
错误详情
Caused by: java.lang.UnsupportedOperationException: Cannot inspect org.apache.hadoop.io.IntWritable
at org.apache.hadoop.hive.ql.io.parquet.serde.primitive.ParquetStringInspector.getPrimitiveWritableObject(ParquetStringInspector.java:52)
at org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:420)
at org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:279)
at org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serializeStruct(LazyBinarySerDe.java:239)
at org.apache.hadoop.hive.serde2.lazybinary.LazyBinarySerDe.serialize(LazyBinarySerDe.java:201)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.makeValueWritable(ReduceSinkOperator.java:563)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:395)
... 13 more
原因
hive 建表语句中的字段类型与 parquet 文件中的类型不一致导致。
查看 parquet 文件 schema
- 下载 parquet-tools-1.9.0.jar
- hadoop jar ./parquet-tools-1.9.0.jar schema hdfs_file_path