[Question][Hive3Sink] kafka data to hive3 table ：.ClassNotFoundException: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat

[leevin101]

Search before asking

• [X] I had searched in the issues and found no similar question.

• [ ] I had googled my question but i didn't get any help.

• [X] I had read the documentation: ChunJun doc but it didn't help me.

Description

我在使用chunjun 1.12.5采集Kafka数据入Hive3 orc表的时候，报如下错误：java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat

我的环境信息： Hadoop：3.1.4 Chunjun：1.12.5 Kafka：2.1 Hive：3.1.4

详细日志: INFO [main] org.apache.flink.configuration.GlobalConfiguration:Loading configuration property: fs.hdfs.hadoopconf, /soft/hadoop-3.1.4/etc/hadoop INFO [main] com.dtstack.chunjun.Main:Register to table configuration:{table.dynamic-table-options.enabled=true, pipeline.name=Flink_Job} INFO [main] com.dtstack.chunjun.util.PluginUtil:ChunJun executionMode: local INFO [main] com.dtstack.chunjun.util.PluginUtil:ChunJun reset pipeline.jars: [/soft/chunjun-dist-v1.12.5/chunjun-dist/chunjun-core.jar, /soft/chunjun-dist-v1.12.5/chunjun-dist/metrics/prometheus/chunjun-metrics-prometheus.jar, /soft/chunjun-dist-v1.12.5/chunjun-dist/dirty-data-collector/log/chunjun-dirty-log.jar, /soft/chunjun-dist-v1.12.5/chunjun-dist/connector/hive3/chunjun-connector-hive3.jar, /soft/chunjun-dist-v1.12.5/chunjun-dist/connector/kafka/chunjun-connector-kafka.jar] INFO [main] com.dtstack.chunjun.classloader.ClassLoaderManager:jarUrl:file:/soft/chunjun-dist-v1.12.5/chunjun-dist/chunjun-core.jar_file:/soft/chunjun-dist-v1.12.5/chunjun-dist/connector/hive3/chunjun-connector-hive3.jar_file:/soft/chunjun-dist-v1.12.5/chunjun-dist/connector/kafka/chunjun-connector-kafka.jar_file:/soft/chunjun-dist-v1.12.5/chunjun-dist/dirty-data-collector/log/chunjun-dirty-log.jar_file:/soft/chunjun-dist-v1.12.5/chunjun-dist/metrics/prometheus/chunjun-metrics-prometheus.jar create ClassLoad successful... Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/orc/OrcOutputFormat at com.dtstack.chunjun.connector.hive3.sink.Hive3OutputFormatBuilder.newBuild(Hive3OutputFormatBuilder.java:37) at com.dtstack.chunjun.connector.hive3.sink.Hive3SinkFactory.createSink(Hive3SinkFactory.java:67) at com.dtstack.chunjun.Main.exeSyncJob(Main.java:222) at com.dtstack.chunjun.Main.main(Main.java:122) at com.dtstack.chunjun.client.local.LocalClusterClientHelper.submit(LocalClusterClientHelper.java:35) at com.dtstack.chunjun.client.Launcher.main(Launcher.java:119) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 6 more

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[zhiqiang-dev]

类冲突了。 可以把工程里面所有 pom 所有 hadoop 和 hive 依赖统一。 如果打包的时候，可以通过参数指定版本。

[lujisen]

chunjun在加载类的时候默认只加载了chun-开头的jar包，其他jar无法加载，如果需要写入hive orc表，只需要在/opt/chunjun-1.12.5/chunjun-dist/connector/hive3下将hive-exec*.jar修改为chunjun-hive-exec*.jar即可，其他jar参照即可