DTStack
master-flink1.16.1 编译完-各种connetor jar 普遍不能用，比之前1.28版本的jar明显小
Search before asking
-
[ ] I had searched in the issues and found no similar question.
-
[ ] I had googled my question but i didn't get any help.
-
[ ] I had read the documentation: ChunJun doc but it didn't help me.
Description
No response
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
问题是chunjun1.28 版本能执行的脚本，master-flink1.16.1，执行报错。。执行任何脚本都是报JdbcDynamicTableFactory 和有关的错误
2023-02-21 16:55:20.781 [main] INFO com.dtstack.chunjun.util.PluginUtil - ChunJun executionMode: standalone 2023-02-21 16:55:20.781 [main] INFO com.dtstack.chunjun.util.PluginUtil - ChunJun reset pipeline.jars: [] 2023-02-21 16:55:20.989 [main] INFO com.dtstack.chunjun.util.PluginUtil - ChunJun executionMode: standalone 2023-02-21 16:55:20.989 [main] INFO com.dtstack.chunjun.util.PluginUtil - ChunJun reset pipeline.jars: [file:/opt/chunjun1.6/chunjun-dist/connector/mysql/chunjun-connector-mysql.jar] Exception in thread "main" org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: com.dtstack.chunjun.throwable.ChunJunSqlParseException: ----------sql start--------- 1> 2> 3> insert into sink select * from source u
----------sql end---------
com/dtstack/chunjun/connector/jdbc/table/JdbcDynamicTableFactory
Unable to create a source for reading table 'default_catalog.default_database.source'.
Table options are:
'connector'='mysql-x' 'password'='******' 'scan.fetch-size'='2' 'scan.query-timeout'='10' 'table-name'='datasource_classify' 'url'='jdbc:mysql://192.168.168.200:3306/iccm_ermp' 'username'='root' at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219) at org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:158) at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:82) at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:117) at com.dtstack.chunjun.client.util.JobGraphUtil.buildJobGraph(JobGraphUtil.java:62) at com.dtstack.chunjun.client.standalone.StandaloneClusterClientHelper.submit(StandaloneClusterClientHelper.java:61) at com.dtstack.chunjun.client.Launcher.main(Launcher.java:119) Caused by: com.dtstack.chunjun.throwable.ChunJunRuntimeException: com.dtstack.chunjun.throwable.ChunJunSqlParseException: ----------sql start--------- 1> 2> 3> insert into sink select * from source u
----------sql end---------
com/dtstack/chunjun/connector/jdbc/table/JdbcDynamicTableFactory
Unable to create a source for reading table 'default_catalog.default_database.source'.
Table options are:
'connector'='mysql-x' 'password'='******' 'scan.fetch-size'='2' 'scan.query-timeout'='10' 'table-name'='datasource_classify' 'url'='jdbc:mysql://192.168.168.200:3306/iccm_ermp' 'username'='root' at com.dtstack.chunjun.Main.exeSqlJob(Main.java:158) at com.dtstack.chunjun.Main.main(Main.java:115) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349) ... 7 more Caused by: com.dtstack.chunjun.throwable.ChunJunSqlParseException: ----------sql start--------- 1> 2> 3> insert into sink select * from source u
----------sql end---------
com/dtstack/chunjun/connector/jdbc/table/JdbcDynamicTableFactory
Unable to create a source for reading table 'default_catalog.default_database.source'.
Table options are:
'connector'='mysql-x' 'password'='******' 'scan.fetch-size'='2' 'scan.query-timeout'='10' 'table-name'='datasource_classify' 'url'='jdbc:mysql://192.168.168.200:3306/iccm_ermp' 'username'='root' at com.dtstack.chunjun.sql.parser.SqlParser.lambda$parseSql$1(SqlParser.java:72) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at com.dtstack.chunjun.sql.parser.SqlParser.parseSql(SqlParser.java:66) at com.dtstack.chunjun.Main.exeSqlJob(Main.java:149) ... 13 more Caused by: org.apache.flink.table.api.ValidationException: com/dtstack/chunjun/connector/jdbc/table/JdbcDynamicTableFactory
Unable to create a source for reading table 'default_catalog.default_database.source'.
Table options are:
'connector'='mysql-x' 'password'='******' 'scan.fetch-size'='2' 'scan.query-timeout'='10' 'table-name'='datasource_classify' 'url'='jdbc:mysql://192.168.168.200:3306/iccm_ermp' 'username'='root' at org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:177) at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.createDynamicTableSource(CatalogSourceTable.java:254) at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:100) at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3585) at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2507) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2144) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2093) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050) at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663) at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644) at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3438) at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:165) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:157) at org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:902) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:871) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:250) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlInsert(SqlToOperationConverter.java:564) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:248) at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:77) at org.apache.flink.table.api.internal.StatementSetImpl.addInsertSql(StatementSetImpl.java:50) at com.dtstack.chunjun.sql.parser.InsertStmtParser.execStmt(InsertStmtParser.java:47) at com.dtstack.chunjun.sql.parser.AbstractStmtParser.handleStmt(AbstractStmtParser.java:50) at com.dtstack.chunjun.sql.parser.AbstractStmtParser.handleStmt(AbstractStmtParser.java:52) at com.dtstack.chunjun.sql.parser.AbstractStmtParser.handleStmt(AbstractStmtParser.java:52) at com.dtstack.chunjun.sql.parser.SqlParser.lambda$parseSql$1(SqlParser.java:69) ... 24 more Caused by: java.lang.NoClassDefFoundError: com/dtstack/chunjun/connector/jdbc/table/JdbcDynamicTableFactory at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:756) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at java.util.Iterator.forEachRemaining(Iterator.java:116) at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:471) at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:282) at org.apache.flink.table.factories.FactoryUtil.getDynamicTableFactory(FactoryUtil.java:420) at org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:173) ... 49 more Caused by: java.lang.ClassNotFoundException: com.dtstack.chunjun.connector.jdbc.table.JdbcDynamicTableFactory at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
问题是chunjun1.28 版本能执行的脚本，master-flink1.16.1，执行报错。。执行任何脚本都是报JdbcDynamicTableFactory 和有关的错误
2023-02-21 16:55:20.781 [main] INFO com.dtstack.chunjun.util.PluginUtil - ChunJun executionMode: standalone 2023-02-21 16:55:20.781 [main] INFO com.dtstack.chunjun.util.PluginUtil - ChunJun reset pipeline.jars: [] 2023-02-21 16:55:20.989 [main] INFO com.dtstack.chunjun.util.PluginUtil - ChunJun executionMode: standalone 2023-02-21 16:55:20.989 [main] INFO com.dtstack.chunjun.util.PluginUtil - ChunJun reset pipeline.jars: [file:/opt/chunjun1.6/chunjun-dist/connector/mysql/chunjun-connector-mysql.jar] Exception in thread "main" org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: com.dtstack.chunjun.throwable.ChunJunSqlParseException: ----------sql start--------- 1> 2> 3> insert into sink select * from source u
----------sql end---------
com/dtstack/chunjun/connector/jdbc/table/JdbcDynamicTableFactory
Unable to create a source for reading table 'default_catalog.default_database.source'.
Table options are:
'connector'='mysql-x' 'password'='******' 'scan.fetch-size'='2' 'scan.query-timeout'='10' 'table-name'='datasource_classify' 'url'='jdbc:mysql://192.168.168.200:3306/iccm_ermp' 'username'='root' at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:366) at org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:219) at org.apache.flink.client.program.PackagedProgramUtils.getPipelineFromProgram(PackagedProgramUtils.java:158) at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:82) at org.apache.flink.client.program.PackagedProgramUtils.createJobGraph(PackagedProgramUtils.java:117) at com.dtstack.chunjun.client.util.JobGraphUtil.buildJobGraph(JobGraphUtil.java:62) at com.dtstack.chunjun.client.standalone.StandaloneClusterClientHelper.submit(StandaloneClusterClientHelper.java:61) at com.dtstack.chunjun.client.Launcher.main(Launcher.java:119) Caused by: com.dtstack.chunjun.throwable.ChunJunRuntimeException: com.dtstack.chunjun.throwable.ChunJunSqlParseException: ----------sql start--------- 1> 2> 3> insert into sink select * from source u
----------sql end---------
com/dtstack/chunjun/connector/jdbc/table/JdbcDynamicTableFactory
Unable to create a source for reading table 'default_catalog.default_database.source'.
Table options are:
'connector'='mysql-x' 'password'='******' 'scan.fetch-size'='2' 'scan.query-timeout'='10' 'table-name'='datasource_classify' 'url'='jdbc:mysql://192.168.168.200:3306/iccm_ermp' 'username'='root' at com.dtstack.chunjun.Main.exeSqlJob(Main.java:158) at com.dtstack.chunjun.Main.main(Main.java:115) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:349) ... 7 more Caused by: com.dtstack.chunjun.throwable.ChunJunSqlParseException: ----------sql start--------- 1> 2> 3> insert into sink select * from source u
----------sql end---------
com/dtstack/chunjun/connector/jdbc/table/JdbcDynamicTableFactory
Unable to create a source for reading table 'default_catalog.default_database.source'.
Table options are:
'connector'='mysql-x' 'password'='******' 'scan.fetch-size'='2' 'scan.query-timeout'='10' 'table-name'='datasource_classify' 'url'='jdbc:mysql://192.168.168.200:3306/iccm_ermp' 'username'='root' at com.dtstack.chunjun.sql.parser.SqlParser.lambda$parseSql$1(SqlParser.java:72) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1384) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at com.dtstack.chunjun.sql.parser.SqlParser.parseSql(SqlParser.java:66) at com.dtstack.chunjun.Main.exeSqlJob(Main.java:149) ... 13 more Caused by: org.apache.flink.table.api.ValidationException: com/dtstack/chunjun/connector/jdbc/table/JdbcDynamicTableFactory
Unable to create a source for reading table 'default_catalog.default_database.source'.
Table options are:
'connector'='mysql-x' 'password'='******' 'scan.fetch-size'='2' 'scan.query-timeout'='10' 'table-name'='datasource_classify' 'url'='jdbc:mysql://192.168.168.200:3306/iccm_ermp' 'username'='root' at org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:177) at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.createDynamicTableSource(CatalogSourceTable.java:254) at org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.java:100) at org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3585) at org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2507) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2144) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2093) at org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2050) at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:663) at org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:644) at org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3438) at org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:570) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:165) at org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:157) at org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:902) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:871) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:250) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlInsert(SqlToOperationConverter.java:564) at org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:248) at org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:77) at org.apache.flink.table.api.internal.StatementSetImpl.addInsertSql(StatementSetImpl.java:50) at com.dtstack.chunjun.sql.parser.InsertStmtParser.execStmt(InsertStmtParser.java:47) at com.dtstack.chunjun.sql.parser.AbstractStmtParser.handleStmt(AbstractStmtParser.java:50) at com.dtstack.chunjun.sql.parser.AbstractStmtParser.handleStmt(AbstractStmtParser.java:52) at com.dtstack.chunjun.sql.parser.AbstractStmtParser.handleStmt(AbstractStmtParser.java:52) at com.dtstack.chunjun.sql.parser.SqlParser.lambda$parseSql$1(SqlParser.java:69) ... 24 more Caused by: java.lang.NoClassDefFoundError: com/dtstack/chunjun/connector/jdbc/table/JdbcDynamicTableFactory at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:756) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:473) at java.net.URLClassLoader.access$100(URLClassLoader.java:74) at java.net.URLClassLoader$1.run(URLClassLoader.java:369) at java.net.URLClassLoader$1.run(URLClassLoader.java:363) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:362) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at java.util.Iterator.forEachRemaining(Iterator.java:116) at org.apache.flink.table.factories.FactoryUtil.discoverFactories(FactoryUtil.java:471) at org.apache.flink.table.factories.FactoryUtil.discoverFactory(FactoryUtil.java:282) at org.apache.flink.table.factories.FactoryUtil.getDynamicTableFactory(FactoryUtil.java:420) at org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:173) ... 49 more Caused by: java.lang.ClassNotFoundException: com.dtstack.chunjun.connector.jdbc.table.JdbcDynamicTableFactory at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClassWithoutExceptionHandling(FlinkUserCodeClassLoader.java:64) at org.apache.flink.util.FlinkUserCodeClassLoader.loadClass(FlinkUserCodeClassLoader.java:48) at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
感觉你这是编译引起的。
我本地centos8 环境，maven 配的