apache
[Bug] [WorkFlow] PostgreSQL Task cannot get Params from upstream task
Search before asking
- [X] I had searched in the issues and found no similar issues.
What happened
I create PostgreSQL Datasource，and I need to get Params from upstream task.
ERROR log is: [INFO] 2023-11-26 16:38:10.034 +0800 - Begin to pulling task [INFO] 2023-11-26 16:38:10.035 +0800 - Begin to initialize task [INFO] 2023-11-26 16:38:10.035 +0800 - Set task startTime: Sun Nov 26 16:38:10 CST 2023 [INFO] 2023-11-26 16:38:10.036 +0800 - Set task envFile: /home/bigdata/module/dolphinscheduler/worker-server/conf/dolphinscheduler_env.sh [INFO] 2023-11-26 16:38:10.036 +0800 - Set task appId: 9_97 [INFO] 2023-11-26 16:38:10.036 +0800 - End initialize task [INFO] 2023-11-26 16:38:10.036 +0800 - Set task status to TaskExecutionStatus{code=1, desc='running'} [INFO] 2023-11-26 16:38:10.036 +0800 - TenantCode:ccit check success [INFO] 2023-11-26 16:38:10.037 +0800 - ProcessExecDir:/tmp/dolphinscheduler/exec/process/ccit/11704971997312/11716796347141_22/9/97 check success [INFO] 2023-11-26 16:38:10.037 +0800 - Resources:{} check success [INFO] 2023-11-26 16:38:10.037 +0800 - Task plugin: SQL create success [INFO] 2023-11-26 16:38:10.037 +0800 - Success initialized task plugin instance success [INFO] 2023-11-26 16:38:10.037 +0800 - Success set taskVarPool: [{"prop":"tblname","direct":"IN","type":"VARCHAR","value":"C20230817"}] [INFO] 2023-11-26 16:38:10.037 +0800 - Full sql parameters: SqlParameters{type='POSTGRESQL', datasource=2, sql='select * from ${tblname}', sqlType=0, sendEmail=null, displayRows=10, limit=0, segmentSeparator=, udfs='null', showType='null', connParams='null', groupId='0', title='null', preStatements=[], postStatements=[]} [INFO] 2023-11-26 16:38:10.037 +0800 - sql type : POSTGRESQL, datasource : 2, sql : select * from ${tblname} , localParams : [],udfs : null,showType : null,connParams : null,varPool : [Property{prop='tblname', direct=IN, type=VARCHAR, value='C20230817'}] ,query max result limit 0 [INFO] 2023-11-26 16:38:10.038 +0800 - setSqlParamsMap: Property with paramName: tblname put in sqlParamsMap of content select * from ${tblname} successfully. [INFO] 2023-11-26 16:38:10.038 +0800 - after replace sql , preparing : select * from ? [INFO] 2023-11-26 16:38:10.038 +0800 - Sql Params are replaced sql , parameters:C20230817(VARCHAR) [INFO] 2023-11-26 16:38:10.038 +0800 - can't find udf function resource [INFO] 2023-11-26 16:38:10.049 +0800 - prepare statement replace sql : select * from ?, sql parameters : {1=Property{prop='tblname', direct=IN, type=VARCHAR, value='C20230817'}} [ERROR] 2023-11-26 16:38:10.050 +0800 - execute sql error: ERROR: syntax error at or near "$1" Position: 15 [ERROR] 2023-11-26 16:38:10.050 +0800 - sql task error org.postgresql.util.PSQLException: ERROR: syntax error at or near "$1" Position: 15 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:181) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:133) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeQuery(SqlTask.java:306) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:209) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:163) at org.apache.dolphinscheduler.server.worker.runner.DefaultWorkerDelayTaskExecuteRunnable.executeTask(DefaultWorkerDelayTaskExecuteRunnable.java:49) at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:174) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [ERROR] 2023-11-26 16:38:10.050 +0800 - Task execute failed, due to meet an exception org.apache.dolphinscheduler.plugin.task.api.TaskException: Execute sql task failed at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:170) at org.apache.dolphinscheduler.server.worker.runner.DefaultWorkerDelayTaskExecuteRunnable.executeTask(DefaultWorkerDelayTaskExecuteRunnable.java:49) at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:174) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74) at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.postgresql.util.PSQLException: ERROR: syntax error at or near "$1" Position: 15 at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2676) at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2366) at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:356) at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:490) at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:408) at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:181) at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:133) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeQuery(SqlTask.java:306) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:209) at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:163) ... 9 common frames omitted [INFO] 2023-11-26 16:38:10.051 +0800 - Get a exception when execute the task, will send the task execute result to master, the current task execute result is TaskExecutionStatus{code=6, desc='failure'}
upstream task is:
current task is:
What you expected to happen
can get taskVarPool from upstream task
How to reproduce
1.create postgresql datasource 2.create workflow with two task. 3.One task is generating tablename,another is selecting the table. The name of table is passing by taskVarPool.
Anything else
No response
Version
3.1.x
Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
