[Bug] [Resource/file-manange] Failed to get file on shell task

[lakako]

Search before asking

• [X] I had searched in the issues and found no similar issues.

What happened

I have uploaded a file to file manage. When I want to use this file on shell task, happened "no such file or directory"

version: 3.2.0 standalone-server

What you expected to happen

use file on shell task correctly.

How to reproduce

upload file and create a shell task. select that file and use it in shell task.

Anything else

No response

Version

3.2.x

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[fuchanghai]

hi @lakako you can try cat /test .txt

[lakako]

[INFO] 2023-12-01 14:14:40.702 +0800 -  -> 
	cat: /test.txt: No such file or directory
	cat: test.txt: No such file or directory

root@295f2097dcc1:/opt/dolphinscheduler# cat /dolphinscheduler/default/resources/test.txt
hhhh
cxzcxz

[git-blame]

I have the same problem. The issue seems to be the file is placed with the wrong path in the execution directory. The entire file:/ URI is treated like a directory path.

[INFO] 2024-01-30 20:06:41.110 +0000 - get resource file from path:file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default/resources/hello.sh
[INFO] 2024-01-30 20:06:41.130 +0000 - Download resources: {file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default/resources/hello.sh=file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default/resources/hello.sh} successfully
...
[INFO] 2024-01-30 20:06:41.162 +0000 - Final Shell file is :
#!/bin/bash
BASEDIR=$(cd `dirname $0`; pwd)
cd $BASEDIR
echo "What's here: $BASEDIR"
find .
bash hello.sh
[INFO] 2024-01-30 20:06:41.164 +0000 - Executing shell command : sudo -u root -i /tmp/dolphinscheduler/exec/process/default/12443475537632/12443490756832_1/1/1/1_1.sh
[INFO] 2024-01-30 20:06:41.206 +0000 - process start, process id is: 43710
[INFO] 2024-01-30 20:06:42.204 +0000 -  ->
    What's here: /tmp/dolphinscheduler/exec/process/default/12443475537632/12443490756832_1/1/1
    .
    ./file:
    ./file:/opt
    ./file:/opt/apache-dolphinscheduler-3.2.0-bin
    ./file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server
    ./file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files
    ./file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default
    ./file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default/resources
    ./file:/opt/apache-dolphinscheduler-3.2.0-bin/standalone-server/files/default/resources/hello.sh
    ./1_1.sh
    bash: hello.sh: No such file or directory

1706645301047.log

[git-blame]

This path problem also affects other task such as Spark Task including the main JAR file. However, since Dolphin will use the full JAR path when constructing the spark-submit command, the task still completes successfully.

But spark options like --files are still affected since specifying --files myfile.txt in the task definition will not work.

[SbloodyS]

Fixed.