[Bug] [helm chart] Bean of StorageOperator is missing in the worker server

[paddyabc]

Search before asking

• [x] I had searched in the issues and found no similar issues.

What happened

I saw a similar issue for a standalone deployment, but it can't help on helm chart According to the document, I am using the minio (installed by dolphin scheduler helm chart) as the storage and already add the configuration to tell the helm chart to use S3 instead

common:
  configmap:
    RESOURCE_STORAGE_TYPE: "S3"

for the other values related to the minio, I am using the helm chart default value The problem only existed in the worker and here is the error message

Description:

Parameter 2 of constructor in org.apache.dolphinscheduler.server.worker.executor.PhysicalTaskExecutorFactory required a bean of type 'org.apache.dolphinscheduler.plugin.storage.api.StorageOperator' that could not be found.

The injection point has the following annotations:
        - @org.springframework.beans.factory.annotation.Autowired(required=true)

The following candidates were found but could not be injected:
        - User-defined bean method 'storageOperate' in 'StorageConfiguration' ignored as the bean value is null


Action:

Consider revisiting the entries above or defining a bean of type 'org.apache.dolphinscheduler.plugin.storage.api.StorageOperator' in your configuration.

Stack Trace:

Caused by: org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'physicalTaskExecutorFactory' defined in URL [jar:file:/opt/dolphinscheduler/libs/dolphinscheduler-worker-3.3.1.jar!/org/apache/dolphinscheduler/server/worker/executor/PhysicalTaskExecutorFactory.class]: Unsatisfied dependency expressed through constructor parameter 2; nested exception is org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.apache.dolphinscheduler.plugin.storage.api.StorageOperator' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:800)
        at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:229)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1372)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1222)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:582)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:542)
        at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
        at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1391)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
        ... 33 common frames omitted
Caused by: org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying bean of type 'org.apache.dolphinscheduler.plugin.storage.api.StorageOperator' available: expected at least 1 bean which qualifies as autowire candidate. Dependency annotations: {}
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.raiseNoMatchingBeanFound(DefaultListableBeanFactory.java:1801)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1396)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1311)
        at org.springframework.beans.factory.support.ConstructorResolver.resolveAutowiredArgument(ConstructorResolver.java:887)
        at org.springframework.beans.factory.support.ConstructorResolver.createArgumentArray(ConstructorResolver.java:791)
        ... 47 common frames omitted

What you expected to happen

I expect I can use the default value in the helm chart for minio, so that I can start the worker

How to reproduce

helm3 install dolphinscheduler oci://registry-1.docker.io/apache/dolphinscheduler-helm --version 3.3.1 -n dolphin --values values.yaml

here is the values. yaml (removed ip/hostname and username/password)

postgresql:
  enabled: false

zookeeper:
  enabled: false

common:
  configmap:
    RESOURCE_STORAGE_TYPE: "S3"

conf:
  common:
    resource.storage.type: "S3"

externalDatabase:
  enabled: true
  host: 
  type: "postgresql"
  port: 
  username: 
  password: 
  database: "dolphin_scheduler"

ingress:
  enabled: true
  host: 
  path: "/"
  tls:
    enabled: true
    secretName: "dolphin-tls"

registryJdbc:
  enabled: true
  hikariConfig:
    enabled: true
    driverClassName: "org.postgresql.Driver"
    jdbcurl: 
    username: 
    password: 

Anything else

No response

Version

3.3.1

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[SebastienTolron]

Same here :/

[zzj-always]

Same here :/

[cheneychoi]

Same here :/

[SbloodyS]

You can try install plugins mannully base on the doc in https://dolphinscheduler.apache.org/en-us/docs/3.3.2/guide/installation/pseudo-cluster for temporary workaround.

[liyin37]

I have resovled the api and worker need to rebuild images like: api dokcerfile is blow:

FROM apache/dolphinscheduler-api:3.3.2
COPY *.jar  /opt/dolphinscheduler/libs/
RUN ln -s /opt/dolphinscheduler/libs/mysql-connector-java-8.0.30.jar /opt/dolphinscheduler/api-server/libs/mysql-connector-java-8.0.30.jar && \
    ln -s /opt/dolphinscheduler/libs/dolphinscheduler-datasource-mysql-3.3.2.jar /opt/dolphinscheduler/api-server/libs/dolphinscheduler-datasource-mysql-3.3.2.jar && \
    ln -s /opt/dolphinscheduler/libs/dolphinscheduler-datasource-postgresql-3.3.2.jar /opt/dolphinscheduler/api-server/libs/dolphinscheduler-datasource-postgresql-3.3.2.jar && \
    ln -s /opt/dolphinscheduler/libs/dolphinscheduler-storage-s3-3.3.2.jar /opt/dolphinscheduler/api-server/libs/dolphinscheduler-storage-s3-3.3.2.jar

worker dockerfile is :

FROM apache/dolphinscheduler-worker:3.3.2
COPY *.jar  /opt/dolphinscheduler/libs/
RUN ln -s /opt/dolphinscheduler/libs/mysql-connector-java-8.0.30.jar /opt/dolphinscheduler/worker-server/libs/mysql-connector-java-8.0.30.jar && \
    ln -s /opt/dolphinscheduler/libs/dolphinscheduler-datasource-mysql-3.3.2.jar /opt/dolphinscheduler/worker-server/libs/dolphinscheduler-datasource-mysql-3.3.2.jar && \
    ln -s /opt/dolphinscheduler/libs/dolphinscheduler-datasource-postgresql-3.3.2.jar /opt/dolphinscheduler/worker-server/libs/dolphinscheduler-datasource-postgresql-3.3.2.jar && \
    ln -s /opt/dolphinscheduler/libs/dolphinscheduler-storage-s3-3.3.2.jar /opt/dolphinscheduler/worker-server/libs/dolphinscheduler-storage-s3-3.3.2.jar