Getting error in the python job

[JyotinP]

{base.py:73} INFO - Using connection ID 'spark-conn' for task execution. {spark_submit.py:351} INFO - Spark-Submit cmd: spark-submit --master spark://spark-master-1:7077 --name arrow-spark jobs/python/wordcountjob.py {spark_submit.py:521} INFO - /home//.local/lib/python3.11/site-packages/pyspark/bin/load-spark-env.sh: line 68: ps: command not found {spark_submit.py:521} INFO - /home//.local/lib/python3.11/site-packages/pyspark/bin/spark-class: line 71: /usr/lib/jvm/java-11-openjdk-arm64/bin/java: No such file or directory {spark_submit.py:521} INFO - /home/***/.local/lib/python3.11/site-packages/pyspark/bin/spark-class: line 97: CMD: bad array subscript {taskinstance.py:1935} ERROR - Task failed with exception Traceback (most recent call last): File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/apache/spark/operators/spark_submit.py", line 160, in execute self._hook.submit(self._application) File "/home/airflow/.local/lib/python3.11/site-packages/airflow/providers/apache/spark/hooks/spark_submit.py", line 452, in submit raise AirflowException( airflow.exceptions.AirflowException: Cannot execute: spark-submit --master spark://spark-master-1:7077 --name arrow-spark jobs/python/wordcountjob.py. Error code is: 1.

[franceZa]

at Dockerfile add " RUN export JAVA_HOME" It works for me

Set JAVA_HOME environment variable ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/ RUN export JAVA_HOME

[weldermartins]

at Dockerfile add " RUN export JAVA_HOME" It works for me

Set JAVA_HOME environment variable ENV JAVA_HOME /usr/lib/jvm/java-11-openjdk-amd64/ RUN export JAVA_HOME

I did as you advised, but in my case I still got the same error.

[2024-01-16, 20:01:50 UTC] {standard_task_runner.py:104} ERROR - Failed to execute job 7 for task python_job (Cannot execute: spark-submit --master spark://spark-master:7077 --name arrow-spark --deploy-mode client jobs/python/wordcountjob.py. Error code is: 1.; 239)

python and java compatible

Versions

variables

docker-compose.yml

Airflow connection

** Airflow Job Error**

On my machine local, spark-submit ran without errors.

[yashraizb]

@weldermartins Please check this issue link since this worked for me. And let me if this works for you.

[yashraizb]

@JyotinP It looks like ps command is not found, try adding procps in the dockerfile where java sdk and other packages are getting installed. Refer the below code:

RUN apt-get update &&
apt-get install -y gcc python3-dev openjdk-11-jdk procps &&
apt-get clean