SparkingFlow
SparkingFlow copied to clipboard
airscholar
This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python, Scala and Java as an example.
{base.py:73} INFO - Using connection ID 'spark-conn' for task execution. {spark_submit.py:351} INFO - Spark-Submit cmd: spark-submit --master spark://spark-master-1:7077 --name arrow-spark jobs/python/wordcountjob.py {spark_submit.py:521} INFO - /home/***/.local/lib/python3.11/site-packages/pyspark/bin/load-spark-env.sh: line 68: ps: command not...
every time i try to submit my spark app i am runing on windows i changed the docker file arm64 to amd64 for the jdk i get this error 
During docker-compose up --build airflow webserver exits and the process is tucked until it not killed and start again when airflow init done. Having `restart: always` will restart webserver building...
`ERROR! Maximum number of retries (20) reached. 2024-02-20 22:47:30 2024-02-20 22:47:30 Last check result: 2024-02-20 22:47:30 $ airflow db check 2024-02-20 22:47:30 Unable to load the config, contains a configuration...
**Latest version of Airflow.**  **Connection Airflow**  **Dag** 
Metadata
Owner
Metadata
This project demonstrates how to use Apache Airflow to submit jobs to Apache spark cluster in different programming laguages using Python, Scala and Java as an example.