Neelesh Salian

Adding MergeInto into the Spark Scala API

9

Context: https://iceberg.apache.org/#spark-writes/#merge-into I see that merge into is the recommended writing pattern instead of Insert Overwrite. But there's no Dataframe/ DataSet Scala API equivalent and we have to resort to...

[CT-809] Improve the docker image setup for Apache Spark

3

### Describe the feature #### Context Currently, the project relies on GoDataDriven's spark images for its base image in the docker-compose as well as the test configuration. The problem here...

Failing to pass arguments to pyspark job.

1

Trying to get arguments passed to a python file ``` exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.128.247.167 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.deploy.PythonRunner .py arg_1 arg_2 arg_3 arg_4 ``` But...