Luca Canali

[SPARK-38098][PYTHON] Add support for ArrayType of nested StructType to arrow-based conversion

7

### What changes were proposed in this pull request? This proposes to add support for ArrayType of nested StructType to arrow-based conversion. This allows Pandas UDFs, mapInArrow UDFs, and toPandas...

[SPARK-40485][SQL] Extend the partitioning options of the JDBC data source

1

### What changes were proposed in this pull request? This proposes to extend the available partitioning options for the JDBC data source. ### Why are the changes needed? Partitioning options...

[SPARK-34265][PYTHON][SQL] Instrument Python UDFs using SQL metrics

8

### What changes are proposed in this pull request? This proposes to add SQLMetrics instrumentation for Python UDF execution, including Pandas UDF, and related operations such as MapInPandas and MapInArrow....

This follows up on https://github.com/swan-cern/jupyter-extensions/commit/31d442852eedab2cb880976673ad0bf821cbcea8 After the move to central Grafana, orgId is now specified in the Spark Metrics bundle parameter.

This is to add configuration for setting spark.kubernetes.container.image The configuration is via an environemnt variable that can be set by the spawner. This will allow to support executor containers built...

[SPARK-48493][PYTHON] Enhance Python Datasource Reader with direct Arrow Batch support for improved performance

3

### What changes were proposed in this pull request? This pull request proposes enhancing the Python Datasource Reader by adding an option to yield Arrow batches directly. This change aims...

e2e tests fail

2

e2e tests fail after recent changes, in particular commit 4346ede This is to track the investigation and fix of the issue. CC: @fjammes