Partho Sarthi
Partho Sarthi
Fixes #8844. This PR adds fix for removing internal metadata information from the schema while using CTAS. This feature was added in Spark 3.5.0. ### Changes: Created a new class...
Multiple integration tests `test_parquet_append_with_downcast, test_parquet_write_column_name_with_dots` etc failed with the following error: ``` [2024-03-13T19:07:42.384Z] E : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2428.0 failed 1 times,...
This PR fixes the internal field metadata leakage by updating CTAS/RTAS commands to remove such metadata before table creation. Our definition of class `GpuCreateDataSourceTableAsSelectCommand` might require similar changes. Apache Commit...
Makes changes in the catalyst to ensure same behaviour of `wider-type-narrow-value` literal in v1 and v2 sources. We should be fine mostly. https://github.com/apache/spark/commit/866779d7cbd
A new trait `ScanFileListing` has been introduced to encapsulate and streamline the handling of file listing results. It replaces `PartitionDirectory`. We should modify its usage in `GpuFileSourceScanExec` and `GpuHiveTableScanExec` classes....
This PR fixes different task attempts writing to the same directory. This might affect our GpuFileFormatWriter. https://github.com/apache/spark/commit/67d11b1992a
Changes in plan output string due to shift operators. Should double check if this affects our tests. https://github.com/apache/spark/commit/a78ef738af0
New stateful field was added in `MapConcat`. We should update `GpuMapConcat` correspondingly. https://github.com/apache/spark/commit/af1ac1edc2a