Arrow error: Cast error: Cannot cast from struct to other types

[kettlelinna]

Describe the bug

it will throw exception when my schema is complex particularly have a complex struct type

To Reproduce Steps to reproduce the behavior:

1. spark.read.parquet("part-00000-8f6e45af-264c-4cea-b09f-0753de014135-c000.snappy.parquet").show()

Expected behavior

it should compatibility of struct type whatever complex

Screenshots

[kettlelinna]

data.json I have convert file format to json, please feel free take a try if u intersting

[lihao712]

we use arrow-cast as base function to cast struct, and it can't change struct to other datatype

[kettlelinna]

we use arrow-cast as base function to cast struct, and it can't change struct to other datatype

but I don't call cast on spark side, I just show the records which have struct type column spark.read.parquet("part-00000-8f6e45af-264c-4cea-b09f-0753de014135-c000.snappy.parquet").show()

[lihao712]

Blaze currently provides comprehensive support for Spark SQL, but there hasn't been sufficient testing for DataFrame-level calls. can you use sql to execute this logical,like behind to get more information?

val parqDF = spark.read.parquet("part-00000-8f6e45af-264c-4cea-b09f-0753de014135-c000.snappy.parquet"); parqDF.createOrReplaceTempView("ParquetTable") val parkSQL = spark.sql("select * from ParquetTable ")