iceberg icon indicating copy to clipboard operation
iceberg copied to clipboard

Spark: Support rewrite file with z-order for nested Struct type

Open xiaoxuandev opened this issue 1 year ago • 1 comments

Notes

This change adds support for rewriting files with Z-ordering for nested Struct types in Iceberg. This allows more efficient data organization and retrieval in scenarios where nested Struct types are present.

The implementation involves extending the existing retrieve z-order columns logic to handle nested Struct types.

Testing

Unit tests

Tested real dataset on EMR 7.0.0, and verified the data was ordered based on nested column.

xiaoxuandev avatar Feb 27 '24 17:02 xiaoxuandev