Datafusion mixes vended credentials between different tables in the same bucket

[SergeiPatiakin]

Steps To Reproduce

• Create two tables with identical data in an Iceberg REST catalog with x_iceberg_access_delegation = Some("vended-credentials"):
  • t1 at s3://some-bucket/t1
  • t2 at s3://some-bucket/t2
• Run the Datafusion query SELECT * FROM t1 UNION ALL SELECT * FROM t2

Expected behavior

• Query should succeed
• Credentials vended for t1 should be used for scanning t1
• Credentials vended for t2 should be used for scanning t2

Observed behavior

• Query fails with permission errors
• Credentials vended for t2 are used for scanning t1

Comments

• The root cause is Datafusion's DefaultObjectStoreRegistry only takes bucket name into account when looking up the key (register_store, get_url_key). This means that the object store instance for t1 is discarded when the object store instance for t2 is registered
• We have been able to work around this with https://github.com/splitgraph/iceberg-rs/pull/9 but we're not sure how well this solution plays with other folks' needs. Let me know if you have a better idea for a fix!

[JanKaul]

I can't really tell the implications of changing the URL for the object_store registry and it feels a bit fragile to me. How well does this work with paths that are hardcoded into the iceberg metadata/manifests? Given the path of a Parquet file from an Iceberg manifest, wouldn't Datafusion still use its original path to get the object-store from the registry?

What do you think about if this crate would provide its own ObjectStoreRegistry.

[gruuya]

How well does this work with paths that are hardcoded into the iceberg metadata/manifests? Given the path of a Parquet file from an Iceberg manifest, wouldn't Datafusion still use its original path to get the object-store from the registry?

It should work fine since all of this is scoped to a table scan method; first the key is defined and the store is registered https://github.com/JanKaul/iceberg-rust/blob/8cb7e0114e19f3347d986fe06c45e1392ba33a1e/datafusion_iceberg/src/table.rs#L280-L285

Subsequently this key is passed to FileScanConfigs https://github.com/JanKaul/iceberg-rust/blob/8cb7e0114e19f3347d986fe06c45e1392ba33a1e/datafusion_iceberg/src/table.rs#L605-L609

In turn DataFusion uses those directly when starting up the stream https://github.com/apache/datafusion/blob/392280020109f049915fb2724a5a0f40d549d447/datafusion/datasource/src/file_scan_config.rs#L453.

Fwiw we had this problem with delta-rs too, and a workaround similar to https://github.com/splitgraph/iceberg-rs/pull/9 is still in place.

[JanKaul]

Okay, makes sense. I think it would be great to also include it in this repo. Thanks for your efforts.

[gruuya]

Sorry for the delay, here's the upstream of the changes we've been using https://github.com/JanKaul/iceberg-rust/pull/178