Robert Pack

chore: bump arrow and datafusion

3

# Description ~~Using the latest arrow (18) and datafusion (10) dependencies.~~ supersedes #666 This PR moves most file path handling to use the `Path` abstraction from `object_store`. We do so...

# Description In the current implementation of vacuum, we do not clean up the associated `.crc` files as well as the possibly existing bloom index files. **Use Case** **Related Issue(s)**

Add ConflictChecker for concurrency control

2

# Description This PR adds a `ConflictChecker` struct for conflict resolution in cases of concurrent commit failures. The implementation is based on the [reference implementation](https://github.com/delta-io/delta/blob/fe36a53f3c70c5f9c9b5052c12cd1703f495da97/core/src/main/scala/org/apache/spark/sql/delta/ConflictChecker.scala). SO far the implementation is...

[discussion] move delta log handling to new struct `DeltaLog` and harmonize operations

4

# Description As we add more and more high level operations (vacuum, optimize, writes, ....) our `DeltaTable` struct is continuously growing in complexity, and maybe it's time for us to...

# Description When committing transactions to the Delta log, make sure we never end up in an inconsistent or corrupt state. **Use Case** When performing mutating operations against a Delta...

High-level writer API

2

# Description Recently I have been trying to understand the current state of writer implementations for `delta-rs` and what could be the desired state for a high level writer API....

At our company we are big fans of type hints to ensure code quality. At the very least in some instances tpye hints could be more permissive here. e g....

Remove cargo.patch for `object_store` crate

1

# Description Currently we patch the version of the `object_store` crate to take advantage of the removed S3 and azure SDKs, once the new version is published (apache/arrow-rs#2620), we need...

# Description Currently fake-gcs does not seem to support copy-if-not-exists functionality (fsouza/fake-gcs-server#909). Once that gets resolved, we should enable the omitted integration tests related to concurrent writers. **Use Case** Knowing...

Support preconditions on copyTo requests

1

As I just started looking into GCS, I may completely have missed something, But after some testing and browsing the code, it seems to me precoditions are currently not supported...