iceberg-python icon indicating copy to clipboard operation
iceberg-python copied to clipboard

Support merge manifests on writes (MergeAppend)

Open HonahX opened this issue 1 year ago • 0 comments

Add MergeAppendFiles. This PR will enable the following configurations:

  • commit.manifest-merge.enabled: Controls whether to automatically merge manifests on writes.
  • commit.manifest.min-count-to-merge: Minimum number of manifests to accumulate before merging.
  • commit.manifest.target-size-bytes: Target size when merging manifest files.

Since commit.manifest-merge.enabled is default to True, we need to make MergeAppend as the default way to append data to align with the property definition and java implementation

HonahX avatar Feb 04 '24 08:02 HonahX