openhouse
openhouse copied to clipboard
Introduce branching and support for spark.wap.branch
Summary
Implement Iceberg table branching functionality in OpenHouse internal catalog with comprehensive testing infrastructure.
The old code was getting unwieldy with nested for-loops, mutable collections, and scattered state management, repeated logic and overall needed a restructure with branching supported first-class. i feel more confident that this is reliable.
Changes
- Branch Operations: Added branch management methods to OpenHouseInternalTableOperations including create, delete, list, and commit reference handling
- Multi-branch Commit Support: Implemented logic to handle commits across different branch references with checks to reject ambiguous commits
- WAP Branch Integration: Added Write-Audit-Publish (WAP) branch support with proper test coverage
- Test Infrastructure: Created comprehensive test suite (BranchTestSpark3_5) covering branch operations, edge cases, and integration scenarios
Changes
- [X] Client-facing API Changes
- [X] Internal API Changes
- [ ] Bug Fixes
- [X] New Features
- [ ] Performance Improvements
- [ ] Code Style
- [X] Refactoring
- [ ] Documentation
- [X] Tests
Testing Done
- [ ] Manually Tested on local docker setup. Please include commands ran, and their output.
- [X] Added new tests for the changes made.
- [X] Updated existing tests to reflect the changes made.
- [ ] No tests added or updated. Please explain why. If unsure, please feel free to ask for help.
- [ ] Some other form of testing like staging or soak time in production. Please explain.
Additional Information
- [ ] Breaking Changes
- [ ] Deprecations
- [ ] Large PR broken into smaller PRs, and PR plan linked in the description.