apache
[FEATURE] Supporting storing column metadata in Gravitino
Describe the feature
Currently, column is managed out of Gravitino, for example, column in a Hive table is gotten from HMS, column in an Iceberg table is gotten from Iceberg's metadata.
The current design choice avoids the duplication of maintaining column metadata both in Gravitino and underlying sources. But in the meantime, it brings some limitations like tagging on columns is hard to achieve in Gravitino.
So we should figure out a solution to handle this scenario: support some operations in Gravitino while keep consistency between two services for column.
Attach the design doc (https://docs.google.com/document/d/1X9kFeckDMrzqHo5fcOO55bnU0GIf2DM0KxheH4Nry54/edit?usp=sharing)
Tasks:
- [x] #4795
- [ ] #4867
- [ ] Implement the column management logic in Gravitino.
- [ ] #4599
Motivation
No response
Describe the solution
No response
Additional context
No response
Here attaches the design doc https://docs.google.com/document/d/1X9kFeckDMrzqHo5fcOO55bnU0GIf2DM0KxheH4Nry54/edit?usp=sharing