[Improvement] Add Iceberg (and minimal Spark DSv2) commands to Authz module
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Search before asking
- [X] I have searched in the issues and found no similar issues.
What would you like to be improved?
We recently integrated Apache Iceberg into our product with Apache Kyuubi. We started from Ranger ACL plugin of Apache Submarine, but it has only DSv1 ACLs. So We add some DSv2 ACL checks to the module for Iceberg tables.
Below is the full list of added commands: Spark DSv2 DDLs
- AddColumns, AlterColumn, DropColumns, RenameColumn, RenameTable,
- SetTableProperties, UnsetTableProperties
- CreateV2Table, DropTable, ReplaceTableAsSelect
- TruncateTable
Spark DSv2 DMLs
- AppendData
Iceberg commands: DDLs
- AddPartitionField/ DropPartitionField
- SetIdentifierFields/ DropIdentifierFields
- SetWriteDistributionAndOrdering
Iceberg commands: DMLs
- MergeIntoIcebergTable
- UpdateIcebergTable
- DeleteFromIcebergTable
We uses Kyuubi already, and now Kyuubi has it's own Authz module. So we're willing to contribute back to Kyuubi.
How should we improve?
PrivilegesBuilder.buildCommand has almost all DSv1 and some DSv2 commands. Create PrivilegesBuilder.buildV2Command and PrivilegesBuilder.buildIcebergCommand for each purpose. Then call these functions from PrivilegesBuilder.build like:
def build(plan: LogicalPlan): (Seq[PrivilegeObject], Seq[PrivilegeObject]) = {
val inputObjs = new ArrayBuffer[PrivilegeObject]
val outputObjs = new ArrayBuffer[PrivilegeObject]
plan match {
// RunnableCommand
case cmd: Command =>
buildCommand(cmd, inputObjs, outputObjs)
buildV2Command(cmd, inputObjs, outputObjs)
buildIcebergCommand(cmd, inputObjs, outputObjs)
// Queries
case _ => buildQuery(plan, inputObjs)
}
(inputObjs, outputObjs)
}
Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
thanks @minyk
@yaooqinn this is getting bigger than I thought. Currently, initial translation to kyuubi was done and start to write unit test.
May I send Working-In-Progress PR for discussion? Any restriction for WIP PR?
feel free to send a WIP PR @minyk
Both V2 commands/plans and Iceberg commands are covered in the current Authz implementation. This issue should be good to close.
thanks @bowenliang123