kyuubi icon indicating copy to clipboard operation
kyuubi copied to clipboard

[Improvement] Add Iceberg (and minimal Spark DSv2) commands to Authz module

Open minyk opened this issue 4 years ago • 3 comments

Code of Conduct

Search before asking

  • [X] I have searched in the issues and found no similar issues.

What would you like to be improved?

We recently integrated Apache Iceberg into our product with Apache Kyuubi. We started from Ranger ACL plugin of Apache Submarine, but it has only DSv1 ACLs. So We add some DSv2 ACL checks to the module for Iceberg tables.

Below is the full list of added commands: Spark DSv2 DDLs

  • AddColumns, AlterColumn, DropColumns, RenameColumn, RenameTable,
  • SetTableProperties, UnsetTableProperties
  • CreateV2Table, DropTable, ReplaceTableAsSelect
  • TruncateTable

Spark DSv2 DMLs

  • AppendData

Iceberg commands: DDLs

  • AddPartitionField/ DropPartitionField
  • SetIdentifierFields/ DropIdentifierFields
  • SetWriteDistributionAndOrdering

Iceberg commands: DMLs

  • MergeIntoIcebergTable
  • UpdateIcebergTable
  • DeleteFromIcebergTable

We uses Kyuubi already, and now Kyuubi has it's own Authz module. So we're willing to contribute back to Kyuubi.

How should we improve?

PrivilegesBuilder.buildCommand has almost all DSv1 and some DSv2 commands. Create PrivilegesBuilder.buildV2Command and PrivilegesBuilder.buildIcebergCommand for each purpose. Then call these functions from PrivilegesBuilder.build like:

  def build(plan: LogicalPlan): (Seq[PrivilegeObject], Seq[PrivilegeObject]) = {
    val inputObjs = new ArrayBuffer[PrivilegeObject]
    val outputObjs = new ArrayBuffer[PrivilegeObject]
    plan match {
      // RunnableCommand
      case cmd: Command => 
          buildCommand(cmd, inputObjs, outputObjs)
          buildV2Command(cmd, inputObjs, outputObjs)
          buildIcebergCommand(cmd, inputObjs, outputObjs)
      // Queries
      case _ => buildQuery(plan, inputObjs)
    }
    (inputObjs, outputObjs)
  }

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

minyk avatar Apr 28 '22 08:04 minyk

thanks @minyk

yaooqinn avatar Apr 28 '22 08:04 yaooqinn

@yaooqinn this is getting bigger than I thought. Currently, initial translation to kyuubi was done and start to write unit test.

May I send Working-In-Progress PR for discussion? Any restriction for WIP PR?

minyk avatar May 03 '22 03:05 minyk

feel free to send a WIP PR @minyk

yaooqinn avatar May 05 '22 02:05 yaooqinn

Both V2 commands/plans and Iceberg commands are covered in the current Authz implementation. This issue should be good to close.

bowenliang123 avatar Jan 10 '23 07:01 bowenliang123

thanks @bowenliang123

yaooqinn avatar Jan 10 '23 07:01 yaooqinn