[Bug] Iceberg ranger test case compatible error

[davidyuan1223]

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

Search before asking

• [x] I have searched in the issues and found no similar issues.

Describe the bug

Currently iceberg ranger test use hadoop type catalog to test the permission, but the hadoop type has some compatible questions with alter table rename command and create view command test case

  test("RENAME TABLE for Iceberg") {
    val table = s"$catalogV2.$namespace1.partitioned_table"
    val newTable = s"$catalogV2.$namespace1.renamed_table"
    withCleanTmpResources(Seq((table, "table"), (newTable, "table"))) {
      doAs(
        admin,
        sql(
          s"CREATE TABLE $table (id int NOT NULL, name string, city string) USING iceberg"))
      val renameTableSql = s"ALTER TABLE $table RENAME TO $newTable"
      interceptEndsWith[AccessControlException] {
        doAs(someone, sql(renameTableSql))
      }(s"does not have [alter] privilege on [$namespace1/partitioned_table]")
      doAs(admin, sql(renameTableSql))
    }
  }

  test("CREATE VIEW for Iceberg") {
    val table = s"$catalogV2.$namespace1.partitioned_table"
    val view = s"$catalogV2.$namespace1.test_view"
    withCleanTmpResources(Seq((table, "table"), (view, "view"))) {
      doAs(
        admin,
        sql(
          s"CREATE TABLE $table (id int NOT NULL, name string, city string) USING iceberg"))
      doAs(admin, sql(s"INSERT INTO $table VALUES (1, 'test', 'city')"))
      val createViewSql = s"CREATE VIEW $view AS SELECT * FROM $table"
      interceptEndsWith[AccessControlException] {
        doAs(someone, sql(createViewSql))
      }(s"does not have [create] privilege on [$namespace1]")
      doAs(admin, sql(createViewSql))
    }
  }

Affects Version(s)

master

Kyuubi Server Log Output

# Hadoop Type Catalog
Cannot rename Hadoop tables
java.lang.UnsupportedOperationException: Cannot rename Hadoop tables
---------------------------
Creating a view is not supported by catalog: local
java.lang.UnsupportedOperationException: Creating a view is not supported by catalog: local

# cancel Hadoop Type Catalog, use hive, code could see additional context
org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive Metastore
....
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
....
Caused by: MetaException(message:Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=/var/folders/dz/wcf2hy112t1_zw142j3lqr6c0000gn/T/hms-f1c6c83d-7435-4f0d-9641-7dbb392eee87;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: Failed to start database '/var/folders/dz/wcf2hy112t1_zw142j3lqr6c0000gn/T/hms-f1c6c83d-7435-4f0d-9641-7dbb392eee87' with class loader sun.misc.Launcher$AppClassLoader@18b4aac2, see the next exception for details.
....
Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /private/var/folders/dz/wcf2hy112t1_zw142j3lqr6c0000gn/T/hms-f1c6c83d-7435-4f0d-9641-7dbb392eee87.

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

remove the hadoop

  override def beforeAll(): Unit = {
    spark.conf.set(
      s"spark.sql.catalog.$catalogV2",
      "org.apache.iceberg.spark.SparkCatalog")
    spark.conf.set(
      s"spark.sql.catalog.$catalogV2.warehouse",
      Utils.createTempDir("iceberg-hadoop").toString)

Are you willing to submit PR?

• [ ] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• [x] No. I cannot submit a PR at this time.

[davidyuan1223]

@bowenliang123 @pan3793 @yaooqinn please check this issue, i cannot resolve it.

[yaooqinn]

Is hive catalog OK to use for this case?

[davidyuan1223]

Is hive catalog OK to use for this case?

I think it should be ok，but the derby(memory or file) will has some compatible errors.

[yaooqinn]

Let's give it a try?

[davidyuan1223]

Let's give it a try?

org.apache.iceberg.hive.RuntimeMetaException: Failed to connect to Hive Metastore .... Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient .... Caused by: MetaException(message:Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=/var/folders/dz/wcf2hy112t1_zw142j3lqr6c0000gn/T/hms-f1c6c83d-7435-4f0d-9641-7dbb392eee87;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------ java.sql.SQLException: Failed to start database '/var/folders/dz/wcf2hy112t1_zw142j3lqr6c0000gn/T/hms-f1c6c83d-7435-4f0d-9641-7dbb392eee87' with class loader sun.misc.Launcher$AppClassLoader@18b4aac2, see the next exception for details. .... Caused by: ERROR XSDB6: Another instance of Derby may have already booted the database /private/var/folders/dz/wcf2hy112t1_zw142j3lqr6c0000gn/T/hms-f1c6c83d-7435-4f0d-9641-7dbb392eee87.

This is the hive error

[yaooqinn]

This Derby means you try to initiate 2 db instances in the same directory, it can be fixed in many ways

[davidyuan1223]

This Derby means you try to initiate 2 db instances in the same directory, it can be fixed in many ways

I think you are right，but i have tried many ways，always report error

[pan3793]

Iceberg has a pool for the Hive client, disabling it might help.

[davidyuan1223]

Iceberg has a pool for the Hive client, disabling it might help.

sorry, i'm not good at derby, i cannot find the root case, there has a pr #7071 , i remove the hadoop type conf, you could see the error report

[baotran306]

Hi team @davidyuan1223 @yaooqinn @pan3793, I hope you're all doing well. Any updates on this issue?

[davidyuan1223]

Hi team @davidyuan1223 @yaooqinn @pan3793, I hope you're all doing well. Any updates on this issue?

Sorry. I'm busy recently. Maybe next month try to resolve it?

[nqvuong1998]

Hi @pan3793 @yaooqinn @davidyuan1223 , I observed that Iceberg-HadoopCatalog does not fully support Iceberg syntax, such as RENAME TABLE and CREATE VIEW. However, I attempted to switch to Iceberg-HiveCatalog (embedded Derby), but encountered a known issue: https://github.com/apache/iceberg/issues/7847#issuecomment-1694428156

[davidyuan1223]

@yaooqinn @pan3793 wanna ask a question，can we use MySQL testcontainers to be metastore to test?

[pan3793]

@davidyuan1223 Actually, I faced the same derby issue when working on https://github.com/apache/kyuubi/pull/7064, it also requires that Spark Hive Session catalog and Iceberg Catalog use the same HMS backend, I think either using a MySQL container or an HMS container is fine.

[davidyuan1223]

@davidyuan1223 Actually, I faced the same derby issue when working on #7064, it also requires that Spark Hive Session catalog and Iceberg Catalog use the same HMS backend, I think either using a MySQL container or an HMS container is fine. OK. I will try this recently