Failed to create a new table thru Accumulo Client library (org.apache.accumulo.core.client.AccumuloClient) in Kerberos-enabled Accumulo cluster

[sjyang18]

Describe the bug Accumulo Client should be able to support the same API in both non-kerberos and kerberos enabled accumulo cluster. I was able to use Kerberos key tab and principal to create a table, inject data, drop the table thru Accumulo shell. When I do the same in Java Accumulo Client, my java program may list user, tables, and inject data to an existing table, but failed to create a new table with the following error in client side.

[main} DEBUG org.apache.accumulo.examples.helloworld.InsertWithKerberos  - Failed to create table
org.apache.accumulo.core.client.AccumuloException: Internal error processing beginFateOperation
        at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:402)
        at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:354)
        at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doTableFateOperation(TableOperationsImpl.java:1692)
        at org.apache.accumulo.core.clientImpl.TableOperationsImpl.create(TableOperationsImpl.java:245)
        at org.apache.accumulo.examples.helloworld.InsertWithKerberos.lambda$main$1(InsertWithKerberos.java:105)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
        at org.apache.accumulo.examples.helloworld.InsertWithKerberos.main(InsertWithKerberos.java:98)
Caused by: org.apache.thrift.TApplicationException: Internal error processing beginFateOperation
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79)
        at org.apache.accumulo.core.manager.thrift.FateService$Client.recv_beginFateOperation(FateService.java:91)
        at org.apache.accumulo.core.manager.thrift.FateService$Client.beginFateOperation(FateService.java:77)
        at org.apache.accumulo.core.clientImpl.TableOperationsImpl.beginFateOperation(TableOperationsImpl.java:258)
        at org.apache.accumulo.core.clientImpl.TableOperationsImpl.doFateOperation(TableOperationsImpl.java:364)
        ... 8 more

On the manager side, I see the matching error with NPE.

2021-04-23T00:26:04,450 [thrift.ProcessFunction] ERROR: Internal error processing beginFateOperation
java.lang.NullPointerException: null
        at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:?]
        at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.getTokenClassFromName(TCredentialsUpdatingInvocationHandler.java:158) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.updateArgs(TCredentialsUpdatingInvocationHandler.java:95) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.server.rpc.TCredentialsUpdatingInvocationHandler.invoke(TCredentialsUpdatingInvocationHandler.java:62) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at com.sun.proxy.$Proxy41.beginFateOperation(Unknown Source) ~[?:?]
        at org.apache.accumulo.core.master.thrift.FateService$Processor$beginFateOperation.getResult(FateService.java:421) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.core.master.thrift.FateService$Processor$beginFateOperation.getResult(FateService.java:400) ~[accumulo-core-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:38) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) ~[libthrift-0.12.0.jar:0.12.0]
        at org.apache.accumulo.server.rpc.UGIAssumingProcessor.process(UGIAssumingProcessor.java:104) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.accumulo.server.rpc.TimedProcessor.process(TimedProcessor.java:63) ~[accumulo-server-base-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:310) ~[libthrift-0.12.0.jar:0.12.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:834) [?:?]

Versions (OS, Maven, Java, and others, as appropriate):

• Affected version(s) of this project: [e.g. Accumulo 2.1.0-SNAPSHOT]
• OS: [e.g. CentOS 7.5]
• Others:

To Reproduce Steps to reproduce the behavior (or a link to an example repository that reproduces the problem):

1. Start with 'https://github.com/apache/accumulo-examples/blob/main/src/main/java/org/apache/accumulo/examples/helloworld/Insert.java'.
2. Change code like the following to use kerberos principal and keytab file you have setup for kerberos authentication.

    try {

      final Configuration hadoopConf = new Configuration(false);
      hadoopConf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHENTICATION, "kerberos");
      UserGroupInformation.setConfiguration(hadoopConf);
      UserGroupInformation.loginUserFromKeytab(principal, keytabPath);

      final File keytabFile = new File(keytabPath);
      final KerberosToken ktoken = new KerberosToken(principal, keytabFile);
      AccumuloClient aclient = Accumulo.newClient().from(opts.getClientPropsPath())
          .as(principal, ktoken).useSasl().build();
      boolean authenticated = aclient.securityOperations().authenticateUser(principal, ktoken);
      log.debug("AccumuloClient instance created with : " + aclient.whoami() + " for Kerberos, "
          + authenticated);

      for (SystemPermission perm : SystemPermission.values()) {
        log.debug("Expected user to have permission: " + perm,
            aclient.securityOperations().hasSystemPermission(aclient.whoami(), perm));
      }
      aclient.tableOperations().create("hellotable");
      aclient.close();
    } catch (Exception e) {
      log.debug("Exception caught in Main....");
      log.debug(e.getMessage());
    }

3. Change accumulo-client.properties to have kerberos authentication case. Test environment and configuration with 'accumulo shell' to verify. After verification, drop the test table.

4. Compile the changed code and run the same while you tail the manager log on your head node. You will see the similar failure exceptions I shared above with NPE.

Expected behavior Accumulo client should be able to create a table in Accumulo.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context I reviewed the Accumulo Shell code and it is also using the same client library, but not sure what additional thing(s) Shell class is doing in order to make the creation work.