HDDS-5043. Can't list volume accurately in secure mode

[zhengchenyu]

This issue aims to fix the incorrect usage of UGI's getUserName in secure mode. In my test and search code, I Found we couldn't list volume accurately in secure mode.

[zhengchenyu]

@jojochuang Yeah, I check it except s3 (I don't know the usage about s3). I also found some wrong usage about ACL, but it doesn't mater, Because acl check both getUserName and getShortUserName. I only correct the code which will faill the unit test. If you think we need to correct all wrong usage, I will do it. Here are all wrong usage:

wrong usage function	description
ObjectStore.listVolumesByUser	fixed in this PR
RpcClient.createVolume	fixed in this PR
OzoneAclUtil.checkAclRights	fixed in this PR
RpcClient.getAclList	usage for set default acl. Becuase this acl user is owner. So it doesn't matter
OmKeyGenerator.createKey	usage for set default acl. Becuase this acl user is owner. So it doesn't matter
s3	I don't know the usage about s3, so I do not correct it.
BenchMark	I think there is no need to set security mode when benchmark before, so I don't correct it.

[zhengchenyu]

@jojochuang Can I ask another question? I don't know why this PR fail when check, though this patch work well in my laptop or server.

[jojochuang]

From a quick peek the failure might be related to the check change. I am trigger a rebuild to verify

[jojochuang]

@bharatviswa504 FYI we should review the getShortUserName usage in s3g.

benchmark -- it's going to be valuable to be able to benchmark in secure mode in the future. But let's do that in a separate jira.

[zhengchenyu]

@jojochuang Can we reopen this issue? Sorry for rebase this branch, so this issue was closed. And how about the progress about benchmark in secure mode?

[jojochuang]

i don't have the privilege to reopen the PR. Could you follow this instruction to reopen? https://gist.github.com/robertpainsi/2c42c15f1ce6dab03a0675348edd4e2c

[zhengchenyu]

i don't have the privilege to reopen the PR. Could you follow this instruction to reopen? https://gist.github.com/robertpainsi/2c42c15f1ce6dab03a0675348edd4e2c

Thank you very much! It works.

[jojochuang]

our tests have been quite flaky these days. i retriggered the build.

[adoroszlai]

@zhengchenyu failure in acceptance (secure) seems related to this change. Can you please check?

Can follow link with read access                                      | FAIL |
'PERMISSION_DENIED User testuser2/[email protected] doesn't have READ permission to access volume 71945-target null null' does not contain 'key-in-readable-bucket'

[zhengchenyu]

@zhengchenyu failure in acceptance (secure) seems related to this change. Can you please check?

Can follow link with read access                                      | FAIL |
'PERMISSION_DENIED User testuser2/[email protected] doesn't have READ permission to access volume 71945-target null null' does not contain 'key-in-readable-bucket'

Yes, I found it. the setup of test like below:

    Execute             ozone sh volume addacl --acl user:testuser2/[email protected]:r ${target}
    Execute             ozone sh volume addacl --acl user:testuser2/[email protected]:rl ${source}
    Execute             ozone sh bucket addacl --acl user:testuser2/[email protected]:rl ${source}/readable-bucket
    Execute             ozone sh bucket addacl --acl user:testuser2/[email protected]:r ${target}/readable-link
    Execute             ozone sh bucket addacl --acl user:testuser2/[email protected]:r ${target}/link-to-unreadable-bucket

I think it is reasonable that set acl with testuser2. But if someone set acl with testuser2/[email protected], I think we need make sure that this setting is only for testuser2/[email protected], but not for testuser2 in other host.

I have fix it, Let's wait to check.

[zhengchenyu]

Still run failed. error like below:

List root                                                             | FAIL |
Parent suite setup failed:
'{
  "metadata" : { },
  "name" : "27000-source",
  "admin" : "root",
  "owner" : "root",
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1,
  "usedNamespace" : 3,
  "creationTime" : "2021-08-06T09:11:43.411Z",
  "modificationTime" : "2021-08-06T09:12:18.682Z",
  "acls" : [ {
    "type" : "USER",
    "name" : "root",
    "aclScope" : "ACCESS",
    "aclList" : [ "ALL" ]
  }, {
    "type" : "GROUP",
    "name" : "root",
    "aclScope" : "ACCESS",
    "aclList" : [ "ALL" ]
    [ Message content over the limit has been removed. ]
  "name" : "vol-kaart",
  "admin" : "root",
  "owner" : "root",
  "quotaInBytes" : -1,
  "quotaInNamespace" : -1,
  "usedNamespace" : 1,
  "creationTime" : "2021-08-06T09:29:29.277Z",
  "modificationTime" : "2021-08-06T09:29:29.277Z",
  "acls" : [ {
    "type" : "USER",
    "name" : "root",
    "aclScope" : "ACCESS",
    "aclList" : [ "ALL" ]
  }, {
    "type" : "GROUP",
    "name" : "root",
    "aclScope" : "ACCESS",
    "aclList" : [ "ALL" ]
  } ]
}' does not match '"admin" : "(hadoop|testuser/scm[^@]*@EXAMPLE.COM)"'

I think here admin should be testuser, but not testuser/[email protected]. But I don't know why admin and owner is root in this test. Can you tell me where is the setting of "acceptance (secure)", then I will set the integeration test in my pc. @adoroszlai @jojochuang

[adoroszlai]

@zhengchenyu If you run acceptance (secure) check locally as

cd hadoop-ozone/dist/target/ozone-1.2.0-SNAPSHOT/compose/ozonesecure
KEEP_RUNNING=true ./test.sh

the docker cluster will be kept alive and you can investigate the problem.

[zhengchenyu]

@adoroszlai I found this config, and the ugi.getShortUserName was replace with root.

CORE-SITE.XML_hadoop.security.auth_to_local=RULE:[2:$1](testuser2.*) RULE:[2:$1@$0](.*)s/.*/root/

I don't know why use this config, I can't find the init commit, this is too old.

In CreateVolumeHandler.java, I changed code. Because I think owner and admin should be shortUserName, but not fullUserName. But the shortUserName was replace with root, so I allow root to check admin user. Please help me review, thanks.

[zhengchenyu]

@jojochuang @adoroszlai can you help reiview this? I have rebase all commit to f70217fa5a89194a54f94c62a0b55baada09d68b.

[kerneltime]

@zhengchenyu do you plan to rebase this PR to the latest code?