HIVE-16913: Support per-session S3 credentials

[zhangbutao]

What changes were proposed in this pull request?

1. HMS use Warehouse to delete/create hdfs dir, so we should pass s3 credentials to Warehouse conf. Adding threadlocal conf in Warehouse so that we can pass session-level conf to Warehouse synchronously. e.g. set metaconf:fs.s3a.secret.key=my_secret_key;
2. Make S3 related credentials param valid in MetastoreConf.ConfVars

Why are the changes needed?

Currently, users can only set S3 credentials on the HMS server side and users cannot set different S3 credentials for multiple S3 buckets. With this change, users can set per-session S3 credentials in client side. e.g, we can set S3 credentials like this from beeline client:

set fs.s3a.secret.key=my_secret_key; set fs.s3a.access.key=my_access.key; set fs.s3a.endpoint=my_endpoint; set metaconf:fs.s3a.secret.key=my_secret_key; set metaconf:fs.s3a.access.key=my_access_key; set metaconf:fs.s3a.endpoint=my_endpoint;

Does this PR introduce any user-facing change?

NO

How was this patch tested?

Test with multiple s3 buckets and credentials in local env.

[shameersss1]

Can we a have a test (either unit test or qtest) for this? with a dummy keys and printing it somewhere

[zhangbutao]

Can we a have a test (either unit test or qtest) for this? with a dummy keys and printing it somewhere

@shameersss1 Thx for your advice. This PR seems difficult to test using current test framework. Although HIVE-14373 has added integration tests for hive on S3, it is mainly tested using global ak & sk. However, this PR is aimed for dynamic ak & sk, and I need set the ak & sk parameters using hive set command. Maybe i need sometime to explore how to test this scenario...

[sonarqubecloud[bot]]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

No Coverage information
No Duplication information

[github-actions[bot]]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Feel free to reach out on the [email protected] list if the patch is in need of reviews.

[sonarqubecloud[bot]]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

No Coverage information
No Duplication information

[Aggarwal-Raghav]

@zhangbutao, can we reborn this PR for master branch. This is a much needed patch. Also, can we expose fs.s3a.security.credential.provider.path in conf as well, it is another way to pass the credentials.

[zhangbutao]

@zhangbutao, can we reborn this PR for master branch. This is a much needed patch. Also, can we expose fs.s3a.security.credential.provider.path in conf as well, it is another way to pass the credentials.

@Aggarwal-Raghav Thank you for your interest in this feature. I will try to submit a new PR based latest master branch ASAP.

[Aggarwal-Raghav]

@zhangbutao, can we reborn this PR for master branch. This is a much needed patch. Also, can we expose fs.s3a.security.credential.provider.path in conf as well, it is another way to pass the credentials.

@Aggarwal-Raghav Thank you for your interest in this feature. I will try to submit a new PR based latest master branch ASPAP.

Thanks

[zhangbutao]

hi @Aggarwal-Raghav , i just submitted a new PR https://github.com/apache/hive/pull/5257. :)