apache
HADOOP-19384. S3A: Add support for ProfileCredentialsProvider (#7284)
Description of PR
This commit adds a wrapper for the AWS ProfileCredentialsProvider.
How was this patch tested?
The patch was tested by running the hadoop-aws integration tests with fs.s3a.aws.credentials.provider and fs.s3a.assumed.role.credentials.provider configured to only include org.apache.hadoop.fs.s3a.ProfileAWSCredentialsProvider. Buckets/endpoints used were in the us-east-1 region. 4 test failures that seem unrelated to this change have details in the JIRA.
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 31m 34s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 0s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 0s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 0s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 0s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 1 new or modified test files. | |
| _ branch-3.4 Compile Tests _ | ||||
| +0 :ok: | mvndep | 1m 58s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 32m 50s | branch-3.4 passed | |
| +1 :green_heart: | compile | 16m 37s | branch-3.4 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | compile | 15m 4s | branch-3.4 passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 | |
| +1 :green_heart: | checkstyle | 4m 51s | branch-3.4 passed | |
| +1 :green_heart: | mvnsite | 2m 39s | branch-3.4 passed | |
| +1 :green_heart: | javadoc | 1m 53s | branch-3.4 passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 41s | branch-3.4 passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 | |
| +1 :green_heart: | spotbugs | 3m 51s | branch-3.4 passed | |
| +1 :green_heart: | shadedclient | 33m 14s | branch has no errors when building and testing our client artifacts. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 34s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 1m 21s | the patch passed | |
| +1 :green_heart: | compile | 16m 41s | the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javac | 16m 41s | the patch passed | |
| +1 :green_heart: | compile | 15m 34s | the patch passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 | |
| +1 :green_heart: | javac | 15m 34s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 3s | the patch passed | |
| +1 :green_heart: | mvnsite | 2m 15s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 36s | the patch passed with JDK Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 | |
| +1 :green_heart: | javadoc | 1m 32s | the patch passed with JDK Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 | |
| +1 :green_heart: | spotbugs | 3m 58s | the patch passed | |
| +1 :green_heart: | shadedclient | 37m 55s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 19m 20s | hadoop-common in the patch passed. | |
| +1 :green_heart: | unit | 3m 8s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 0s | The patch does not generate ASF License warnings. | |
| 259m 18s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.49 ServerAPI=1.49 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7690/1/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/7690 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 6ed3d234e957 5.15.0-139-generic #149-Ubuntu SMP Fri Apr 11 22:06:13 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / dd133014e95ded6e35c6a8a62ead0ca4b7b9d416 |
| Default Java | Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.27+6-post-Ubuntu-0ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_452-8u452-ga~us1-0ubuntu1~20.04-b09 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7690/1/testReport/ |
| Max. process+thread count | 1259 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7690/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
I am not seeing the stack trace you reported, FWIW. just ITestS3AContractOpen>AbstractContractOpenTest.testInputStreamReadNegativePosition, which is known about
@steveloughran Then it might just be an issue with my local setup? There is nothing else from my end to report if the automated tests are all passing.
put those stack traces up. failues in different configurations are always interesting -and finding them during development is better than finding them in production.
do they go away when you use the normal credential providers?
@steveloughran I did post the stacktraces in the JIRA earlier, I'd assumed you were responding after seeing those, guess it fell between the cracks. I'll have to change my setup to use the normal credentials providers, we can continue the conversation in the JIRA once I have those tests run.
thanks. FWIW you can just put the traces into this PR, helps give a history...that is probably why I missed them.
If it isn't credentials or permissions, then it'll be configuration. But we need to understand what, whether some changes can make the problems go away, whether the troubleshooting.md doc needs expansion and more.
@steveloughran Managed to run the ITests with the SimpleAWSCredentialsProvider, and the aforementioned failures don't show up (aside from the timeout error). There is a different test that fails with this stacktrace:
[ERROR] testIfMatchOverwriteWithOutdatedEtag(org.apache.hadoop.fs.s3a.impl.ITestS3APutIfMatchAndIfNoneMatch) Time elapsed: 3.164 s <<< FAILURE! java.lang.AssertionError: Expected a org.apache.hadoop.fs.s3a.RemoteFileChangedException to be thrown, but got the result: : S3AFileStatus{path=s3a://vn7-test-bucket/job-00-fork-0008/test/testIfMatchOverwriteWithOutdatedEtag; isDirectory=false; length=1024; replication=1; blocksize=33554432; modification_time=1750712397000; access_time=0; owner=vnarayanan; group=vnarayanan; permission=rw-rw-rw-; isSymlink=false; hasAcl=false; isEncrypted=true; isErasureCoded=false} isEmptyDirectory=FALSE eTag="4340256b04f80df42c1a89c65a60d35d" versionId=null at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:501) at org.apache.hadoop.test.LambdaTestUtils.intercept(LambdaTestUtils.java:386) at org.apache.hadoop.fs.s3a.impl.ITestS3APutIfMatchAndIfNoneMatch.testIfMatchOverwriteWithOutdatedEtag(ITestS3APutIfMatchAndIfNoneMatch.java:476) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.lang.Thread.run(Thread.java:750)