apache
HADOOP-18073. Adds in s3 client config and updates list.
Description of PR
Draft PR for first set of sdk upgrade changes
How was this patch tested?
Thanks @dannycjones, have moved those things to constants.
RE using eu-west-1 and the region logic in general:
Not using us-east-1 as headBucket() fails with that region. This is because us-east-1 uses the endpoint s3.amazonaws.com, which resolves bucket.s3.amazonaws.com to the actual region the bucket is in. As the request is signed with us-east-1 and not the bucket's region, it fails. For more info, see this issue.
I'm not sure how the region logic should behave and how it should handle failures. Returning us-east-1 at the end here is not of much use, as without cross region access, if the region is configured incorrectly, any request to S3 will fail. Instead, to handle network failures etc, we should probably add some retry logic in this method.
I'm also not sure if this new region/endpoint logic is sufficient to handle third party stores, keen to know what other people think.
@steveloughran / @mukund-thakur , I've opened a PR with the first set of changes for the SDK upgrade, could you please take a look?
:confetti_ball: +1 overall
| Vote | Subsystem | Runtime | Logfile | Comment |
|---|---|---|---|---|
| +0 :ok: | reexec | 0m 53s | Docker mode activated. | |
| _ Prechecks _ | ||||
| +1 :green_heart: | dupname | 0m 1s | No case conflicting files found. | |
| +0 :ok: | codespell | 0m 1s | codespell was not available. | |
| +0 :ok: | detsecrets | 0m 1s | detect-secrets was not available. | |
| +0 :ok: | xmllint | 0m 1s | xmllint was not available. | |
| +1 :green_heart: | @author | 0m 0s | The patch does not contain any @author tags. | |
| +1 :green_heart: | test4tests | 0m 0s | The patch appears to include 5 new or modified test files. | |
| _ feature-HADOOP-18073-s3a-sdk-upgrade Compile Tests _ | ||||
| +0 :ok: | mvndep | 14m 32s | Maven dependency ordering for branch | |
| +1 :green_heart: | mvninstall | 28m 26s | feature-HADOOP-18073-s3a-sdk-upgrade passed | |
| +1 :green_heart: | compile | 25m 26s | feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | compile | 22m 0s | feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | checkstyle | 4m 25s | feature-HADOOP-18073-s3a-sdk-upgrade passed | |
| +1 :green_heart: | mvnsite | 1m 51s | feature-HADOOP-18073-s3a-sdk-upgrade passed | |
| +1 :green_heart: | javadoc | 1m 35s | feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 1m 37s | feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +0 :ok: | spotbugs | 0m 59s | branch/hadoop-project no spotbugs output file (spotbugsXml.xml) | |
| +1 :green_heart: | shadedclient | 24m 9s | branch has no errors when building and testing our client artifacts. | |
| -0 :warning: | patch | 24m 36s | Used diff version of patch file. Binary files and potentially other changes not applied. Please rebase and squash commits if necessary. | |
| _ Patch Compile Tests _ | ||||
| +0 :ok: | mvndep | 0m 41s | Maven dependency ordering for patch | |
| +1 :green_heart: | mvninstall | 0m 50s | the patch passed | |
| +1 :green_heart: | compile | 24m 33s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javac | 24m 33s | the patch passed | |
| +1 :green_heart: | compile | 22m 18s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +1 :green_heart: | javac | 22m 18s | the patch passed | |
| +1 :green_heart: | blanks | 0m 0s | The patch has no blanks issues. | |
| +1 :green_heart: | checkstyle | 4m 17s | the patch passed | |
| +1 :green_heart: | mvnsite | 1m 51s | the patch passed | |
| +1 :green_heart: | javadoc | 1m 29s | the patch passed with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 | |
| +1 :green_heart: | javadoc | 1m 38s | the patch passed with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 | |
| +0 :ok: | spotbugs | 0m 45s | hadoop-project has no data from spotbugs | |
| +1 :green_heart: | shadedclient | 24m 14s | patch has no errors when building and testing our client artifacts. | |
| _ Other Tests _ | ||||
| +1 :green_heart: | unit | 0m 44s | hadoop-project in the patch passed. | |
| +1 :green_heart: | unit | 3m 1s | hadoop-aws in the patch passed. | |
| +1 :green_heart: | asflicense | 1m 9s | The patch does not generate ASF License warnings. | |
| 220m 4s |
| Subsystem | Report/Notes |
|---|---|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4706/11/artifact/out/Dockerfile |
| GITHUB PR | https://github.com/apache/hadoop/pull/4706 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux bc4dbb7be277 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4 01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18073-s3a-sdk-upgrade / 2ad9c81486e334ea644041e2735dd3c030e49d69 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4706/11/testReport/ |
| Max. process+thread count | 590 (vs. ulimit of 5500) |
| modules | C: hadoop-project hadoop-tools/hadoop-aws U: . |
| Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4706/11/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
closing this, these changes will be part of a new PR I'll open once the feature branch is in sync with trunk.