aws
AWSSecurityTokenServiceClient.doInvoke goes into infinite loop
Describe the bug
I am trying to access a aws secret from a spring boot application with assumedRole setup . The application is ran from databricks.
Using dependency :
<dependency>
<groupId>io.awspring.cloud</groupId>
<artifactId>spring-cloud-starter-aws-secrets-manager-config</artifactId>
<version>2.4.1</version>
</dependency>
aws-java-sdk-sts:1.12.195
STSProfileCredentialsService with null webindentity token.
Expected Behavior
Ideally it should have assumed the role fetch the creds.
Current Behavior
It goes into infinite loop resulting into stackOverflowError
Getting an infinite loop error.
java.lang.StackOverflowError
at java.util.LinkedList$ListItr.
Reproduction Steps
/**
* Invoke the request using the http client. Assumes credentials (or lack thereof) have been configured in the
* ExecutionContext beforehand.
**/
private <X, Y extends AmazonWebServiceRequest> Response<X> doInvoke(Request<Y> request, HttpResponseHandler<AmazonWebServiceResponse<X>> responseHandler,
ExecutionContext executionContext, URI discoveredEndpoint, URI uriFromEndpointTrait) {
if (discoveredEndpoint != null) {
request.setEndpoint(discoveredEndpoint);
request.getOriginalRequest().getRequestClientOptions().appendUserAgent("endpoint-discovery");
} else if (uriFromEndpointTrait != null) {
request.setEndpoint(uriFromEndpointTrait);
} else {
request.setEndpoint(endpoint);
}
request.setTimeOffset(timeOffset);
DefaultErrorResponseHandler errorResponseHandler = new DefaultErrorResponseHandler(exceptionUnmarshallers);
return client.execute(request, responseHandler, errorResponseHandler, executionContext);
}
in AWSSecurityTokenServiceClient class
Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
1.12.195
JDK version used
Jdk1.8
Operating System and version
Databricks
Note that this issue is reproducible with pure AWS SDK without Spring Cloud AWS involved.
https://github.com/maciej-scratches/aws-sts-infinite-loop
Perhaps there is some configuration missing?
Hi @DineshNO @maciejwalkowiak thank you for reaching out, I can reproduce. Will investigate.
Some findings:
- The error does not occur when all three Web Identity Provider environment variables are set (AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_SESSION_NAME)
- The error does not happen in Java SDK 2.x.
- The error started in Java SDK version
1.11.704, which according to the changelog:
1.11.704 - 2020-01-09 AWS SDK for Java Bugfixes
- Increase the priority of the AWS_WEB_IDENTITY_TOKEN_FILE/AWS_ROLE_ARN/AWS_ROLE_SESSION_NAME environment variables when loading credentials so that they are considered before web_identity_token_file/role_arn/role_session_name profile properties. This is consistent with the other AWS SDKs, including the CLI.
@DineshNO Is there any particular reason why only AWS_ROLE_ARN is set and not the other variables? The three parameters are needed to call AssumeRoleWithWebIdentity.
Hi @debora-ito My application is not deployed to any EC2 instance.Its just an executable jar which i am trying to run from Databricks where Instance profile is set and i am trying to assume a role.
I do not have any details reg. the AWS_WEB_IDENTITY_TOKEN_FILE .
So when i checked the code i found AWS_WEB_IDENTITY_TOKEN_FILE not required for STSAssumeRoleSessionCredentialsProvider which is the most popular way of setting up assume role when i checked online .So i believed this should work.
STSProfileCredentialsService.java
public class STSProfileCredentialsService implements ProfileCredentialsService { @Override public AWSCredentialsProvider getAssumeRoleCredentialsProvider(RoleInfo targetRoleInfo) { if (targetRoleInfo.getWebIdentityTokenFilePath() == null) { return new STSAssumeRoleSessionCredentialsProvider.Builder(targetRoleInfo.getRoleArn(), targetRoleInfo.getRoleSessionName()) .withLongLivedCredentialsProvider(targetRoleInfo.getLongLivedCredentialsProvider()) .withExternalId(targetRoleInfo.getExternalId()) .build(); } else { return new STSAssumeRoleWithWebIdentitySessionCredentialsProvider.Builder( targetRoleInfo.getRoleArn(), targetRoleInfo.getRoleSessionName(), targetRoleInfo.getWebIdentityTokenFilePath()) .build(); } } }
Hi @debora-ito , just another observation . In version 1.11.700, the order of providers changed in DefaultAWSCredentialsProviderChain . ProfileCredentialsProvider is coming before WebIdentityTokenCredentialsProvider.create().
this changed in 704 , which i think commented as priority changed.
Upgrading to the latest aws-java-sdk-sts 1.x version fixed this for us based on https://github.com/aws/aws-sdk-java/issues/2136
UPDATE: This was fixed by remembering to add the aws-java-sdk-sts dependency (but the failure mode could have been more informative!)
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sts</artifactId>
<version>${aws-sdk.version}</version>
</dependency>
Managed to repro this with 1.11.699 running in an EKS pod with an IAM assumable role and the following Java code:
WebIdentityTokenCredentialsProvider credentialsProvider = WebIdentityTokenCredentialsProvider.create();
log.debug("Successfully fetched credentials");
log.debug("Credential details: AWS_ACCESS_KEY_ID: "
+ credentialsProvider.getCredentials().getAWSAccessKeyId());
In the logs I see the following:
15:36:00.437 [main] DEBUG c.i.e.config.SecretsJsonConfig - Successfully fetched credentials
15:36:00.450 [main] DEBUG com.amazonaws.AmazonWebServiceClient - Internal logging successfully configured to commons logger: true
15:36:00.527 [main] DEBUG c.a.internal.config.InternalConfig - Configuration override awssdk_config_override.json not found.
15:36:00.779 [main] DEBUG c.a.m.CsmConfigurationProviderChain - Unable to load configuration from com.amazonaws.monitoring.EnvironmentVariableCsmConfigurationProvider@17b016ac: Unable to load Client Side Monitoring configurations from environment variables!
15:36:00.779 [main] DEBUG c.a.m.CsmConfigurationProviderChain - Unable to load configuration from com.amazonaws.monitoring.SystemPropertyCsmConfigurationProvider@733ec58b: Unable to load Client Side Monitoring configurations from system properties variables!
15:36:00.780 [main] DEBUG c.a.m.CsmConfigurationProviderChain - Unable to load configuration from com.amazonaws.monitoring.ProfileCsmConfigurationProvider@415ef4d8: Unable to load config file
15:36:00.813 [main] DEBUG com.amazonaws.metrics.AwsSdkMetrics - Admin mbean registered under com.amazonaws.management:type=AwsSdkMetrics
15:36:00.833 [main] DEBUG c.a.auth.AWSCredentialsProviderChain - Unable to load credentials from EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
15:36:00.834 [main] DEBUG c.a.auth.AWSCredentialsProviderChain - Unable to load credentials from SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey)
15:36:00.834 [main] DEBUG c.a.auth.AWSCredentialsProviderChain - Unable to load credentials from com.amazonaws.auth.profile.ProfileCredentialsProvider@1d0cac30: profile file cannot be null
15:36:00.835 [main] DEBUG c.a.m.CsmConfigurationProviderChain - Unable to load configuration from com.amazonaws.monitoring.EnvironmentVariableCsmConfigurationProvider@17b016ac: Unable to load Client Side Monitoring configurations from environment variables!
15:36:00.835 [main] DEBUG c.a.m.CsmConfigurationProviderChain - Unable to load configuration from com.amazonaws.monitoring.SystemPropertyCsmConfigurationProvider@733ec58b: Unable to load Client Side Monitoring configurations from system properties variables!
15:36:00.835 [main] DEBUG c.a.m.CsmConfigurationProviderChain - Unable to load configuration from com.amazonaws.monitoring.ProfileCsmConfigurationProvider@415ef4d8: Unable to load config file
15:36:00.838 [main] DEBUG c.a.auth.AWSCredentialsProviderChain - Unable to load credentials from EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
And then the following is repeated many many times:
15:36:00.838 [main] DEBUG c.a.auth.AWSCredentialsProviderChain - Unable to load credentials from SystemPropertiesCredentialsProvider: Unable to load AWS credentials from Java system properties (aws.accessKeyId and aws.secretKey)
15:36:00.838 [main] DEBUG c.a.auth.AWSCredentialsProviderChain - Unable to load credentials from com.amazonaws.auth.profile.ProfileCredentialsProvider@1d0cac30: profile file cannot be null
15:36:00.839 [main] DEBUG c.a.auth.AWSCredentialsProviderChain - Unable to load credentials from EnvironmentVariableCredentialsProvider: Unable to load AWS credentials from environment variables (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY))
And then the relevant part of the stack trace is this:
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:129)
followed by this repeated a bunch of times:
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.getCredentialsFromContext(AmazonHttpClient.java:1239)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.runBeforeRequestHandlers(AmazonHttpClient.java:815)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:765)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:752)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:726)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:668)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:532)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:512)
at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1271)
at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1247)
at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:454)
at com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:431)
at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:319)
at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:34)
at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:74)
at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:71)
at com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:260)
at com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:216)
at com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:157)
at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:297)
at com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:33)
at com.amazonaws.auth.profile.internal.securitytoken.STSProfileCredentialsServiceProvider.getCredentials(STSProfileCredentialsServiceProvider.java:71)
at com.amazonaws.auth.WebIdentityTokenCredentialsProvider.getCredentials(WebIdentityTokenCredentialsProvider.java:72)
at com.amazonaws.auth.AWSCredentialsProviderChain.getCredentials(AWSCredentialsProviderChain.java:117)
