Deprecate Usage of `sslComonName` in Endpoint Creation

[dlm6693]

The Problem

Currently when creating a service client, an sslCommonName attribute may be used for endpoint construction in unique cases. The format of sslCommonName is typically {region}.{service}.{dnsSuffix}, as opposed to the more common {service}.{region}.{dnsSuffix}. This usage originated from a time where Python versions (<2.7) didn't supply an SSL module, requiring specific certificate formats.

Now that the library only support Python 3.7+, we'll be deprecating the usage of sslCommonName to standardize Boto3 with all other AWS SDKs. This will also resolve long running issues of services such as SQS and GuardDuty being incompatible with certain VPC endpoint configurations.

Required Actions

In the immediate term, we will start raising a deprecation warning when sslCommonName is used. This is to alert customers of the upcoming change and provide time to make any required changes.

For most users, this will not require any changes. The URL will automatically update when the next minor version (1.28.0) is released, and clients will continue to operate the same. For any users with strict network rules, explicitly allow listing domains, you will need to add support for {service}.{region}.{dnsSuffix} as demonstrated below:

Old Format: https://us-west-2.sqs.amazonaws.com New Format: https://sqs.us-west-2.amazonaws.com

Warning Mitigation Strategy

1. If you wish to ensure that your application does not use sslCommonName now or test the impending deprecation, we have created a new environment variable BOTO_DISABLE_COMMONNAME. Setting this to true will suppress the warning and convert to the new hostname format.
2. If you are concerned about this change causing disruptions, you can pin your version of botocore to <1.28.0 until you are ready to migrate.
3. If you are only concerned about silencing the warning in your logs, use warnings.filterwarnings when instantiating a new service client.

import warnings
warnings.filterwarnings('ignore', category=FutureWarning, module='botocore.client')

Other Information

Endpoint Docs: https://docs.aws.amazon.com/general/latest/gr/rande.html Related Issues: https://github.com/boto/botocore/issues/2376, https://github.com/boto/boto3/issues/1900, https://github.com/boto/boto3/issues/3311, https://github.com/boto/botocore/issues/2683

[bblommers]

The EMR client now gives the following warning:

The elasticmapreduce client is currently using a deprecated endpoint: elasticmapreduce.us-east-1.amazonaws.com. In the next minor version this will be moved to elasticmapreduce.us-east-1.amazonaws.com.

Note that the URL's are the same. What is the actual change for this service?

[dlm6693]

Oh interesting! Thank you for raising this. I guess for whatever reason the SSL common name for EMR is the same as the standard host name. It will follow the {service}.{region}.{dnsSuffix} convention as outlined above so it won't actually change. I don't think we will spend engineering resources to cherry pick common names that follow this pattern as they will be fully deprecated in the next minor version bump. Feel free to suppress this one or any others like it if needed. A decent workaround would be to add a regular expression to match the message in the message parameter of warnings.filterwarnings. Something like the following should do the trick.

warnings.filterwarnings(
    'ignore', 
    category=FutureWarning, 
    module='botocore.client', 
    message='.*elasticmapreduce.us-east-1.amazonaws.com.*'
)

[bblommers]

Awesome - thanks for the info @dlm6693!

[NotSoShaby]

Please notice that boto3 sqs client is creating queues with the soon to be deprecated format. We started using this:

    `queue = sqs_client.create_queue(QueueName=f"{org.name.lower()}-queue", tags={'ClientName': org.name})`

to create queues for our clients, but the returned queue is giving us this deprecaion warning:

FutureWarning: The sqs client is currently using a deprecated endpoint: queue.amazonaws.com. In the next minor version this will be moved to sqs.us-east-1.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.

boto3 should create a properly formatted queues a while before you break the old ones IMO

[nateprewitt]

Hi @NotSoShaby, both formats will continue to work for the foreseeable future. The only difference with this change is the hostname used for routing requests from Boto3. The new format can be accessed now with the BOTO_DISABLE_COMMONNAME environment variable if you wish to transition early.

The primary reason for the warning is to make sure users can make any firewall adjustments if needed since the domain is changing.

[dlm6693]

Hi @NotSoShaby if I'm understanding you correctly, you can create a SQS endpoint with the non-deprecated format by enabling the environment variable described above.

[NotSoShaby]

@nateprewitt thanks, will try it out. assuming it should be set to true? @dlm6693 no, I wasn't using the env variable flag will update once i tested it.

thanks for the quick response :)

[dlm6693]

Yep set to true

[lehmat]

I'm getting the deprecation warning even though the DNS name is taken from CloudFormation output of an AWS::SQS:Queue Ref value and have confirmed this environment variable to be set as: https://sqs.eu-north-1.amazonaws.com/XXXXX/FILTERED-SQSQueue-1MANR5OR9C65A on lambda which is then correctly read by our code. Is this by design?

[tikicoder]

I am getting the warning on rds FutureWarning: The rds client is currently using a deprecated endpoint: rds.amazonaws.com. In the next minor version this will be moved to rds.us-east-1.amazonaws.com. See https://github.com/boto/botocore/issues/2705 for more details.

I verified when I create the client I pass in the region

[dlm6693]

Hi @tikicoder. That is expected behavior. The sslCommonName template for this service and region is defined as {service}.{dnsSuffix} in endpoints.json. You can confirm the actual endpoint being used for operation calls by running the following.

import boto3
rds_client = boto3.client('rds', region_name='us-east-1')
rds_client._endpoint
> rds(https://rds.amazonaws.com)

[tikicoder]

@dlm6693 Thanks. I am presuming boto will be updated to fix the issue? It looks like I could by passing in an extra config, but that seems like overkill on something that I would presume should work since I am passing in all the required values. It is also the only one of the clients, I have used, so far that does this.

[dlm6693]

@tikicoder of course hence the deprecation warning and the creation of this issue. As documented at the top of this issue, you can disable the behavior now by setting the environment variable BOTO_DISABLE_COMMONNAME to true

[tikicoder]

@dlm6693 The warning doesn't bother me as long as I can explain it. Instead of setting an environment variable, I would have preferred a flag. I either have to manually set the environment variable or I have to tell people how to do that. Fortunately, I have spun up my own boto helper system so it's not that big of an issue.

Thanks for the quick replies

[gheorghina]

hello, I am new to python botocore and the ecosystem; I am trying to run localstack and celery workers via docker compose, and I get similar error: botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://eu-east-1.queue.amazonaws.com/" in /usr/local/lib/python3.8/site-packages/celery/worker/consumer/consumer.py:367: CPendingDeprecationWarning

how would you advice to adjust the configurations?

Thank you

[dlm6693]

@gheorghina I suspect that is because the eu-east-1 region does not exist. Here is a list of available regions https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions

[gheorghina]

for sure I am missing some configurations, as you are right, I expected it to be eu-central-1, I have no idea how botocore solves it to eu-east-1

[dlm6693]

@gheorghina if you continue to run into problems feel free to open up a new issue. It seems to be unrelated to this one. I have no issue connecting to SQS using eu-central-1. Please add boto3.set_stream_logger() to your script and share the output.

[vbhjckfd]

I got a warning

/usr/local/lib/python3.10/site-packages/botocore/client.py:625: FutureWarning: The sqs client is currently using a deprecated endpoint: eu-west-1.queue.amazonaws.com. In the next minor version this will be moved to sqs.eu-west-1.amazonaws.com.

But all of my endpoints are already using service.zone.amazonaws.com format and specifically eu-west-1.queue.amazonaws.com isn't findable anywhere :(

[dlm6693]

@vbhjckfd have you tried using the environment variable described above?

[vbhjckfd]

@vbhjckfd have you tried using the environment variable described above?

Setting env variable BOTO_DISABLE_COMMONNAME to true eliminates warning and still keeps my script working.

[Taragolis]

Just wondering is botocore>=1.28.0 still use sslCommonName by default? And for disable it still required to set env BOTO_DISABLE_COMMONNAME=true

[dlm6693]

@Taragolis yes. We'll need to update this document, but only a select group of services (s3 and s3-control) have had the new methodology of endpoint resolution enabled. The rest will come soon, but until that time, sslCommonName will be used by default when defined for all other services.

[dlm6693]

After conferring with the team, BOTO_DISABLE_COMMNAME will no longer be needed to suppress the deprecation warning for the NEXT minor version 1.29.0. That will be released in the near future. I've updated the issue accordingly.

[Taragolis]

@dlm6693 many thanks for detail explanation!

[nateprewitt]

Resolving now that we've had botocore 1.29.x out for a couple weeks. Please feel free to let us know if you encounter any issues but the migration should be complete at this point.