WARNING: Connection pool is full

[lightiverson]

Issue Summary

We're using the twilio-python library to send out thousands of SMS messages concurrently using Python's concurrent features library. During usage the Logging module gives us a warning saying: urllib3.connectionpool:_put_conn:304 - Connection pool is full, discarding connection: api.twilio.com.

Steps to Reproduce

1. Get an array of valid phonenumbers
2. Map the client.message.create function using concurrent.futures
3. Run the program with Logging

Code Snippet

def send_session_message():
      message = client.messages.create(
          body=body,
          from_=from_,
          to=[recipient],
          persistent_action=coordinates,
      )

with concurrent.futures.ThreadPoolExecutor() as executor:
      results = executor.map(
          send_session_message,
          recipients,
          repeat(body),
          repeat(from_),
          repeat(messaging_service_sid),
      )

Exception/Log

# WARNING  | urllib3.connectionpool:_put_conn:304 - Connection pool is full, discarding connection: api.twilio.com

Technical details:

• twilio-python version: 7.1.0
• python version: 3.8.12

[charan678]

hello @lightiverson The issue is with your implementation of connection pool logic nothing to do with twilio-python. you can try blocking new connection when connection pool is full. you can look into python/urllib documentation and find the solution to handle connection pool full issue. This link might be helpful to you.

[charan678]

@lightiverson can you please update to latest version of twilio-python. We are not supporting 7.X version.

[lightiverson]

hi @charan678 thanks for the link! We came across this information and handled the connection pool full issue in the TwilioHttpClient like so:

This seems to have solved the problem. However, you mentioned that the issue has nothing to do with twilio-python. Is there a better solution which does not involve editing the twilio-python package?

[lightiverson]

@charan678 as for the package version, you're right we will update to the latest version asap. Thank you.

[charan678]

@lightiverson We can't remove this snippet, It limits maximum pool size. you can set max_worker parameter in concurrent.futures.ThreadPoolExecutor.

[lightiverson]

@charan678 it seems there's a misunderstanding. The highlighted snippet, is what we have added locally to the twilio-python source code to solve this issue. It currently does not exist in your repository.

Would you like us to make a pull request to have this snippet added?

[charan678]

@lightiverson Sorry for misunderstanding, you can raise the pull request. It will take next 10-11 days to be visible in next release. Meanwhile, you can change max_workers to equal or less than default pool size of urllib which 10.

[lightiverson]

@charan678 No worries! Got it, we'll raise the pull request shortly ~~,with the max_workers set to equal or less than 10~~. Thanks for your help.

Edit: Realized the max workers bit was about ThreadPoolExecutor() and not about HTTPAdapter().