SecretManagerServiceAsyncClient.create_secret: Task got Future attached to a different loop

[rdong8]

Determine this is the right repository

• [X] I determined this is the correct repository in which to report this bug.

Summary of the issue

Context

I am trying to use async client for the GCP Secret Manager in my Telegram bot. I want to create, store and access the secret. I am running this bot in GKE autopilot.

Expected Behavior:

I expect to be able to create the secret asynchronously.

Actual Behavior:

When attempting to call create_secret, you get an error. This error only occurs when I use the secret manager together with the python-telegram-bot library, but both work on their own. This leads me to believe that one of them is not handling the asyncio loop correctly or making a new loop when it shouldn't be.

API client name and version

google-cloud-secret-manager 2.20.2

Reproduction steps: code

See here: https://github.com/rdong8/experiments

Reproduction steps: supporting files

Prerequisites:

• GKE autopilot cluster, or another cluster that has enough permissions to access GCP secret manager
• kubectl and helm, pointing to the cluster

1. git clone https://github.com/rdong8/experiments && cd experiments
2. Create a .env file with a Telegram BOT_TOKEN and GCP_PROJECT_ID - the schema is in the .env.template file. You can get a Telegram bot token by messaging BotFather. Do not use quotes around the values.
3. Create a secret using that file: kubectl create secret generic experiments-secret -n test --create-namespace --from-env-file=.env
4. Run from the repo root: helm install -n test --set image.tag=f1eede3 experiments charts/experiments
5. Get the pod ID: kubectl get pods -n test
6. Watch the logs of the bot: kubectl logs -n test -f pod/<POD ID>
7. Start a conversation with the bot you created on Telegram
8. Try to set a secret, ie. send the message /set mysecret
9. Observe the exception traceback in the logs

Reproduction steps: actual results

No error handlers are registered, logging exception.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/telegram/ext/_application.py", line 1335, in process_update
    await coroutine
  File "/usr/local/lib/python3.12/site-packages/telegram/ext/_handlers/basehandler.py", line 158, in handle_update
    return await self.callback(update, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/experiments/__main__.py", line 40, in set_secret
    await client.create_secret(secret_id=secret_id, parent=parent, secret=secret)
  File "/usr/local/lib/python3.12/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/async_client.py", line 539, in create_secret
    response = await rpc(
               ^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/google/api_core/grpc_helpers_async.py", line 85, in __await__
    response = yield from self._call.__await__()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/grpc/aio/_call.py", line 308, in __await__
    response = yield from self._call_response
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Task <Task pending name='Application:7717576672:update_fetcher' coro=<Application._update_fetcher() running at /usr/local/lib/python3.12/site-packages/telegram/ext/_application.py:1258>> got Future <Task pending name='Task-19' coro=<UnaryUnaryCall._invoke() running at /usr/local/lib/python3.12/site-packages/grpc/aio/_call.py:577>> attached to a different loop

Reproduction steps: expected results

No output (successfully create secret)

OS & version + platform

General-purpose compute class on GKE autopilot

Python environment

Python 3.12.6

Python dependencies

Package                     Version   Editable project location
--------------------------- --------- ------------------------------
anyio                       4.6.0
build                       1.2.2
cachetools                  5.5.0
certifi                     2024.8.30
cffi                        1.17.1
charset-normalizer          3.3.2
cryptography                43.0.1
docutils                    0.21.2
google-api-core             2.20.0
google-auth                 2.35.0
google-cloud-secret-manager 2.20.2
googleapis-common-protos    1.65.0
grpc-google-iam-v1          0.13.1
grpcio                      1.66.2
grpcio-status               1.66.2
h11                         0.14.0
httpcore                    1.0.6
httpx                       0.27.2
idna                        3.10
importlib_metadata          8.5.0
jaraco.classes              3.4.0
jaraco.context              6.0.1
jaraco.functools            4.1.0
jeepney                     0.8.0
keyring                     25.4.1
markdown-it-py              3.0.0
mdurl                       0.1.2
more-itertools              10.5.0
nh3                         0.2.18
packaging                   24.1
pip                         24.2
pkginfo                     1.10.0
proto-plus                  1.24.0
protobuf                    5.28.2
pyasn1                      0.6.1
pyasn1_modules              0.4.1
pycparser                   2.22
Pygments                    2.18.0
pyproject_hooks             1.2.0
python-telegram-bot         21.6
readme_renderer             44.0
requests                    2.32.3
requests-toolbelt           1.0.0
rfc3986                     2.0.0
rich                        13.9.1
rsa                         4.9
SecretStorage               3.3.3
sniffio                     1.3.1
twine                       5.1.1
urllib3                     2.2.3
zipp                        3.20.2