Cloud Secrets Manager often errors DEADLINE Exceeded

[0xterran]

⏳ Secrets Manager retrieved in 31.092 seconds

So slow!

---

Yes I have tried the quick fixes and google cloud docs & github/stackoverflow search.

Environment details

• which product (packages/*): @google-cloud/secret-manager
• OS: Linux Google Cloud Run
• Node.js version: v18
• npm version: 9.5
• google-cloud-node version: 5.0.1

Steps to reproduce

About 25% of the time, cloud secrets manager will timeout when attempting to access a secret. I have logged avg duration to get a secret and its around 10-30 seconds on first attempt (and <1 sec thereafter). The code is run within firebase functions and cloud run on us-central1. All my secrets managers are set to auto-replication policy.

What is causing this issue and how can I fix it? It's severely critical.

The pattern I am using:

const secret = await getSecretFromGCloud() // this takes 10+ seconds
axios.post(url, payload, headers)

Error:

Error: 4 DEADLINE_EXCEEDED: Deadline exceeded at callErrorFromStatus (/workspace/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/call.js:31:19) at Object.onReceiveStatus (/workspace/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/client.js:192:76) at Object.onReceiveStatus (/workspace/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:360:141) at Object.onReceiveStatus (/workspace/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/client-interceptors.js:323:181) at /workspace/node_modules/@google-cloud/secret-manager/node_modules/@grpc/grpc-js/build/src/resolving-call.js:99:78 at processTicksAndRejections (node:internal/process/task_queues:77:11) at runNextTicks (node:internal/process/task_queues:64:3) at process.processTimers (node:internal/timers:509:9)

[0xterran]

My most recent logs:

Secrets Manager retrieved in 11.064 seconds
Secrets Manager retrieved in 18.103 seconds 
Secrets Manager retrieved in 31.092 seconds

The pattern I am using:

const secret = await getSecretFromGCloud() // this takes 10+ seconds
axios.post(url, payload, headers)

[nullobject]

Same problem here 😞

[0xterran]

Anyone from gcloud know if this is just a region issue? It's severely critical

My settings:

• no custom encryption (gcloud managed)
• automatic region (invokations from us-central1)

[antongildebrand]

I have the same problem. Loading a secret takes ~10 seconds for me. The weird thing is that it's only slow for me, not my colleagues, even though we're on the same network, hardware etc.

[david1542]

I also seem to experience the same problem. I have an express.js app which tries to retrieve some secrets

[fwermus]

happening here

[fwermus]

how do you manage to use secrets in a cloud function in case of cold start?