(Exactly once delivery) BatchError: Failed to "modifyAckDeadline" Reason: 3 INVALID_ARGUMENT.

[JaegarSarauer]

1. Is this a client library issue or a product issue? Client Library Issue.

2. Did someone already solve this?

• [x] Search the issues already opened: https://github.com/googleapis/nodejs-pubsub/issues
• [x] Search the issues on our "catch-all" repository: https://github.com/googleapis/google-cloud-node
• [x] Search or ask on StackOverflow (engineers monitor these tags): http://stackoverflow.com/questions/tagged/google-cloud-platform+node.js

3. Do you have a support contract? No.

If the support paths suggested above still do not result in a resolution, please provide the following details.

Environment details

• OS: Mac OS Monterey 12.3.1 (M1)
• Node.js version: v14.19.1
• npm version: 6.14.16
• @google-cloud/pubsub version: ^2.19.0

Steps to reproduce

1. Create a Topic and Subscription for a project in the PubSub console. a) Assign the google API scope: [email protected] to the topic and subscription. b) Enable "Exactly once delivery" for the subscription.
2. Create a project which watches a mailbox for unread messages and gets the subscription just created:

const pubsub = new PubSub({ projectId: 'example-project' });
const sub = await pubsub.subscription('example-project-sub');

sub.on('error', async (error) => {
  console.info('Pubsub subscribe error: ', error);
});

sub.on('message', async (message) => {
  try {
    message.ack();
  } catch (err) {
    console.info(err);
  }
});

await gmail.users.watch({
    userId: 'me',
    requestBody: {
      labelIds: ['UNREAD'],
      topicName: `projects/example-project/topics/example-project-topic`,
    },
  })

(Authentication Code):

  const { google } = require('googleapis');

  const JWT = google.auth.JWT;
  const auth = new JWT({
    keyFile: 'credentials-service.json',
    scopes: ['https://www.googleapis.com/auth/gmail.readonly', 'https://mail.google.com/'],
    subject: '[email protected]',
  });

  await auth.authorize();

  const gmail = google.gmail({
    auth,
    version: 'v1',
  });

4. Run the project concurrently with at least 2 instances.
5. Send 10 unique emails to the email matched to the topic in fast succession.
6. Witness the error appear:

BatchError: Failed to "modifyAckDeadline" for 1 message(s). Reason: 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.
   at ./example-project/node_modules/@google-cloud/pubsub/src/message-queues.ts:258:15
   at async Promise.all (index 0)
   at async ModAckQueue._sendBatch (./example-project/node_modules/@google-cloud/pubsub/src/message-queues.ts:262:5)
   at async ModAckQueue.flush (./example-project/node_modules/@google-cloud/pubsub/src/message-queues.ts:142:7) {
 ackIds: [
   'BhYsXUZIUTcZCGhRDk9eIz81IChFEgADZ3kiHkYfYmhcXHUHUAUZei5hcmNbRwkIR1Z0X3MrFZ22qtYxUa3P8PXZdUh6FwULRVp_WV8ZC2tfX3YPUyWguYL5j-KZJBs-faaUqfgtLZu7y682Zi09WhJLLD5-NTZFQV5AEkw-A0RJUytDCypYEU4EISE-MD5FU0RQ'
 ],
 code: 3,
 details: 'Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.',
 metadata: Metadata {
   internalRepr: Map(2) {
     'google.rpc.errorinfo-bin' => "EXACTLY_ONCE_ACKID_FAILUREpubsub.googleapis.com�
�BhYsXUZIUTcZCGhRDk9eIz81IChFEgADZ3kiHkYfYmhcXHUHUAUZei5iI2gPG1AIRlp0X3MrFcCaqtYxUa3P8PXZdUh6FwULRVp9X10TDWlVX3IGVCWguYL5j-KZJBs-ffu4qfgtLYDZlqg2Zi09WhJLLD5-NTZFQV5AEkw-A0RJUytDCypYEU4EISE-MD5FU0RQ PERMANENT_FAILURE_INVALID_ACK_ID",
     'grpc-status-details-bin' => "�Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.�
(type.googleapis.com/google.rpc.ErrorInfo�
EXACTLY_ONCE_ACKID_FAILUREpubsub.googleapis.com�
�BhYsXUZIUTcZCGhRDk9eIz81IChFEgADZ3kiHkYfYmhcXHUHUAUZei5iI2gPG1AIRlp0X3MrFcCaqtYxUa3P8PXZdUh6FwULRVp9X10TDWlVX3IGVCWguYL5j-KZJBs-ffu4qfgtLYDZlqg2Zi09WhJLLD5-NTZFQV5AEkw-A0RJUytDCypYEU4EISE-MD5FU0RQ PERMANENT_FAILURE_INVALID_ACK_ID"
   },
   options: {}
 }
}

Additional Details:

• There is no online record of this extended error log: "Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.".
  • Except for this PR in the pubsub .net repo.

[JaegarSarauer]

Closing for now as I am able to remove the error by removing more of the code from my project. At first I had thought the problem stemmed from the ack() call on the subscriber message, but it may be due to how I am handling history and messages further into the project.

Will reopen once the issue is narrowed down more, otherwise I will post what caused the issue.

[JaegarSarauer]

I have reopened as I was able to verify the bug is reproducible with the above code.

[feywind]

@JaegarSarauer Thanks for the report and all the info. The Node library for Pub/Sub doesn't currently support Exact Once, but it's a feature that's currently in progress.

[feywind]

Well, slight qualification: 2.19.3 has a patch that should keep it from crashing when dealing with EO ack IDs (which are much larger), but there isn't functionality to deal with ack() responses yet.

https://github.com/googleapis/nodejs-pubsub/issues/1504

[JaegarSarauer]

No problem, thanks for the reply.

For anyone who may come across this problem and need a solution, I ended up using a distributed lock (using redis).

With dist locks, I was able to lock the resource I was using on one of my program instances, so I didn't mind getting excess messages from Pub/Sub.

[feywind]

@JaegarSarauer Ah, thanks for the follow-up (sorry for my slow follow-up follow-up). I'm hoping we'll have proper exactly-once support in Node soon but it's good you found a solution.

[shoxsz]

I don't get it, why should the nodejs library have support for it? shouldn't it be a behaviour from the pubsub backend side?

[feywind]

@shoxsz It is, but it requires a fair amount of client library support to work gracefully and correctly. Specifically, we need to allow users to get a response from calling ack() on a message, to see if they actually claimed it or not.

[shoxsz]

Thanks for the quick reply, can you tell me if this have been solved in the current version 3.1.0?

[feywind]

@shoxsz It hasn't been merged, but it's in the later stages of development right now, fingers crossed that it won't be too long.

[shoxsz]

Waiting for it, thanks for the help.

[Abjuk]

3.2.0 issue still actual error: OTHER : 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed. at /Users/capgemini/projects/medable/screenshot-service/node_modules/@google-cloud/pubsub/src/message-queues.ts:419:21 at Array.forEach () at ModAckQueue.handleAckFailures (/Users/capgemini/projects/medable/screenshot-service/node_modules/@google-cloud/pubsub/src/message-queues.ts:418:12) at /Users/capgemini/projects/medable/screenshot-service/node_modules/@google-cloud/pubsub/src/message-queues.ts:563:33 at async Promise.all (index 0) at async ModAckQueue._sendBatch (/Users/capgemini/projects/medable/screenshot-service/node_modules/@google-cloud/pubsub/src/message-queues.ts:574:45) at async ModAckQueue.flush (/Users/capgemini/projects/medable/screenshot-service/node_modules/@google-cloud/pubsub/src/message-queues.ts:269:23) { errorCode: 'OTHER'

[mkorpar]

Have been experiencing the same issue. Furthermore it is crashing my app. Is there some kind of missed error propagation missing and is thrown "into the void"? I've made everything according to examples available on google docs pages.

[modestaspruckus]

@feywind Same issue appears after upgrade to 3.2

Error: INVALID : 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.
    at <anonymous> (/releases/975/node_modules/@google-cloud/pubsub/src/message-queues.ts:419:21)
    at Array.forEach (<anonymous>)
    at ModAckQueue.handleAckFailures (/releases/975/node_modules/@google-cloud/pubsub/src/message-queues.ts:418:12)
    at <anonymous> (/releases/975/node_modules/@google-cloud/pubsub/src/message-queues.ts:563:33)
    at async Promise.all (index 0)
    at ModAckQueue._sendBatch (/releases/975/node_modules/@google-cloud/pubsub/src/message-queues.ts:574:45)
    at ModAckQueue.flush (/releases/975/node_modules/@google-cloud/pubsub/src/message-queues.ts:269:23)

[feywind]

I'll reopen this.

[Abjuk]

@feywind Hello! Could you help - how can I intercept and suppress this error so that it does not lead to restarting pods in Kubernetes. if i put a call of message.ack(); inside try/catch doesn't help. subscription.on("error", () => {}) doesn't catch this error either

[Abjuk]

In my case, I think this error occurs because ackDeadline > 600, what happens if I use modAck(0) instead of ack() ?

And I can't find in the source code where this error is generated: "error: OTHER : 3 INVALID_ARGUMENT: Some acknowledgment ids in the request were invalid. This could be because the acknowledgment ids have expired or the acknowledgment ids were malformed." Tell me, please. I want to investigate this part of the code and understand the causes of the error

[mkorpar]

I think the error itself comes from pubsub backend. As far as supressing goes, haven't found any way except editing the library itself.

[Abjuk]

@mkorpar Hello! Could you please share for me your solution

[bonzzy]

Is there any progress made on this part? I get the same error when using exactly once delivery and NodeJs. ackError: INVALID : 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid

Is the solution for now not to use exactly once delivery with NodeJs?

[udnes99]

Occurred for us today as well (NOT using exactly-once delivery), leading to pods in K8s having to be restarted. This persisted for a while until it seemingly resolved on its own. Still detrimental to the availability of our services.

 "AckError: INVALID : 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.
    at /app/node_modules/@google-cloud/pubsub/build/src/message-queues.js:330:29
    at Array.forEach (<anonymous>)
    at ModAckQueue.handleAckFailures (/app/node_modules/@google-cloud/pubsub/build/src/message-queues.js:328:18)
    at /app/node_modules/@google-cloud/pubsub/build/src/message-queues.js:470:43
    at async Promise.all (index 1)
    at async ModAckQueue._sendBatch (/app/node_modules/@google-cloud/pubsub/build/src/message-queues.js:476:31)
    at async ModAckQueue.flush (/app/node_modules/@google-cloud/pubsub/build/src/message-queues.js:195:29) {"

[Ganitzsh]

Bumped into the same issue yesterday using the latest version 3.2.1. Downgrading to 3.2.0 reduced the amount of errors, and acknowledging messages works fine. So for now I have catch all on uncaughtException that logs the error when this particular one occurs, otherwise I just exit the process. This is a workaround and it's not ideal, hoping for a fix soon

[youb93]

Hello, Same issue over there, is there any news about it ?

[Ori-b]

Hello, Any news? getting the same issues:

1. 'uncaughtException' instead of a properly propagated exception (that can be caught on.('error')
2. Exactly once delivery - i am holding a message unacked and still get re-deliveries of un-acked messages within each hour....what is the client library logic regarding that duration? (it seems that the ackDeadline has no effect whether it is 10 seconds, or 600 seconds) - see images below (timestamps of redelivered message after a hour - single listener).

3. Is the Exactly Once delivery could be said to be operational in any sense?

[garymathews]

I'm also running into this issue.

From what I've found it's caused by this deferred promise. https://github.com/googleapis/nodejs-pubsub/blob/main/src/message-queues.ts#L191

...
const responsePromise = defer<void>();

// TC1. Doing this still causes the 'unhandledException'.
responsePromise.promise.catch(() => {});

// TC2. Doing this prevents the 'unhandledException'. Note the existing promise in 'responsePromise' is being updated.
responsePromise.promise = responsePromise.promise.catch(() => {});

this._requests.push({
...

So this .catch handler is being ignored. https://github.com/googleapis/nodejs-pubsub/blob/main/src/subscriber.ts#L531-L533

For the time being, my workaround is to use an uncaughtException handler.

[modestaspruckus]

Hey @feywind , Issue still persists on 3.3.0 version

INVALID : 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.
/home/builder/backend/releases/543/node_modules/@google-cloud/pubsub/src/message-queues.ts in <anonymous> at line 419:21
/home/builder/backend/releases/543/node_modules/@google-cloud/pubsub/src/message-queues.ts in Array.forEach
/home/builder/backend/releases/543/node_modules/@google-cloud/pubsub/src/message-queues.ts in ModAckQueue.handleAckFailures at line 418:12
/home/builder/backend/releases/543/node_modules/@google-cloud/pubsub/src/message-queues.ts in <anonymous> at line 563:33
/home/builder/backend/releases/543/node_modules/@google-cloud/pubsub/src/message-queues.ts in Promise.all
/home/builder/backend/releases/543/node_modules/@google-cloud/pubsub/src/message-queues.ts in ModAckQueue._sendBatch at line 574:45
/home/builder/backend/releases/543/node_modules/@google-cloud/pubsub/src/message-queues.ts in ModAckQueue.flush at line 269:23

[Abjuk]

Actual: INVALID : 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.

[fabioszymanski]

Hey guys, is there any news in this issue? Any workaround? I'm facing the same here when consuming a message from a subscription with exactly-once delivery activated, this is intermittent, can't find the root cause exactly.

UnhandledPromiseRejectionWarning: Error: INVALID : 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid

the whole error is: (node:1) UnhandledPromiseRejectionWarning: Error: INVALID : 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed. at /app/node_modules/@google-cloud/pubsub/build/src/message-queues.js:330:29 at Array.forEach (<anonymous>) at AckQueue.handleAckFailures (/app/node_modules/@google-cloud/pubsub/build/src/message-queues.js:328:18) at AckQueue._sendBatch (/app/node_modules/@google-cloud/pubsub/build/src/message-queues.js:401:42) at async AckQueue.flush (/app/node_modules/@google-cloud/pubsub/build/src/message-queues.js:195:29)

[feywind]

Those look like errors coming from the Pub/Sub service itself, and there's not a lot we can do from the client library end outside of entering a support ticket (to be able to look at the backend to try to match up times and errors). :(

If these are resulting in an error when exactly once delivery is disabled, or an unhandled rejection that user code can't catch, that's something I can look at library-side. Without EOD enabled, this error shouldn't occur at all. And if you've got a try/catch around the ack() call, it shouldn't turn into an unhandled rejection.

[anthony-langford]

Experiencing the same error here with other packages as well (e.g. @algoan/pub-sub).

2023-04-24 07:04:25 | Node.js v19.8.1
2023-04-24 07:04:25 | }
2023-04-24 07:04:25 | errorCode: 'INVALID'
2023-04-24 07:04:25 | at async ModAckQueue.flush (/usr/src/app/silver/node_modules/@google-cloud/pubsub/build/src/message-queues.js:195:29) {
2023-04-24 07:04:25 | at async ModAckQueue._sendBatch (/usr/src/app/silver/node_modules/@google-cloud/pubsub/build/src/message-queues.js:476:31)
2023-04-24 07:04:25 | at async Promise.all (index 0)
2023-04-24 07:04:25 | at /usr/src/app/silver/node_modules/@google-cloud/pubsub/build/src/message-queues.js:470:43
2023-04-24 07:04:25 | at ModAckQueue.handleAckFailures (/usr/src/app/silver/node_modules/@google-cloud/pubsub/build/src/message-queues.js:328:18)
2023-04-24 07:04:25 | at Array.forEach (<anonymous>)
2023-04-24 07:04:25 | at /usr/src/app/silver/node_modules/@google-cloud/pubsub/build/src/message-queues.js:330:29
2023-04-24 07:04:25 | AckError: INVALID : 3 INVALID_ARGUMENT: Some acknowledgement ids in the request were invalid. This could be because the acknowledgement ids have expired or the acknowledgement ids were malformed.
2023-04-24 07:04:25 |  
2023-04-24 07:04:25 | ^
2023-04-24 07:04:25 | const exc = new subscriber_1.AckError(e[0], rpcError.message);
2023-04-24 07:04:25 | /usr/src/app/silver/node_modules/@google-cloud/pubsub/build/src/message-queues.js:330