waha icon indicating copy to clipboard operation
waha copied to clipboard

[Feature Request] Add support for metadata in "Send messages" methods and webhook responses

Open Sabyrzhanuly opened this issue 1 year ago • 11 comments

Is your feature request related to a problem? Please describe.
I'm always frustrated when I can't identify which specific message was sent from the database when a webhook is triggered. This makes it difficult to mark messages as delivered.


Describe the solution you'd like
I would like to have the ability to send additional custom fields in the "Send messages" methods. For example:

{
  "chatId": "[email protected]",
  "text": "Hi there!",
  "session": "mysession",
  "custom_fields": {
    "field_1": "val_1",
    "field_2": "val_2",
    "field_3": "val_3"
  }
}

When the webhook request is sent, these custom fields should be returned. This will allow me to easily track and identify which specific message was sent and update the database accordingly.


Describe alternatives you've considered
I considered using the response from the /api/sendText method, but I am worried that the webhook might arrive first, and my script might not be ready in time. Another alternative could be using custom headers in the "Send messages" methods that would be returned with each request, allowing for unique identification of requests.


Additional context
I mentioned the "Send messages" example because this is where I specifically need this feature. However, it could be beneficial to add this functionality to all requests.

patron:PLUS

Sabyrzhanuly avatar Sep 22 '24 21:09 Sabyrzhanuly

Hi! One question - do you want to receive this metadata in WAHA, the same session or different third party system/different session? Because we can add it in the project, so it'll track "outgoing" and "incoming" message and based on metadata fields we can add from outgoing message we can add it to incoming and send via webhook.

But we can not add it in WA message payload itself, so you can get it from another WAHA session or different software that uses WA as well. Something like this https://github.com/devlikeapro/waha/issues/272 - it's not possible (or possible, but Meta can track such messages pretty quickly with custom payload, we don't wanna do it)

patron:PRO

devlikepro avatar Sep 23 '24 07:09 devlikepro

Hi! One question - do you want to receive this metadata in WAHA, the same session or different third party system/different session? Because we can add it in the project, so it'll track "outgoing" and "incoming" message and based on metadata fields we can add from outgoing message we can add it to incoming and send via webhook.

But we can not add it in WA message payload itself, so you can get it from another WAHA session or different software that uses WA as well. Something like this #272 - it's not possible (or possible, but Meta can track such messages pretty quickly with custom payload, we don't wanna do it)

patron:PRO

Hi

I want to implement the following workflow. In my database, I will create messages that are ready for sending, but I want to send them at my convenience. When sending messages through the "Send messages" methods, I would like to include additional metadata that pertains specifically to each message. When I receive webhooks from WAHA, I want to get back the same metadata that I specified during the message sending process. This would be very useful in the context of the message.ack webhook.

Please note, there’s no need to send this additional metadata to WhatsApp itself. I just want to use the metadata for correct identification of the sent message in another system when the message.ack webhook is received.

For example:

{
  "event": "message.ack",
  "session": "default",
  "engine": "WEBJS",
  "payload": {
    "id": "[email protected]_4CC5EDD64BC22EBA6D639F2AF571346C",
    "from": "[email protected]",
    "participant": null,
    "fromMe": true,
    "ack": 3,
    "ackName": "READ"
  },
  "metadata": {
    "field_1": "val_1",
    "field_2": "val_2"
  }
}

patron:PLUS

Sabyrzhanuly avatar Sep 23 '24 08:09 Sabyrzhanuly

got it, thank you for the detailed information! :bow: Yeah, that would be a good addition to the session.metadata probably and we could add this for sure. Will plan it, hopefully in this year :crossed_fingers:

patron:PRO

devlikepro avatar Sep 23 '24 09:09 devlikepro

got it, thank you for the detailed information! 🙇 Yeah, that would be a good addition to the session.metadata probably and we could add this for sure. Will plan it, hopefully in this year 🤞

patron:PRO

How can I handle this for now? Are there any similar alternatives?

Also, how can I track the progress of this feature?

patron:PLUS

Sabyrzhanuly avatar Sep 23 '24 09:09 Sabyrzhanuly

The way you described and what we'll do:

  1. When you send a message via HTTP /api/sendSomething - get the message.id from the response ( it depends on engine probably) and save chatId + messageId in your database along side with custom metadata fields
  2. When you got a message (message.any - lookup the database for "chatId + messageId" pair and get metadata from it
  3. If there's no metadata - add a loop like "wait 0.5 seconds, try to find metadata again for 2-5 seconds, repeat" and if there's no metadata after 2-5 seconds - process it like there's no metadata (may be something bad happened during this)

This is the logic we'll add. Also likely we'll have like 1 minute cache in memory so if database doesn't work - we can get it from memory, but we'll save it in the database anyway for restart purposes.

It'll give you like 99% chance of getting metadata. For the rest... well, it's the price of distributed systems :)

patron:PRO

devlikepro avatar Sep 23 '24 09:09 devlikepro

The way you described and what we'll do:

  1. When you send a message via HTTP /api/sendSomething - get the message.id from the response ( it depends on engine probably) and save chatId + messageId in your database along side with custom metadata fields
  2. When you got a message (message.any - lookup the database for "chatId + messageId" pair and get metadata from it
  3. If there's no metadata - add a loop like "wait 0.5 seconds, try to find metadata again for 2-5 seconds, repeat" and if there's no metadata after 2-5 seconds - process it like there's no metadata (may be something bad happened during this)

This is the logic we'll add. Also likely we'll have like 1 minute cache in memory so if database doesn't work - we can get it from memory, but we'll save it in the database anyway for restart purposes.

It'll give you like 99% chance of getting metadata. For the rest... well, it's the price of distributed systems :)

patron:PRO

Thank you very much!

patron:PLUS

Sabyrzhanuly avatar Sep 23 '24 09:09 Sabyrzhanuly

The way you described and what we'll do:

  1. When you send a message via HTTP /api/sendSomething - get the message.id from the response ( it depends on engine probably) and save chatId + messageId in your database along side with custom metadata fields
  2. When you got a message (message.any - lookup the database for "chatId + messageId" pair and get metadata from it
  3. If there's no metadata - add a loop like "wait 0.5 seconds, try to find metadata again for 2-5 seconds, repeat" and if there's no metadata after 2-5 seconds - process it like there's no metadata (may be something bad happened during this)

This is the logic we'll add. Also likely we'll have like 1 minute cache in memory so if database doesn't work - we can get it from memory, but we'll save it in the database anyway for restart purposes.

It'll give you like 99% chance of getting metadata. For the rest... well, it's the price of distributed systems :)

patron:PRO

Can you answer a few questions? After calling the HTTP /api/sendSomething, will I have enough time to save the message.id from the response in my database? How soon will the webhook arrive? Is there a possibility that the webhook will arrive immediately and I won't have enough time to save it? Is there an option to set an artificial delay?

patron:PLUS

Sabyrzhanuly avatar Sep 24 '24 05:09 Sabyrzhanuly

After calling the HTTP /api/sendSomething, will I have enough time to save the message.id from the response in my database? How soon will the webhook arrive?

It depends on many things - how WA servers fast, how fast your database connection.

Is there a possibility that the webhook will arrive immediately and I won't have enough time to save it?

yes, this is why there's a loop in the logic with "wait 0.5 seconds to get metadata and loop it for 2-5 seconds" from your database when you got a new webhook event.

Is there an option to set an artificial delay?

Probably yes, but what delay is enough? May be the database got stuck, networks sucks and etc, etc, many things could happen on that way.

I'd go with artificial delay in "get metadata" logic instead of adding delay in sending.


It's basically well-known problem for distributed systems - sync the state between 2 and more systems. Like solving CAP theorem - 2 phase commit protocol, reconciling records - all can be useful for that.

We don't think adding such a complex thing will be worse in 1% of cases, in 99% of cases - the delay on the receiver side is enough.

patron:PRO

devlikepro avatar Sep 24 '24 06:09 devlikepro

After calling the HTTP /api/sendSomething, will I have enough time to save the message.id from the response in my database? How soon will the webhook arrive?

It depends on many things - how WA servers fast, how fast your database connection.

Is there a possibility that the webhook will arrive immediately and I won't have enough time to save it?

yes, this is why there's a loop in the logic with "wait 0.5 seconds to get metadata and loop it for 2-5 seconds" from your database when you got a new webhook event.

Is there an option to set an artificial delay?

Probably yes, but what delay is enough? May be the database got stuck, networks sucks and etc, etc, many things could happen on that way.

I'd go with artificial delay in "get metadata" logic instead of adding delay in sending.

It's basically well-known problem for distributed systems - sync the state between 2 and more systems. Like solving CAP theorem - 2 phase commit protocol, reconciling records - all can be useful for that.

We don't think adding such a complex thing will be worse in 1% of cases, in 99% of cases - the delay on the receiver side is enough.

patron:PRO

Could you please tell me how I can make the webhook retry the request to my server? If I respond with a 404 or 500, will that trigger a retry? According to the documentation, it should retry 15 times with a 2-second delay.

patron:PLUS

Sabyrzhanuly avatar Sep 26 '24 03:09 Sabyrzhanuly

Hi! Via API https://waha.devlike.pro/docs/how-to/webhooks/#retries

{
  "name": "default",
  "config": {
    "webhooks": [
      {
        "url": "https://webhook.site/11111111-1111-1111-1111-11111111",
        "events": [
          "message"
        ],
        "retries": {
          "delaySeconds": 2,
          "attempts": 15
        }
      }
    ]
  }
}

Via Dashboard

image

If I respond with a 404 or 500, will that trigger a retry?

Yeah, it will.


We've added few improvements in webhook retries internals in 2024.10.1 version, will release it 1 Oct!

patron:PRO

devlikepro avatar Sep 26 '24 08:09 devlikepro

Hi! We've added source: app|api in 2025.3.1, so you can identify in webhooks the message source (either WAHA or other devices/app/integrations)

https://waha.devlike.pro/docs/how-to/events/#message

  • source: app|api - can be api for message.any event if you send a message via WAHA API. Otherwise, it’s app.

As alternative for metadata we could allow sending pre-generating message.id in /api/sendText (we'll provide API for that as well), so you can store message.id in your db with metadata and then fetch it on message.any webhooks.

patron:PRO

devlikepro avatar Mar 27 '25 05:03 devlikepro