Webhook signature verification fails
Environment
SaaS (https://sentry.io/)
Steps to Reproduce
Expected Result
The signature matches
Actual Result
The signature doesn't match in ~10% of the messages
Product Area
Settings - Integrations
Link
No response
DSN
No response
Version
No response
Assigning to @getsentry/support for routing ⏲️
Routing to @getsentry/product-owners-settings-integrations for triage ⏲️
Thanks for this report, we will investigate this cc @sentaur-athena
This issue is fixed now. We investigated and the reason was a new json serializer we started using since May 9. We reverted the changes to the previous serializer and looking at dashboards the 400 and 401 responses are eliminated.
@quintasda as we're adding back the new serializer wanted to make sure we're not breaking customer's experience. Can you please provide me with code snippet of how you're validating signature?
You mentioned Verify the Signature from our docs but I was wondering if you're using the python or js example? Also what you use to dump json.
@sentaur-athena I'm having issues with this using the exact code from the Sentry docs:
const hmac = crypto.createHmac('sha256', clientSecret);
hmac.update(JSON.stringify(body), 'utf8');
const digest = hmac.digest('hex');
return digest === signature;
This is on NodeJS 19.
Frankly, I don't think this solution is technically sound. JSON.stringify does not produce stable, deterministic output across different Node versions and even if it did there's no guarantee that serializers for Python, Ruby, Go, etc. will produce the same output. There are dedicated libraries that attempt to do this but I don't know of any standard that is widely supported on different platforms.
See the discussion here: https://github.com/nodejs/node/issues/15628
@jasonyonker I agree with you. We should look into something more stable here. Let me look into it and get back to you.
@jasonyonker I read the discussion you sent and I don't think any string serializer would work here since any customer might use a different language and framework. I'll update the docs and suggest to do the verification on raw bytes of body. The catch with that is that some frameworks don't provide raw body bytes at all. Looks like fastify and express might be some examples.
Can you try with hmac.update(raw_request_body)?
Update: We'll update the docs to bring awareness to this issue