getsentry
sdk should send request data as a structured format if possible
Expected Behavior
GO SDK should send request payload as a json if the request has the right content-type
https://develop.sentry.dev/sdk/event-payloads/request/#examples, as per the doc
data Optional. Submitted data in a format that makes the most sense. SDKs should discard large bodies by default. Can be given as string or structural data of any format.
right now the SDK is sending data as a string no matter if the actual content can be a structured data, which lead to the scrubbing to remove everything.
Screenshots
SDK
-
sentry-goversion: 0.12.0 - Go version: 1.17.5
- Using Go Modules? yes
Sentry
- Using hosted Sentry in sentry.io? yes
PR for this can be found here https://github.com/getsentry/sentry-go/pull/403
Hi @emilien-puget. Thanks for the report/PR.
I talked about this with the team and learned that our ingestion pipeline / data scrubbing system already tries to parse JSON bodies before filtering. What could be happening to lead to the full body being filtered as in your screenshot is in the case of the request body being too large and being truncated by the SDK before sending (or, less likely, by an intermediary Relay).
Could you please provide a link to an event in Sentry.io?
If body size / truncation is the culprit, then the current state of #403 would not solve it, as it retains the body as a string in case parsing JSON fails. If truncation is not happening, then we need to double check our data scrubbing code to see if something else is causing the whole body to be filtered.
hi @rhcarvalho thanks for the reply
Here is an example of the body completely filtered https://sentry.io/organizations/splio_test/performance/http-go-playground:846fb51c977843ee933796ff2ebbbd8f/ You can see the result of the linked PR in this event https://sentry.io/organizations/splio_test/performance/http-go-playground:39cc49b905984cbc85145489e3e7b55d/
Thanks, that helps. Clearly it's a small body and no truncation should be happening.
~@emilien-puget in the broken transaction there's an associated error event that makes it seem as if JSON parsing failed in your app code as well. so i am not sure what behavior you want from sentry here. this data is sensitive, and also invalid json.~
screenshots of UI
oh disregard previous comment, i misread that you already tested with a patched SDK. is that a test application and can you be sure that the request is actually the same? if so we'll investigate internally why relay doesn't parse this json, but we would need to know the exact payload the unpatched sdk sends. for all we know there could be differences in Go's JSON parser vs the one we use on the server.
@untitaker the error is a dummy error not related to the json, you can see on the third screenshot that "post failed" is not related to a json decode, please look at the the second event with the modified SDK
i managed to intercept this, https://gist.github.com/emilien-puget/8d37fe5094992f7d6cef7b5c0b5c00b1 unfortunately, i cannot check the event on sentry side because my organization account has reached the limit of transactions sent
Thank you, i will create a ticket for this so we (server team) can start looking into this case, but no ETA.
i got the payload of the modified SDK as well https://gist.github.com/emilien-puget/87d6a150506a5cc870ee25b64c1a5182
@untitaker where could we track the server-side of this issue? Shall we open a new issue in another repo? IIUC the expectation is that data scrubbing should work such that it will parse JSON and operate on inner fields even if the SDK sends the body as a string, could you confirm that?
In the meantime, I'm working on an opt-in and backwards-compatible patch to replace #403.
hi, sorry. we are tracking this server-side in JIRA. consider this issue assigned to ingest team. you can also transfer it to relay if you want.