runtime icon indicating copy to clipboard operation
runtime copied to clipboard

Create an endpoint to get the size of a transaction for a step

Open Hooghof opened this issue 1 year ago • 0 comments

Background: At WCG they sometimes want to know how muc data (bytes) have been processed to see where problems are. Norman said: As a user, I want to be able to find what inbound and outbound endpoint (flow_route_id) passed through in amount (size of the full message) and at what time (timestamp).

The problem is that both Camel, nor Java in general, don't provide mechanism to get the size of a message. What is possible is to get the size of a document (that contains body and headers) in Elasticsearch:

https://stackoverflow.com/questions/27691758/elasticsearch-get-size-stats-of-document-given-document-id

What we like is an endpoint that has the following request:

FlowID Date range An elastic query than returns

the number of documents per step the total size of size of documents per step the total number of documents found + total size flow processed

Related to https://github.com/dovetailworld/front-end/issues/3698

Hooghof avatar Apr 09 '24 06:04 Hooghof