Inject a list of Spans in a message in a Flink application running on KDA
What does the application do I have an Apache Flink application running on KDA which consumes messages from kinesis, groups them and write it to a file. After writing, it sends the file metadata as a SQS message which is processed by downstream applications.
Goal To introduce tracing in this application I am using datadog tracer (https://github.com/DataDog/dd-trace-java) to inject and extract spans.Every kinesis message has an injected span. The application extracts these spans per message, groups them in a list as per the grouping condition. I want to send this list of span(context)s along with the file metadata injected into an SQS message. But I cannot inject all these span(context)s in an SQS message because
- Can inject exactly one Span in the message
- Size limitations on SQS message
It seems there is one option to attach this span list to a parent span and then send it to SQS, but since I cannot run a datadog agent in KDA I am looking for other options.
What solution can I use in this use case?