Create child span when parent span is from another service.

[JianTray]

Hi there,

I'm trying to follow a trace through SQS: Service1 -> SQS -> Service2 I've extracted the span_id and trace_id from the span in Service1 and sent it through SQS, where I've picked it up in Service2. I'm now trying to create a new span in Service2 that presents itself a child of the span in Service1 in the flame graph.

I have the java agent running alongside both services, and I have code along the lines of (this is in Scala) in Service2:

val trace_id = message.getMessageAttributes.get("dd.trace_id").getStringValue
val span_id = message.getMessageAttributes.get("dd.span_id").getStringValue

val tracer = GlobalTracer.get()
val parentContext = new CustomSpanContext(trace_id, span_id)

val span = tracer
    .buildSpan("sqs.trigger")
    .asChildOf(parentContext)
    .start()

... do some stuff...

span.close()

I'm essentially having trouble constructing the parent context. I'm tried implementing my own CustomSpanContext (just with span_id, and trace_id), but this left the parent_id of the new span as 0. I can see in from the code that it requires the context to be a OTSpanContext class. I've also tried to create an ExtractedContext, but can't seem to be able to cast the tracer to DDTracer or CoreTracer.

val tracer = GlobalTracer.get().asInstanceOf[DDTracer]
val parentContext = new ExtractedContext(
    DDId.from(traceID.get),
    DDId.from(spanID.get),
    PrioritySampling.UNSET,
    "sniper",
    new util.HashMap[String, String],
    new util.HashMap[String, String]
)

I can't seem to get this to work - I'm thinking this should be simpler than I'm making it, can anyone give me any help here?

Thanks!!

[randomanderson]

The best way to accomplish this would be to use the OpenTracing Tracer.inject() and Tracer.extract().

To inject on the producer side: tracer.inject(span.context(), Format.Builtin.HTTP_HEADERS, message);

to extract on the consumer side:

SpanContext extractedContext = tracer.extract(Format.Builtin.HTTP_HEADERS, message);

Where message implements io.opentracing.propagation.TextMap. You can use the io.opentracing.propagation.TextMapAdapter class to wrap a Map or implement on your own class.

There is an example with Kinesis at https://github.com/DataDog/dd-trace-java-kinesis-example with a few different producer and consumer implementations.

[JianTray]

Thanks for the quick reply @randomanderson!! Super useful and after some investigation I've got it to work now :)

[randomanderson]

@JianTray no problem. SQS support is definitely something we want to support natively and is already on our radar. I added #1823 to track.

Closing this issue as resolved.

[jorgebo10]

Hey, guys i noticed too that there is no span being generated for SQS consumers. I was able to extract and push the headers to SQS on the producer side, however, although i was able to get headers then on the consumer side i can not see yet the spans being created under the same traceId for the consumer. Actually no span is being created. I tried with the following. Do yuo think you could point out any work around? Thanks

   @SqsListener(value = "my-queue", deletionPolicy = ON_SUCCESS)
    public void updateTransaction(@Headers Map<String, String> header,  final UpdateTransactionDto updateTransactionDto) {
            log.info("HEADERS:" + header);
        Mono.just(updateTransactionDto)
                .log("TransactionUpdatesListener.updateTransactionDto")
                .flatMap(updateTransactionService::update)
                .log("TransactionUpdatesListener.update")
                .subscribeOn(Schedulers.boundedElastic())
                .contextWrite(context -> {
                    Tracer tracer = GlobalTracer.get();
                    Span span = getSpan(tracer.extract(Format.Builtin.HTTP_HEADERS, new TextMapAdapter(header)));
                    context.put("span", getSpan(span.context()).context());
                    tracer.activateSpan(span);
                    return context;
                })
                .contextWrite(context -> context.delete("span"))
                .doOnError(e -> Mono.deferContextual(Mono::just)
                        .map(context -> context.get("span"))
                        .doOnNext(o -> {
                            Span span = (Span) o;
                            span.setTag(Tags.ERROR, true);
                            span.log(Collections.singletonMap(Fields.ERROR_OBJECT, e));
                            span.finish();
                        }))
                .doOnSuccess(e -> Mono.deferContextual(Mono::just)
                        .map(context -> context.get("span"))
                        .doOnNext(o -> {
                            Span span = (Span) o;
                            span.finish();
                        }))
                .subscribe();
    }
}