Timestamp skew between Datadog SDK spans / System.Diagnostics spans

[IlSocio]

Describe the bug A clear and concise description of what the bug is. The timestamps of the spans generated by the Datadog SDK is "delayed" respect to the one generated by the ActivitySource.StartActivity()

To Reproduce Steps to reproduce the behavior:

1. create a project mixing Datadog SDK spans and ActivitySource spans
2. make sure you set the DD_TRACE_OTEL_ENABLED=true

Expected behavior The spans timestamp should be aligned.

Screenshots In the image above you can see 4 spans, using both Datadog SDK and System.Diagnostics:

• 1st ActivitySource.StartActivity()
• 2nd Tracer.Instance.StartActive
• 3rd ActivitySource.StartActivity()
• 4th SQL integration

Both the spans 2nd and 4th, generated by Datadog (using SDK and with Automatic instrumentation) have a delay of some ms.

Runtime environment (please complete the following information):

• Instrumentation mode: manual with NuGet package.
• Datadog.Trace.Bundle v3.3.1
• OS: Windows Server 2022 Version21H2
• CLR: NET 6.0

Additional context Add any other context about the problem here.

[bouwkast]

@IlSocio

Thanks for reporting this, I think I can see what is going on.

When we have Activity created Datadog Spans we use the Activity.StartTimeUtc (and Activity.Duration) for the Span.StartTime and Span.Duration. When we have Datadog-only Spans created we use our internal TraceClock to manage the start/end/duration times.

But since the TraceClock isn't being synchronized with the Activity timings it appears starts to mess up all of the timings when we have traces with both Span and Activity.

I'll see what we can do to synchronize this to fix that.

We'll keep you posted on this. Thanks again!