[feature request] Real-time update of Exporter's header configuration #181

[jjyangbiu]

Package

None

Is your feature request related to a problem?

I found that the default Exporter’s Headers in opentelemetry cannot be updated in real time.

Background

My trace and metric data is collected and reported to the cloud, and the upload requires using the cloud provider’s AccessKey. According to the provider’s protocol, the AccessKey needs to be included in the Exporter’s Headers. The AccessKey may change when permissions are updated, so it needs to be updated in real time within the Exporter to ensure that log reporting does not fail.

What is the expected behavior?

I hope there is a way to update the Exporter’s Headers, rather than having to re-implement the entire Exporter initialization process at the application level.

Which alternative solutions or features have you considered?

I tried customizing the Exporter, but this approach requires rewriting many processes.

Additional context

No response

[martincostello]

Link to the specific code from the screenshot (please link to code in future, screenshots aren't accessible):

https://github.com/open-telemetry/opentelemetry-dotnet/blob/3f86f47ac3e067f03dc98620f1c1be22e9d9f9fb/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/OtlpExporterOptionsExtensions.cs#L25-L69

[martincostello]

Looking at the code, to support this any instances of IExportClient would need to change to using IOptionsMonitor<OtlpExporterOptions>.

For example:

https://github.com/open-telemetry/opentelemetry-dotnet/blob/3f86f47ac3e067f03dc98620f1c1be22e9d9f9fb/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/Implementation/ExportClient/OtlpExportClient.cs#L24-L24

Something like:

protected OtlpExportClient(IOptionsMonitor<OtlpExporterOptions> options, HttpClient httpClient, string signalPath)
{
    // ... existing code, then:
    options.OnChange((o) =>
    {
        this.Headers = o.GetHeaders<Dictionary<string, string>>((d, k, v) => d.Add(k, v));
    });
}

[surenderssm]

Just checking the latest here.

[alanwest]

Looking at the code, to support this any instances of IExportClient would need to change to using IOptionsMonitor<OtlpExporterOptions>.

Discussed during today's meeting. I support leveraging IOptionsMonitor. @jjyangbiu is this something you'd be interested in taking on? Otherwise, @martincostello mentioned he might be able to take this on at some point.

[martincostello]

I just spent a little time looking at this, and unfortunately it's not quite as simple as it seems.

While it's easy enough to change the client to take an IOptionsMonitor<T> to observe the changes to the HTTP headers, where that instance comes from ultimately traces up to here:

https://github.com/open-telemetry/opentelemetry-dotnet/blob/a2bb510f792a616e368dea535f79e92392a082a1/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/OtlpMetricExporterExtensions.cs#L65-L90

The branch for name == null makes things difficult to do in a way that actually also observes configuration loading. Even before I got to this point I needed to introduce an internal "fake" of IOptionsMonitor<T> to wrap a T in to keep binary compatibility for the public exporter constructors.

Thoughts before I go down to deep a rabbit hole refactoring?

[CodeBlanch]

@martincostello I'm OK with creating a wrapper IOptionsMonitor<T> which no-ops for this use case.

We have shipped public APIs which accept naked OtlpExporterOptions ex: https://github.com/open-telemetry/opentelemetry-dotnet/blob/1479b236f7eb6adba440d53cee4f4ac6fd0036e5/src/OpenTelemetry.Exporter.OpenTelemetryProtocol/OtlpLogExporter.cs#L37

My guess is you would need it there too.

    public OtlpLogExporter(OtlpExporterOptions options)
-       : this(options, sdkLimitOptions: new(), experimentalOptions: new(), transmissionHandler: null)
+       : this(new NoopOptionsMonitor<OtlpLogExporter>(options), sdkLimitOptions: new(), experimentalOptions: new(), transmissionHandler: null)
    {
    }

    internal OtlpLogExporter(
-       OtlpExporterOptions exporterOptions,
+       IOptionsMonitor<OtlpExporterOptions> exporterOptions,

[martincostello]

Yeah I already have that in a local branch from when I started the work, but it's the other piece that's given me more pause as how to accomplish.

[martincostello]

I've pushed up the initial branch here with the work done so far: https://github.com/martincostello/opentelemetry-dotnet/commit/c5f310966abc29bffb0e30c46a743b4f748100a9