Support for hot-reloading of TLS credentials used in exporters

[evantorrie]

The OpenTelemetry Collector supports hot-reloading of TLS credentials sourced from local files, via a reload_interval available in both receiver and exporters configurations.

The opentelemetry-java standard exporters (OTLP in particular, but also Zipkin and Jaeger) do not support hot reloading of TLS credentials.

This hampers connecting from a Java Otel client using this library to a hot-reload enabled OpenTelemetry Collector server, where server cert rotation may be relatively frequent (once a day, for example - usually much shorter than the average time in between restarts of client applications).

Describe the solution you'd like

Support for optional hot-reload of TLS credentials for exporters. Ideally also Autoconfigure SDK support via a parameter (may require spec change?).

Describe alternatives you've considered Currently our alternative is to add a localhost based collector (e.g. k8s daemonset pods) and connect over a non mutual TLS connection to the localhost Otel Collector on an open host port. The Otel Collector can then make use of its own built-in TLS credential hot reload to ensure that long-running connections between local collector and remote collector are enabled with mutual TLS.

[jack-berg]

While there's no feature for this today, you could implement it yourself in a pretty straight forward manner with something like the following:

  public class RefreshingMetricExporter implements MetricExporter {

    private final Supplier<MetricExporter> metricExporterSupplier;
    private final Duration refreshInterval;
    private final Object lock = new Object();

    @GuardedBy("lock") private long lastRefresh;
    @GuardedBy("lock") private MetricExporter metricExporter;

    public RefreshingMetricExporter(Supplier<MetricExporter> metricExporterSupplier, Duration refreshInterval) {
      this.metricExporterSupplier = metricExporterSupplier;
      this.refreshInterval = refreshInterval;
    }

    private MetricExporter metricExporter() {
      synchronized (lock) {
        if (lastRefresh == 0 || (System.currentTimeMillis() - lastRefresh) > refreshInterval.toMillis()) {
          if (metricExporter != null) {
            metricExporter.shutdown();
          }
          metricExporter = metricExporterSupplier.get();
          lastRefresh = System.currentTimeMillis();
        }
        return metricExporter;
      }
    }

    @Override
    public AggregationTemporality getAggregationTemporality(InstrumentType instrumentType) {
      return metricExporter().getAggregationTemporality(instrumentType);
    }

    @Override
    public CompletableResultCode export(Collection<MetricData> metrics) {
      return metricExporter().export(metrics);
    }

    @Override
    public CompletableResultCode flush() {
      return metricExporter().flush();
    }

    @Override
    public CompletableResultCode shutdown() {
      return metricExporter().shutdown();
    }
  }

That code delegates to MetricExporter (but it could be LogExporter or SpanExporter) which it obtains from a Supplier after some configurable interval has exceeded. If you implement a supplier which when invoked builds an OTLP exporter with the current TLS credentials from some file, I believe you'll accomplish your goal.

(Note: I put that together in a few minutes - there may be a bug or a better way to accomplish the same thing! Use at your own risk 🙂 )

[abhinavk96]

Hi @jack-berg Is this feature currently being tracked/worked on? If not would you be open to me contributing to this? Thank you.

[jack-berg]

Some things have changed since my 8/22/22 comment.. Notably, the OTLP exporters now allow you to directly set SSLContext / X509TrustManager, e.g. OtlpGrpcSpanExporterBuilder#setSslContext.

You should be able to provide an SSLContext / X509TrustManager implementation which reloads credentials from local files. However, I'm not sure its a common enough use case to justify including / maintain the API surface area in this repo. Perhaps you could consider implementing it in opentelemetry-java-contrib?

[abhinavk96]

This is great, I did not realize passing a custom X509TrustManager is now possible which is what I was looking for. Thank you.

[jack-berg]

I'm actually going to go ahead an close this issue, since I as I mentioned in this comment, I don't think we would want to host the utility code needed to do the hot-swapping in this repository, and the setSslContext methods allow users to configure their own functionality around this.