prometheus
logback instrumentation module for 1.X
In Prometheus client 1.X, an equivalent for simpleclient_logback is missing.
simpleclient_logback contained only one class InstrumentedAppender.
It is easy to come up with an equivalent for 1.X (code below) but it would be great to have official support for both logback and log4j2.
I think a prominent use case is a high number of error logs from a given logger in a given time window.
Code for 1.X
package io.prometheus.metrics.instrumentation.logback;
import ch.qos.logback.classic.Level;
import ch.qos.logback.classic.spi.ILoggingEvent;
import ch.qos.logback.core.UnsynchronizedAppenderBase;
import io.prometheus.metrics.core.metrics.Counter;
public class InstrumentedAppender extends UnsynchronizedAppenderBase<ILoggingEvent> {
public static final String METER_NAME = "logback_appender_total";
private static final Counter defaultCounter = Counter.builder().name(METER_NAME)
.help("Logback log statements at various log levels")
.labelNames("level")
.register();
private final Counter counter = defaultCounter;
@Override
protected void append(ILoggingEvent event) {
switch (event.getLevel().toInt()) {
case Level.TRACE_INT:
this.counter.labelValues("level", "trace").inc();
break;
case Level.DEBUG_INT:
this.counter.labelValues("level", "debug").inc();
break;
case Level.INFO_INT:
this.counter.labelValues("level", "info").inc();
break;
case Level.WARN_INT:
this.counter.labelValues("level", "warn").inc();
break;
case Level.ERROR_INT:
this.counter.labelValues("level", "error").inc();
break;
default:
break;
}
}
}
As 1.X is a good opportunity to redesign, is this one-to-one approach a good one and can be submitted as an MR? I've seen an alternative in form of TurboFilter in Micrometer LogbackMetrics but appender approach seems more straightforward to me.
@zeitlinger Thanks for assigning me! I'll start working on it and update you soon
@Tharanishwaran if you would like to take the code from my original post as a starting point, I think the questions that need some attention are:
- is a Turbo filter superior in any aspect to an appender? From my point of view appender fits better semantically (we are processing lugging events in a data sink) and functionally - we can add filters to an appender.
- performance comparison between the approaches would be great
- The code in original post uses
UnsynchronizedAppenderBase<ILoggingEvent>as a base class. It has some internal synchronization, which is likely unnecessary - Counter class is thread-safe. Again, this will gain us a few performance points. - Currently the appender would need to be added in Logback config. Maybe it would be more consistent if we could register it in Prometheus registry via register method.
@lesiak Thank you for your detailed suggestions. They're extremely helpful as I start working on this issue. Here's my plan:
- I'll implement both the appender and Turbo filter approaches for comparison.
- I'll create benchmarks to compare their performance.
- I'll look into optimizing synchronization in the base class.
- I'll explore implementing registration via the Prometheus registry method.
Additionally, I'll consider handling custom log levels and potential configuration options. Do you have any thoughts on which configuration options might be most valuable to users?
I'll provide updates as I progress. Thank you again for your guidance
I don't see a lot of configuration options.
If you go with MeterBinder approach to implement registration with Prometheus registry, then some options available in Logback config could be helpful - like adding a custom filter.
@lesiak Thanks for the tips! I'll check out the MeterBinder approach and custom Logback filters. I'll keep it simple otherwise. I'll let you know how it goes as I work on it.