enhancement(source metrics): Adding a histogram for event byte size

[pabloem]

Useful when debugging low-level sources like UDP, TCP sockets

If this metric looks good, I can add docs and we can prepare to merge. Thanks!

[bits-bot]

All committers have signed the CLA.

[hhromic]

This is an interesting addition, we use histograms for this kind of data too in our non-Vector components.

However, with histograms you often need to pay attention to the pre-defined buckets which depend on the measured data. For example, latencies and byte sizes often use different bucket configurations. Hence, I wonder how this implementation allows configuration of the buckets for the histograms for each metric?

From https://www.robustperception.io/how-does-a-prometheus-histogram-work/:

So why not always use histograms? There's a long answer, but the short version is that with histograms you have to pre-choose your buckets, and the costs moves from the client to Prometheus itself due to bucket cardinality. The default ten buckets cover a typical web service with latency in the millisecond to second range, and on occasion you will want to adjust them. Here for example they have been overridden to better help track requests for PromQL, which have a two minute default timeout. Having more than ten buckets will give more accurate results, however it can also add up to a lot of time series. Particularly when combined with other labels.

[pabloem]

yeah good question - I believe users would create an internal_metrics source, and among these internal metrics we'd have this particular histogram. (newbie assumption): They can probably define histogram buckets in a remap stage, or if they are using prometheus_exporter sinks, then configure that sink's default buckets - though that either requires extra custom config for each histogram or affects all histograms respectively.

Thoughts?

[pabloem]

I'll fix up failed CI this evening (EST tzone)

[pabloem]

@jszwedko thanks for the review! Hopefully changes are enough!

[pabloem]

I've actually changed these because I was intending to cover the lower-level protocols (UDP, TCP-based ones), and was actually covering higher-level ones (HTTP-based ones) - please take another look? @jszwedko

[github-actions[bot]]

Regression Detector Results

Run ID: c3e7b8bf-2afa-48f1-b37a-38eb9e9e060e Baseline: 2bb51342882b0640c5332c06229a51a72fd902d6 Comparison: ce35b80ad2a52deebfc45af2ddb68a2f61600fea Total CPUs: 7

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

Significant changes in experiment optimization goals

Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%

perf	experiment	goal	Δ mean %	Δ mean % CI
✅	otlp_http_to_blackhole	ingress throughput	+5.69	[+5.54, +5.84]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
✅	otlp_http_to_blackhole	ingress throughput	+5.69	[+5.54, +5.84]
➖	socket_to_socket_blackhole	ingress throughput	+4.93	[+4.87, +4.99]
➖	syslog_log2metric_tag_cardinality_limit_blackhole	ingress throughput	+4.54	[+4.38, +4.71]
➖	syslog_humio_logs	ingress throughput	+4.07	[+3.87, +4.28]
➖	http_text_to_http_json	ingress throughput	+2.53	[+2.34, +2.72]
➖	fluent_elasticsearch	ingress throughput	+1.98	[+1.47, +2.48]
➖	syslog_log2metric_humio_metrics	ingress throughput	+1.95	[+1.78, +2.12]
➖	syslog_loki	ingress throughput	+1.14	[+1.08, +1.20]
➖	http_elasticsearch	ingress throughput	+0.94	[+0.85, +1.02]
➖	file_to_blackhole	egress throughput	+0.64	[-1.98, +3.26]
➖	datadog_agent_remap_blackhole	ingress throughput	+0.35	[+0.25, +0.44]
➖	datadog_agent_remap_blackhole_acks	ingress throughput	+0.31	[+0.17, +0.45]
➖	http_to_http_noack	ingress throughput	+0.08	[-0.02, +0.18]
➖	otlp_grpc_to_blackhole	ingress throughput	+0.07	[-0.03, +0.16]
➖	http_to_http_json	ingress throughput	+0.04	[-0.03, +0.12]
➖	splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-0.00	[-0.14, +0.14]
➖	splunk_hec_indexer_ack_blackhole	ingress throughput	-0.01	[-0.15, +0.13]
➖	splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.05	[-0.16, +0.06]
➖	enterprise_http_to_http	ingress throughput	-0.13	[-0.20, -0.05]
➖	http_to_s3	ingress throughput	-0.14	[-0.43, +0.14]
➖	datadog_agent_remap_datadog_logs	ingress throughput	-0.37	[-0.48, -0.25]
➖	datadog_agent_remap_datadog_logs_acks	ingress throughput	-0.81	[-0.91, -0.72]
➖	splunk_hec_route_s3	ingress throughput	-0.82	[-1.31, -0.33]
➖	syslog_splunk_hec_logs	ingress throughput	-1.31	[-1.47, -1.16]
➖	http_to_http_acks	ingress throughput	-1.40	[-2.75, -0.05]
➖	syslog_regex_logs2metric_ddmetrics	ingress throughput	-2.34	[-2.51, -2.17]
➖	syslog_log2metric_splunk_hec_metrics	ingress throughput	-2.74	[-2.93, -2.55]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[github-actions[bot]]

Regression Detector Results

Run ID: d6b95f0d-6bd2-4d70-bd07-d30f8c206fa4 Baseline: e4e7321e7e0a872612458d6f9d1ec083126b76be Comparison: 2f9ded0cd6f36a635e23d803efdbf8a0ca905031 Total CPUs: 7

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

Significant changes in experiment optimization goals

Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%

perf	experiment	goal	Δ mean %	Δ mean % CI
❌	syslog_regex_logs2metric_ddmetrics	ingress throughput	-6.84	[-7.00, -6.69]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	syslog_humio_logs	ingress throughput	+4.22	[+4.09, +4.36]
➖	syslog_log2metric_humio_metrics	ingress throughput	+3.60	[+3.47, +3.74]
➖	file_to_blackhole	egress throughput	+2.69	[+0.19, +5.18]
➖	splunk_hec_route_s3	ingress throughput	+2.38	[+1.88, +2.88]
➖	datadog_agent_remap_blackhole	ingress throughput	+1.60	[+1.48, +1.73]
➖	http_to_http_acks	ingress throughput	+1.19	[-0.18, +2.55]
➖	syslog_splunk_hec_logs	ingress throughput	+1.16	[+1.07, +1.25]
➖	http_text_to_http_json	ingress throughput	+0.32	[+0.18, +0.46]
➖	http_to_s3	ingress throughput	+0.29	[+0.01, +0.57]
➖	http_to_http_noack	ingress throughput	+0.10	[+0.02, +0.19]
➖	http_to_http_json	ingress throughput	+0.05	[-0.03, +0.12]
➖	splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-0.00	[-0.14, +0.14]
➖	splunk_hec_indexer_ack_blackhole	ingress throughput	-0.00	[-0.15, +0.14]
➖	splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.02	[-0.14, +0.09]
➖	syslog_log2metric_splunk_hec_metrics	ingress throughput	-0.04	[-0.22, +0.14]
➖	enterprise_http_to_http	ingress throughput	-0.05	[-0.11, +0.01]
➖	syslog_log2metric_tag_cardinality_limit_blackhole	ingress throughput	-0.13	[-0.28, +0.03]
➖	http_elasticsearch	ingress throughput	-0.37	[-0.46, -0.28]
➖	datadog_agent_remap_datadog_logs_acks	ingress throughput	-0.41	[-0.51, -0.31]
➖	datadog_agent_remap_blackhole_acks	ingress throughput	-0.84	[-0.96, -0.72]
➖	otlp_grpc_to_blackhole	ingress throughput	-0.91	[-0.99, -0.82]
➖	syslog_loki	ingress throughput	-1.05	[-1.16, -0.93]
➖	datadog_agent_remap_datadog_logs	ingress throughput	-1.08	[-1.20, -0.96]
➖	fluent_elasticsearch	ingress throughput	-1.17	[-1.64, -0.69]
➖	socket_to_socket_blackhole	ingress throughput	-2.85	[-2.94, -2.76]
➖	otlp_http_to_blackhole	ingress throughput	-4.98	[-5.12, -4.84]
❌	syslog_regex_logs2metric_ddmetrics	ingress throughput	-6.84	[-7.00, -6.69]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[github-actions[bot]]

Regression Detector Results

Run ID: 9895491f-00d2-419f-8b43-2045927ac8f3 Baseline: 665ab39dce1cfcad46cbae746c82973df957bfdf Comparison: 4c0e1527d5f1144d1ec4dcb634e2f0e45db52ee7 Total CPUs: 7

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

Significant changes in experiment optimization goals

Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%

perf	experiment	goal	Δ mean %	Δ mean % CI
❌	otlp_http_to_blackhole	ingress throughput	-5.21	[-5.34, -5.07]

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	syslog_log2metric_humio_metrics	ingress throughput	+4.05	[+3.92, +4.18]
➖	syslog_humio_logs	ingress throughput	+2.80	[+2.66, +2.94]
➖	file_to_blackhole	egress throughput	+2.16	[-0.37, +4.69]
➖	http_elasticsearch	ingress throughput	+1.93	[+1.84, +2.01]
➖	splunk_hec_route_s3	ingress throughput	+1.31	[+0.86, +1.77]
➖	http_text_to_http_json	ingress throughput	+1.04	[+0.89, +1.18]
➖	syslog_regex_logs2metric_ddmetrics	ingress throughput	+0.78	[+0.69, +0.88]
➖	datadog_agent_remap_blackhole	ingress throughput	+0.43	[+0.34, +0.52]
➖	otlp_grpc_to_blackhole	ingress throughput	+0.34	[+0.25, +0.44]
➖	syslog_splunk_hec_logs	ingress throughput	+0.18	[+0.06, +0.29]
➖	http_to_http_noack	ingress throughput	+0.15	[+0.06, +0.24]
➖	http_to_http_json	ingress throughput	+0.05	[-0.03, +0.13]
➖	splunk_hec_indexer_ack_blackhole	ingress throughput	+0.00	[-0.14, +0.15]
➖	splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-0.00	[-0.14, +0.14]
➖	http_to_s3	ingress throughput	-0.03	[-0.31, +0.25]
➖	splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.04	[-0.16, +0.07]
➖	datadog_agent_remap_datadog_logs_acks	ingress throughput	-0.07	[-0.16, +0.02]
➖	http_to_http_acks	ingress throughput	-0.11	[-1.47, +1.26]
➖	enterprise_http_to_http	ingress throughput	-0.12	[-0.19, -0.05]
➖	datadog_agent_remap_blackhole_acks	ingress throughput	-0.21	[-0.33, -0.10]
➖	syslog_log2metric_splunk_hec_metrics	ingress throughput	-0.75	[-0.90, -0.60]
➖	syslog_loki	ingress throughput	-0.88	[-0.93, -0.83]
➖	datadog_agent_remap_datadog_logs	ingress throughput	-1.05	[-1.15, -0.94]
➖	syslog_log2metric_tag_cardinality_limit_blackhole	ingress throughput	-1.70	[-1.84, -1.56]
➖	fluent_elasticsearch	ingress throughput	-2.03	[-2.51, -1.55]
➖	socket_to_socket_blackhole	ingress throughput	-2.70	[-2.78, -2.61]
❌	otlp_http_to_blackhole	ingress throughput	-5.21	[-5.34, -5.07]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".

[pabloem]

fwiw the fact that the possible regression appears in two different tests in two different runs (and those two tests are not correlated between runs) makes me think the regressions are not caused by this change : )

[jszwedko]

fwiw the fact that the possible regression appears in two different tests in two different runs (and those two tests are not correlated between runs) makes me think the regressions are not caused by this change : )

It does seem suspicious that the runs didn't match up. I could see a change like this having some overhead but it should be relatively negligible. It also wouldn't affect otlp_http_to_blackhole though it would have potentially affected syslog_regex_logs2metric_ddmetrics.

[github-actions[bot]]

Regression Detector Results

Run ID: 72a62763-307f-41ff-9cb7-201e3d14b87c Baseline: 7d7b1a2620c65540183298aac804aa4f7abe48ec Comparison: f1439bc42e8a9498b169c14eb030d8d6f2530ac8 Total CPUs: 7

Performance changes are noted in the perf column of each table:

• ✅ = significantly better comparison variant performance
• ❌ = significantly worse comparison variant performance
• ➖ = no significant change in performance

No significant changes in experiment optimization goals

Confidence level: 90.00% Effect size tolerance: |Δ mean %| ≥ 5.00%

There were no significant changes in experiment optimization goals at this confidence level and effect size tolerance.

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI
➖	syslog_regex_logs2metric_ddmetrics	ingress throughput	+2.50	[+2.37, +2.62]
➖	syslog_log2metric_splunk_hec_metrics	ingress throughput	+1.85	[+1.69, +2.01]
➖	otlp_http_to_blackhole	ingress throughput	+1.78	[+1.65, +1.92]
➖	http_text_to_http_json	ingress throughput	+1.71	[+1.59, +1.84]
➖	syslog_log2metric_humio_metrics	ingress throughput	+1.66	[+1.52, +1.81]
➖	fluent_elasticsearch	ingress throughput	+1.48	[+0.99, +1.96]
➖	syslog_loki	ingress throughput	+1.29	[+1.23, +1.34]
➖	syslog_splunk_hec_logs	ingress throughput	+1.23	[+1.16, +1.31]
➖	datadog_agent_remap_blackhole_acks	ingress throughput	+0.74	[+0.65, +0.83]
➖	otlp_grpc_to_blackhole	ingress throughput	+0.63	[+0.54, +0.73]
➖	http_to_http_acks	ingress throughput	+0.63	[-0.74, +2.00]
➖	http_to_http_noack	ingress throughput	+0.17	[+0.08, +0.27]
➖	socket_to_socket_blackhole	ingress throughput	+0.09	[+0.03, +0.16]
➖	splunk_hec_route_s3	ingress throughput	+0.05	[-0.40, +0.50]
➖	http_to_http_json	ingress throughput	+0.00	[-0.07, +0.07]
➖	splunk_hec_to_splunk_hec_logs_acks	ingress throughput	-0.00	[-0.14, +0.14]
➖	splunk_hec_indexer_ack_blackhole	ingress throughput	-0.00	[-0.15, +0.14]
➖	splunk_hec_to_splunk_hec_logs_noack	ingress throughput	-0.03	[-0.15, +0.08]
➖	http_to_s3	ingress throughput	-0.06	[-0.34, +0.22]
➖	enterprise_http_to_http	ingress throughput	-0.13	[-0.21, -0.04]
➖	http_elasticsearch	ingress throughput	-0.71	[-0.78, -0.64]
➖	datadog_agent_remap_blackhole	ingress throughput	-0.88	[-0.98, -0.79]
➖	datadog_agent_remap_datadog_logs_acks	ingress throughput	-0.89	[-0.98, -0.80]
➖	syslog_humio_logs	ingress throughput	-1.47	[-1.58, -1.35]
➖	syslog_log2metric_tag_cardinality_limit_blackhole	ingress throughput	-1.57	[-1.70, -1.45]
➖	datadog_agent_remap_datadog_logs	ingress throughput	-1.93	[-2.04, -1.82]
➖	file_to_blackhole	egress throughput	-2.59	[-4.98, -0.20]

Explanation

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

3. Its configuration does not mark it "erratic".