Closes #3649

Implements multiple host feature for elasticsearch sink with failover. Where the amount of inflight requests for each endpoint is an estimate of its load.

Possible extensions

Related to splitting ARC layer, https://github.com/vectordotdev/vector/issues/3649#issuecomment-1148033374, it's possible to split of from ARC controller statistic gathering component although some changes/refactoring will be needed. Once extracted, that component can be added to HealthService or even better can be it's own layer which would provide load estimate for a specific endpoint where the load would be something like in_flight/current_limit as float.
Sniffing feature can be added by implementing Stream for it and replacing fixed list of services with it at https://github.com/ktff/vector/blob/07bbc61be01ca16b4396b4554c144c9bbadec6c4/src/sinks/elasticsearch/config.rs#L358
Reusing distribution layer should be relatively straightforward for sinks that use TowerRequestSettings.

Todo

[x] Internal metrics
[x] Internal logs
[x] Documentation

Jun 20 '22 13:06 ktff

Deploy Preview for vector-project canceled.

Name	Link
Latest commit	02d9add57437ddcb20bbf25e3c2b3a44f1b128bb
Latest deploy log	https://app.netlify.com/sites/vector-project/deploys/62f2515d5fa8050009761c5a

Jun 20 '22 13:06 netlify[bot]

Soak Test Results

Baseline: f296e018ec9077f28b912ab0f9928388e9e9b5fa Comparison: a4801723be0f3a58d98953b5dd040dbeb36ff7d5 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_log2metric_splunk_hec_metrics	413.75KiB	2.3	100.00%	17.53MiB	770.48KiB	15.68KiB	0.0429051	17.94MiB	638.6KiB	13.01KiB	0.0347604	False	False
splunk_hec_route_s3	431.44KiB	2.22	100.00%	18.99MiB	2.36MiB	49.4KiB	0.124408	19.42MiB	2.2MiB	46.01KiB	0.11332	False	False
syslog_log2metric_humio_metrics	302.58KiB	2.14	100.00%	13.81MiB	410.0KiB	8.36KiB	0.0289956	14.1MiB	154.39KiB	3.16KiB	0.0106897	False	False
syslog_loki	327.22KiB	2.1	100.00%	15.19MiB	561.32KiB	11.44KiB	0.0360821	15.51MiB	326.85KiB	6.67KiB	0.0205772	False	False
syslog_splunk_hec_logs	269.41KiB	1.64	100.00%	16.0MiB	140.72KiB	2.87KiB	0.00858443	16.27MiB	141.44KiB	2.89KiB	0.00848889	False	False
http_pipelines_blackhole_acks	52.87KiB	1.2	99.87%	4.29MiB	574.67KiB	11.75KiB	0.130645	4.35MiB	560.63KiB	11.45KiB	0.125939	False	False
syslog_humio_logs	189.09KiB	1.11	100.00%	16.7MiB	134.53KiB	2.76KiB	0.00786623	16.88MiB	141.45KiB	2.89KiB	0.0081804	False	False
syslog_regex_logs2metric_ddmetrics	122.6KiB	0.96	100.00%	12.49MiB	588.01KiB	11.98KiB	0.0459779	12.61MiB	699.98KiB	14.24KiB	0.0542133	False	False
http_pipelines_no_grok_blackhole	160.75KiB	0.94	99.98%	16.63MiB	1.3MiB	27.3KiB	0.0783276	16.78MiB	1.64MiB	34.15KiB	0.0976064	False	False
http_to_http_noack	96.25KiB	0.4	100.00%	23.75MiB	1.05MiB	21.89KiB	0.0441933	23.85MiB	240.86KiB	4.92KiB	0.00986198	False	False
datadog_agent_remap_datadog_logs	89.16KiB	0.13	77.26%	67.64MiB	3.01MiB	62.73KiB	0.0444299	67.73MiB	1.86MiB	38.98KiB	0.0274449	False	False
fluent_elasticsearch	32.58KiB	0.04	97.75%	79.44MiB	707.39KiB	14.22KiB	0.00869404	79.47MiB	54.7KiB	1.1KiB	0.000671959	False	False
http_to_http_json	758.99B	0	6.14%	23.84MiB	333.64KiB	6.82KiB	0.0136615	23.85MiB	332.11KiB	6.78KiB	0.0135983	False	False
splunk_hec_to_splunk_hec_logs_noack	477.97B	0	3.98%	23.84MiB	321.61KiB	6.58KiB	0.0131732	23.84MiB	323.82KiB	6.63KiB	0.0132635	False	False
splunk_hec_indexer_ack_blackhole	-10.46KiB	-0.04	36.08%	23.77MiB	751.05KiB	15.28KiB	0.0308522	23.76MiB	798.67KiB	16.25KiB	0.0328223	False	False
splunk_hec_to_splunk_hec_logs_acks	-15.04KiB	-0.06	49.73%	23.77MiB	750.54KiB	15.27KiB	0.0308319	23.75MiB	808.4KiB	16.44KiB	0.0332295	False	False
socket_to_socket_blackhole	-16.39KiB	-0.11	73.66%	14.04MiB	548.38KiB	11.17KiB	0.0381477	14.02MiB	464.63KiB	9.48KiB	0.0323591	False	False
file_to_blackhole	-196.73KiB	-0.2	95.26%	95.29MiB	3.3MiB	68.49KiB	0.0346013	95.1MiB	3.48MiB	71.77KiB	0.0365442	False	False
http_pipelines_blackhole	-16.24KiB	-0.36	65.43%	4.44MiB	580.42KiB	11.9KiB	0.127748	4.42MiB	610.59KiB	12.45KiB	0.134869	False	False
datadog_agent_remap_blackhole_acks	-273.2KiB	-0.39	99.80%	68.1MiB	3.29MiB	68.59KiB	0.0482674	67.83MiB	2.66MiB	55.51KiB	0.039171	False	False
datadog_agent_remap_datadog_logs_acks	-423.31KiB	-0.61	100.00%	67.94MiB	1.02MiB	21.44KiB	0.0150723	67.53MiB	3.08MiB	64.33KiB	0.0455558	False	False
datadog_agent_remap_blackhole	-563.22KiB	-0.84	100.00%	65.5MiB	844.08KiB	17.28KiB	0.0125828	64.95MiB	951.54KiB	19.47KiB	0.0143049	False	False
http_to_http_acks	-478.9KiB	-2.57	96.59%	18.23MiB	7.3MiB	152.6KiB	0.40025	17.77MiB	7.97MiB	166.63KiB	0.448641	True	True

Jun 20 '22 15:06 github-actions[bot]

Soak Test Results

Baseline: f296e018ec9077f28b912ab0f9928388e9e9b5fa Comparison: 0309d8a7a9c1dfbdf6211323f3194f34f583f35e Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

Changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

experiment	Δ mean	Δ mean %	confidence
datadog_agent_remap_blackhole	-6.4MiB	-9.94	100.00%

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_log2metric_humio_metrics	499.72KiB	3.54	100.00%	13.77MiB	653.76KiB	13.33KiB	0.0463627	14.26MiB	126.05KiB	2.58KiB	0.00863294	False	False
syslog_loki	351.03KiB	2.38	100.00%	14.38MiB	997.42KiB	20.34KiB	0.067711	14.73MiB	1.06MiB	22.11KiB	0.0717791	False	False
http_pipelines_no_grok_blackhole	258.85KiB	1.46	100.00%	17.27MiB	1.08MiB	22.53KiB	0.0622676	17.52MiB	1.15MiB	23.97KiB	0.0655232	False	False
syslog_log2metric_splunk_hec_metrics	250.0KiB	1.39	100.00%	17.55MiB	858.46KiB	17.45KiB	0.0477458	17.8MiB	447.47KiB	9.12KiB	0.0245461	False	False
http_pipelines_blackhole_acks	40.6KiB	0.96	96.71%	4.13MiB	670.74KiB	13.71KiB	0.158401	4.17MiB	645.96KiB	13.19KiB	0.151101	False	False
datadog_agent_remap_blackhole_acks	618.35KiB	0.9	100.00%	67.12MiB	3.71MiB	77.39KiB	0.0552892	67.72MiB	2.24MiB	46.74KiB	0.0330411	False	False
syslog_splunk_hec_logs	106.82KiB	0.64	100.00%	16.4MiB	412.61KiB	8.42KiB	0.0245587	16.51MiB	385.16KiB	7.87KiB	0.02278	False	False
syslog_humio_logs	109.83KiB	0.64	100.00%	16.66MiB	121.64KiB	2.49KiB	0.00712825	16.77MiB	132.89KiB	2.72KiB	0.00773753	False	False
syslog_regex_logs2metric_ddmetrics	78.78KiB	0.62	100.00%	12.44MiB	624.91KiB	12.72KiB	0.0490622	12.51MiB	651.93KiB	13.27KiB	0.0508686	False	False
http_to_http_noack	95.38KiB	0.39	100.00%	23.75MiB	1.05MiB	21.88KiB	0.0441676	23.85MiB	240.06KiB	4.9KiB	0.00982921	False	False
http_to_http_acks	65.57KiB	0.36	21.40%	17.62MiB	8.14MiB	170.22KiB	0.462077	17.68MiB	8.19MiB	171.28KiB	0.463271	True	True
socket_to_socket_blackhole	49.74KiB	0.35	98.16%	13.86MiB	667.81KiB	13.61KiB	0.0470294	13.91MiB	790.29KiB	16.11KiB	0.0554601	False	False
http_pipelines_blackhole	9.84KiB	0.22	42.88%	4.46MiB	591.73KiB	12.11KiB	0.129444	4.47MiB	610.8KiB	12.46KiB	0.133328	False	False
splunk_hec_indexer_ack_blackhole	13.66KiB	0.06	46.42%	23.76MiB	790.24KiB	16.07KiB	0.0324755	23.77MiB	742.87KiB	15.12KiB	0.0305116	False	False
splunk_hec_to_splunk_hec_logs_acks	9.41KiB	0.04	32.87%	23.76MiB	789.5KiB	16.06KiB	0.0324449	23.77MiB	751.15KiB	15.29KiB	0.0308571	False	False
fluent_elasticsearch	18.89KiB	0.02	92.72%	79.45MiB	519.84KiB	10.47KiB	0.00638798	79.47MiB	53.33KiB	1.07KiB	0.000655168	False	False
splunk_hec_route_s3	2.8KiB	0.01	3.47%	19.62MiB	2.17MiB	45.45KiB	0.110816	19.62MiB	2.18MiB	45.62KiB	0.111202	False	False
splunk_hec_to_splunk_hec_logs_noack	400.99B	0	3.32%	23.83MiB	324.21KiB	6.64KiB	0.0132807	23.84MiB	325.46KiB	6.67KiB	0.0133316	False	False
http_to_http_json	937.66B	0	7.77%	23.85MiB	328.69KiB	6.71KiB	0.0134584	23.85MiB	321.67KiB	6.57KiB	0.0131703	False	False
datadog_agent_remap_datadog_logs_acks	-132.19KiB	-0.2	81.72%	65.96MiB	2.94MiB	61.42KiB	0.0445195	65.84MiB	3.73MiB	77.9KiB	0.0566113	False	False
file_to_blackhole	-315.92KiB	-0.32	98.81%	95.28MiB	3.42MiB	70.98KiB	0.0358594	94.98MiB	5.02MiB	103.56KiB	0.0528547	False	False
datadog_agent_remap_datadog_logs	-364.72KiB	-0.52	99.99%	68.22MiB	2.41MiB	50.35KiB	0.0353403	67.86MiB	3.85MiB	80.37KiB	0.0567189	False	False
datadog_agent_remap_blackhole	-6.4MiB	-9.94	100.00%	64.39MiB	6.97MiB	146.02KiB	0.108239	57.99MiB	9.76MiB	204.44KiB	0.168248	False	False

Jun 20 '22 23:06 github-actions[bot]

@jszwedko this is ready for review.

cc. @sim0nx

Jun 21 '22 12:06 ktff

Soak Test Results

Baseline: f296e018ec9077f28b912ab0f9928388e9e9b5fa Comparison: 383b28d1c7b913a89b10a73cda21c397bda09df5 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_log2metric_humio_metrics	272.24KiB	1.96	100.00%	13.54MiB	548.05KiB	11.18KiB	0.0395222	13.8MiB	512.84KiB	10.51KiB	0.0362711	False	False
http_pipelines_no_grok_blackhole	239.79KiB	1.37	100.00%	17.07MiB	1.14MiB	23.87KiB	0.0667839	17.3MiB	1.18MiB	24.63KiB	0.0681743	False	False
socket_to_socket_blackhole	146.08KiB	1.03	100.00%	13.89MiB	639.55KiB	13.02KiB	0.0449666	14.03MiB	486.45KiB	9.92KiB	0.0338546	False	False
splunk_hec_route_s3	147.97KiB	0.78	93.76%	18.52MiB	2.69MiB	56.32KiB	0.145458	18.66MiB	2.68MiB	55.96KiB	0.143413	False	False
syslog_log2metric_splunk_hec_metrics	76.24KiB	0.42	99.93%	17.6MiB	842.76KiB	17.14KiB	0.0467461	17.68MiB	705.8KiB	14.37KiB	0.0389846	False	False
datadog_agent_remap_blackhole	252.13KiB	0.37	100.00%	67.14MiB	719.26KiB	14.72KiB	0.0104589	67.39MiB	650.52KiB	13.31KiB	0.00942465	False	False
syslog_loki	56.25KiB	0.36	100.00%	15.11MiB	566.73KiB	11.55KiB	0.0366223	15.16MiB	226.7KiB	4.63KiB	0.0145965	False	False
http_to_http_noack	69.1KiB	0.28	99.96%	23.78MiB	931.55KiB	18.98KiB	0.0382507	23.85MiB	237.77KiB	4.85KiB	0.00973564	False	False
http_pipelines_blackhole	10.82KiB	0.25	45.99%	4.31MiB	608.09KiB	12.46KiB	0.137836	4.32MiB	613.29KiB	12.51KiB	0.138674	False	False
datadog_agent_remap_datadog_logs_acks	140.65KiB	0.22	60.55%	62.02MiB	5.46MiB	114.27KiB	0.0879946	62.16MiB	5.7MiB	119.24KiB	0.0916	False	False
fluent_elasticsearch	40.56KiB	0.05	95.70%	79.43MiB	996.95KiB	20.0KiB	0.0122541	79.47MiB	56.12KiB	1.13KiB	0.000689499	False	False
splunk_hec_indexer_ack_blackhole	4.71KiB	0.02	16.98%	23.76MiB	772.43KiB	15.71KiB	0.0317377	23.77MiB	753.14KiB	15.33KiB	0.030939	False	False
http_to_http_json	53.31B	0	0.42%	23.84MiB	345.82KiB	7.06KiB	0.0141615	23.84MiB	340.89KiB	6.96KiB	0.0139594	False	False
splunk_hec_to_splunk_hec_logs_noack	-787.83B	-0	6.52%	23.84MiB	323.24KiB	6.62KiB	0.0132404	23.84MiB	326.19KiB	6.68KiB	0.0133616	False	False
syslog_regex_logs2metric_ddmetrics	-7.86KiB	-0.06	32.35%	12.28MiB	637.08KiB	12.97KiB	0.0506475	12.27MiB	672.39KiB	13.67KiB	0.0534877	False	False
splunk_hec_to_splunk_hec_logs_acks	-18.74KiB	-0.08	60.08%	23.77MiB	731.97KiB	14.9KiB	0.0300631	23.75MiB	810.95KiB	16.49KiB	0.0333325	False	False
file_to_blackhole	-233.21KiB	-0.24	98.81%	95.38MiB	2.95MiB	61.36KiB	0.030886	95.16MiB	3.37MiB	69.5KiB	0.0353701	False	False
syslog_humio_logs	-42.21KiB	-0.25	100.00%	16.44MiB	218.44KiB	4.48KiB	0.0129743	16.4MiB	242.74KiB	4.97KiB	0.0144539	False	False
syslog_splunk_hec_logs	-45.91KiB	-0.27	99.99%	16.4MiB	388.77KiB	7.94KiB	0.0231486	16.35MiB	396.8KiB	8.11KiB	0.0236914	False	False
datadog_agent_remap_datadog_logs	-502.93KiB	-0.72	100.00%	67.96MiB	2.63MiB	54.91KiB	0.0386912	67.47MiB	4.1MiB	85.54KiB	0.0607635	False	False
http_pipelines_blackhole_acks	-58.58KiB	-1.39	99.71%	4.11MiB	659.61KiB	13.49KiB	0.156642	4.05MiB	701.08KiB	14.32KiB	0.168838	False	False
http_to_http_acks	-315.43KiB	-1.71	83.18%	17.99MiB	7.56MiB	158.01KiB	0.420028	17.68MiB	7.92MiB	165.58KiB	0.447908	True	True
datadog_agent_remap_blackhole_acks	-1.84MiB	-2.71	100.00%	67.67MiB	3.38MiB	70.46KiB	0.0498698	65.84MiB	4.96MiB	103.66KiB	0.0753092	False	False

Jun 21 '22 14:06 github-actions[bot]

Soak Test Results

Baseline: f296e018ec9077f28b912ab0f9928388e9e9b5fa Comparison: 9c791f830cf09a5b72c69300450132e3cd79bd4c Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_log2metric_humio_metrics	350.26KiB	2.47	100.00%	13.82MiB	551.88KiB	11.25KiB	0.0389872	14.16MiB	138.2KiB	2.83KiB	0.00952712	False	False
socket_to_socket_blackhole	164.97KiB	1.17	100.00%	13.81MiB	662.66KiB	13.5KiB	0.0468431	13.97MiB	551.35KiB	11.25KiB	0.0385258	False	False
http_pipelines_blackhole_acks	36.97KiB	0.82	97.82%	4.38MiB	564.04KiB	11.53KiB	0.125601	4.42MiB	550.89KiB	11.25KiB	0.121671	False	False
http_pipelines_blackhole	28.73KiB	0.7	79.75%	4.03MiB	777.41KiB	15.94KiB	0.188568	4.05MiB	781.88KiB	15.94KiB	0.18834	False	False
datadog_agent_remap_blackhole_acks	354.11KiB	0.53	99.98%	65.34MiB	3.64MiB	75.87KiB	0.0556307	65.69MiB	2.84MiB	59.32KiB	0.0432176	False	False
splunk_hec_route_s3	102.21KiB	0.51	90.14%	19.65MiB	2.1MiB	43.87KiB	0.106783	19.75MiB	2.09MiB	43.64KiB	0.10567	False	False
syslog_log2metric_splunk_hec_metrics	81.61KiB	0.45	99.99%	17.61MiB	777.56KiB	15.82KiB	0.0431019	17.69MiB	618.03KiB	12.59KiB	0.0341045	False	False
http_to_http_noack	78.87KiB	0.32	99.99%	23.77MiB	969.58KiB	19.75KiB	0.0398272	23.85MiB	249.97KiB	5.1KiB	0.0102348	False	False
http_to_http_acks	48.02KiB	0.26	15.76%	17.8MiB	8.09MiB	169.07KiB	0.454177	17.84MiB	8.25MiB	172.37KiB	0.462018	True	True
fluent_elasticsearch	84.74KiB	0.1	97.42%	79.39MiB	1.84MiB	37.98KiB	0.0232086	79.47MiB	52.16KiB	1.05KiB	0.000640825	False	False
datadog_agent_remap_blackhole	63.01KiB	0.1	55.86%	63.41MiB	2.89MiB	60.49KiB	0.0455268	63.47MiB	2.63MiB	55.11KiB	0.0414788	False	False
syslog_loki	11.71KiB	0.08	56.92%	14.71MiB	595.04KiB	12.13KiB	0.0395029	14.72MiB	419.77KiB	8.58KiB	0.0278456	False	False
syslog_regex_logs2metric_ddmetrics	4.03KiB	0.03	20.60%	12.0MiB	486.1KiB	9.9KiB	0.039558	12.0MiB	581.64KiB	11.83KiB	0.0473168	False	False
http_to_http_json	2.45KiB	0.01	19.85%	23.84MiB	338.04KiB	6.91KiB	0.0138414	23.85MiB	336.37KiB	6.87KiB	0.0137718	False	False
splunk_hec_to_splunk_hec_logs_noack	1017.71B	0	8.44%	23.84MiB	322.12KiB	6.6KiB	0.0131949	23.84MiB	325.77KiB	6.67KiB	0.0133439	False	False
syslog_splunk_hec_logs	-11.67KiB	-0.07	56.69%	16.12MiB	535.89KiB	10.94KiB	0.0324538	16.11MiB	494.71KiB	10.1KiB	0.0299812	False	False
splunk_hec_indexer_ack_blackhole	-19.8KiB	-0.08	63.88%	23.78MiB	713.33KiB	14.52KiB	0.0292916	23.76MiB	791.54KiB	16.1KiB	0.0325298	False	False
splunk_hec_to_splunk_hec_logs_acks	-28.82KiB	-0.12	81.71%	23.78MiB	687.81KiB	14.0KiB	0.0282371	23.75MiB	810.81KiB	16.49KiB	0.0333258	False	False
syslog_humio_logs	-29.7KiB	-0.17	100.00%	16.6MiB	124.29KiB	2.55KiB	0.0073086	16.57MiB	125.07KiB	2.56KiB	0.00736754	False	False
file_to_blackhole	-232.67KiB	-0.24	97.86%	95.37MiB	3.29MiB	68.37KiB	0.0344482	95.14MiB	3.61MiB	74.45KiB	0.0378957	False	False
datadog_agent_remap_datadog_logs	-185.18KiB	-0.27	97.88%	66.71MiB	2.83MiB	59.01KiB	0.042359	66.53MiB	2.6MiB	54.5KiB	0.0390996	False	False
datadog_agent_remap_datadog_logs_acks	-465.67KiB	-0.67	100.00%	67.87MiB	643.61KiB	13.15KiB	0.0092584	67.42MiB	2.69MiB	56.22KiB	0.0398442	False	False
http_pipelines_no_grok_blackhole	-176.75KiB	-1	100.00%	17.2MiB	1.22MiB	25.49KiB	0.070737	17.03MiB	1.63MiB	34.02KiB	0.0957281	False	False

Jun 21 '22 14:06 github-actions[bot]

Soak Test Results

Baseline: 00d9de7d50335f84b6bbddad64180c3f8f9dc690 Comparison: ed9ca3002ce58cf3f7d7da89c37b8381db520b8d Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_log2metric_humio_metrics	326.85KiB	2.38	100.00%	13.44MiB	383.02KiB	7.82KiB	0.0278327	13.76MiB	581.68KiB	11.85KiB	0.0412881	False	False
syslog_log2metric_splunk_hec_metrics	255.35KiB	1.53	100.00%	16.29MiB	1.56MiB	32.59KiB	0.0958238	16.54MiB	1.63MiB	33.89KiB	0.0983137	False	False
datadog_agent_remap_blackhole_acks	754.02KiB	1.12	100.00%	65.83MiB	4.99MiB	103.83KiB	0.0757549	66.56MiB	2.85MiB	59.64KiB	0.0427822	False	False
http_pipelines_blackhole_acks	13.22KiB	1.07	100.00%	1.2MiB	106.63KiB	2.17KiB	0.0866735	1.21MiB	54.38KiB	1.11KiB	0.0437331	False	False
syslog_splunk_hec_logs	165.79KiB	0.98	100.00%	16.6MiB	710.8KiB	14.48KiB	0.0417952	16.77MiB	619.48KiB	12.63KiB	0.0360738	False	False
syslog_humio_logs	139.03KiB	0.81	100.00%	16.7MiB	114.87KiB	2.35KiB	0.00671447	16.84MiB	114.01KiB	2.33KiB	0.00661034	False	False
syslog_regex_logs2metric_ddmetrics	100.3KiB	0.78	100.00%	12.57MiB	539.65KiB	11.0KiB	0.0419079	12.67MiB	566.31KiB	11.54KiB	0.0436385	False	False
datadog_agent_remap_blackhole	305.99KiB	0.47	96.93%	63.38MiB	5.3MiB	110.46KiB	0.0836703	63.68MiB	4.24MiB	88.48KiB	0.0665515	False	False
http_pipelines_blackhole	2.32KiB	0.13	72.32%	1.68MiB	10.9KiB	228.16B	0.0063364	1.68MiB	104.14KiB	2.12KiB	0.0604458	False	False
splunk_hec_to_splunk_hec_logs_noack	15.83KiB	0.06	82.19%	23.82MiB	472.72KiB	9.65KiB	0.0193751	23.84MiB	328.65KiB	6.71KiB	0.0134615	False	False
splunk_hec_indexer_ack_blackhole	9.31KiB	0.04	28.36%	23.75MiB	906.95KiB	18.45KiB	0.037285	23.76MiB	874.85KiB	17.8KiB	0.0359517	False	False
http_to_http_acks	1.76KiB	0.01	0.59%	17.76MiB	8.02MiB	167.62KiB	0.451323	17.76MiB	8.0MiB	166.94KiB	0.450392	True	True
splunk_hec_to_splunk_hec_logs_acks	-3.22KiB	-0.01	10.16%	23.75MiB	871.88KiB	17.73KiB	0.0358486	23.74MiB	884.51KiB	17.99KiB	0.0363728	False	False
file_to_blackhole	-61.6KiB	-0.06	46.86%	95.37MiB	3.28MiB	67.9KiB	0.0343398	95.31MiB	3.42MiB	71.25KiB	0.035924	False	False
http_to_http_json	-53.67KiB	-0.22	99.99%	23.85MiB	335.23KiB	6.84KiB	0.0137252	23.79MiB	604.61KiB	12.33KiB	0.0248087	False	False
fluent_elasticsearch	-206.49KiB	-0.25	100.00%	79.47MiB	54.53KiB	1.1KiB	0.000669923	79.27MiB	2.25MiB	46.23KiB	0.028383	False	False
http_to_http_noack	-71.89KiB	-0.29	99.95%	23.84MiB	360.17KiB	7.37KiB	0.0147513	23.77MiB	946.2KiB	19.27KiB	0.0388675	False	False
syslog_loki	-55.42KiB	-0.37	99.50%	14.47MiB	615.93KiB	12.6KiB	0.0415574	14.42MiB	748.66KiB	15.22KiB	0.0507028	False	False
socket_to_socket_blackhole	-80.26KiB	-0.57	100.00%	13.71MiB	117.24KiB	2.39KiB	0.00834713	13.64MiB	185.06KiB	3.78KiB	0.0132507	False	False
splunk_hec_route_s3	-177.57KiB	-0.91	99.17%	19.06MiB	2.29MiB	47.64KiB	0.119881	18.89MiB	2.27MiB	47.45KiB	0.120238	False	False
datadog_agent_remap_datadog_logs_acks	-690.74KiB	-1.03	100.00%	65.56MiB	3.22MiB	67.32KiB	0.0490761	64.89MiB	4.62MiB	96.22KiB	0.0712099	False	False
datadog_agent_remap_datadog_logs	-851.41KiB	-1.24	100.00%	66.81MiB	925.61KiB	18.96KiB	0.0135264	65.98MiB	4.37MiB	90.93KiB	0.0661485	False	False
http_pipelines_no_grok_blackhole	-155.66KiB	-1.28	100.00%	11.86MiB	81.75KiB	1.67KiB	0.00673292	11.7MiB	1.09MiB	22.72KiB	0.0931666	False	False
http_text_to_http_json	-620.54KiB	-1.54	100.00%	39.44MiB	807.16KiB	16.48KiB	0.0199835	38.83MiB	807.82KiB	16.5KiB	0.020312	False	False

Jun 22 '22 12:06 github-actions[bot]

Soak Test Results

Baseline: ddb71f87e203fcd1b48f2491c7a9b46a64603416 Comparison: 4bec635c374780b637771ee350301d209e847d94 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_pipelines_blackhole_acks	13.88KiB	1.13	100.00%	1.2MiB	102.08KiB	2.08KiB	0.0832987	1.21MiB	62.61KiB	1.28KiB	0.0505171	False	False
syslog_regex_logs2metric_ddmetrics	80.55KiB	0.63	100.00%	12.47MiB	598.55KiB	12.19KiB	0.0468481	12.55MiB	559.05KiB	11.39KiB	0.0434822	False	False
syslog_log2metric_splunk_hec_metrics	47.26KiB	0.27	92.70%	17.19MiB	884.44KiB	18.03KiB	0.0502279	17.24MiB	943.73KiB	19.22KiB	0.0534516	False	False
splunk_hec_to_splunk_hec_logs_noack	10.3KiB	0.04	65.36%	23.83MiB	423.64KiB	8.65KiB	0.0173593	23.84MiB	327.83KiB	6.69KiB	0.0134277	False	False
splunk_hec_indexer_ack_blackhole	6.79KiB	0.03	21.00%	23.74MiB	906.88KiB	18.44KiB	0.0372944	23.75MiB	866.43KiB	17.62KiB	0.0356211	False	False
file_to_blackhole	-35.88KiB	-0.04	33.72%	95.37MiB	2.88MiB	59.6KiB	0.0301434	95.33MiB	2.73MiB	56.69KiB	0.0285881	False	False
splunk_hec_to_splunk_hec_logs_acks	-10.97KiB	-0.05	34.28%	23.75MiB	837.55KiB	17.04KiB	0.0344251	23.74MiB	881.14KiB	17.92KiB	0.036233	False	False
http_to_http_json	-32.82KiB	-0.13	99.06%	23.85MiB	340.45KiB	6.95KiB	0.0139399	23.81MiB	515.94KiB	10.54KiB	0.0211537	False	False
http_text_to_http_json	-58.51KiB	-0.15	98.77%	37.6MiB	819.74KiB	16.73KiB	0.021285	37.54MiB	798.62KiB	16.31KiB	0.0207682	False	False
fluent_elasticsearch	-188.51KiB	-0.23	100.00%	79.47MiB	53.78KiB	1.09KiB	0.00066073	79.29MiB	1.76MiB	36.2KiB	0.0221605	False	False
http_pipelines_blackhole	-6.21KiB	-0.37	98.09%	1.65MiB	41.39KiB	866.66B	0.0244249	1.65MiB	123.17KiB	2.51KiB	0.0729441	False	False
syslog_splunk_hec_logs	-65.02KiB	-0.38	99.96%	16.58MiB	704.95KiB	14.36KiB	0.0415188	16.51MiB	572.35KiB	11.68KiB	0.0338384	False	False
http_to_http_noack	-95.11KiB	-0.39	100.00%	23.85MiB	249.26KiB	5.1KiB	0.0102062	23.75MiB	1.05MiB	21.99KiB	0.0443864	False	False
socket_to_socket_blackhole	-73.32KiB	-0.56	100.00%	12.84MiB	419.35KiB	8.56KiB	0.0318843	12.77MiB	451.89KiB	9.23KiB	0.0345512	False	False
datadog_agent_remap_datadog_logs	-386.49KiB	-0.56	100.00%	66.8MiB	481.74KiB	9.86KiB	0.00704103	66.42MiB	4.23MiB	88.1KiB	0.0636523	False	False
syslog_humio_logs	-102.87KiB	-0.62	100.00%	16.26MiB	408.34KiB	8.34KiB	0.024519	16.16MiB	393.73KiB	8.06KiB	0.0237889	False	False
datadog_agent_remap_blackhole	-660.81KiB	-0.97	100.00%	66.63MiB	4.52MiB	94.23KiB	0.0678851	65.98MiB	3.54MiB	73.9KiB	0.0536925	False	False
datadog_agent_remap_datadog_logs_acks	-673.74KiB	-1.07	99.76%	61.46MiB	7.22MiB	151.03KiB	0.117447	60.8MiB	7.81MiB	162.59KiB	0.128442	False	False
http_to_http_acks	-204.75KiB	-1.11	62.03%	17.95MiB	7.8MiB	163.19KiB	0.434734	17.75MiB	7.97MiB	166.42KiB	0.449176	True	True
syslog_loki	-194.24KiB	-1.24	100.00%	15.27MiB	285.33KiB	5.85KiB	0.0182462	15.08MiB	777.27KiB	15.8KiB	0.0503306	False	False
http_pipelines_no_grok_blackhole	-194.59KiB	-1.78	99.97%	10.67MiB	1.68MiB	35.04KiB	0.157013	10.48MiB	1.92MiB	40.01KiB	0.183266	False	False
splunk_hec_route_s3	-445.71KiB	-2.32	100.00%	18.73MiB	2.43MiB	50.66KiB	0.129917	18.29MiB	2.42MiB	50.54KiB	0.132203	False	False
syslog_log2metric_humio_metrics	-377.0KiB	-2.74	100.00%	13.46MiB	634.54KiB	12.96KiB	0.0460305	13.09MiB	701.45KiB	14.29KiB	0.0523157	False	False
datadog_agent_remap_blackhole_acks	-2.32MiB	-3.82	100.00%	60.66MiB	7.47MiB	155.5KiB	0.123107	58.34MiB	6.14MiB	128.61KiB	0.105288	False	False

Jun 22 '22 19:06 github-actions[bot]

Soak Test Results

Baseline: 91d88e8540bfe154038ca79265d28b31e6242c05 Comparison: 5a91f8987428a28330d34fff0583f238fde51123 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_pipelines_blackhole_acks	7.54KiB	0.61	99.22%	1.2MiB	115.23KiB	2.34KiB	0.0936754	1.21MiB	78.22KiB	1.59KiB	0.0632017	False	False
datadog_agent_remap_datadog_logs_acks	355.4KiB	0.54	98.27%	64.85MiB	4.89MiB	102.12KiB	0.0753824	65.2MiB	5.23MiB	108.92KiB	0.0802428	False	False
http_to_http_acks	76.77KiB	0.42	25.70%	17.68MiB	7.95MiB	166.17KiB	0.449596	17.75MiB	7.9MiB	164.91KiB	0.444855	True	True
splunk_hec_route_s3	13.24KiB	0.07	16.03%	19.37MiB	2.28MiB	47.53KiB	0.117893	19.38MiB	2.15MiB	44.99KiB	0.110929	False	False
splunk_hec_to_splunk_hec_logs_noack	12.1KiB	0.05	72.79%	23.83MiB	429.96KiB	8.77KiB	0.0176186	23.84MiB	326.25KiB	6.66KiB	0.0133621	False	False
splunk_hec_indexer_ack_blackhole	11.6KiB	0.05	36.33%	23.75MiB	863.46KiB	17.57KiB	0.0354908	23.77MiB	842.56KiB	17.15KiB	0.0346153	False	False
http_pipelines_blackhole	735.22B	0.04	30.31%	1.68MiB	8.89KiB	185.99B	0.00515938	1.68MiB	89.95KiB	1.83KiB	0.0521816	False	False
splunk_hec_to_splunk_hec_logs_acks	4.13KiB	0.02	13.70%	23.75MiB	833.24KiB	16.95KiB	0.0342545	23.75MiB	830.15KiB	16.89KiB	0.0341215	False	False
file_to_blackhole	-49.35KiB	-0.05	35.49%	95.34MiB	3.44MiB	71.39KiB	0.036116	95.29MiB	3.84MiB	79.92KiB	0.0403162	False	False
syslog_regex_logs2metric_ddmetrics	-13.35KiB	-0.11	55.30%	12.32MiB	636.17KiB	12.96KiB	0.0504264	12.3MiB	581.39KiB	11.85KiB	0.0461329	False	False
http_to_http_json	-49.14KiB	-0.2	99.96%	23.85MiB	339.91KiB	6.94KiB	0.0139171	23.8MiB	592.74KiB	12.09KiB	0.024318	False	False
syslog_log2metric_humio_metrics	-30.72KiB	-0.21	98.97%	13.98MiB	264.7KiB	5.4KiB	0.0184913	13.95MiB	523.87KiB	10.67KiB	0.0366752	False	False
http_text_to_http_json	-88.72KiB	-0.22	99.99%	38.69MiB	801.05KiB	16.35KiB	0.0202135	38.61MiB	809.17KiB	16.53KiB	0.020464	False	False
fluent_elasticsearch	-206.52KiB	-0.25	99.97%	79.47MiB	52.95KiB	1.07KiB	0.000650567	79.27MiB	2.74MiB	56.33KiB	0.0345315	False	False
http_to_http_noack	-80.1KiB	-0.33	99.99%	23.85MiB	242.23KiB	4.95KiB	0.00991765	23.77MiB	984.42KiB	20.06KiB	0.0404382	False	False
syslog_humio_logs	-72.12KiB	-0.42	100.00%	16.75MiB	233.03KiB	4.76KiB	0.0135853	16.68MiB	131.02KiB	2.68KiB	0.00767049	False	False
datadog_agent_remap_datadog_logs	-386.92KiB	-0.56	100.00%	67.12MiB	733.45KiB	15.02KiB	0.010669	66.74MiB	3.97MiB	82.8KiB	0.0595102	False	False
syslog_splunk_hec_logs	-104.85KiB	-0.62	100.00%	16.54MiB	796.51KiB	16.21KiB	0.0470096	16.44MiB	707.74KiB	14.42KiB	0.0420306	False	False
http_pipelines_no_grok_blackhole	-82.1KiB	-0.73	97.55%	10.99MiB	1010.32KiB	20.63KiB	0.0897837	10.91MiB	1.45MiB	30.1KiB	0.132482	False	False
datadog_agent_remap_blackhole_acks	-578.96KiB	-0.83	100.00%	68.08MiB	4.65MiB	96.76KiB	0.0682363	67.52MiB	2.94MiB	61.55KiB	0.0435486	False	False
datadog_agent_remap_blackhole	-548.29KiB	-0.87	99.85%	61.45MiB	6.31MiB	131.45KiB	0.102697	60.92MiB	5.4MiB	112.66KiB	0.0886282	False	False
syslog_log2metric_splunk_hec_metrics	-170.35KiB	-0.94	100.00%	17.7MiB	581.59KiB	11.85KiB	0.0320798	17.53MiB	671.65KiB	13.68KiB	0.0373984	False	False
syslog_loki	-162.79KiB	-1.04	100.00%	15.33MiB	366.93KiB	7.51KiB	0.0233761	15.17MiB	782.94KiB	15.92KiB	0.0504022	False	False
socket_to_socket_blackhole	-199.42KiB	-1.56	100.00%	12.51MiB	974.59KiB	19.89KiB	0.0760812	12.31MiB	913.15KiB	18.65KiB	0.0724123	False	False

Jun 23 '22 12:06 github-actions[bot]

Soak Test Results

Baseline: 91d88e8540bfe154038ca79265d28b31e6242c05 Comparison: 37aa3a2f22654197a91641fc17f2c3caf90647c4 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
datadog_agent_remap_blackhole_acks	1.45MiB	2.22	100.00%	65.0MiB	5.57MiB	116.08KiB	0.085706	66.44MiB	2.5MiB	52.32KiB	0.0376122	False	False
http_to_http_acks	232.44KiB	1.28	69.16%	17.7MiB	7.77MiB	162.46KiB	0.438759	17.93MiB	7.68MiB	160.2KiB	0.428051	True	True
http_pipelines_blackhole_acks	14.76KiB	1.2	100.00%	1.2MiB	100.09KiB	2.04KiB	0.0815193	1.21MiB	64.97KiB	1.33KiB	0.0522862	False	False
http_text_to_http_json	246.76KiB	0.62	100.00%	38.68MiB	1.1MiB	23.0KiB	0.028435	38.92MiB	782.32KiB	15.97KiB	0.019623	False	False
http_pipelines_blackhole	9.22KiB	0.54	99.97%	1.65MiB	59.64KiB	1.22KiB	0.0352124	1.66MiB	110.46KiB	2.25KiB	0.0648628	False	False
syslog_regex_logs2metric_ddmetrics	67.15KiB	0.53	100.00%	12.45MiB	597.18KiB	12.16KiB	0.0468442	12.51MiB	489.03KiB	9.97KiB	0.0381595	False	False
splunk_hec_to_splunk_hec_logs_noack	36.56KiB	0.15	98.98%	23.8MiB	616.12KiB	12.56KiB	0.0252753	23.84MiB	327.39KiB	6.68KiB	0.0134103	False	False
splunk_hec_indexer_ack_blackhole	-2.8KiB	-0.01	9.40%	23.76MiB	818.6KiB	16.65KiB	0.0336324	23.76MiB	827.42KiB	16.83KiB	0.0339986	False	False
splunk_hec_to_splunk_hec_logs_acks	-6.95KiB	-0.03	21.90%	23.76MiB	889.48KiB	18.09KiB	0.0365529	23.75MiB	849.12KiB	17.28KiB	0.0349043	False	False
splunk_hec_route_s3	-6.37KiB	-0.03	8.04%	19.61MiB	2.16MiB	44.95KiB	0.110081	19.6MiB	2.12MiB	44.26KiB	0.107973	False	False
file_to_blackhole	-57.16KiB	-0.06	41.59%	95.34MiB	3.21MiB	66.61KiB	0.0336932	95.28MiB	3.87MiB	80.42KiB	0.0405547	False	False
syslog_splunk_hec_logs	-24.49KiB	-0.15	78.05%	16.38MiB	829.71KiB	16.88KiB	0.0494681	16.35MiB	519.5KiB	10.61KiB	0.0310181	False	False
http_to_http_json	-51.14KiB	-0.21	99.98%	23.85MiB	336.28KiB	6.86KiB	0.0137677	23.8MiB	593.27KiB	12.1KiB	0.0243397	False	False
datadog_agent_remap_datadog_logs_acks	-237.06KiB	-0.34	94.66%	67.64MiB	3.57MiB	74.52KiB	0.0527155	67.41MiB	4.68MiB	97.45KiB	0.0694151	False	False
http_to_http_noack	-86.94KiB	-0.36	99.99%	23.84MiB	253.29KiB	5.18KiB	0.0103716	23.76MiB	1.02MiB	21.25KiB	0.0428596	False	False
fluent_elasticsearch	-433.17KiB	-0.53	100.00%	79.47MiB	150.56KiB	3.04KiB	0.00184969	79.05MiB	4.18MiB	85.91KiB	0.0529152	False	False
syslog_humio_logs	-91.39KiB	-0.54	100.00%	16.61MiB	164.84KiB	3.36KiB	0.00968701	16.52MiB	166.63KiB	3.41KiB	0.00984522	False	False
syslog_log2metric_splunk_hec_metrics	-113.21KiB	-0.64	100.00%	17.36MiB	674.48KiB	13.74KiB	0.0379407	17.25MiB	801.37KiB	16.31KiB	0.045367	False	False
datadog_agent_remap_datadog_logs	-486.72KiB	-0.7	100.00%	67.69MiB	328.67KiB	6.73KiB	0.00474048	67.22MiB	4.46MiB	92.87KiB	0.0663322	False	False
datadog_agent_remap_blackhole	-637.39KiB	-0.95	100.00%	65.48MiB	4.82MiB	100.38KiB	0.0736174	64.86MiB	3.44MiB	71.69KiB	0.0529728	False	False
syslog_loki	-157.92KiB	-1.02	100.00%	15.18MiB	315.64KiB	6.47KiB	0.0202982	15.03MiB	773.97KiB	15.73KiB	0.0502838	False	False
syslog_log2metric_humio_metrics	-178.52KiB	-1.29	100.00%	13.52MiB	423.64KiB	8.64KiB	0.0306024	13.34MiB	678.82KiB	13.81KiB	0.0496763	False	False
http_pipelines_no_grok_blackhole	-193.44KiB	-1.61	100.00%	11.74MiB	226.53KiB	4.62KiB	0.0188318	11.56MiB	1.17MiB	24.37KiB	0.101278	False	False
socket_to_socket_blackhole	-227.7KiB	-1.63	100.00%	13.67MiB	128.69KiB	2.63KiB	0.00918965	13.45MiB	110.42KiB	2.25KiB	0.00801507	False	False

Jun 23 '22 12:06 github-actions[bot]

Soak Test Results

Baseline: 91d88e8540bfe154038ca79265d28b31e6242c05 Comparison: 82b65193909be5b2ecd3ce1535fcaa80ddfbc8ce Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_pipelines_blackhole_acks	9.21KiB	0.74	99.93%	1.21MiB	108.91KiB	2.22KiB	0.0877148	1.22MiB	77.71KiB	1.58KiB	0.0621268	False	False
syslog_regex_logs2metric_ddmetrics	49.64KiB	0.4	99.96%	12.01MiB	481.64KiB	9.81KiB	0.0391397	12.06MiB	488.77KiB	9.96KiB	0.0395594	False	False
splunk_hec_indexer_ack_blackhole	18.1KiB	0.07	55.62%	23.74MiB	848.96KiB	17.26KiB	0.0349091	23.76MiB	793.39KiB	16.14KiB	0.0325995	False	False
file_to_blackhole	9.03KiB	0.01	7.27%	95.31MiB	3.17MiB	65.65KiB	0.0332484	95.32MiB	3.56MiB	74.06KiB	0.0373592	False	False
splunk_hec_to_splunk_hec_logs_noack	516.05B	0	4.22%	23.84MiB	330.8KiB	6.76KiB	0.01355	23.84MiB	327.85KiB	6.69KiB	0.013429	False	False
splunk_hec_to_splunk_hec_logs_acks	-6.32KiB	-0.03	19.84%	23.75MiB	862.97KiB	17.55KiB	0.0354787	23.74MiB	884.61KiB	17.99KiB	0.0363778	False	False
http_pipelines_blackhole	-2.19KiB	-0.13	70.62%	1.67MiB	9.7KiB	203.16B	0.00565459	1.67MiB	102.05KiB	2.08KiB	0.059572	False	False
http_to_http_json	-31.15KiB	-0.13	98.88%	23.85MiB	334.83KiB	6.83KiB	0.0137088	23.82MiB	498.57KiB	10.19KiB	0.0204387	False	False
syslog_log2metric_humio_metrics	-30.11KiB	-0.21	98.03%	13.78MiB	279.85KiB	5.71KiB	0.0198251	13.75MiB	568.1KiB	11.57KiB	0.0403305	False	False
http_text_to_http_json	-87.26KiB	-0.22	99.98%	38.79MiB	796.66KiB	16.26KiB	0.0200548	38.7MiB	812.67KiB	16.6KiB	0.0205027	False	False
http_to_http_noack	-96.26KiB	-0.39	100.00%	23.84MiB	252.04KiB	5.15KiB	0.0103201	23.75MiB	1.07MiB	22.21KiB	0.0448398	False	False
splunk_hec_route_s3	-79.88KiB	-0.4	79.00%	19.59MiB	2.16MiB	45.06KiB	0.110482	19.51MiB	2.16MiB	45.05KiB	0.110419	False	False
syslog_humio_logs	-84.82KiB	-0.5	100.00%	16.72MiB	131.72KiB	2.69KiB	0.00769311	16.63MiB	159.95KiB	3.28KiB	0.0093884	False	False
syslog_log2metric_splunk_hec_metrics	-95.21KiB	-0.54	99.99%	17.37MiB	741.11KiB	15.1KiB	0.0416598	17.28MiB	886.82KiB	18.05KiB	0.0501187	False	False
datadog_agent_remap_datadog_logs	-368.47KiB	-0.54	99.99%	66.61MiB	1.18MiB	24.75KiB	0.0177283	66.25MiB	4.45MiB	92.66KiB	0.0671382	False	False
datadog_agent_remap_datadog_logs_acks	-386.38KiB	-0.56	99.94%	67.72MiB	2.85MiB	59.71KiB	0.0421201	67.35MiB	4.55MiB	94.65KiB	0.0674742	False	False
datadog_agent_remap_blackhole_acks	-458.01KiB	-0.68	99.35%	66.16MiB	6.15MiB	128.19KiB	0.0930089	65.71MiB	5.21MiB	108.96KiB	0.0792764	False	False
syslog_loki	-108.2KiB	-0.73	100.00%	14.53MiB	480.25KiB	9.83KiB	0.0322676	14.43MiB	759.43KiB	15.44KiB	0.0514001	False	False
syslog_splunk_hec_logs	-126.36KiB	-0.75	100.00%	16.43MiB	845.89KiB	17.26KiB	0.0502828	16.3MiB	812.64KiB	16.59KiB	0.0486715	False	False
fluent_elasticsearch	-821.25KiB	-1.01	100.00%	79.47MiB	53.8KiB	1.08KiB	0.000660969	78.67MiB	6.82MiB	139.7KiB	0.0866541	False	False
http_pipelines_no_grok_blackhole	-147.87KiB	-1.22	100.00%	11.83MiB	120.9KiB	2.47KiB	0.00997673	11.69MiB	1.1MiB	22.99KiB	0.0944261	False	False
socket_to_socket_blackhole	-190.93KiB	-1.37	100.00%	13.63MiB	105.65KiB	2.16KiB	0.00756535	13.45MiB	112.57KiB	2.3KiB	0.0081727	False	False
datadog_agent_remap_blackhole	-962.49KiB	-1.43	100.00%	65.86MiB	4.47MiB	93.06KiB	0.0677857	64.92MiB	3.32MiB	69.26KiB	0.0511178	False	False
http_to_http_acks	-581.89KiB	-3.2	98.95%	17.77MiB	7.6MiB	158.85KiB	0.427587	17.2MiB	7.78MiB	162.45KiB	0.451933	True	True

Jun 23 '22 13:06 github-actions[bot]

Soak Test Results

Baseline: 91d88e8540bfe154038ca79265d28b31e6242c05 Comparison: 7fdaec8d2bc940d220020629c14a981d76a59ec3 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_loki	79.81KiB	0.56	95.83%	14.01MiB	1.29MiB	27.16KiB	0.0923597	14.09MiB	1.36MiB	28.23KiB	0.0962457	False	False
http_pipelines_blackhole_acks	6.24KiB	0.52	98.19%	1.17MiB	106.6KiB	2.17KiB	0.0886714	1.18MiB	73.66KiB	1.5KiB	0.0609598	False	False
http_pipelines_blackhole	2.44KiB	0.15	61.33%	1.59MiB	73.53KiB	1.5KiB	0.0451076	1.59MiB	117.32KiB	2.39KiB	0.0718581	False	False
splunk_hec_route_s3	18.62KiB	0.1	21.49%	18.87MiB	2.36MiB	49.1KiB	0.124954	18.89MiB	2.27MiB	47.48KiB	0.120095	False	False
splunk_hec_indexer_ack_blackhole	15.32KiB	0.06	46.44%	23.74MiB	888.11KiB	18.06KiB	0.0365229	23.76MiB	830.35KiB	16.89KiB	0.034126	False	False
splunk_hec_to_splunk_hec_logs_acks	5.69KiB	0.02	20.03%	23.76MiB	792.03KiB	16.12KiB	0.0325452	23.77MiB	766.92KiB	15.61KiB	0.031506	False	False
splunk_hec_to_splunk_hec_logs_noack	5.7KiB	0.02	41.72%	23.83MiB	385.27KiB	7.87KiB	0.0157837	23.84MiB	331.17KiB	6.76KiB	0.0135643	False	False
syslog_regex_logs2metric_ddmetrics	-13.91KiB	-0.11	59.35%	12.47MiB	557.24KiB	11.36KiB	0.0436458	12.45MiB	605.16KiB	12.33KiB	0.047451	False	False
http_to_http_json	-35.29KiB	-0.14	99.48%	23.85MiB	334.41KiB	6.83KiB	0.0136912	23.81MiB	519.15KiB	10.61KiB	0.0212856	False	False
file_to_blackhole	-214.67KiB	-0.22	89.78%	95.36MiB	3.67MiB	75.98KiB	0.0384915	95.15MiB	5.17MiB	107.09KiB	0.0542831	False	False
syslog_splunk_hec_logs	-46.32KiB	-0.27	99.08%	16.6MiB	726.6KiB	14.79KiB	0.0427324	16.56MiB	483.63KiB	9.88KiB	0.0285202	False	False
syslog_humio_logs	-56.12KiB	-0.33	100.00%	16.6MiB	145.9KiB	2.98KiB	0.00858353	16.54MiB	116.1KiB	2.38KiB	0.00685299	False	False
syslog_log2metric_humio_metrics	-56.54KiB	-0.4	99.94%	13.69MiB	466.35KiB	9.52KiB	0.0332657	13.63MiB	654.27KiB	13.32KiB	0.0468597	False	False
http_to_http_noack	-104.79KiB	-0.43	99.99%	23.83MiB	517.05KiB	10.56KiB	0.0211863	23.73MiB	1.18MiB	24.65KiB	0.0498357	False	False
http_text_to_http_json	-188.26KiB	-0.47	100.00%	38.97MiB	761.21KiB	15.54KiB	0.0190734	38.78MiB	781.06KiB	15.94KiB	0.0196634	False	False
datadog_agent_remap_datadog_logs_acks	-333.99KiB	-0.48	99.54%	67.37MiB	3.04MiB	63.63KiB	0.0451398	67.04MiB	4.76MiB	99.0KiB	0.0709238	False	False
syslog_log2metric_splunk_hec_metrics	-112.37KiB	-0.63	100.00%	17.36MiB	773.74KiB	15.76KiB	0.0435169	17.25MiB	858.04KiB	17.46KiB	0.0485646	False	False
datadog_agent_remap_blackhole_acks	-510.41KiB	-0.73	100.00%	68.46MiB	4.65MiB	96.8KiB	0.067888	67.96MiB	3.13MiB	65.44KiB	0.0460484	False	False
fluent_elasticsearch	-606.7KiB	-0.75	100.00%	79.47MiB	53.21KiB	1.07KiB	0.000653738	78.88MiB	5.88MiB	120.72KiB	0.0745754	False	False
datadog_agent_remap_blackhole	-741.54KiB	-1.09	100.00%	66.63MiB	4.57MiB	95.2KiB	0.0685844	65.91MiB	3.01MiB	62.83KiB	0.0456636	False	False
socket_to_socket_blackhole	-169.82KiB	-1.22	100.00%	13.62MiB	110.88KiB	2.26KiB	0.0079456	13.46MiB	107.72KiB	2.2KiB	0.00781407	False	False
http_to_http_acks	-248.56KiB	-1.37	73.43%	17.76MiB	7.64MiB	159.6KiB	0.429822	17.52MiB	7.48MiB	156.16KiB	0.426923	True	True
datadog_agent_remap_datadog_logs	-858.54KiB	-1.38	99.83%	60.65MiB	8.5MiB	178.2KiB	0.140162	59.81MiB	9.91MiB	206.51KiB	0.165737	False	False
http_pipelines_no_grok_blackhole	-195.56KiB	-1.6	100.00%	11.9MiB	47.25KiB	987.53B	0.00387599	11.71MiB	1.19MiB	24.84KiB	0.101888	False	False

Jun 23 '22 13:06 github-actions[bot]

Soak Test Results

Baseline: 91d88e8540bfe154038ca79265d28b31e6242c05 Comparison: 69f31545a161e6b2f409266a5e513a2d0fa58d6a Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_log2metric_splunk_hec_metrics	230.82KiB	1.3	100.00%	17.37MiB	798.28KiB	16.27KiB	0.0448665	17.6MiB	780.43KiB	15.89KiB	0.0433015	False	False
http_pipelines_blackhole_acks	8.9KiB	0.75	99.90%	1.16MiB	106.77KiB	2.17KiB	0.0901432	1.17MiB	79.27KiB	1.62KiB	0.0664263	False	False
syslog_regex_logs2metric_ddmetrics	66.06KiB	0.52	100.00%	12.48MiB	592.09KiB	12.06KiB	0.0463148	12.55MiB	529.61KiB	10.8KiB	0.0412146	False	False
http_pipelines_blackhole	5.36KiB	0.33	99.65%	1.59MiB	17.35KiB	363.27B	0.0106418	1.6MiB	88.07KiB	1.8KiB	0.053838	False	False
syslog_splunk_hec_logs	20.7KiB	0.12	66.58%	16.57MiB	852.76KiB	17.36KiB	0.0502325	16.6MiB	616.5KiB	12.57KiB	0.0362708	False	False
splunk_hec_to_splunk_hec_logs_noack	21.87KiB	0.09	92.21%	23.82MiB	512.22KiB	10.45KiB	0.0209994	23.84MiB	326.82KiB	6.67KiB	0.0133865	False	False
splunk_hec_indexer_ack_blackhole	10.2KiB	0.04	32.43%	23.75MiB	855.0KiB	17.39KiB	0.0351521	23.76MiB	839.8KiB	17.09KiB	0.0345125	False	False
file_to_blackhole	-35.42KiB	-0.04	32.92%	95.35MiB	2.77MiB	57.33KiB	0.0289999	95.32MiB	2.91MiB	60.48KiB	0.0304813	False	False
splunk_hec_to_splunk_hec_logs_acks	-15.69KiB	-0.06	49.55%	23.77MiB	776.08KiB	15.8KiB	0.0318799	23.75MiB	855.71KiB	17.41KiB	0.0351736	False	False
syslog_log2metric_humio_metrics	-9.37KiB	-0.07	59.60%	13.94MiB	273.42KiB	5.58KiB	0.0191455	13.93MiB	478.19KiB	9.74KiB	0.0335055	False	False
http_to_http_json	-38.93KiB	-0.16	99.74%	23.85MiB	333.99KiB	6.82KiB	0.0136742	23.81MiB	537.99KiB	10.98KiB	0.0220614	False	False
http_text_to_http_json	-84.33KiB	-0.21	99.83%	38.61MiB	938.91KiB	19.17KiB	0.0237453	38.52MiB	925.53KiB	18.89KiB	0.0234569	False	False
splunk_hec_route_s3	-47.85KiB	-0.25	51.73%	18.91MiB	2.36MiB	49.05KiB	0.12453	18.86MiB	2.26MiB	47.34KiB	0.120029	False	False
http_to_http_noack	-64.14KiB	-0.26	99.80%	23.84MiB	403.39KiB	8.24KiB	0.0165223	23.78MiB	934.78KiB	19.05KiB	0.0383879	False	False
datadog_agent_remap_blackhole_acks	-239.59KiB	-0.38	98.30%	62.21MiB	4.01MiB	83.57KiB	0.0644752	61.97MiB	2.65MiB	55.47KiB	0.0427672	False	False
datadog_agent_remap_datadog_logs_acks	-276.88KiB	-0.4	97.80%	66.89MiB	3.36MiB	70.2KiB	0.0501963	66.62MiB	4.73MiB	98.4KiB	0.0709449	False	False
syslog_humio_logs	-81.82KiB	-0.48	100.00%	16.78MiB	112.94KiB	2.31KiB	0.00657174	16.7MiB	108.25KiB	2.22KiB	0.00632892	False	False
fluent_elasticsearch	-486.17KiB	-0.6	100.00%	79.47MiB	52.01KiB	1.05KiB	0.000638924	79.0MiB	4.7MiB	96.46KiB	0.0594685	False	False
datadog_agent_remap_datadog_logs	-487.23KiB	-0.71	100.00%	66.56MiB	447.31KiB	9.15KiB	0.00656191	66.08MiB	4.62MiB	96.32KiB	0.0699679	False	False
socket_to_socket_blackhole	-188.68KiB	-1.36	100.00%	13.59MiB	110.61KiB	2.26KiB	0.00794594	13.41MiB	114.61KiB	2.34KiB	0.00834671	False	False
syslog_loki	-233.28KiB	-1.49	100.00%	15.25MiB	360.04KiB	7.37KiB	0.023048	15.02MiB	812.7KiB	16.52KiB	0.0528143	False	False
http_pipelines_no_grok_blackhole	-188.29KiB	-1.57	100.00%	11.68MiB	241.45KiB	4.93KiB	0.0201845	11.5MiB	1.17MiB	24.37KiB	0.101806	False	False
datadog_agent_remap_blackhole	-1007.35KiB	-1.6	100.00%	61.42MiB	3.85MiB	80.13KiB	0.0626155	60.43MiB	3.52MiB	73.29KiB	0.0581658	False	False
http_to_http_acks	-298.81KiB	-1.63	81.82%	17.85MiB	7.31MiB	152.87KiB	0.409538	17.56MiB	7.82MiB	163.37KiB	0.4454	True	True

Jun 23 '22 13:06 github-actions[bot]

Thanks @ktff ! We'll get this reviewed.

Jun 23 '22 19:06 jszwedko

Soak Test Results

Baseline: e14b2c1796d35118eef967bce1a94d38df9958f6 Comparison: 8e76b54a5fcff80b11f95668b267d0eecb3766be Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
splunk_hec_route_s3	1.28MiB	6.92	100.00%	18.49MiB	2.69MiB	56.02KiB	0.145534	19.77MiB	2.1MiB	43.85KiB	0.10605	False	False
syslog_regex_logs2metric_ddmetrics	121.07KiB	0.98	100.00%	12.06MiB	674.17KiB	13.72KiB	0.0546004	12.17MiB	579.57KiB	11.82KiB	0.0464826	False	False
http_pipelines_blackhole_acks	8.7KiB	0.7	99.91%	1.21MiB	114.92KiB	2.34KiB	0.0931134	1.21MiB	56.94KiB	1.16KiB	0.0458097	False	False
http_text_to_http_json	281.73KiB	0.67	100.00%	41.0MiB	817.54KiB	16.69KiB	0.0194689	41.27MiB	822.83KiB	16.81KiB	0.0194643	False	False
splunk_hec_to_splunk_hec_logs_noack	59.37KiB	0.24	99.99%	23.78MiB	660.46KiB	13.44KiB	0.027116	23.84MiB	334.95KiB	6.84KiB	0.0137182	False	False
splunk_hec_indexer_ack_blackhole	20.35KiB	0.08	55.72%	23.74MiB	967.34KiB	19.67KiB	0.0397859	23.76MiB	873.49KiB	17.77KiB	0.0358958	False	False
datadog_agent_remap_blackhole_acks	33.78KiB	0.05	25.98%	61.33MiB	3.93MiB	81.89KiB	0.0640529	61.36MiB	2.9MiB	60.59KiB	0.0471796	False	False
file_to_blackhole	-63.03KiB	-0.06	43.09%	95.34MiB	3.57MiB	73.9KiB	0.0373865	95.27MiB	3.96MiB	82.4KiB	0.0415857	False	False
splunk_hec_to_splunk_hec_logs_acks	-18.12KiB	-0.07	50.88%	23.77MiB	877.81KiB	17.86KiB	0.036057	23.75MiB	950.9KiB	19.33KiB	0.0390887	False	False
http_pipelines_blackhole	-1.42KiB	-0.08	41.59%	1.67MiB	15.82KiB	331.27B	0.00926335	1.67MiB	125.88KiB	2.56KiB	0.0737586	False	False
syslog_humio_logs	-19.99KiB	-0.12	100.00%	16.56MiB	108.86KiB	2.22KiB	0.00641648	16.54MiB	117.61KiB	2.41KiB	0.00694096	False	False
syslog_log2metric_splunk_hec_metrics	-27.27KiB	-0.16	70.34%	17.04MiB	818.87KiB	16.7KiB	0.0469267	17.01MiB	986.7KiB	20.09KiB	0.0566329	False	False
fluent_elasticsearch	-145.32KiB	-0.18	100.00%	79.47MiB	52.93KiB	1.07KiB	0.000650319	79.33MiB	1.44MiB	29.73KiB	0.0181818	False	False
http_to_http_json	-52.42KiB	-0.21	99.98%	23.85MiB	328.35KiB	6.7KiB	0.0134438	23.8MiB	595.49KiB	12.14KiB	0.024434	False	False
syslog_splunk_hec_logs	-36.02KiB	-0.22	93.55%	16.22MiB	738.92KiB	15.04KiB	0.0444894	16.18MiB	607.21KiB	12.38KiB	0.0366383	False	False
datadog_agent_remap_blackhole	-184.6KiB	-0.28	85.98%	64.61MiB	4.77MiB	99.32KiB	0.0738066	64.43MiB	3.65MiB	76.11KiB	0.0566255	False	False
http_to_http_noack	-113.48KiB	-0.46	100.00%	23.85MiB	245.78KiB	5.03KiB	0.0100635	23.73MiB	1.15MiB	23.92KiB	0.0483452	False	False
syslog_loki	-78.03KiB	-0.51	100.00%	15.05MiB	417.69KiB	8.55KiB	0.0271006	14.97MiB	813.83KiB	16.54KiB	0.0530722	False	False
datadog_agent_remap_datadog_logs_acks	-346.32KiB	-0.52	99.59%	65.53MiB	3.49MiB	72.89KiB	0.0532254	65.19MiB	4.62MiB	96.15KiB	0.0708392	False	False
datadog_agent_remap_datadog_logs	-470.2KiB	-0.7	100.00%	65.69MiB	982.35KiB	20.1KiB	0.0146017	65.23MiB	4.43MiB	92.17KiB	0.0678413	False	False
http_to_http_acks	-137.03KiB	-0.75	43.15%	17.91MiB	8.25MiB	172.48KiB	0.46043	17.78MiB	8.02MiB	167.28KiB	0.450848	True	True
http_pipelines_no_grok_blackhole	-87.96KiB	-0.77	99.99%	11.19MiB	332.2KiB	6.78KiB	0.0289853	11.1MiB	1.06MiB	22.02KiB	0.0951714	False	False
syslog_log2metric_humio_metrics	-139.11KiB	-1.07	100.00%	12.73MiB	1.04MiB	21.84KiB	0.0820944	12.59MiB	1.19MiB	24.79KiB	0.0944781	False	False
socket_to_socket_blackhole	-227.57KiB	-1.6	100.00%	13.87MiB	136.15KiB	2.78KiB	0.00958058	13.65MiB	114.18KiB	2.33KiB	0.00816555	False	False

Jul 05 '22 13:07 github-actions[bot]

Ping @bruceg

Jul 05 '22 14:07 ktff

Soak Test Results

Baseline: 2b2259ce2a966296efdca8e9ec99591dde683f62 Comparison: 5f5d50cf50034634ebc17b0e0c4f459f27edcf19 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_to_http_acks	139.94KiB	0.77	46.30%	17.69MiB	7.48MiB	156.46KiB	0.423049	17.82MiB	7.86MiB	164.01KiB	0.440799	True	True
syslog_loki	56.14KiB	0.36	99.91%	15.13MiB	227.58KiB	4.66KiB	0.014684	15.19MiB	800.66KiB	16.28KiB	0.0514751	False	False
datadog_agent_remap_blackhole_acks	64.65KiB	0.1	36.91%	65.82MiB	4.99MiB	103.83KiB	0.0757528	65.88MiB	4.09MiB	85.54KiB	0.0621386	False	False
splunk_hec_to_splunk_hec_logs_acks	20.27KiB	0.08	61.24%	23.75MiB	854.05KiB	17.37KiB	0.035113	23.77MiB	775.18KiB	15.78KiB	0.031844	False	False
splunk_hec_to_splunk_hec_logs_noack	6.98KiB	0.03	50.06%	23.83MiB	382.19KiB	7.81KiB	0.0156582	23.84MiB	331.75KiB	6.77KiB	0.0135879	False	False
splunk_hec_indexer_ack_blackhole	-588.26B	-0	1.71%	23.75MiB	930.55KiB	18.93KiB	0.0382611	23.75MiB	937.8KiB	19.08KiB	0.0385601	False	False
file_to_blackhole	-34.97KiB	-0.04	25.44%	95.34MiB	3.58MiB	74.31KiB	0.0375929	95.3MiB	3.75MiB	78.04KiB	0.0393461	False	False
syslog_log2metric_humio_metrics	-9.4KiB	-0.07	41.06%	12.92MiB	547.83KiB	11.18KiB	0.0414015	12.91MiB	655.62KiB	13.35KiB	0.0495828	False	False
syslog_regex_logs2metric_ddmetrics	-9.16KiB	-0.07	43.12%	12.39MiB	581.81KiB	11.85KiB	0.0458646	12.38MiB	532.73KiB	10.86KiB	0.042026	False	False
http_pipelines_blackhole_acks	-1011.44B	-0.09	27.20%	1.13MiB	116.99KiB	2.38KiB	0.101063	1.13MiB	75.96KiB	1.55KiB	0.0656738	False	False
http_to_http_json	-31.08KiB	-0.13	98.81%	23.85MiB	335.59KiB	6.85KiB	0.0137401	23.82MiB	503.11KiB	10.28KiB	0.0206255	False	False
fluent_elasticsearch	-193.56KiB	-0.24	100.00%	79.47MiB	53.02KiB	1.07KiB	0.00065132	79.28MiB	1.75MiB	36.0KiB	0.0220545	False	False
http_to_http_noack	-85.15KiB	-0.35	99.98%	23.84MiB	403.56KiB	8.25KiB	0.0165294	23.75MiB	1.04MiB	21.67KiB	0.0437299	False	False
http_pipelines_blackhole	-7.73KiB	-0.47	99.82%	1.62MiB	38.98KiB	815.32B	0.0234848	1.61MiB	114.7KiB	2.34KiB	0.0694244	False	False
datadog_agent_remap_blackhole	-389.62KiB	-0.58	99.90%	65.83MiB	4.36MiB	90.83KiB	0.0661604	65.45MiB	3.67MiB	76.49KiB	0.0560281	False	False
syslog_log2metric_splunk_hec_metrics	-169.37KiB	-0.92	100.00%	17.89MiB	634.88KiB	12.93KiB	0.0346402	17.73MiB	749.28KiB	15.25KiB	0.0412633	False	False
http_pipelines_no_grok_blackhole	-131.37KiB	-1.12	100.00%	11.5MiB	277.07KiB	5.65KiB	0.0235273	11.37MiB	1.13MiB	23.45KiB	0.0990586	False	False
syslog_splunk_hec_logs	-210.65KiB	-1.23	100.00%	16.69MiB	714.69KiB	14.56KiB	0.0418024	16.49MiB	633.34KiB	12.91KiB	0.0375068	False	False
datadog_agent_remap_datadog_logs_acks	-845.2KiB	-1.23	100.00%	67.16MiB	3.21MiB	67.13KiB	0.0477926	66.33MiB	4.71MiB	98.0KiB	0.0709604	False	False
datadog_agent_remap_datadog_logs	-1020.39KiB	-1.56	100.00%	63.75MiB	2.75MiB	57.57KiB	0.0430831	62.76MiB	5.1MiB	106.23KiB	0.0812865	False	False
splunk_hec_route_s3	-363.03KiB	-1.87	100.00%	18.93MiB	2.32MiB	48.31KiB	0.1225	18.58MiB	2.3MiB	48.12KiB	0.123811	False	False
socket_to_socket_blackhole	-307.9KiB	-2.16	100.00%	13.93MiB	111.04KiB	2.27KiB	0.00778355	13.63MiB	102.43KiB	2.09KiB	0.0073385	False	False
syslog_humio_logs	-503.0KiB	-3.02	100.00%	16.25MiB	544.55KiB	11.12KiB	0.0327091	15.76MiB	570.59KiB	11.69KiB	0.0353407	False	False
http_text_to_http_json	-1.65MiB	-3.9	100.00%	42.2MiB	811.17KiB	16.56KiB	0.0187672	40.55MiB	833.78KiB	17.03KiB	0.0200738	False	False

Jul 06 '22 13:07 github-actions[bot]

That makes sense. Then I'll change it to use Buffer.

Jul 18 '22 18:07 ktff

Soak Test Results

Baseline: da0076250e4d6110bfbd64426c674b12d87e8db2 Comparison: cbfb4068dd6c28d8e1f4cb25fe719a02fdb22291 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_pipelines_blackhole_acks	58.39KiB	5.17	100.00%	1.1MiB	117.58KiB	2.4KiB	0.104116	1.16MiB	95.59KiB	1.95KiB	0.0804835	False	False
syslog_log2metric_splunk_hec_metrics	865.21KiB	4.65	100.00%	18.16MiB	1.41MiB	29.45KiB	0.0777311	19.0MiB	925.83KiB	18.83KiB	0.0475646	False	False
syslog_regex_logs2metric_ddmetrics	123.86KiB	0.96	100.00%	12.62MiB	642.04KiB	13.09KiB	0.0496825	12.74MiB	647.16KiB	13.2KiB	0.0496031	False	False
syslog_log2metric_humio_metrics	70.15KiB	0.51	100.00%	13.4MiB	234.69KiB	4.79KiB	0.0170971	13.47MiB	563.58KiB	11.48KiB	0.0408487	False	False
splunk_hec_route_s3	98.45KiB	0.5	87.09%	19.4MiB	2.26MiB	47.14KiB	0.11646	19.5MiB	2.13MiB	44.54KiB	0.109206	False	False
syslog_splunk_hec_logs	50.21KiB	0.29	98.67%	16.96MiB	796.02KiB	16.2KiB	0.0458135	17.01MiB	598.2KiB	12.2KiB	0.0343289	False	False
http_to_http_acks	33.39KiB	0.18	11.09%	18.05MiB	8.13MiB	169.94KiB	0.450057	18.09MiB	8.08MiB	168.78KiB	0.446831	True	True
syslog_loki	20.92KiB	0.14	78.77%	14.68MiB	362.02KiB	7.41KiB	0.0240842	14.7MiB	740.27KiB	15.05KiB	0.04918	False	False
splunk_hec_to_splunk_hec_logs_acks	23.73KiB	0.1	68.36%	23.75MiB	869.4KiB	17.68KiB	0.0357461	23.77MiB	774.16KiB	15.76KiB	0.0317993	False	False
splunk_hec_indexer_ack_blackhole	13.68KiB	0.06	38.26%	23.73MiB	964.51KiB	19.61KiB	0.0396811	23.75MiB	940.23KiB	19.12KiB	0.0386604	False	False
splunk_hec_to_splunk_hec_logs_noack	12.3KiB	0.05	73.23%	23.83MiB	431.58KiB	8.81KiB	0.0176852	23.84MiB	330.45KiB	6.75KiB	0.0135341	False	False
enterprise_http_to_http	-1.93KiB	-0.01	21.64%	23.85MiB	240.66KiB	4.91KiB	0.00985338	23.84MiB	244.31KiB	5.0KiB	0.0100037	False	False
file_to_blackhole	-94.17KiB	-0.1	56.34%	95.33MiB	3.81MiB	78.91KiB	0.0399266	95.24MiB	4.42MiB	91.76KiB	0.0463733	False	False
http_to_http_json	-23.84KiB	-0.1	95.17%	23.84MiB	349.76KiB	7.14KiB	0.0143219	23.82MiB	476.06KiB	9.73KiB	0.0195128	False	False
socket_to_socket_blackhole	-27.01KiB	-0.19	97.67%	13.59MiB	469.12KiB	9.58KiB	0.0337075	13.56MiB	346.08KiB	7.07KiB	0.0249148	False	False
fluent_elasticsearch	-154.19KiB	-0.19	100.00%	79.47MiB	53.93KiB	1.09KiB	0.000662551	79.32MiB	1.48MiB	30.48KiB	0.0186511	False	False
http_to_http_noack	-84.6KiB	-0.35	99.97%	23.84MiB	408.26KiB	8.35KiB	0.0167223	23.75MiB	1.03MiB	21.51KiB	0.0434199	False	False
http_pipelines_no_grok_blackhole	-43.35KiB	-0.38	92.18%	11.2MiB	295.73KiB	6.04KiB	0.0257804	11.16MiB	1.15MiB	23.85KiB	0.102687	False	False
http_pipelines_blackhole	-10.37KiB	-0.65	100.00%	1.56MiB	32.98KiB	690.2B	0.0206753	1.55MiB	92.65KiB	1.89KiB	0.0584673	False	False
http_text_to_http_json	-278.3KiB	-0.7	100.00%	39.1MiB	848.07KiB	17.31KiB	0.0211763	38.83MiB	793.68KiB	16.2KiB	0.019957	False	False
datadog_agent_remap_blackhole	-546.08KiB	-0.83	100.00%	64.63MiB	4.05MiB	84.41KiB	0.062604	64.1MiB	2.98MiB	62.17KiB	0.0464727	False	False
datadog_agent_remap_datadog_logs_acks	-655.59KiB	-1.01	100.00%	63.67MiB	3.53MiB	73.63KiB	0.0553598	63.03MiB	4.48MiB	93.29KiB	0.071085	False	False
datadog_agent_remap_datadog_logs	-883.1KiB	-1.37	100.00%	63.17MiB	1.2MiB	25.05KiB	0.0189227	62.3MiB	4.42MiB	92.11KiB	0.0709934	False	False
syslog_humio_logs	-284.57KiB	-1.58	100.00%	17.61MiB	220.77KiB	4.51KiB	0.0122417	17.33MiB	691.43KiB	14.16KiB	0.0389539	False	False
datadog_agent_remap_blackhole_acks	-3.1MiB	-4.86	100.00%	63.86MiB	5.76MiB	119.93KiB	0.0902	60.76MiB	4.84MiB	101.19KiB	0.07959	False	False

Jul 19 '22 13:07 github-actions[bot]

Soak Test Results

Baseline: 788078497177b949d8983bee43e782e43ed4d34c Comparison: 9af3671d5e50fd38e68a524aff14bf44c693f0b0 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_to_http_acks	277.77KiB	1.51	79.13%	17.92MiB	7.84MiB	163.9KiB	0.437597	18.19MiB	7.09MiB	148.12KiB	0.39004	True	True
syslog_loki	158.9KiB	1.05	100.00%	14.81MiB	449.8KiB	9.19KiB	0.0296563	14.96MiB	739.4KiB	15.03KiB	0.0482443	False	False
syslog_log2metric_humio_metrics	132.85KiB	0.96	100.00%	13.48MiB	328.18KiB	6.7KiB	0.0237744	13.61MiB	558.34KiB	11.37KiB	0.0400618	False	False
syslog_regex_logs2metric_ddmetrics	97.3KiB	0.73	100.00%	13.08MiB	661.81KiB	13.48KiB	0.0493943	13.18MiB	620.17KiB	12.64KiB	0.0459525	False	False
datadog_agent_remap_blackhole	400.74KiB	0.62	99.40%	63.23MiB	5.68MiB	118.31KiB	0.0898423	63.62MiB	4.09MiB	85.34KiB	0.0642869	False	False
splunk_hec_to_splunk_hec_logs_noack	43.26KiB	0.18	99.65%	23.79MiB	647.18KiB	13.19KiB	0.0265558	23.84MiB	326.7KiB	6.67KiB	0.0133817	False	False
splunk_hec_route_s3	30.29KiB	0.16	35.50%	18.87MiB	2.26MiB	47.06KiB	0.119731	18.9MiB	2.19MiB	45.9KiB	0.116121	False	False
syslog_humio_logs	26.21KiB	0.15	100.00%	17.57MiB	231.87KiB	4.74KiB	0.0128872	17.59MiB	121.17KiB	2.48KiB	0.00672447	False	False
splunk_hec_to_splunk_hec_logs_acks	15.25KiB	0.06	47.36%	23.75MiB	863.25KiB	17.56KiB	0.0354919	23.76MiB	809.14KiB	16.47KiB	0.0332463	False	False
syslog_splunk_hec_logs	4.78KiB	0.03	17.93%	17.21MiB	806.15KiB	16.4KiB	0.0457413	17.21MiB	650.53KiB	13.27KiB	0.0369017	False	False
splunk_hec_indexer_ack_blackhole	5.61KiB	0.02	17.61%	23.75MiB	880.98KiB	17.92KiB	0.0362111	23.76MiB	870.17KiB	17.71KiB	0.0357586	False	False
enterprise_http_to_http	-2.26KiB	-0.01	24.21%	23.85MiB	253.64KiB	5.18KiB	0.0103848	23.84MiB	254.29KiB	5.2KiB	0.0104124	False	False
file_to_blackhole	-35.17KiB	-0.04	32.85%	95.34MiB	2.59MiB	53.71KiB	0.0271692	95.31MiB	3.04MiB	63.19KiB	0.0318484	False	False
fluent_elasticsearch	-52.04KiB	-0.06	100.00%	79.47MiB	54.7KiB	1.11KiB	0.00067203	79.42MiB	474.31KiB	9.54KiB	0.00583089	False	False
http_to_http_json	-39.23KiB	-0.16	99.76%	23.85MiB	331.99KiB	6.78KiB	0.0135919	23.81MiB	537.9KiB	10.98KiB	0.0220577	False	False
syslog_log2metric_splunk_hec_metrics	-43.95KiB	-0.23	96.85%	18.72MiB	633.84KiB	12.92KiB	0.0330665	18.67MiB	777.8KiB	15.83KiB	0.0406698	False	False
http_pipelines_blackhole_acks	-4.21KiB	-0.35	88.13%	1.16MiB	103.2KiB	2.1KiB	0.0868327	1.16MiB	82.96KiB	1.69KiB	0.0700523	False	False
http_to_http_noack	-94.02KiB	-0.39	99.99%	23.84MiB	401.07KiB	8.2KiB	0.0164281	23.74MiB	1.09MiB	22.73KiB	0.0459006	False	False
socket_to_socket_blackhole	-107.14KiB	-0.77	100.00%	13.63MiB	238.21KiB	4.86KiB	0.0170642	13.52MiB	228.23KiB	4.66KiB	0.0164759	False	False
http_pipelines_blackhole	-15.84KiB	-0.94	100.00%	1.64MiB	15.35KiB	321.16B	0.00912311	1.63MiB	112.26KiB	2.29KiB	0.0673564	False	False
http_text_to_http_json	-410.73KiB	-1.03	100.00%	38.96MiB	868.77KiB	17.73KiB	0.0217693	38.56MiB	813.35KiB	16.61KiB	0.0205925	False	False
http_pipelines_no_grok_blackhole	-128.13KiB	-1.09	100.00%	11.48MiB	85.63KiB	1.75KiB	0.00728303	11.35MiB	1.13MiB	23.48KiB	0.0993146	False	False
datadog_agent_remap_datadog_logs	-1007.64KiB	-1.57	100.00%	62.78MiB	2.93MiB	61.48KiB	0.0467209	61.8MiB	4.75MiB	98.94KiB	0.0768542	False	False
datadog_agent_remap_datadog_logs_acks	-1.05MiB	-1.66	100.00%	63.47MiB	2.87MiB	60.04KiB	0.0452094	62.41MiB	4.46MiB	92.92KiB	0.0715039	False	False
datadog_agent_remap_blackhole_acks	-1.4MiB	-2.16	100.00%	64.86MiB	4.63MiB	96.37KiB	0.0713568	63.46MiB	3.23MiB	67.47KiB	0.0508341	False	False

Jul 19 '22 15:07 github-actions[bot]

Soak Test Results

Baseline: 788078497177b949d8983bee43e782e43ed4d34c Comparison: c5f22670b1800b32425ba8fb661e5651d134b7fd Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
http_to_http_acks	317.97KiB	1.72	79.54%	18.01MiB	8.64MiB	180.63KiB	0.479466	18.32MiB	8.32MiB	173.75KiB	0.454155	True	True
syslog_log2metric_humio_metrics	181.13KiB	1.32	100.00%	13.35MiB	165.65KiB	3.38KiB	0.0121105	13.53MiB	597.12KiB	12.15KiB	0.0430843	False	False
syslog_regex_logs2metric_ddmetrics	113.19KiB	0.87	100.00%	12.72MiB	729.03KiB	14.85KiB	0.0559633	12.83MiB	656.61KiB	13.39KiB	0.0499696	False	False
http_pipelines_blackhole	7.87KiB	0.49	96.49%	1.58MiB	116.38KiB	2.38KiB	0.0719286	1.59MiB	141.08KiB	2.88KiB	0.0867721	False	False
syslog_loki	52.84KiB	0.35	99.83%	14.76MiB	366.51KiB	7.5KiB	0.0242458	14.81MiB	742.22KiB	15.09KiB	0.0489295	False	False
splunk_hec_route_s3	40.42KiB	0.21	46.43%	19.15MiB	2.25MiB	46.91KiB	0.117629	19.19MiB	2.17MiB	45.38KiB	0.11294	False	False
splunk_hec_indexer_ack_blackhole	36.47KiB	0.15	83.48%	23.73MiB	982.34KiB	19.97KiB	0.0404177	23.77MiB	838.93KiB	17.07KiB	0.0344654	False	False
splunk_hec_to_splunk_hec_logs_noack	18.69KiB	0.08	87.70%	23.82MiB	487.83KiB	9.95KiB	0.0199964	23.84MiB	338.24KiB	6.91KiB	0.013854	False	False
syslog_humio_logs	8.58KiB	0.05	91.17%	17.49MiB	164.23KiB	3.35KiB	0.00917043	17.49MiB	183.14KiB	3.75KiB	0.0102216	False	False
splunk_hec_to_splunk_hec_logs_acks	-536.69B	-0	1.74%	23.75MiB	833.71KiB	16.96KiB	0.0342668	23.75MiB	839.04KiB	17.07KiB	0.0344867	False	False
enterprise_http_to_http	-2.41KiB	-0.01	25.91%	23.85MiB	249.1KiB	5.08KiB	0.0101993	23.84MiB	256.01KiB	5.24KiB	0.010483	False	False
http_to_http_json	-18.63KiB	-0.08	89.83%	23.85MiB	341.87KiB	6.98KiB	0.0139972	23.83MiB	439.23KiB	8.99KiB	0.0179971	False	False
syslog_splunk_hec_logs	-15.11KiB	-0.09	53.95%	17.05MiB	755.84KiB	15.38KiB	0.0432954	17.03MiB	662.22KiB	13.5KiB	0.0379654	False	False
file_to_blackhole	-100.04KiB	-0.1	58.50%	95.33MiB	3.7MiB	76.7KiB	0.0388	95.24MiB	4.61MiB	95.8KiB	0.0483866	False	False
fluent_elasticsearch	-201.18KiB	-0.25	100.00%	79.47MiB	52.61KiB	1.06KiB	0.000646372	79.28MiB	1.71MiB	35.09KiB	0.0215196	False	False
datadog_agent_remap_blackhole_acks	-287.47KiB	-0.43	99.45%	65.15MiB	4.2MiB	87.55KiB	0.064489	64.87MiB	2.63MiB	55.06KiB	0.040554	False	False
datadog_agent_remap_blackhole	-263.17KiB	-0.43	97.22%	60.19MiB	4.64MiB	96.65KiB	0.0770666	59.93MiB	3.37MiB	70.34KiB	0.0562226	False	False
http_to_http_noack	-111.68KiB	-0.46	100.00%	23.85MiB	249.2KiB	5.1KiB	0.0102031	23.74MiB	1.11MiB	23.23KiB	0.0469301	False	False
syslog_log2metric_splunk_hec_metrics	-94.01KiB	-0.49	100.00%	18.89MiB	496.98KiB	10.13KiB	0.0256914	18.79MiB	851.18KiB	17.32KiB	0.0442169	False	False
http_pipelines_no_grok_blackhole	-72.12KiB	-0.63	99.76%	11.16MiB	71.04KiB	1.45KiB	0.00621205	11.09MiB	1.14MiB	23.67KiB	0.10249	False	False
socket_to_socket_blackhole	-89.99KiB	-0.64	100.00%	13.75MiB	157.82KiB	3.22KiB	0.0112041	13.67MiB	141.45KiB	2.89KiB	0.0101064	False	False
http_text_to_http_json	-396.14KiB	-1	100.00%	38.79MiB	826.55KiB	16.87KiB	0.0208064	38.4MiB	839.0KiB	17.13KiB	0.0213328	False	False
http_pipelines_blackhole_acks	-13.52KiB	-1.14	100.00%	1.16MiB	118.03KiB	2.4KiB	0.0996599	1.14MiB	96.77KiB	1.97KiB	0.082654	False	False
datadog_agent_remap_datadog_logs	-1.36MiB	-2.16	100.00%	63.07MiB	2.11MiB	44.3KiB	0.0335178	61.7MiB	4.68MiB	97.47KiB	0.0758434	False	False
datadog_agent_remap_datadog_logs_acks	-2.18MiB	-3.47	100.00%	62.81MiB	3.94MiB	82.31KiB	0.062653	60.64MiB	5.47MiB	113.92KiB	0.0902347	False	False

Jul 19 '22 16:07 github-actions[bot]

Soak Test Results

Baseline: 788078497177b949d8983bee43e782e43ed4d34c Comparison: a1a324dc31a18f5aeebca36fb79a7b1f039ff19b Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_loki	301.53KiB	2.09	100.00%	14.12MiB	214.21KiB	4.39KiB	0.0148151	14.41MiB	741.08KiB	15.06KiB	0.0502059	False	False
http_pipelines_no_grok_blackhole	147.31KiB	1.32	100.00%	10.91MiB	392.31KiB	8.01KiB	0.0351049	11.06MiB	1.11MiB	23.09KiB	0.100289	False	False
syslog_log2metric_humio_metrics	173.34KiB	1.28	100.00%	13.2MiB	390.84KiB	7.98KiB	0.0289101	13.37MiB	597.92KiB	12.17KiB	0.0436679	False	False
syslog_splunk_hec_logs	148.5KiB	0.87	100.00%	16.73MiB	794.54KiB	16.16KiB	0.0463678	16.88MiB	693.17KiB	14.14KiB	0.0401043	False	False
syslog_regex_logs2metric_ddmetrics	104.18KiB	0.79	100.00%	12.81MiB	565.38KiB	11.53KiB	0.0431058	12.91MiB	597.53KiB	12.18KiB	0.0451978	False	False
splunk_hec_route_s3	38.98KiB	0.2	45.17%	18.7MiB	2.21MiB	45.99KiB	0.118113	18.73MiB	2.19MiB	45.83KiB	0.11696	False	False
http_text_to_http_json	70.25KiB	0.18	95.03%	38.3MiB	1.31MiB	27.32KiB	0.0341115	38.37MiB	1.11MiB	23.12KiB	0.0288206	False	False
syslog_humio_logs	18.8KiB	0.11	99.64%	17.15MiB	226.49KiB	4.62KiB	0.0128967	17.17MiB	219.33KiB	4.49KiB	0.0124754	False	False
splunk_hec_to_splunk_hec_logs_noack	21.97KiB	0.09	92.71%	23.82MiB	493.91KiB	10.08KiB	0.0202458	23.84MiB	340.72KiB	6.96KiB	0.0139538	False	False
datadog_agent_remap_datadog_logs	38.52KiB	0.06	30.97%	63.13MiB	2.22MiB	46.49KiB	0.0351447	63.17MiB	4.07MiB	84.75KiB	0.0643857	False	False
splunk_hec_to_splunk_hec_logs_acks	10.0KiB	0.04	32.09%	23.75MiB	853.81KiB	17.37KiB	0.0351006	23.76MiB	825.35KiB	16.8KiB	0.0339167	False	False
syslog_log2metric_splunk_hec_metrics	296.68B	0	1.04%	18.89MiB	701.59KiB	14.29KiB	0.0362577	18.89MiB	836.77KiB	17.03KiB	0.043243	False	False
enterprise_http_to_http	308.7B	0	3.38%	23.85MiB	247.07KiB	5.04KiB	0.0101166	23.85MiB	245.61KiB	5.03KiB	0.0100566	False	False
splunk_hec_indexer_ack_blackhole	-6.26KiB	-0.03	19.41%	23.76MiB	868.91KiB	17.68KiB	0.0357122	23.75MiB	903.48KiB	18.38KiB	0.0371423	False	False
file_to_blackhole	-42.28KiB	-0.04	42.41%	95.37MiB	2.46MiB	50.98KiB	0.025779	95.33MiB	2.68MiB	55.79KiB	0.0281137	False	False
http_to_http_json	-43.34KiB	-0.18	99.90%	23.85MiB	337.69KiB	6.89KiB	0.0138254	23.81MiB	550.96KiB	11.25KiB	0.0225968	False	False
http_pipelines_blackhole_acks	-2.61KiB	-0.23	57.07%	1.1MiB	118.16KiB	2.41KiB	0.10509	1.1MiB	110.48KiB	2.25KiB	0.0984831	False	False
http_to_http_noack	-75.49KiB	-0.31	99.85%	23.83MiB	507.54KiB	10.38KiB	0.020796	23.75MiB	1.02MiB	21.37KiB	0.043128	False	False
http_to_http_acks	-71.3KiB	-0.39	22.80%	17.99MiB	8.37MiB	175.02KiB	0.465261	17.92MiB	8.29MiB	172.98KiB	0.462383	True	True
fluent_elasticsearch	-314.74KiB	-0.39	99.98%	79.47MiB	53.65KiB	1.08KiB	0.00065913	79.17MiB	4.12MiB	84.72KiB	0.0520871	False	False
datadog_agent_remap_blackhole_acks	-263.0KiB	-0.42	93.69%	60.51MiB	5.19MiB	108.02KiB	0.0856881	60.25MiB	4.37MiB	91.35KiB	0.0725312	False	False
socket_to_socket_blackhole	-80.19KiB	-0.57	100.00%	13.76MiB	158.71KiB	3.24KiB	0.0112594	13.68MiB	171.23KiB	3.5KiB	0.0122171	False	False
http_pipelines_blackhole	-11.82KiB	-0.73	99.64%	1.58MiB	111.97KiB	2.29KiB	0.0690393	1.57MiB	164.39KiB	3.35KiB	0.102104	False	False
datadog_agent_remap_blackhole	-535.14KiB	-0.81	100.00%	64.87MiB	4.25MiB	88.47KiB	0.0654352	64.35MiB	3.1MiB	64.62KiB	0.0481201	False	False
datadog_agent_remap_datadog_logs_acks	-746.39KiB	-1.19	100.00%	61.11MiB	3.91MiB	81.6KiB	0.0639212	60.39MiB	4.69MiB	97.58KiB	0.0776179	False	False

Jul 19 '22 17:07 github-actions[bot]

@tobz I've changed it use Buffer so feel free to check it.

As a note, there is a bug I'm currently debugging that's related to this PR. When a retry request is made after a node failed sometimes ES can return error version conflict, document already exists even though we aren't setting _id.

Jul 20 '22 13:07 ktff

@tobz @bruceg this is ready for review.

Regarding the error version conflict, document already exists it's a transient error on the side of Elasticsearch. There is a report of it here https://discuss.elastic.co/t/version-conflict-while-restarting-cluster-node/271484.

The error occurs on requests to data nodes at the time of master node disconnection. Once triggered the data node will return response with errors like:

{"took":38138,"errors":true,"items":[
{"create":{"_index":"test","_type":"_doc","_id":"YN73MIIBO6U5bdOAwuy_","status":409,"error": "type":"version_conflict_engine_exception","reason":"[YN73MIIBO6U5bdOAwuy_]: version conflict, document already exists (current version [1])","index_uuid":"_MOdBFqEQy6rRUpa3xfi2w","shard":"0","index":"test"}}},
{"create":{"_index":"test","_type":"_doc","_id":"Yd73MIIBO6U5bdOAwuy_","status":409,"error":{"type":"version_conflict_engine_exception","reason":"[Yd73MIIBO6U5bdOAwuy_]: version conflict, document already exists (current version [1])","index_uuid":"_MOdBFqEQy6rRUpa3xfi2w","shard":"0","index":"test"}}},
...

This can be triggered by Vector + distribution but it can also be triggered with well timed curl.

How to reproduce

Start two node cluster, one master, one data. Send large bulk requests to data node in a loop to increase the chance of triggering the error:

while true; do
curl --data-binary "@/some/path/data.txt" -H "Content-Type: application/json" -X POST http://localhost:9202/_bulk?timeout=60s;
echo "";
done;

Where data.txt has size counted in MB and has content like:

{ "create":{ "_index": "test"} }
{ "message": "..." "@timestamp": "2099-05-06T16:21:15.000Z"}
{ "create":{ "_index": "test"} }
{ "message": "..." "@timestamp": "2099-05-06T16:21:15.000Z"}
...

Restart master node several times to trigger the error.

To trigger it with Vector, instead of curl loop run Vector with config:

data_dir = "/var/lib/vector/"

[sources.demo]
  type = "demo_logs" # required
  format = "shuffle" # required 
  interval = 0.0
  lines = ["test"]
  
[sinks.es]
inputs = ["demo"]
type = "elasticsearch"
endpoint = "http://localhost:9201"
distribution.endpoints = ["http://localhost:9202"]
mode = "data_stream"
suppress_type_name = true

Problem

Currently we don't retry on response OK 200 that has errors. Because of it, events in that request are dropped.

A solution needs to cover partial failures of this kind since it's not clear if all of the events failed. There is an issue for this, #140.

This is out of scope of this PR.

I'll open an issue for this once this PR gets merged.

A possible fix for users is to use use bulk mode with index action.

cc. @sim0nx

Jul 24 '22 16:07 ktff

Soak Test Results

Baseline: 05a318729cbcd3733957da7c3f6cf1d9bf70283b Comparison: aca4015d9ca45b300fee7ed991adbdd96971b500 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_log2metric_humio_metrics	438.43KiB	3.29	100.00%	13.0MiB	369.21KiB	7.53KiB	0.0277252	13.43MiB	506.29KiB	10.32KiB	0.0368073	False	False
syslog_loki	433.39KiB	2.92	100.00%	14.49MiB	342.72KiB	7.01KiB	0.0230942	14.91MiB	738.02KiB	15.0KiB	0.0483206	False	False
splunk_hec_route_s3	438.51KiB	2.33	100.00%	18.35MiB	2.37MiB	49.38KiB	0.129236	18.78MiB	2.24MiB	46.75KiB	0.119034	False	False
http_text_to_http_json	835.5KiB	2.14	100.00%	38.13MiB	848.83KiB	17.33KiB	0.021736	38.94MiB	774.82KiB	15.82KiB	0.019425	False	False
socket_to_socket_blackhole	228.29KiB	1.65	100.00%	13.48MiB	132.61KiB	2.71KiB	0.00960605	13.7MiB	118.55KiB	2.42KiB	0.00844735	False	False
syslog_regex_logs2metric_ddmetrics	78.13KiB	0.59	100.00%	12.93MiB	674.0KiB	13.73KiB	0.0508772	13.01MiB	635.71KiB	12.96KiB	0.047705	False	False
http_pipelines_blackhole_acks	6.03KiB	0.5	98.84%	1.18MiB	99.99KiB	2.04KiB	0.0826029	1.19MiB	61.15KiB	1.25KiB	0.050261	False	False
datadog_agent_remap_blackhole	214.99KiB	0.34	93.53%	62.04MiB	4.34MiB	90.52KiB	0.0699653	62.25MiB	3.5MiB	73.12KiB	0.0562806	False	False
datadog_agent_remap_blackhole_acks	124.35KiB	0.19	71.42%	63.15MiB	4.71MiB	98.1KiB	0.074593	63.28MiB	3.0MiB	62.83KiB	0.0474474	False	False
splunk_hec_to_splunk_hec_logs_noack	23.43KiB	0.1	93.71%	23.81MiB	516.93KiB	10.54KiB	0.0211933	23.84MiB	337.87KiB	6.9KiB	0.0138386	False	False
enterprise_http_to_http	-2.33KiB	-0.01	25.72%	23.85MiB	241.85KiB	4.94KiB	0.00990196	23.84MiB	249.34KiB	5.1KiB	0.0102095	False	False
splunk_hec_indexer_ack_blackhole	-13.52KiB	-0.06	42.65%	23.76MiB	794.48KiB	16.17KiB	0.0326426	23.75MiB	873.21KiB	17.76KiB	0.0358977	False	False
file_to_blackhole	-60.52KiB	-0.06	42.91%	95.34MiB	3.41MiB	70.74KiB	0.0357856	95.28MiB	3.85MiB	79.98KiB	0.0403691	False	False
http_to_http_json	-26.33KiB	-0.11	97.28%	23.85MiB	335.3KiB	6.84KiB	0.0137283	23.82MiB	477.03KiB	9.75KiB	0.0195523	False	False
splunk_hec_to_splunk_hec_logs_acks	-27.28KiB	-0.11	73.85%	23.77MiB	797.05KiB	16.22KiB	0.0327443	23.74MiB	889.22KiB	18.08KiB	0.0365719	False	False
fluent_elasticsearch	-176.12KiB	-0.22	100.00%	79.47MiB	52.89KiB	1.07KiB	0.000649821	79.3MiB	1.69MiB	34.75KiB	0.0212916	False	False
http_pipelines_blackhole	-5.88KiB	-0.37	99.92%	1.53MiB	20.45KiB	427.92B	0.0130169	1.53MiB	83.65KiB	1.71KiB	0.0534458	False	False
http_pipelines_no_grok_blackhole	-46.09KiB	-0.4	95.59%	11.25MiB	129.62KiB	2.65KiB	0.0112486	11.21MiB	1.09MiB	22.73KiB	0.0973694	False	False
http_to_http_noack	-113.56KiB	-0.47	100.00%	23.85MiB	248.0KiB	5.07KiB	0.0101542	23.73MiB	1.14MiB	23.8KiB	0.0480993	False	False
http_to_http_acks	-90.7KiB	-0.48	32.14%	18.44MiB	7.48MiB	156.26KiB	0.40525	18.35MiB	7.33MiB	153.28KiB	0.399524	True	True
syslog_splunk_hec_logs	-105.96KiB	-0.59	100.00%	17.43MiB	742.99KiB	15.13KiB	0.0416079	17.33MiB	677.25KiB	13.8KiB	0.0381529	False	False
syslog_log2metric_splunk_hec_metrics	-149.6KiB	-0.77	100.00%	19.08MiB	653.97KiB	13.32KiB	0.0334716	18.93MiB	849.4KiB	17.28KiB	0.0438095	False	False
syslog_humio_logs	-142.41KiB	-0.82	100.00%	17.02MiB	166.0KiB	3.39KiB	0.00952406	16.88MiB	134.48KiB	2.75KiB	0.007779	False	False
datadog_agent_remap_datadog_logs_acks	-976.17KiB	-1.5	100.00%	63.68MiB	3.06MiB	63.94KiB	0.0480063	62.72MiB	4.45MiB	92.55KiB	0.0708719	False	False
datadog_agent_remap_datadog_logs	-1.21MiB	-1.87	100.00%	64.54MiB	305.78KiB	6.26KiB	0.00462593	63.33MiB	4.26MiB	88.79KiB	0.0673102	False	False

Jul 24 '22 20:07 github-actions[bot]

Soak Test Results

Baseline: 05a318729cbcd3733957da7c3f6cf1d9bf70283b Comparison: 6bf060a3f577a8b97b4c0cc2312ced288a2e67be Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.

experiment	Δ mean	Δ mean %	confidence	baseline mean	baseline stdev	baseline stderr	baseline CoV	comparison mean	comparison stdev	comparison stderr	comparison CoV	erratic	declared erratic
syslog_loki	477.48KiB	3.45	100.00%	13.53MiB	1.02MiB	21.36KiB	0.0753676	13.99MiB	1.17MiB	24.26KiB	0.0832886	False	False
splunk_hec_route_s3	460.09KiB	2.43	100.00%	18.49MiB	2.34MiB	48.67KiB	0.126347	18.94MiB	2.22MiB	46.42KiB	0.117103	False	False
syslog_log2metric_humio_metrics	275.22KiB	2.05	100.00%	13.09MiB	353.78KiB	7.22KiB	0.0263922	13.36MiB	589.37KiB	12.0KiB	0.0430828	False	False
socket_to_socket_blackhole	246.4KiB	1.79	100.00%	13.44MiB	158.98KiB	3.25KiB	0.0115485	13.68MiB	130.34KiB	2.66KiB	0.00930176	False	False
datadog_agent_remap_blackhole_acks	535.38KiB	0.81	100.00%	64.87MiB	4.17MiB	86.89KiB	0.0642864	65.39MiB	2.21MiB	46.17KiB	0.0337204	False	False
datadog_agent_remap_blackhole	463.64KiB	0.71	100.00%	64.12MiB	4.47MiB	92.98KiB	0.0696296	64.58MiB	3.12MiB	65.15KiB	0.0483466	False	False
syslog_regex_logs2metric_ddmetrics	54.21KiB	0.41	99.27%	12.83MiB	762.92KiB	15.52KiB	0.0580537	12.88MiB	632.4KiB	12.89KiB	0.047924	False	False
splunk_hec_to_splunk_hec_logs_noack	17.64KiB	0.07	87.29%	23.82MiB	457.53KiB	9.35KiB	0.0187514	23.84MiB	333.05KiB	6.8KiB	0.0136399	False	False
splunk_hec_indexer_ack_blackhole	3.21KiB	0.01	10.16%	23.75MiB	881.79KiB	17.93KiB	0.0362551	23.75MiB	864.1KiB	17.58KiB	0.0355229	False	False
http_pipelines_no_grok_blackhole	-282.51B	-0	0.87%	11.19MiB	369.1KiB	7.53KiB	0.0322064	11.19MiB	1.16MiB	24.08KiB	0.10335	False	False
enterprise_http_to_http	-3.24KiB	-0.01	33.74%	23.85MiB	250.18KiB	5.11KiB	0.0102433	23.84MiB	263.45KiB	5.39KiB	0.0107881	False	False
splunk_hec_to_splunk_hec_logs_acks	-5.24KiB	-0.02	17.48%	23.76MiB	814.07KiB	16.57KiB	0.0334533	23.75MiB	834.82KiB	16.98KiB	0.0343131	False	False
file_to_blackhole	-69.71KiB	-0.07	41.67%	95.36MiB	4.52MiB	93.48KiB	0.047346	95.29MiB	4.14MiB	86.05KiB	0.0434295	False	False
http_to_http_json	-39.45KiB	-0.16	99.75%	23.85MiB	338.82KiB	6.92KiB	0.0138717	23.81MiB	541.64KiB	11.06KiB	0.0222113	False	False
fluent_elasticsearch	-191.4KiB	-0.24	100.00%	79.47MiB	53.49KiB	1.08KiB	0.000657178	79.29MiB	1.74MiB	35.78KiB	0.0219161	False	False
http_to_http_noack	-83.05KiB	-0.34	99.96%	23.84MiB	412.06KiB	8.43KiB	0.0168788	23.75MiB	1.04MiB	21.76KiB	0.0439164	False	False
syslog_humio_logs	-75.86KiB	-0.42	100.00%	17.51MiB	278.67KiB	5.69KiB	0.0155362	17.44MiB	311.77KiB	6.39KiB	0.0174552	False	False
http_pipelines_blackhole	-7.61KiB	-0.46	99.82%	1.61MiB	42.67KiB	893.1B	0.0259093	1.6MiB	111.44KiB	2.27KiB	0.067977	False	False
syslog_splunk_hec_logs	-82.55KiB	-0.47	99.98%	17.19MiB	790.0KiB	16.08KiB	0.0448584	17.11MiB	742.99KiB	15.14KiB	0.0423878	False	False
syslog_log2metric_splunk_hec_metrics	-132.97KiB	-0.68	100.00%	19.15MiB	491.7KiB	10.03KiB	0.0250626	19.03MiB	840.76KiB	17.11KiB	0.0431476	False	False
http_text_to_http_json	-534.83KiB	-1.39	100.00%	37.45MiB	1.95MiB	40.81KiB	0.052161	36.93MiB	2.76MiB	57.66KiB	0.0746481	False	False
http_to_http_acks	-274.66KiB	-1.45	76.74%	18.48MiB	8.19MiB	171.24KiB	0.443129	18.22MiB	7.36MiB	153.65KiB	0.403969	True	True
datadog_agent_remap_datadog_logs_acks	-927.03KiB	-1.46	100.00%	62.05MiB	3.23MiB	67.42KiB	0.0519852	61.15MiB	4.28MiB	89.11KiB	0.0699911	False	False
datadog_agent_remap_datadog_logs	-1.13MiB	-1.83	100.00%	61.82MiB	1.0MiB	21.01KiB	0.0162201	60.69MiB	4.12MiB	85.79KiB	0.0678761	False	False
http_pipelines_blackhole_acks	-40.08KiB	-3.34	100.00%	1.17MiB	109.49KiB	2.23KiB	0.0913415	1.13MiB	85.28KiB	1.74KiB	0.0736088	False	False

Jul 24 '22 23:07 github-actions[bot]

@ktff Admittedly, seeing Currently we don't retry on response OK 200 that has errors. Because of it, events in that request are dropped. doesn't make me feel very good.

I guess my two questions would be:

it sounds like this already a possible issue to hit with Vector without this PR?
does this PR dramatically increase the chance of hitting this problem?

Jul 25 '22 17:07 tobz

@tobz

it sounds like this already a possible issue to hit with Vector without this PR?

Indeed:

https://github.com/vectordotdev/vector/issues/2968
https://discord.com/channels/742820443487993987/746070591097798688/775478035738525707
https://github.com/vectordotdev/vector/issues/10146
https://github.com/vectordotdev/vector/issues/11359

does this PR dramatically increase the chance of hitting this problem?

Not dramatically. The basic condition is for master node to fail, that's rare in it self, and that's not enough. A request must be sent to a data node before failure and data node must be processing it during master node failure, without a large several MB sized request, that's not likely to happen and even then it wont necessary happen. Although having multiple nodes increases that chance. And finally bulk.action needs to be create, which is set when data_stream mode is used, and _id isn't used. So yea, it should be rare and some users can avoid it.

Also as a note, I'm planning to tackle #140 which is becoming a necessity for this sink.

Jul 25 '22 17:07 ktff

Hmmm, alright. I'll/we'll take another pass this week.

Apologies overall for the slow review here: we're trying to avoid getting stuck with code patterns that we aren't confident in, and want to make sure this change is going to be maintainable long-term.

Jul 25 '22 18:07 tobz

This version conflict can happen without using distribution. It's possible to trigger this error with a single endpoint. So the only thing distribution changes is that it can increases throughput which proportionally increases the chance of triggering the error.

cc @tobz, @sim0nx

Jul 25 '22 18:07 ktff

enhancement(elasticsearch sink): Multiple hosts

Possible extensions

Todo

✅ Deploy Preview for vector-project canceled.

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

Soak Test Results

How to reproduce

Problem

Soak Test Results

Soak Test Results

Deploy Preview for vector-project canceled.