vector icon indicating copy to clipboard operation
vector copied to clipboard
verified publisher icon vectordotdev

enhancement(socket sink): Adhere to instrumentation spec

Open neuronull opened this issue 3 years ago • 4 comments

Closes: #14211

Epic: https://github.com/vectordotdev/vector/issues/13995

  • Added ComponentEventsDropped where it seemed correct to me.
  • Combined nearly identical Error events.
  • General cleanup on Error events

neuronull avatar Sep 14 '22 20:09 neuronull

Deploy Preview for vector-project canceled.

Name Link
Latest commit 63731b8c947ad012f0df79f9489ef0c146b6b271
Latest deploy log https://app.netlify.com/sites/vector-project/deploys/63498fd09a6e3e00087ddc4c

netlify[bot] avatar Sep 14 '22 20:09 netlify[bot]

Soak Test Results

Baseline: 66441f3eeb0f56fa8f1aefe9883c9e563b01598b Comparison: 5d1336c5f8dbca2f348af83ccc31595df9d07c58 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_humio_logs 306.79KiB 1.92 100.00% 15.56MiB 1.05MiB 22.0KiB 0 0.0675554 15.86MiB 774.33KiB 15.86KiB 0 0.0476566 False False
http_to_http_acks 297.03KiB 1.69 80.22% 17.13MiB 8.04MiB 168.01KiB 0 0.468974 17.42MiB 7.57MiB 158.0KiB 0 0.434414 True True
splunk_hec_route_s3 203.67KiB 1.06 99.82% 18.75MiB 2.26MiB 47.13KiB 0 0.120684 18.95MiB 2.16MiB 45.26KiB 0 0.114127 False False
datadog_agent_remap_blackhole_acks 415.98KiB 0.66 100.00% 61.46MiB 4.1MiB 85.49KiB 0 0.0667677 61.87MiB 2.44MiB 50.99KiB 0 0.0393647 False False
datadog_agent_remap_blackhole 243.21KiB 0.4 98.96% 60.0MiB 3.61MiB 75.28KiB 0 0.0601606 60.24MiB 2.76MiB 57.66KiB 0 0.045854 False False
splunk_hec_to_splunk_hec_logs_noack 20.79KiB 0.09 90.49% 23.82MiB 512.4KiB 10.45KiB 0 0.0210062 23.84MiB 331.67KiB 6.77KiB 0 0.0135853 False False
splunk_hec_to_splunk_hec_logs_acks 14.73KiB 0.06 43.28% 23.74MiB 912.47KiB 18.55KiB 0 0.0375241 23.76MiB 876.97KiB 17.84KiB 0 0.0360421 False False
enterprise_http_to_http -1.04KiB -0 10.96% 23.85MiB 262.03KiB 5.35KiB 0 0.0107283 23.85MiB 258.67KiB 5.29KiB 0 0.0105911 False False
syslog_regex_logs2metric_ddmetrics -3.94KiB -0.03 20.99% 12.45MiB 527.79KiB 10.76KiB 0 0.0413797 12.45MiB 499.44KiB 10.18KiB 0 0.0391688 False False
splunk_hec_indexer_ack_blackhole -15.65KiB -0.06 49.36% 23.77MiB 788.07KiB 16.04KiB 0 0.0323729 23.75MiB 848.13KiB 17.25KiB 0 0.0348623 False False
file_to_blackhole -91.01KiB -0.09 59.80% 95.34MiB 3.4MiB 70.51KiB 0 0.0356673 95.25MiB 3.97MiB 82.58KiB 0 0.0417114 False False
http_to_http_json -28.83KiB -0.12 98.21% 23.85MiB 335.23KiB 6.84KiB 0 0.0137258 23.82MiB 492.73KiB 10.07KiB 0 0.0201981 False False
fluent_elasticsearch -117.05KiB -0.14 100.00% 79.47MiB 52.13KiB 1.05KiB 0 0.000640464 79.36MiB 1.16MiB 23.82KiB 0 0.0145706 False False
http_to_http_noack -52.36KiB -0.21 97.25% 23.82MiB 608.55KiB 12.44KiB 0 0.0249439 23.77MiB 992.7KiB 20.23KiB 0 0.0407777 False False
datadog_agent_remap_datadog_logs_acks -173.04KiB -0.29 91.27% 58.92MiB 2.61MiB 54.58KiB 0 0.0442738 58.75MiB 4.09MiB 85.18KiB 0 0.0696376 False False
syslog_log2metric_splunk_hec_metrics -71.97KiB -0.41 99.70% 17.03MiB 772.24KiB 15.75KiB 0 0.0442663 16.96MiB 906.46KiB 18.46KiB 0 0.0521755 False False
syslog_loki -80.81KiB -0.53 100.00% 14.77MiB 356.14KiB 7.29KiB 0 0.0235361 14.69MiB 737.86KiB 15.0KiB 0 0.0490248 False False
datadog_agent_remap_datadog_logs -406.02KiB -0.64 100.00% 62.25MiB 489.1KiB 10.02KiB 0 0.00767076 61.86MiB 4.4MiB 91.65KiB 0 0.0711644 False False
http_text_to_http_json -315.7KiB -0.8 100.00% 38.36MiB 839.82KiB 17.14KiB 0 0.021376 38.05MiB 891.12KiB 18.19KiB 0 0.0228655 False False
syslog_splunk_hec_logs -152.04KiB -0.91 100.00% 16.41MiB 705.42KiB 14.36KiB 0 0.0419796 16.26MiB 691.24KiB 14.09KiB 0 0.0415119 False False
http_pipelines_blackhole -17.32KiB -0.99 100.00% 1.7MiB 55.05KiB 1.12KiB 0 0.0315695 1.69MiB 114.94KiB 2.34KiB 0 0.0665737 False False
syslog_log2metric_humio_metrics -233.42KiB -1.84 100.00% 12.36MiB 333.88KiB 6.82KiB 0 0.0263721 12.13MiB 615.76KiB 12.52KiB 0 0.0495497 False False
http_pipelines_no_grok_blackhole -341.46KiB -3.08 100.00% 10.84MiB 264.42KiB 5.4KiB 0 0.0238175 10.51MiB 1.07MiB 22.38KiB 0 0.102292 False False
http_pipelines_blackhole_acks -44.02KiB -3.6 100.00% 1.19MiB 119.75KiB 2.44KiB 0 0.0979376 1.15MiB 100.3KiB 2.04KiB 0 0.0850974 False False
socket_to_socket_blackhole -924.51KiB -3.86 100.00% 23.39MiB 970.34KiB 19.81KiB 0 0.0404988 22.49MiB 901.07KiB 18.4KiB 0 0.0391176 False False

github-actions[bot] avatar Sep 14 '22 21:09 github-actions[bot]

Soak Test Results

Baseline: a3de63325ca611bb383bb197ba5a31bc81b9ea76 Comparison: adc74e0b061dc950a6ab0a9f2e2ec0d6ea486805 Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
datadog_agent_remap_blackhole 1.05MiB 1.74 100.00% 59.95MiB 4.05MiB 84.24KiB 0 0.0674547 61.0MiB 2.04MiB 42.55KiB 0 0.0333919 False False
datadog_agent_remap_blackhole_acks 501.94KiB 0.82 99.99% 60.1MiB 4.6MiB 95.85KiB 0 0.0765196 60.59MiB 4.05MiB 84.74KiB 0 0.0668962 False False
http_text_to_http_json 304.04KiB 0.78 100.00% 38.05MiB 870.03KiB 17.76KiB 0 0.0223261 38.34MiB 868.43KiB 17.73KiB 0 0.0221126 False False
syslog_loki 69.07KiB 0.47 99.99% 14.36MiB 427.78KiB 8.76KiB 0 0.0290872 14.43MiB 721.49KiB 14.67KiB 0 0.0488287 False False
http_pipelines_blackhole_acks 1.67KiB 0.13 47.54% 1.21MiB 105.08KiB 2.14KiB 0 0.0845286 1.22MiB 74.67KiB 1.52KiB 0 0.0599857 False False
splunk_hec_to_splunk_hec_logs_noack 21.66KiB 0.09 91.50% 23.82MiB 517.46KiB 10.56KiB 0 0.0212133 23.84MiB 334.1KiB 6.82KiB 0 0.0136844 False False
datadog_agent_remap_datadog_logs_acks 31.77KiB 0.05 23.16% 61.36MiB 2.81MiB 58.84KiB 0 0.0458297 61.39MiB 4.34MiB 90.43KiB 0 0.0707549 False False
splunk_hec_indexer_ack_blackhole 2.97KiB 0.01 9.36% 23.75MiB 892.17KiB 18.15KiB 0 0.0366799 23.75MiB 862.29KiB 17.54KiB 0 0.0354468 False False
syslog_log2metric_humio_metrics -169.21B -0 1.21% 12.94MiB 193.19KiB 3.95KiB 0 0.0145718 12.94MiB 498.27KiB 10.15KiB 0 0.0375837 False False
splunk_hec_to_splunk_hec_logs_acks -1.15KiB -0 3.68% 23.75MiB 862.56KiB 17.54KiB 0 0.0354582 23.75MiB 869.42KiB 17.68KiB 0 0.035742 False False
enterprise_http_to_http -990.27B -0 10.71% 23.85MiB 247.87KiB 5.06KiB 0 0.0101485 23.85MiB 249.08KiB 5.09KiB 0 0.0101986 False False
file_to_blackhole -59.53KiB -0.06 42.23% 95.33MiB 3.44MiB 71.4KiB 0 0.036121 95.28MiB 3.83MiB 79.59KiB 0 0.0401707 False False
http_to_http_json -27.96KiB -0.11 98.07% 23.85MiB 339.96KiB 6.94KiB 0 0.013918 23.82MiB 475.74KiB 9.73KiB 0 0.0194993 False False
datadog_agent_remap_datadog_logs -123.03KiB -0.2 84.91% 61.22MiB 1.04MiB 21.82KiB 0 0.0170095 61.1MiB 3.97MiB 82.81KiB 0 0.065044 False False
fluent_elasticsearch -188.87KiB -0.23 100.00% 79.47MiB 52.45KiB 1.06KiB 0 0.000644342 79.29MiB 1.61MiB 33.21KiB 0 0.0203572 False False
http_to_http_noack -86.0KiB -0.35 99.98% 23.84MiB 405.39KiB 8.29KiB 0 0.0166047 23.75MiB 1.04MiB 21.73KiB 0 0.0438536 False False
syslog_splunk_hec_logs -66.97KiB -0.4 99.77% 16.19MiB 807.49KiB 16.43KiB 0 0.0487079 16.12MiB 711.71KiB 14.51KiB 0 0.0431049 False False
syslog_regex_logs2metric_ddmetrics -69.55KiB -0.54 100.00% 12.64MiB 534.08KiB 10.89KiB 0 0.0412436 12.58MiB 636.81KiB 12.97KiB 0 0.0494419 False False
syslog_humio_logs -94.63KiB -0.56 100.00% 16.41MiB 260.28KiB 5.31KiB 0 0.0154829 16.32MiB 203.6KiB 4.17KiB 0 0.0121802 False False
http_pipelines_blackhole -10.53KiB -0.62 100.00% 1.65MiB 38.03KiB 795.89B 0 0.0225253 1.64MiB 103.73KiB 2.12KiB 0 0.061825 False False
splunk_hec_route_s3 -146.26KiB -0.77 96.94% 18.61MiB 2.34MiB 48.69KiB 0 0.125647 18.47MiB 2.24MiB 46.92KiB 0 0.121467 False False
socket_to_socket_blackhole -182.6KiB -0.79 100.00% 22.55MiB 293.86KiB 6.0KiB 0 0.0127231 22.37MiB 292.68KiB 5.98KiB 0 0.0127729 False False
syslog_log2metric_splunk_hec_metrics -156.7KiB -0.85 100.00% 18.01MiB 668.09KiB 13.61KiB 0 0.0362088 17.86MiB 810.92KiB 16.5KiB 0 0.0443261 False False
http_to_http_acks -192.44KiB -1.08 57.84% 17.43MiB 8.1MiB 169.27KiB 0 0.464539 17.24MiB 8.1MiB 169.32KiB 0 0.469563 True True
http_pipelines_no_grok_blackhole -366.89KiB -3.27 100.00% 10.96MiB 53.91KiB 1.1KiB 0 0.00480346 10.6MiB 1.04MiB 21.61KiB 0 0.0978543 False False

github-actions[bot] avatar Sep 20 '22 03:09 github-actions[bot]

Soak Test Results

Baseline: 14fefee7ebef098fdb2b5d02ea37d5f759ade48d Comparison: 90ec41059d22cd6e78c70620a462b675a9eff5df Total Vector CPUs: 4

Explanation

A soak test is an integrated performance test for vector in a repeatable rig, with varying configuration for vector. What follows is a statistical summary of a brief vector run for each configuration across SHAs given above. The goal of these tests are to determine, quickly, if vector performance is changed and to what degree by a pull request. Where appropriate units are scaled per-core.

The table below, if present, lists those experiments that have experienced a statistically significant change in their throughput performance between baseline and comparision SHAs, with 90.0% confidence OR have been detected as newly erratic. Negative values mean that baseline is faster, positive comparison. Results that do not exhibit more than a ±8.87% change in mean throughput are discarded. An experiment is erratic if its coefficient of variation is greater than 0.3. The abbreviated table will be omitted if no interesting changes are observed.

No interesting changes in throughput with confidence ≥ 90.00% and absolute Δ mean >= ±8.87%:

Fine details of change detection per experiment.
experiment Δ mean Δ mean % confidence baseline mean baseline stdev baseline stderr baseline outlier % baseline CoV comparison mean comparison stdev comparison stderr comparison outlier % comparison CoV erratic declared erratic
syslog_loki 426.4KiB 2.96 100.00% 14.07MiB 622.63KiB 12.75KiB 0 0.0432147 14.48MiB 877.5KiB 17.84KiB 0 0.0591539 False False
splunk_hec_route_s3 401.12KiB 2.15 100.00% 18.2MiB 2.28MiB 47.41KiB 0 0.125093 18.6MiB 2.19MiB 45.75KiB 0 0.117617 False False
socket_to_socket_blackhole 295.44KiB 1.21 100.00% 23.86MiB 177.77KiB 3.63KiB 0 0.00727382 24.15MiB 276.01KiB 5.63KiB 0 0.0111586 False False
datadog_agent_remap_blackhole 693.03KiB 1.16 100.00% 58.42MiB 4.35MiB 90.63KiB 0 0.0744179 59.1MiB 3.33MiB 69.59KiB 0 0.0564021 False False
syslog_humio_logs 167.93KiB 1.01 100.00% 16.26MiB 150.29KiB 3.07KiB 0 0.00902468 16.42MiB 161.48KiB 3.31KiB 0 0.00959971 False False
http_pipelines_blackhole_acks 11.8KiB 0.97 100.00% 1.18MiB 103.42KiB 2.11KiB 0 0.0854003 1.19MiB 81.49KiB 1.66KiB 0 0.0666452 False False
syslog_splunk_hec_logs 70.74KiB 0.42 99.99% 16.31MiB 711.89KiB 14.5KiB 0 0.0426242 16.38MiB 544.15KiB 11.11KiB 0 0.0324433 False False
datadog_agent_remap_blackhole_acks 195.41KiB 0.31 93.82% 61.85MiB 3.96MiB 82.6KiB 0 0.0640921 62.04MiB 3.07MiB 64.18KiB 0 0.0494916 False False
syslog_log2metric_humio_metrics 19.88KiB 0.15 90.69% 12.78MiB 266.2KiB 5.43KiB 0 0.0203381 12.8MiB 516.38KiB 10.51KiB 0 0.039393 False False
syslog_regex_logs2metric_ddmetrics 18.99KiB 0.15 71.97% 12.58MiB 663.29KiB 13.5KiB 0 0.0514932 12.6MiB 553.27KiB 11.28KiB 0 0.042889 False False
splunk_hec_indexer_ack_blackhole 17.04KiB 0.07 48.67% 23.74MiB 936.02KiB 19.03KiB 0 0.0384991 23.75MiB 875.79KiB 17.82KiB 0 0.0359968 False False
splunk_hec_to_splunk_hec_logs_noack 16.83KiB 0.07 84.80% 23.82MiB 467.65KiB 9.54KiB 0 0.0191665 23.84MiB 335.45KiB 6.85KiB 0 0.013739 False False
syslog_log2metric_splunk_hec_metrics 6.35KiB 0.04 26.27% 17.66MiB 570.08KiB 11.62KiB 0 0.031514 17.67MiB 733.75KiB 14.95KiB 0 0.0405477 False False
splunk_hec_to_splunk_hec_logs_acks 8.2KiB 0.03 26.81% 23.75MiB 849.32KiB 17.28KiB 0 0.0349121 23.76MiB 813.45KiB 16.55KiB 0 0.0334262 False False
enterprise_http_to_http -2.12KiB -0.01 23.04% 23.85MiB 247.39KiB 5.05KiB 0 0.0101287 23.84MiB 253.47KiB 5.19KiB 0 0.0103787 False False
file_to_blackhole -49.82KiB -0.05 42.05% 95.35MiB 2.92MiB 60.54KiB 0 0.0306229 95.3MiB 3.19MiB 66.45KiB 0 0.0335045 False False
http_to_http_json -17.69KiB -0.07 88.05% 23.85MiB 347.43KiB 7.09KiB 0 0.0142256 23.83MiB 433.63KiB 8.87KiB 0 0.0177679 False False
fluent_elasticsearch -174.7KiB -0.21 100.00% 79.47MiB 53.45KiB 1.08KiB 0 0.000656714 79.3MiB 1.53MiB 31.54KiB 0 0.0193212 False False
http_to_http_noack -69.92KiB -0.29 99.86% 23.84MiB 410.87KiB 8.4KiB 0 0.0168287 23.77MiB 991.29KiB 20.2KiB 0 0.0407185 False False
http_text_to_http_json -148.74KiB -0.37 100.00% 39.78MiB 791.75KiB 16.16KiB 0 0.0194346 39.63MiB 1.0MiB 20.97KiB 0 0.025311 False False
http_to_http_acks -95.34KiB -0.54 32.38% 17.37MiB 7.52MiB 157.21KiB 0 0.432911 17.28MiB 7.92MiB 165.45KiB 0 0.458522 True True
datadog_agent_remap_datadog_logs -391.31KiB -0.61 100.00% 63.09MiB 447.13KiB 9.16KiB 0 0.00691969 62.71MiB 3.88MiB 80.75KiB 0 0.0617861 False False
http_pipelines_blackhole -11.36KiB -0.65 100.00% 1.7MiB 10.82KiB 226.49B 0 0.00621417 1.69MiB 110.26KiB 2.25KiB 0 0.0637289 False False
http_pipelines_no_grok_blackhole -86.75KiB -0.79 99.99% 10.75MiB 98.76KiB 2.02KiB 0 0.00896669 10.67MiB 1.07MiB 22.19KiB 0 0.0998818 False False
datadog_agent_remap_datadog_logs_acks -657.23KiB -1.05 100.00% 61.15MiB 3.22MiB 67.27KiB 0 0.0526213 60.51MiB 4.37MiB 90.89KiB 0 0.0721492 False False

github-actions[bot] avatar Sep 20 '22 15:09 github-actions[bot]