add recipe for handling flaky tests
What Does This Do
This change adds some documentation explaining how to handle flaky tests when they occur on pull requests. It also removes a few outdated references to CircleCI.
Motivation
The motivation is to make it easier for new contributors to get up to speed on the project.
Jira ticket: LANGPLAT-352
Hi! 👋 Thanks for your pull request! 🎉
To help us review it, please make sure to:
- Remove the tag from the pull request title
If you need help, please check our contributing guidelines.
Benchmarks
Startup
Parameters
| Baseline | Candidate | |
|---|---|---|
| baseline_or_candidate | baseline | candidate |
| git_branch | master | emmett.butler/flaky-guidance |
| git_commit_date | 1750427857 | 1750447681 |
| git_commit_sha | 60c30c276e | f8ec06c202 |
| release_version | 1.51.0-SNAPSHOT~60c30c276e | 1.51.0-SNAPSHOT~f8ec06c202 |
See matching parameters
| Baseline | Candidate | |
|---|---|---|
| application | insecure-bank | insecure-bank |
| ci_job_date | 1750449582 | 1750449582 |
| ci_job_id | 992078701 | 992078701 |
| ci_pipeline_id | 68361068 | 68361068 |
| cpu_model | Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz |
| kernel_version | Linux runner-clrnfvyp-project-304-concurrent-0-4979j6gk 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux | Linux runner-clrnfvyp-project-304-concurrent-0-4979j6gk 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| module | Agent | Agent |
| parent | None | None |
Summary
Found 0 performance improvements and 0 performance regressions! Performance is the same for 43 metrics, 10 unstable metrics.
Startup time reports for petclinic
gantt
title petclinic - global startup overhead: candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.026 s) : 0, 1025897
Total [baseline] (10.504 s) : 0, 10503768
Agent [candidate] (1.025 s) : 0, 1025436
Total [candidate] (10.643 s) : 0, 10642590
section appsec
Agent [baseline] (1.188 s) : 0, 1187607
Total [baseline] (10.833 s) : 0, 10832560
Agent [candidate] (1.185 s) : 0, 1185209
Total [candidate] (10.773 s) : 0, 10772905
section iast
Agent [baseline] (1.151 s) : 0, 1151015
Total [baseline] (10.848 s) : 0, 10847997
Agent [candidate] (1.161 s) : 0, 1160964
Total [candidate] (10.864 s) : 0, 10863621
section profiling
Agent [baseline] (1.268 s) : 0, 1268441
Total [baseline] (10.952 s) : 0, 10951651
Agent [candidate] (1.273 s) : 0, 1272572
Total [candidate] (10.991 s) : 0, 10990852
- baseline results
| Module | Variant | Duration | Δ tracing |
|---|---|---|---|
| Agent | tracing | 1.026 s | - |
| Agent | appsec | 1.188 s | 161.71 ms (15.8%) |
| Agent | iast | 1.151 s | 125.118 ms (12.2%) |
| Agent | profiling | 1.268 s | 242.543 ms (23.6%) |
| Total | tracing | 10.504 s | - |
| Total | appsec | 10.833 s | 328.792 ms (3.1%) |
| Total | iast | 10.848 s | 344.229 ms (3.3%) |
| Total | profiling | 10.952 s | 447.883 ms (4.3%) |
- candidate results
| Module | Variant | Duration | Δ tracing |
|---|---|---|---|
| Agent | tracing | 1.025 s | - |
| Agent | appsec | 1.185 s | 159.773 ms (15.6%) |
| Agent | iast | 1.161 s | 135.528 ms (13.2%) |
| Agent | profiling | 1.273 s | 247.136 ms (24.1%) |
| Total | tracing | 10.643 s | - |
| Total | appsec | 10.773 s | 130.315 ms (1.2%) |
| Total | iast | 10.864 s | 221.031 ms (2.1%) |
| Total | profiling | 10.991 s | 348.263 ms (3.3%) |
gantt
title petclinic - break down per module: candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.68 ms) : 0, 686680
BytebuddyAgent [candidate] (685.804 ms) : 0, 685804
GlobalTracer [baseline] (242.344 ms) : 0, 242344
GlobalTracer [candidate] (242.107 ms) : 0, 242107
AppSec [baseline] (61.238 ms) : 0, 61238
AppSec [candidate] (58.359 ms) : 0, 58359
Debugger [baseline] (6.097 ms) : 0, 6097
Debugger [candidate] (7.007 ms) : 0, 7007
Remote Config [baseline] (645.145 µs) : 0, 645
Remote Config [candidate] (669.307 µs) : 0, 669
Telemetry [baseline] (8.003 ms) : 0, 8003
Telemetry [candidate] (10.641 ms) : 0, 10641
section appsec
BytebuddyAgent [baseline] (714.341 ms) : 0, 714341
BytebuddyAgent [candidate] (713.031 ms) : 0, 713031
GlobalTracer [baseline] (236.799 ms) : 0, 236799
GlobalTracer [candidate] (236.308 ms) : 0, 236308
IAST [baseline] (22.015 ms) : 0, 22015
IAST [candidate] (22.129 ms) : 0, 22129
AppSec [baseline] (179.877 ms) : 0, 179877
AppSec [candidate] (179.283 ms) : 0, 179283
Debugger [baseline] (5.809 ms) : 0, 5809
Debugger [candidate] (5.77 ms) : 0, 5770
Remote Config [baseline] (604.759 µs) : 0, 605
Remote Config [candidate] (606.005 µs) : 0, 606
Telemetry [baseline] (7.265 ms) : 0, 7265
Telemetry [candidate] (7.216 ms) : 0, 7216
section iast
BytebuddyAgent [baseline] (804.712 ms) : 0, 804712
BytebuddyAgent [candidate] (812.268 ms) : 0, 812268
GlobalTracer [baseline] (231.61 ms) : 0, 231610
GlobalTracer [candidate] (233.392 ms) : 0, 233392
IAST [baseline] (25.285 ms) : 0, 25285
IAST [candidate] (27.029 ms) : 0, 27029
AppSec [baseline] (54.034 ms) : 0, 54034
AppSec [candidate] (52.633 ms) : 0, 52633
Debugger [baseline] (5.927 ms) : 0, 5927
Debugger [candidate] (5.999 ms) : 0, 5999
Remote Config [baseline] (601.135 µs) : 0, 601
Remote Config [candidate] (629.193 µs) : 0, 629
Telemetry [baseline] (7.972 ms) : 0, 7972
Telemetry [candidate] (8.044 ms) : 0, 8044
section profiling
BytebuddyAgent [baseline] (676.044 ms) : 0, 676044
BytebuddyAgent [candidate] (677.589 ms) : 0, 677589
GlobalTracer [baseline] (360.592 ms) : 0, 360592
GlobalTracer [candidate] (361.396 ms) : 0, 361396
AppSec [baseline] (62.258 ms) : 0, 62258
AppSec [candidate] (62.303 ms) : 0, 62303
Debugger [baseline] (6.09 ms) : 0, 6090
Debugger [candidate] (6.097 ms) : 0, 6097
Remote Config [baseline] (705.341 µs) : 0, 705
Remote Config [candidate] (706.053 µs) : 0, 706
Telemetry [baseline] (8.06 ms) : 0, 8060
Telemetry [candidate] (8.135 ms) : 0, 8135
ProfilingAgent [baseline] (106.325 ms) : 0, 106325
ProfilingAgent [candidate] (107.586 ms) : 0, 107586
Profiling [baseline] (106.35 ms) : 0, 106350
Profiling [candidate] (107.612 ms) : 0, 107612
Startup time reports for insecure-bank
gantt
title insecure-bank - global startup overhead: candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.03 s) : 0, 1030421
Total [baseline] (8.552 s) : 0, 8551769
Agent [candidate] (1.025 s) : 0, 1025058
Total [candidate] (8.568 s) : 0, 8567647
section iast
Agent [baseline] (1.148 s) : 0, 1147624
Total [baseline] (9.204 s) : 0, 9203944
Agent [candidate] (1.152 s) : 0, 1152074
Total [candidate] (9.219 s) : 0, 9219466
- baseline results
| Module | Variant | Duration | Δ tracing |
|---|---|---|---|
| Agent | tracing | 1.03 s | - |
| Agent | iast | 1.148 s | 117.203 ms (11.4%) |
| Total | tracing | 8.552 s | - |
| Total | iast | 9.204 s | 652.176 ms (7.6%) |
- candidate results
| Module | Variant | Duration | Δ tracing |
|---|---|---|---|
| Agent | tracing | 1.025 s | - |
| Agent | iast | 1.152 s | 127.015 ms (12.4%) |
| Total | tracing | 8.568 s | - |
| Total | iast | 9.219 s | 651.819 ms (7.6%) |
gantt
title insecure-bank - break down per module: candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (691.091 ms) : 0, 691091
BytebuddyAgent [candidate] (685.648 ms) : 0, 685648
GlobalTracer [baseline] (242.007 ms) : 0, 242007
GlobalTracer [candidate] (242.091 ms) : 0, 242091
AppSec [baseline] (55.663 ms) : 0, 55663
AppSec [candidate] (58.542 ms) : 0, 58542
Debugger [baseline] (7.053 ms) : 0, 7053
Debugger [candidate] (7.611 ms) : 0, 7611
Remote Config [baseline] (697.43 µs) : 0, 697
Remote Config [candidate] (657.787 µs) : 0, 658
Telemetry [baseline] (12.886 ms) : 0, 12886
Telemetry [candidate] (9.628 ms) : 0, 9628
section iast
BytebuddyAgent [baseline] (802.759 ms) : 0, 802759
BytebuddyAgent [candidate] (805.568 ms) : 0, 805568
GlobalTracer [baseline] (231.144 ms) : 0, 231144
GlobalTracer [candidate] (232.076 ms) : 0, 232076
IAST [baseline] (27.331 ms) : 0, 27331
IAST [candidate] (26.119 ms) : 0, 26119
AppSec [baseline] (51.178 ms) : 0, 51178
AppSec [candidate] (53.023 ms) : 0, 53023
Debugger [baseline] (5.835 ms) : 0, 5835
Debugger [candidate] (5.932 ms) : 0, 5932
Remote Config [baseline] (602.001 µs) : 0, 602
Remote Config [candidate] (597.377 µs) : 0, 597
Telemetry [baseline] (7.973 ms) : 0, 7973
Telemetry [candidate] (7.96 ms) : 0, 7960
Load
Parameters
| Baseline | Candidate | |
|---|---|---|
| baseline_or_candidate | baseline | candidate |
| git_branch | master | emmett.butler/flaky-guidance |
| git_commit_date | 1750427857 | 1750447681 |
| git_commit_sha | 60c30c276e | f8ec06c202 |
| release_version | 1.51.0-SNAPSHOT~60c30c276e | 1.51.0-SNAPSHOT~f8ec06c202 |
See matching parameters
| Baseline | Candidate | |
|---|---|---|
| application | insecure-bank | insecure-bank |
| ci_job_date | 1750449252 | 1750449252 |
| ci_job_id | 992078702 | 992078702 |
| ci_pipeline_id | 68361068 | 68361068 |
| cpu_model | Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz |
| kernel_version | Linux runner-bziamzy-project-304-concurrent-0-kviyhrps 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux | Linux runner-bziamzy-project-304-concurrent-0-kviyhrps 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
Summary
Found 3 performance improvements and 1 performance regressions! Performance is the same for 8 metrics, 12 unstable metrics.
| scenario | Δ mean http_req_duration | Δ mean throughput | candidate mean http_req_duration | candidate mean throughput | baseline mean http_req_duration | baseline mean throughput |
|---|---|---|---|---|---|---|
| scenario:load:insecure-bank:iast:high_load | better [-560.820µs; -235.932µs] or [-5.950%; -2.503%] |
unstable [-48.791op/s; +91.354op/s] or [-9.906%; +18.547%] |
9.027ms | 513.844op/s | 9.425ms | 492.562op/s |
| scenario:load:insecure-bank:tracing:high_load | better [-605.283µs; -349.509µs] or [-7.449%; -4.301%] |
unstable [-46.962op/s; +116.524op/s] or [-8.232%; +20.426%] |
7.648ms | 605.250op/s | 8.125ms | 570.469op/s |
| scenario:load:petclinic:appsec:high_load | worse [+1.484ms; +2.357ms] or [+3.231%; +5.131%] |
unstable [-10.528op/s; +2.353op/s] or [-10.339%; +2.311%] |
47.869ms | 97.737op/s | 45.948ms | 101.825op/s |
| scenario:load:petclinic:iast:high_load | better [-2.365ms; -1.548ms] or [-5.258%; -3.443%] |
unstable [-2.544op/s; +11.969op/s] or [-2.446%; +11.509%] |
43.020ms | 108.713op/s | 44.977ms | 104.000op/s |
Request duration reports for petclinic
gantt
title petclinic - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section baseline
no_agent (37.529 ms) : 37223, 37835
. : milestone, 37529,
appsec (45.948 ms) : 45558, 46338
. : milestone, 45948,
code_origins (44.91 ms) : 44527, 45293
. : milestone, 44910,
iast (44.977 ms) : 44581, 45373
. : milestone, 44977,
profiling (48.746 ms) : 48309, 49182
. : milestone, 48746,
tracing (42.558 ms) : 42226, 42891
. : milestone, 42558,
section candidate
no_agent (36.583 ms) : 36293, 36873
. : milestone, 36583,
appsec (47.869 ms) : 47448, 48290
. : milestone, 47869,
code_origins (44.016 ms) : 43649, 44383
. : milestone, 44016,
iast (43.02 ms) : 42658, 43382
. : milestone, 43020,
profiling (48.052 ms) : 47584, 48521
. : milestone, 48052,
tracing (42.79 ms) : 42450, 43130
. : milestone, 42790,
- baseline results
| Variant | Request duration [CI 0.99] | Δ no_agent |
|---|---|---|
| no_agent | 37.529 ms [37.223 ms, 37.835 ms] | - |
| appsec | 45.948 ms [45.558 ms, 46.338 ms] | 8.419 ms (22.4%) |
| code_origins | 44.91 ms [44.527 ms, 45.293 ms] | 7.381 ms (19.7%) |
| iast | 44.977 ms [44.581 ms, 45.373 ms] | 7.448 ms (19.8%) |
| profiling | 48.746 ms [48.309 ms, 49.182 ms] | 11.217 ms (29.9%) |
| tracing | 42.558 ms [42.226 ms, 42.891 ms] | 5.03 ms (13.4%) |
- candidate results
| Variant | Request duration [CI 0.99] | Δ no_agent |
|---|---|---|
| no_agent | 36.583 ms [36.293 ms, 36.873 ms] | - |
| appsec | 47.869 ms [47.448 ms, 48.29 ms] | 11.286 ms (30.9%) |
| code_origins | 44.016 ms [43.649 ms, 44.383 ms] | 7.433 ms (20.3%) |
| iast | 43.02 ms [42.658 ms, 43.382 ms] | 6.437 ms (17.6%) |
| profiling | 48.052 ms [47.584 ms, 48.521 ms] | 11.469 ms (31.4%) |
| tracing | 42.79 ms [42.45 ms, 43.13 ms] | 6.207 ms (17.0%) |
Request duration reports for insecure-bank
gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section baseline
no_agent (4.387 ms) : 4337, 4436
. : milestone, 4387,
iast (9.425 ms) : 9269, 9581
. : milestone, 9425,
iast_FULL (13.818 ms) : 13544, 14091
. : milestone, 13818,
iast_GLOBAL (9.898 ms) : 9726, 10070
. : milestone, 9898,
profiling (8.476 ms) : 8336, 8615
. : milestone, 8476,
tracing (8.125 ms) : 8000, 8251
. : milestone, 8125,
section candidate
no_agent (4.468 ms) : 4414, 4523
. : milestone, 4468,
iast (9.027 ms) : 8880, 9173
. : milestone, 9027,
iast_FULL (14.096 ms) : 13812, 14379
. : milestone, 14096,
iast_GLOBAL (10.207 ms) : 10025, 10389
. : milestone, 10207,
profiling (8.772 ms) : 8632, 8911
. : milestone, 8772,
tracing (7.648 ms) : 7536, 7759
. : milestone, 7648,
- baseline results
| Variant | Request duration [CI 0.99] | Δ no_agent |
|---|---|---|
| no_agent | 4.387 ms [4.337 ms, 4.436 ms] | - |
| iast | 9.425 ms [9.269 ms, 9.581 ms] | 5.038 ms (114.9%) |
| iast_FULL | 13.818 ms [13.544 ms, 14.091 ms] | 9.431 ms (215.0%) |
| iast_GLOBAL | 9.898 ms [9.726 ms, 10.07 ms] | 5.511 ms (125.6%) |
| profiling | 8.476 ms [8.336 ms, 8.615 ms] | 4.089 ms (93.2%) |
| tracing | 8.125 ms [8.0 ms, 8.251 ms] | 3.739 ms (85.2%) |
- candidate results
| Variant | Request duration [CI 0.99] | Δ no_agent |
|---|---|---|
| no_agent | 4.468 ms [4.414 ms, 4.523 ms] | - |
| iast | 9.027 ms [8.88 ms, 9.173 ms] | 4.558 ms (102.0%) |
| iast_FULL | 14.096 ms [13.812 ms, 14.379 ms] | 9.628 ms (215.5%) |
| iast_GLOBAL | 10.207 ms [10.025 ms, 10.389 ms] | 5.739 ms (128.4%) |
| profiling | 8.772 ms [8.632 ms, 8.911 ms] | 4.304 ms (96.3%) |
| tracing | 7.648 ms [7.536 ms, 7.759 ms] | 3.18 ms (71.2%) |
Dacapo
Parameters
| Baseline | Candidate | |
|---|---|---|
| baseline_or_candidate | baseline | candidate |
| git_branch | master | emmett.butler/flaky-guidance |
| git_commit_date | 1750427857 | 1750447681 |
| git_commit_sha | 60c30c276e | f8ec06c202 |
| release_version | 1.51.0-SNAPSHOT~60c30c276e | 1.51.0-SNAPSHOT~f8ec06c202 |
See matching parameters
| Baseline | Candidate | |
|---|---|---|
| application | biojava | biojava |
| ci_job_date | 1750449763 | 1750449763 |
| ci_job_id | 992078703 | 992078703 |
| ci_pipeline_id | 68361068 | 68361068 |
| cpu_model | Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz | Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz |
| kernel_version | Linux runner-bziamzy-project-304-concurrent-1-fv58j1gk 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux | Linux runner-bziamzy-project-304-concurrent-1-fv58j1gk 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
Summary
Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.
Execution time for biojava
gantt
title biojava - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section baseline
no_agent (15.59 s) : 15590000, 15590000
. : milestone, 15590000,
appsec (15.101 s) : 15101000, 15101000
. : milestone, 15101000,
iast (18.597 s) : 18597000, 18597000
. : milestone, 18597000,
iast_GLOBAL (18.075 s) : 18075000, 18075000
. : milestone, 18075000,
profiling (15.369 s) : 15369000, 15369000
. : milestone, 15369000,
tracing (15.156 s) : 15156000, 15156000
. : milestone, 15156000,
section candidate
no_agent (14.768 s) : 14768000, 14768000
. : milestone, 14768000,
appsec (14.961 s) : 14961000, 14961000
. : milestone, 14961000,
iast (18.938 s) : 18938000, 18938000
. : milestone, 18938000,
iast_GLOBAL (17.98 s) : 17980000, 17980000
. : milestone, 17980000,
profiling (15.309 s) : 15309000, 15309000
. : milestone, 15309000,
tracing (14.932 s) : 14932000, 14932000
. : milestone, 14932000,
- baseline results
| Variant | Execution Time [CI 0.99] | Δ no_agent |
|---|---|---|
| no_agent | 15.59 s [15.59 s, 15.59 s] | - |
| appsec | 15.101 s [15.101 s, 15.101 s] | -489.0 ms (-3.1%) |
| iast | 18.597 s [18.597 s, 18.597 s] | 3.007 s (19.3%) |
| iast_GLOBAL | 18.075 s [18.075 s, 18.075 s] | 2.485 s (15.9%) |
| profiling | 15.369 s [15.369 s, 15.369 s] | -221.0 ms (-1.4%) |
| tracing | 15.156 s [15.156 s, 15.156 s] | -434.0 ms (-2.8%) |
- candidate results
| Variant | Execution Time [CI 0.99] | Δ no_agent |
|---|---|---|
| no_agent | 14.768 s [14.768 s, 14.768 s] | - |
| appsec | 14.961 s [14.961 s, 14.961 s] | 193.0 ms (1.3%) |
| iast | 18.938 s [18.938 s, 18.938 s] | 4.17 s (28.2%) |
| iast_GLOBAL | 17.98 s [17.98 s, 17.98 s] | 3.212 s (21.7%) |
| profiling | 15.309 s [15.309 s, 15.309 s] | 541.0 ms (3.7%) |
| tracing | 14.932 s [14.932 s, 14.932 s] | 164.0 ms (1.1%) |
Execution time for tomcat
gantt
title tomcat - execution time [CI 0.99] : candidate=1.51.0-SNAPSHOT~f8ec06c202, baseline=1.51.0-SNAPSHOT~60c30c276e
dateFormat X
axisFormat %s
section baseline
no_agent (1.475 ms) : 1463, 1486
. : milestone, 1475,
appsec (2.396 ms) : 2347, 2444
. : milestone, 2396,
iast (2.194 ms) : 2132, 2255
. : milestone, 2194,
iast_GLOBAL (2.222 ms) : 2160, 2283
. : milestone, 2222,
profiling (2.048 ms) : 1998, 2099
. : milestone, 2048,
tracing (2.009 ms) : 1961, 2056
. : milestone, 2009,
section candidate
no_agent (1.475 ms) : 1464, 1487
. : milestone, 1475,
appsec (2.393 ms) : 2345, 2442
. : milestone, 2393,
iast (2.192 ms) : 2130, 2253
. : milestone, 2192,
iast_GLOBAL (2.227 ms) : 2165, 2289
. : milestone, 2227,
profiling (2.035 ms) : 1986, 2085
. : milestone, 2035,
tracing (1.996 ms) : 1949, 2043
. : milestone, 1996,
- baseline results
| Variant | Execution Time [CI 0.99] | Δ no_agent |
|---|---|---|
| no_agent | 1.475 ms [1.463 ms, 1.486 ms] | - |
| appsec | 2.396 ms [2.347 ms, 2.444 ms] | 921.382 µs (62.5%) |
| iast | 2.194 ms [2.132 ms, 2.255 ms] | 719.104 µs (48.8%) |
| iast_GLOBAL | 2.222 ms [2.16 ms, 2.283 ms] | 747.106 µs (50.7%) |
| profiling | 2.048 ms [1.998 ms, 2.099 ms] | 573.892 µs (38.9%) |
| tracing | 2.009 ms [1.961 ms, 2.056 ms] | 534.218 µs (36.2%) |
- candidate results
| Variant | Execution Time [CI 0.99] | Δ no_agent |
|---|---|---|
| no_agent | 1.475 ms [1.464 ms, 1.487 ms] | - |
| appsec | 2.393 ms [2.345 ms, 2.442 ms] | 918.427 µs (62.3%) |
| iast | 2.192 ms [2.13 ms, 2.253 ms] | 716.564 µs (48.6%) |
| iast_GLOBAL | 2.227 ms [2.165 ms, 2.289 ms] | 751.884 µs (51.0%) |
| profiling | 2.035 ms [1.986 ms, 2.085 ms] | 560.378 µs (38.0%) |
| tracing | 1.996 ms [1.949 ms, 2.043 ms] | 521.097 µs (35.3%) |
I also wonder if we can share links to our confluence docs about how to use / leverage our GitLab CI?
@PerfectSlayer that's a great idea. I'll open another pull request to add those.
Just realized, but #9088 has a lot of overlapping changes
Closing PR to preview review reminder. Feel free to re-open it if needed!