dd-trace-js icon indicating copy to clipboard operation
dd-trace-js copied to clipboard

test: speed up DI integration tests even more

Open watson opened this issue 8 months ago • 9 comments

What does this PR do?

Don't wait for script loading to stabilize in Dynamic Instrumentation integration tests.

This might make them temporarily set the test breakpoint in the wrong location, but it will quickly get moved to the right location and the test should still trigger as expected.

Debugger CI run performance improvement:

Before After
image image

Motivation

Plugin Checklist

Additional Notes

watson avatar Jun 10 '25 10:06 watson

This stack of pull requests is managed by Graphite. Learn more about stacking.

watson avatar Jun 10 '25 10:06 watson

Overall package size

Self size: 9.63 MB Deduped: 104.57 MB No deduping: 105.09 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------| | @datadog/libdatadog | 0.6.0 | 30.47 MB | 30.47 MB | | @datadog/native-appsec | 8.5.2 | 19.33 MB | 19.34 MB | | @datadog/pprof | 5.8.0 | 12.55 MB | 12.92 MB | | @datadog/native-iast-taint-tracking | 4.0.0 | 11.72 MB | 11.73 MB | | @opentelemetry/core | 1.30.1 | 908.66 kB | 7.16 MB | | protobufjs | 7.5.3 | 2.95 MB | 5.6 MB | | @datadog/wasm-js-rewriter | 4.0.1 | 2.85 MB | 3.58 MB | | @datadog/native-metrics | 3.1.1 | 1.02 MB | 1.43 MB | | @opentelemetry/api | 1.8.0 | 1.21 MB | 1.21 MB | | import-in-the-middle | 1.14.0 | 120.58 kB | 841.68 kB | | source-map | 0.7.4 | 226 kB | 226 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | lru-cache | 7.18.3 | 133.92 kB | 133.92 kB | | pprof-format | 2.1.0 | 111.69 kB | 111.69 kB | | @datadog/sketches-js | 2.1.1 | 109.9 kB | 109.9 kB | | lodash.sortby | 4.7.0 | 75.76 kB | 75.76 kB | | ignore | 5.3.2 | 53.63 kB | 53.63 kB | | istanbul-lib-coverage | 3.2.2 | 34.37 kB | 34.37 kB | | rfdc | 1.4.1 | 27.15 kB | 27.15 kB | | @isaacs/ttlcache | 1.4.1 | 25.2 kB | 25.2 kB | | dc-polyfill | 0.1.9 | 25.11 kB | 25.11 kB | | tlhunter-sorted-set | 0.1.0 | 24.94 kB | 24.94 kB | | shell-quote | 1.8.2 | 23.54 kB | 23.54 kB | | limiter | 1.1.5 | 23.17 kB | 23.17 kB | | retry | 0.13.1 | 18.85 kB | 18.85 kB | | semifies | 1.0.0 | 15.84 kB | 15.84 kB | | jest-docblock | 29.7.0 | 8.99 kB | 12.76 kB | | crypto-randomuuid | 1.0.0 | 11.18 kB | 11.18 kB | | ttl-set | 1.0.0 | 4.61 kB | 9.69 kB | | mutexify | 1.4.0 | 5.71 kB | 8.74 kB | | path-to-regexp | 0.1.12 | 6.6 kB | 6.6 kB | | koalas | 1.0.2 | 6.47 kB | 6.47 kB | | module-details-from-path | 1.0.4 | 3.96 kB | 3.96 kB |

🤖 This report was automatically generated by heaviest-objects-in-the-universe

github-actions[bot] avatar Jun 10 '25 10:06 github-actions[bot]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 80.75%. Comparing base (784197d) to head (a73650f). Report is 78 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #5864   +/-   ##
=======================================
  Coverage   80.75%   80.75%           
=======================================
  Files         464      464           
  Lines       19910    19910           
=======================================
  Hits        16078    16078           
  Misses       3832     3832           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

codecov[bot] avatar Jun 10 '25 10:06 codecov[bot]

Benchmarks

Benchmark execution time: 2025-06-10 19:44:23

Comparing candidate commit a73650f2d9718358d553a19b025ef022ef6505d7 in PR branch watson/speed-up-di-tests2 with baseline commit 784197db4d3c3254ad46e8e10a2d79a8524970dd in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 1267 metrics, 56 unstable metrics.

pr-commenter[bot] avatar Jun 10 '25 10:06 pr-commenter[bot]

Datadog Report

Branch report: watson/speed-up-di-tests2 Commit report: e699f99 Test service: dd-trace-js-integration-tests

:white_check_mark: 0 Failed, 1086 Passed, 0 Skipped, 15m 0.62s Total Time :snowflake: 1 New Flaky

New Flaky Tests (1)

  • [email protected] commonJS flaky test retries retries flaky tests - integration-tests/cypress/cypress.spec.js - Last Failure

    Expand for error
    xpected +0 to equal 1
    

Actually what does this even do exactly? According to the description it seems like the right breakpoint will eventually happen anyway, so why have the delay to begin with?

rochdev avatar Jun 10 '25 19:06 rochdev

Any way to do this without creating such a variable?

I haven't been able to think of anything, as it has to be configurable from outside the sandbox 🤷

Actually what does this even do exactly? According to the description it seems like the right breakpoint will eventually happen anyway, so why have the delay to begin with?

It's to avoid the false positive. Both the data collected would not be what was expected and confuse users, and the incorrectly set breakpoint might be in a hot code-path, which could be a big problem (worst case scenario)

watson avatar Jun 10 '25 19:06 watson

Seems to me like a design flaw more than something that should be configurable. I'm going to agree with @BridgeAR here. This is equivalent to adding a sleep which wouldn't fix the issue but only make it less likely, and I think we should figure out an actual fix for the underlying issue instead.

rochdev avatar Jun 10 '25 21:06 rochdev

Haha, yeah I knew this change was going to be controversial - and to be honest I don't really like it myself either. But I'm really happy for the discussion. I'll mark this as draft for now and think a bit more about it. Also, it's not super important to get this merged. It would just be good DevEx ☺️

watson avatar Jun 11 '25 06:06 watson

I've decided it's not worth it, as it's better to test closer to the real world experience

watson avatar Jun 27 '25 11:06 watson