`ncu-ci walk pr` handling of very broken CI runs

Open richardlau opened this issue 3 years ago • 0 comments
Today's CI reliability GitHub actions run failed running ncu-ci walk pr --stats=true --markdown $PWD/results.md: https://github.com/nodejs/reliability/actions/runs/1904541710
The log is too big to be displayed in the web UI but downloading the log shows that ncu-ci ran out of memory
2022-02-27T00:11:52.5187361Z ---------------------------- alpine-last-latest-x64 ----------------------------
2022-02-27T00:11:52.5187837Z URL        https://ci.nodejs.org/job/node-test-commit-linux/nodes=alpine-last-latest-x64/44808/console
2022-02-27T00:11:52.5187963Z Type       JS_TEST_FAILURE
2022-02-27T00:11:52.5188260Z Built On   test-softlayer-alpine311_container-x64-1
2022-02-27T00:11:52.5188350Z Reason     
2022-02-27T00:11:52.5188614Z   not ok 340 parallel/test-buffer-of-no-deprecation
2022-02-27T00:11:52.5188724Z     ---
2022-02-27T00:11:52.5188825Z     duration_ms: 0.138
2022-02-27T00:11:52.5188920Z     severity: fail
2022-02-27T00:11:52.5189008Z     exitcode: 1
2022-02-27T00:11:52.5189091Z     stack: |-
2022-02-27T00:11:52.5189297Z       You may need to run `make test/common/knownGlobals.json`.
2022-02-27T00:11:52.5189631Z       /home/iojs/build/workspace/node-test-commit-linux/test/common/index.js:260
2022-02-27T00:11:52.5189708Z         throw err;
2022-02-27T00:11:52.5189793Z         ^
2022-02-27T00:11:52.5189872Z       
2022-02-27T00:11:52.5190127Z       Error: Cannot find module './knownGlobals.json'
2022-02-27T00:11:52.5190223Z       Require stack:
2022-02-27T00:11:52.5190544Z       - /home/iojs/build/workspace/node-test-commit-linux/test/common/index.js
2022-02-27T00:11:52.5190958Z       - /home/iojs/build/workspace/node-test-commit-linux/test/parallel/test-buffer-of-no-deprecation.js
2022-02-27T00:11:52.5191204Z           at Function.Module._resolveFilename (node:internal/modules/cjs/loader:933:15)
2022-02-27T00:11:52.5191423Z           at Function.Module._load (node:internal/modules/cjs/loader:778:27)
2022-02-27T00:11:52.5191628Z           at Module.require (node:internal/modules/cjs/loader:999:19)
2022-02-27T00:11:52.5191818Z           at require (node:internal/modules/cjs/helpers:102:18)
2022-02-27T00:11:52.5192301Z           at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux/test/common/index.js:257:18)
2022-02-27T00:11:52.5192496Z           at Module._compile (node:internal/modules/cjs/loader:1097:14)
2022-02-27T00:11:52.5192813Z           at Object.Module._extensions..js (node:internal/modules/cjs/loader:1151:10)
2022-02-27T00:11:52.5193018Z           at Module.load (node:internal/modules/cjs/loader:975:32)
2022-02-27T00:11:52.5193236Z           at Function.Module._load (node:internal/modules/cjs/loader:822:12)
2022-02-27T00:11:52.5193443Z           at Module.require (node:internal/modules/cjs/loader:999:19) {
2022-02-27T00:11:52.5193629Z         code: 'MODULE_NOT_FOUND',
2022-02-27T00:11:52.5193730Z         requireStack: [
2022-02-27T00:11:52.5194191Z FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
2022-02-27T00:11:52.5194380Z  1: 0xa1a640 node::Abort() [node]
2022-02-27T00:11:52.5194583Z  2: 0xa1aa4c node::OnFatalError(char const*, char const*) [node]
2022-02-27T00:11:52.5194837Z  3: 0xb9a9fe v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
2022-02-27T00:11:52.5195123Z  4: 0xb9ad79 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
2022-02-27T00:11:52.5195222Z  5: 0xd57ff5  [node]
2022-02-27T00:11:52.5195470Z  6: 0xd58686 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
2022-02-27T00:11:52.5195789Z  7: 0xd64f45 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
2022-02-27T00:11:52.5196157Z  8: 0xd65df5 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
2022-02-27T00:11:52.5196571Z  9: 0xd688ac v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
2022-02-27T00:11:52.5196899Z 10: 0xd3674c v8::internal::Factory::NewRawOneByteString(int, v8::internal::AllocationType) [node]
2022-02-27T00:11:52.5197346Z 11: 0xe4598b v8::internal::JsonParser<unsigned short>::MakeString(v8::internal::JsonString const&, v8::internal::Handle<v8::internal::String>) [node]
2022-02-27T00:11:52.5197632Z 12: 0xe4820e v8::internal::JsonParser<unsigned short>::ParseJsonValue() [node]
2022-02-27T00:11:52.5197899Z 13: 0xe48a1f v8::internal::JsonParser<unsigned short>::ParseJson() [node]
2022-02-27T00:11:52.5198215Z 14: 0xc4bf75 v8::internal::Builtin_JsonParse(int, unsigned long*, v8::internal::Isolate*) [node]
2022-02-27T00:11:52.5198326Z 15: 0x140e0f9  [node]
2022-02-27T00:11:52.5199009Z /home/runner/work/_temp/7f84222e-d06a-45db-9f36-c117354c1d5d.sh: line 1:  1546 Aborted                 (core dumped) ncu-ci walk pr --stats=true --markdown $PWD/results.md
2022-02-27T00:11:52.5199067Z 
2022-02-27T00:11:52.5199241Z <--- Last few GCs --->
2022-02-27T00:11:52.5199254Z 
2022-02-27T00:11:52.5199931Z [1546:0x3fe4130]    74251 ms: Mark-sweep 2049.3 (2051.9) -> 2049.3 (2053.9) MB, 1487.9 / 0.0 ms  (average mu = 0.068, current mu = 0.003) allocation failure scavenge might not succeed
2022-02-27T00:11:52.5200622Z [1546:0x3fe4130]    75708 ms: Mark-sweep 2051.1 (2053.9) -> 2051.1 (2055.1) MB, 1451.5 / 0.0 ms  (average mu = 0.037, current mu = 0.004) allocation failure scavenge might not succeed
2022-02-27T00:11:52.5200634Z 
2022-02-27T00:11:52.5200642Z 
2022-02-27T00:11:52.5200808Z <--- JS stacktrace --->
2022-02-27T00:11:52.5200817Z 
2022-02-27T00:11:52.5201019Z ==== JS stack trace =========================================
2022-02-27T00:11:52.5201028Z 
2022-02-27T00:11:52.5201213Z Security context: 0x0e3a480008d1 <JSObject>
2022-02-27T00:11:52.5201574Z     0: builtin exit frame: parse(this=0x0e3a4801ee79 <Object map = 0x2623d3703639>,0x30aed0880119 <Very long string[239420894]>,0x0e3a4801ee79 <Object map = 0x2623d3703639>)
2022-02-27T00:11:52.5201583Z 
2022-02-27T00:11:52.5202280Z     1: formatAsJson [0x2f4a7a6e6f61] [/opt/hostedtoolcache/node/12.22.10/x64/lib/node_modules/node-core-utils/lib/ci/build-types/test_build.js:185] [bytecode=0x14b3dd5cf491 offset=78](this=0x34038bbc2201 <Job map = 0x25f543...
2022-02-27T00:11:52.5202356Z 
2022-02-27T00:11:52.5213551Z ##[error]Process completed with exit code 134.
2022-02-27T00:11:52.5377172Z Cleaning up orphan processes
I'm speculating here that this is because the URL's being parsed are for CI jobs with a large number (3600) of test failures.
Feb 27 '22 16:02 richardlau