ncu-ci: output a summary list with all failing tests #396
Description
So it's easier to know which tests are failing across all platforms. Something like:
------------------------------- Failures Summary -------------------------------
Build failures:
- win-vs2017
- cross-compiler-armv7-gcc-6
- osx1011
Test failures:
- sequential/test-inspector-break-when-eval (4 platforms)
- sequential/test-perf-hooks (2 platforms)
- parallel/test-memory-usage (1 platform)
- parallel/test-repl-history-navigation (1 platform)
- sequential/test-heap-prof (1 platform)
- sequential/test-worker-prof (1 platform)
Example:
$ ncu-ci url ci.nodejs.org/job/node-test-pull-request/29794
--------------------------------------------------------------------------------
[1/1] Running PR: 29597
--------------------------------------------------------------------------------
✔ Build data downloaded
✔ Data downloaded
----------------------------------- Summary ------------------------------------
Result FAILURE
URL https://ci.nodejs.org/job/node-test-pull-request/29597/
Source https://github.com/nodejs/node/pull/32116/
Commit [1ab2b73b513c] test: warn when inspector process crashes
Date 2020-03-06 16:25:26 -0800
Author Matheus Marchini <[email protected]>
------------------------------- Failures Summary -------------------------------
Build failures:
- win-vs2017
- cross-compiler-armv7-gcc-6
- osx1011
Test failures:
- sequential/test-inspector-break-when-eval (4 platforms)
- sequential/test-perf-hooks (2 platforms)
- parallel/test-memory-usage (1 platform)
- parallel/test-repl-history-navigation (1 platform)
- sequential/test-heap-prof (1 platform)
- sequential/test-worker-prof (1 platform)
----------------------------------- osx1011 ------------------------------------
URL https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx1011/32407/console
Type BUILD_FAILURE
Built On test-macstadium-macos10.11-x64-1
Reason fatal: No rebase in progress?
----------------------------------- osx1015 ------------------------------------
URL https://ci.nodejs.org/job/node-test-commit-osx/nodes=osx1015/32407/console
Type JS_TEST_FAILURE
Built On test-nearform-macos10.15-x64-3
Reason
not ok 2777 sequential/test-inspector-break-when-eval
---
duration_ms: 15.390
severity: fail
exitcode: 1
stack: |-
[test] Connecting to a child Node process
[test] Testing /json/list
[err] Debugger listening on ws://127.0.0.1:63158/870bb3cd-ad48-4622-8a87-53091991bef8
[err] For help, see: https://nodejs.org/en/docs/inspector
[err]
[test] Setting up a debugger
[err] Debugger attached.
[err]
[test] Breaking in the code
[err] child process crashed, signal SIGSEGV
Timed out waiting for matching notification (break on file:///Users/iojs/build/workspace/node-test-commit-osx/nodes/osx1015/test/fixtures/inspector-global-function.js:9))
1
...
------------------------------- ppcle-ubuntu1404 -------------------------------
URL https://ci.nodejs.org/job/node-test-commit-plinux/nodes=ppcle-ubuntu1404/31338/console
Type JS_TEST_FAILURE
Built On test-osuosl-ubuntu1404-ppc64_le-2
Reason
not ok 2792 sequential/test-inspector-break-when-eval
---
duration_ms: 15.341
severity: fail
exitcode: 1
stack: |-
[test] Connecting to a child Node process
[test] Testing /json/list
[err] Debugger listening on ws://127.0.0.1:52357/f0bedc84-050f-4b9e-a641-286ac55dcb28
[err] For help, see: https://nodejs.org/en/docs/inspector
[err]
[test] Setting up a debugger
[err] Debugger attached.
[err]
[test] Breaking in the code
[err] child process crashed, signal SIGSEGV
Timed out waiting for matching notification (break on file:///home/iojs/build/workspace/node-test-commit-plinux/nodes/ppcle-ubuntu1404/test/fixtures/inspector-global-function.js:9))
1
...
--------------------------------- smartos18-64 ---------------------------------
URL https://ci.nodejs.org/job/node-test-commit-smartos/nodes=smartos18-64/31307/console
Type JS_TEST_FAILURE
Built On test-joyent-smartos18-x64-3
Reason
not ok 2847 sequential/test-inspector-break-when-eval
---
duration_ms: 15.745
severity: fail
exitcode: 1
stack: |-
[test] Connecting to a child Node process
[test] Testing /json/list
[err] Debugger listening on ws://127.0.0.1:44215/d810e836-4d14-4696-a415-2aa7f92714e5
[err] For help, see: https://nodejs.org/en/docs/inspector
[err]
[test] Setting up a debugger
[err] Debugger attached.
[err]
[test] Breaking in the code
[err] child process crashed, signal SIGSEGV
Timed out waiting for matching notification (break on file:///home/iojs/build/workspace/node-test-commit-smartos/nodes/smartos18-64/test/fixtures/inspector-global-function.js:9))
1
...
--------------------- ubuntu1804_sharedlibs_withoutssl_x64 ---------------------
URL https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_withoutssl_x64/18497/console
Type JS_TEST_FAILURE
Built On test-joyent-ubuntu1804_sharedlibs_container-x64-1
Reason
not ok 1393 parallel/test-memory-usage
---
duration_ms: 1.351
severity: fail
exitcode: 1
stack: |-
assert.js:384
throw err;
^
AssertionError [ERR_ASSERTION]: 11571790 - 1102879 >= 10485760
at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/parallel/test-memory-usage.js:42:3)
at Module._compile (internal/modules/cjs/loader.js:1202:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1222:10)
at Module.load (internal/modules/cjs/loader.js:1051:32)
at Function.Module._load (internal/modules/cjs/loader.js:947:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
at internal/main/run_main_module.js:17:47 {
generatedMessage: false,
code: 'ERR_ASSERTION',
actual: false,
expected: true,
operator: '=='
}
...
--------------------- ubuntu1804_sharedlibs_withoutssl_x64 ---------------------
URL https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_withoutssl_x64/18497/console
Type JS_TEST_FAILURE
Built On test-joyent-ubuntu1804_sharedlibs_container-x64-1
Reason
not ok 2768 sequential/test-perf-hooks
---
duration_ms: 1.83
severity: fail
exitcode: 1
stack: |-
{
name: 'node',
entryType: 'node',
startTime: 0,
duration: { around: 468.111141204834 },
nodeStart: { around: 0 },
v8Start: { around: 0 },
bootstrapComplete: { around: 457.42479133605957, delay: 2500 },
environment: { around: 0 },
loopStart: -1,
loopExit: -1
}
assert.js:384
throw err;
^
AssertionError [ERR_ASSERTION]: v8Start: 275.37767028808594 >= 250
at checkNodeTiming (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-perf-hooks.js:67:7)
at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-perf-hooks.js:79:1)
at Module._compile (internal/modules/cjs/loader.js:1202:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1222:10)
at Module.load (internal/modules/cjs/loader.js:1051:32)
at Function.Module._load (internal/modules/cjs/loader.js:947:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
at internal/main/run_main_module.js:17:47 {
generatedMessage: false,
code: 'ERR_ASSERTION',
actual: false,
expected: true,
operator: '=='
}
...
------------------------ ubuntu1804_sharedlibs_zlib_x64 ------------------------
URL https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/18497/console
Type JS_TEST_FAILURE
Built On test-joyent-ubuntu1804_sharedlibs_container-x64-2
Reason
not ok 1648 parallel/test-repl-history-navigation
---
duration_ms: 12.367
severity: fail
exitcode: 1
stack: |-
assert.js:102
throw new AssertionError(obj);
^
AssertionError [ERR_ASSERTION]: Expected values to be strictly deep-equal:
+ actual - expected ... Lines skipped
[
'\x1B[1G',
...
------------------------ ubuntu1804_sharedlibs_zlib_x64 ------------------------
URL https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/18497/console
Type JS_TEST_FAILURE
Built On test-joyent-ubuntu1804_sharedlibs_container-x64-2
Reason
not ok 2741 sequential/test-heap-prof
---
duration_ms: 120.338
severity: fail
exitcode: -15
stack: |-
timeout
...
------------------------ ubuntu1804_sharedlibs_zlib_x64 ------------------------
URL https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/18497/console
Type JS_TEST_FAILURE
Built On test-joyent-ubuntu1804_sharedlibs_container-x64-2
Reason
not ok 2768 sequential/test-inspector-break-when-eval
---
duration_ms: 6.359
severity: fail
exitcode: 1
stack: |-
[test] Connecting to a child Node process
[test] Testing /json/list
[err] Debugger listening on ws://127.0.0.1:43601/97b597e2-1f2f-470a-b4f6-183159499193
[err] For help, see: https://nodejs.org/en/docs/inspector
[err]
[out] Ready!
[out]
[test] Setting up a debugger
[err] Debugger attached.
[err]
[test] Breaking in the code
[test] Step over console statement and test output
[out] 0 3
[out]
[test] Verify node waits for the frontend to disconnect
[err] Waiting for the debugger to disconnect...
[err]
[err] internal/timers.js:404
[err] node.priorityQueuePosition = pos;
[err] ^
[err]
[err] TypeError: Cannot set property 'priorityQueuePosition' of undefined
[err] at setPosition (internal/timers.js:404:30)
[err] at PriorityQueue.percolateDown (internal/priority_queue.js:70:7)
[err] at listOnTimeout (internal/timers.js:512:24)
[err] at processTimers (internal/timers.js:492:7)
[err]
AssertionError [ERR_ASSERTION]: Expected values to be strictly equal:
1 !== 0
at runTests (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-inspector-break-when-eval.js:65:10)
at processTicksAndRejections (internal/process/task_queues.js:97:5) {
generatedMessage: true,
code: 'ERR_ASSERTION',
actual: 1,
expected: 0,
operator: 'strictEqual'
}
1
...
------------------------ ubuntu1804_sharedlibs_zlib_x64 ------------------------
URL https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/18497/console
Type JS_TEST_FAILURE
Built On test-joyent-ubuntu1804_sharedlibs_container-x64-2
Reason
not ok 2802 sequential/test-perf-hooks
---
duration_ms: 0.788
severity: fail
exitcode: 1
stack: |-
{
name: 'node',
entryType: 'node',
startTime: 0,
duration: { around: 426.24277687072754 },
nodeStart: { around: 0 },
v8Start: { around: 0 },
bootstrapComplete: { around: 421.70141792297363, delay: 2500 },
environment: { around: 0 },
loopStart: -1,
loopExit: -1
}
assert.js:384
throw err;
^
AssertionError [ERR_ASSERTION]: v8Start: 250.21231079101562 >= 250
at checkNodeTiming (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-perf-hooks.js:67:7)
at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-perf-hooks.js:79:1)
at Module._compile (internal/modules/cjs/loader.js:1202:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1222:10)
at Module.load (internal/modules/cjs/loader.js:1051:32)
at Function.Module._load (internal/modules/cjs/loader.js:947:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
at internal/main/run_main_module.js:17:47 {
generatedMessage: false,
code: 'ERR_ASSERTION',
actual: false,
expected: true,
operator: '=='
}
...
------------------------ ubuntu1804_sharedlibs_zlib_x64 ------------------------
URL https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu1804_sharedlibs_zlib_x64/18497/console
Type JS_TEST_FAILURE
Built On test-joyent-ubuntu1804_sharedlibs_container-x64-2
Reason
not ok 2821 sequential/test-worker-prof
---
duration_ms: 31.410
severity: fail
exitcode: 1
stack: |-
assert.js:102
throw new AssertionError(obj);
^
AssertionError [ERR_ASSERTION]: child exited with signal: {
error: Error: spawnSync /home/iojs/build/workspace/node-test-commit-linux-containered/out/Release/node ETIMEDOUT
at Object.spawnSync (internal/child_process.js:1070:20)
at spawnSync (child_process.js:597:24)
at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-worker-prof.js:51:23)
at Module._compile (internal/modules/cjs/loader.js:1202:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1222:10)
at Module.load (internal/modules/cjs/loader.js:1051:32)
at Function.Module._load (internal/modules/cjs/loader.js:947:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
at internal/main/run_main_module.js:17:47 {
errno: -110,
code: 'ETIMEDOUT',
syscall: 'spawnSync /home/iojs/build/workspace/node-test-commit-linux-containered/out/Release/node',
path: '/home/iojs/build/workspace/node-test-commit-linux-containered/out/Release/node',
spawnargs: [
'--prof',
'/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-worker-prof.js',
'child'
]
},
status: null,
signal: 'SIGTERM',
output: [ null, '', '' ],
pid: 4629,
stdout: '',
stderr: ''
}
at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-linux-containered/test/sequential/test-worker-prof.js:57:10)
at Module._compile (internal/modules/cjs/loader.js:1202:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:1222:10)
at Module.load (internal/modules/cjs/loader.js:1051:32)
at Function.Module._load (internal/modules/cjs/loader.js:947:14)
at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)
at internal/main/run_main_module.js:17:47 {
generatedMessage: false,
code: 'ERR_ASSERTION',
actual: 'SIGTERM',
expected: null,
operator: 'strictEqual'
}
...
------------------------------ centos7-arm64-gcc6 ------------------------------
URL https://ci.nodejs.org/job/node-test-commit-arm/nodes=centos7-arm64-gcc6/29838/console
Type JS_TEST_FAILURE
Built On test-packetnet-centos7-arm64-2
Reason
not ok 2866 sequential/test-inspector-break-when-eval
---
duration_ms: 15.633
severity: fail
exitcode: 1
stack: |-
[test] Connecting to a child Node process
[test] Testing /json/list
[err] Debugger listening on ws://127.0.0.1:41871/5e35f89c-2fda-4abb-ac8f-9234ebdca44c
[err] For help, see: https://nodejs.org/en/docs/inspector
[err]
[test] Setting up a debugger
[err] Debugger attached.
[err]
[test] Breaking in the code
[err] child process crashed, signal SIGSEGV
Timed out waiting for matching notification (break on file:///home/iojs/build/workspace/node-test-commit-arm/nodes/centos7-arm64-gcc6/test/fixtures/inspector-global-function.js:9))
1
...
---------------------------------- win-vs2017 ----------------------------------
URL https://ci.nodejs.org/job/node-compile-windows/nodes=win-vs2017/32291/console
Type BUILD_FAILURE
Built On test-rackspace-win2012r2_vs2017-x64-4
Reason
error C2139: 'v8::internal::SmallOrderedHashSet': an undefined class is not allowed as an argument to compiler intrinsic type trait '__is_convertible_to' [C:\workspace\node-compile-windows\node\tools\v8_gypfiles\v8_initializers.vcxproj]
c:\workspace\node-compile-windows\node\out\release\obj\global_intermediate\torque-output-root\torque-generated\class-definitions-tq.h(151): note: see declaration of 'v8::internal::SmallOrderedHashSet'
c:\workspace\node-compile-windows\node\deps\v8\src\codegen\tnode.h(250): note: see reference to class template instantiation 'std::is_convertible<T,v8::internal::Object>' being compiled
with
[
-------------------------- cross-compiler-armv7-gcc-6 --------------------------
URL https://ci.nodejs.org/job/node-cross-compile/nodes=cross-compiler-armv7-gcc-6/28127/console
Type BUILD_FAILURE
Built On test-joyent-ubuntu1604_arm_cross-x64-1
Reason
error: unable to delete 'jenkins-node-test-commit-arm-fanned-1ab2b73b513cc7aa3908c0bd10f1c8238607a4e4-binary-pi1p/cc-armv7': remote ref does not exist
error: failed to push some refs to '[email protected]:binary_tmp.git'
+ true
+ JOBS=2
+ CONFIG_FLAGS=' --dest-cpu=arm'
+ make build-ci -j2
--------------------------------- Other builds ---------------------------------