ci(verify-tests): flag specs no CI invocation reaches#8543
Merged
BridgeAR merged 5 commits intoMay 21, 2026
Conversation
The existing "spec is matched by some script glob" check passes whenever a `package.json` test script could in principle run a spec. It misses the inverse failure: a glob exists but no workflow ever sets the env (typically `PLUGINS=<name>`) that would expand the glob to reach the spec. That is how PR #8516's `packages/datadog-instrumentations/test/fastify.spec.js` slipped through with no CI invocation running it, and why codecov reported 8 new lines as patch misses despite the spec covering them locally. Two prerequisites ride along so the joint check produces truthful results: 1. `normalizeScriptGlob` collapsed every `@(${PLUGINS})` to `*` even in `preserveEnv` mode, so a single-plugin job (e.g. `PLUGINS=bluebird`) looked like it exercised every spec in the same directory. Unwrap the extglob to `${PLUGINS}` before the env-aware expansion runs. 2. `instrumentation-http` wraps `yarn test:instrumentations:ci` in `nick-fields/retry@*`'s `command:` input. Treat that input like an inline `run:` so the verifier sees the underlying invocation. On `master` the new check surfaces 14 pre-existing orphans (instrumentation specs with no matching `PLUGINS=` job, plus two plugin integration-test specs and `datadog-plugin-jest/test/util.spec.js`); closing each is out of scope and tracked in follow-up PRs.
Contributor
Overall package sizeSelf size: 5.84 MB Dependency sizes| name | version | self size | total size | |------|---------|-----------|------------| | import-in-the-middle | 3.0.1 | 82.56 kB | 817.39 kB | | opentracing | 0.14.7 | 194.81 kB | 194.81 kB | | dc-polyfill | 0.1.11 | 25.74 kB | 25.74 kB |🤖 This report was automatically generated by heaviest-objects-in-the-universe |
|
BenchmarksBenchmark execution time: 2026-05-20 11:35:18 Comparing candidate commit 988e26e in PR branch Found 0 performance improvements and 0 performance regressions! Performance is the same for 1499 metrics, 94 unstable metrics. |
`packages/datadog-instrumentations/test/{ai,aws-sdk,couchbase,crypto,
express-multi-version,fetch,hono,http-client-options,otel-sdk-trace,
stripe,zlib}.spec.js` are matched by `test:instrumentations`' glob but
no workflow set `PLUGINS=<name>` for `test:instrumentations:ci`, so the
new joint-coverage check in `scripts/verify-exercised-tests.js` flagged
each as never reached by a CI invocation. Add the matching
`instrumentation-<name>` jobs alongside the existing siblings; nine of
the eleven are plain Node unit specs (proxyquire / agent harness) and
need no service containers, so the existing `instrumentations/test`
composite action runs them as-is.
Two specs need a custom job shape on top of that:
1. `couchbase` ships its own libcouchbase whose 3.x line only builds on
Node 18 and whose 4.2+ line only builds on Node 20+; the composite
action runs both oldest (18) and latest (24), so neither range
survives the install on its own. Use a matrix that pins one Node
version per range, mirroring the existing `couchbase` plugin job.
2. `http-client-options` listened on `127.0.0.1` only, but
`http.request(undefined, callback)` resolves to `localhost`, which
Node 18+ may resolve to `::1` first -- the request then fails with
`ECONNREFUSED ::1:<port>` before the spec can assert. Bind the test
server to all interfaces.
`packages/datadog-plugin-{axios,body-parser}/test/integration-test/client.spec.js`
are matched by `test:integration:plugins`' glob but no workflow set
`PLUGINS=<name>` for `test:integration:plugins:coverage`; the existing
`axios:` and `body-parser:` jobs only invoked `plugins/upstream`, which
runs the non-glob `test:plugins:upstream` suite-runner. The new joint
check flagged both specs as never reached.
Append `plugins/integration-test` to each job (which runs
`test:integration:plugins:coverage` with the same `PLUGINS=` env) rather
than spinning up sibling jobs; both specs are self-contained
(`FakeAgent` + `useSandbox`) and slot into the existing harness without
extra service setup.
The axios sandbox previously called `axios.get('/foo')`, which throws
synchronously without producing an http.request, so the spec timed out
waiting for a span. Switch to a full URL pointing at an unused port to
match the existing fetch sandbox; the request can fail and the
instrumentation still ships a span.
`packages/datadog-plugin-jest/test/util.spec.js` is matched by `test:plugins`' glob but no workflow set `PLUGINS=jest` for `test:plugins:ci`; the existing `integration-jest` matrix runs `integration-tests/jest/*.spec.js` only, which does not cover the plugin package's own unit tests. The new joint check flagged it as never reached. Add a single `jest:` job invoking the `plugins/test` composite action. `util.spec.js` is a pure unit test on `getFormattedJestTestParameters` and `getJestSuitesToRun`, so no service containers or matrix axes are needed beyond what `plugins/test` already covers (oldest-maintenance and latest Node). Restore the three fixture files (`test-to-run.js`, `test-to-skip.js`, `test-unskippable.js`) under `packages/datadog-plugin-jest/test/fixtures/` that `getJestSuitesToRun` reads via `isMarkedAsUnskippable`. They were deleted alongside the other jest plugin specs when those moved to integration tests, but `util.spec.js` was kept and silently broke because no CI invocation reached it.
BridgeAR
force-pushed
the
BridgeAR/2026-05-19-verify-exercised-tests-joint
branch
from
de95cd7 to
3f9b76e
Compare
ci(instrumentation): run 11 orphan instrumentation specs
`packages/datadog-instrumentations/test/{ai,aws-sdk,couchbase,crypto,
express-multi-version,fetch,hono,http-client-options,otel-sdk-trace,
stripe,zlib}.spec.js` are matched by `test:instrumentations`' glob but
no workflow set `PLUGINS=<name>` for `test:instrumentations:ci`, so the
new joint-coverage check in `scripts/verify-exercised-tests.js` flagged
each as never reached by a CI invocation. Add the matching
`instrumentation-<name>` jobs alongside the existing siblings; nine of
the eleven are plain Node unit specs (proxyquire / agent harness) and
need no service containers, so the existing `instrumentations/test`
composite action runs them as-is.
Two specs need a custom job shape on top of that:
1. `couchbase` ships its own libcouchbase: the 3.x line has no prebuilt
binary for Node 18+ and its bundled sources do not compile under
modern gcc (missing <cstdint> for std::uint8_t). The composite
action runs both oldest (18) and latest (24), so neither survives
the install on its own. Use a matrix that pins one Node version
per range, mirroring the existing `couchbase` plugin job's `eol`
(16) for `^3.0.7` and Node 18 for `>=4.2.0`.
2. `http-client-options` listened on `127.0.0.1` only, but
`http.request(undefined, callback)` resolves to `localhost`, which
Node 18+ may resolve to `::1` first -- the request then fails with
`ECONNREFUSED ::1:<port>` before the spec can assert. Bind the test
server to all interfaces.
tlhunter
approved these changes
May 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a joint-condition check to
scripts/verify-exercised-tests.js: every spec file must be matched by somepackage.jsonscript glob and reached by at least one workflow invocation that expands the glob with the right env. The existing check only enforced the first half.Two prerequisites land alongside the joint check so it produces truthful results:
normalizeScriptGlobno longer collapses@(${PLUGINS})to*inpreserveEnvmode, so a single-plugin CI job no longer looks like it exercises every spec in the same directory.collectWorkflowRunsunwrapsnick-fields/retry@*'scommand:input like an inlinerun:, soinstrumentation-http's retry-wrappedtest:instrumentations:ciinvocation is visible.The check would have caught the silent CI gap that motivated PR #8516:
packages/datadog-instrumentations/test/fastify.spec.jsmatchedtest:instrumentations' glob, but no workflow setPLUGINS=fastifyfortest:instrumentations:ci, so codecov reported 8 new lines as patch misses despite the spec covering them locally.CI gaps closed in this PR
Running the verifier with the new check on
mastersurfaced 14 pre-existing orphans. All are addressed by the follow-up commits:packages/datadog-instrumentations/test/ai.spec.jsinstrumentation.ymlinstrumentation-aipackages/datadog-instrumentations/test/aws-sdk.spec.jsinstrumentation.ymlinstrumentation-aws-sdkpackages/datadog-instrumentations/test/couchbase.spec.jsinstrumentation.ymlinstrumentation-couchbasepackages/datadog-instrumentations/test/crypto.spec.jsinstrumentation.ymlinstrumentation-cryptopackages/datadog-instrumentations/test/express-multi-version.spec.jsinstrumentation.ymlinstrumentation-express-multi-versionpackages/datadog-instrumentations/test/fetch.spec.jsinstrumentation.ymlinstrumentation-fetchpackages/datadog-instrumentations/test/hono.spec.jsinstrumentation.ymlinstrumentation-honopackages/datadog-instrumentations/test/http-client-options.spec.jsinstrumentation.ymlinstrumentation-http-client-optionspackages/datadog-instrumentations/test/otel-sdk-trace.spec.jsinstrumentation.ymlinstrumentation-otel-sdk-tracepackages/datadog-instrumentations/test/stripe.spec.jsinstrumentation.ymlinstrumentation-stripepackages/datadog-instrumentations/test/zlib.spec.jsinstrumentation.ymlinstrumentation-zlibpackages/datadog-plugin-axios/test/integration-test/client.spec.jsapm-integrations.ymlaxios:(addedplugins/integration-teststep)packages/datadog-plugin-body-parser/test/integration-test/client.spec.jsapm-integrations.ymlbody-parser:(addedplugins/integration-teststep)packages/datadog-plugin-jest/test/util.spec.jstest-optimization.ymljest:fastify.spec.jsis intentionally absent from the list — PR #8516 closes it.Test plan
node scripts/verify-exercised-tests.json this branch: zero errors.instrumentation-fastifyworkflow entry: verifier flagged it as expected; with the entry, it passed. Confirms the joint check is functional.npm run verify:workflow-job-namesclean.eslint scripts/verify-exercised-tests.jsclean.