Commit c46fb91
authored
fix: wrong coordinates, npm v7+ lockfile & audit compatibility, SPDX evaluation & license additions (AGPL-3.0-only, Python-2.0, Ruby, Sleepycat, UPL-1.0, Vim, W3C, X11, Zlib) (#253)
* Fix: wrong package coordinates sent to NCM API
# Fix: wrong package coordinates sent to NCM API
## Summary
| # | Report | Status | Where |
|---|---|---|---|
| 1 | "sending odd packages versions to the API like this one jwt version 0.0.0" | Fixed | `lib/ncm-analyze-tree.js`, `commands/report.js` |
| 2 | `NCM_TOKEN=… NCM_API=… ncm` "with no success" | Not a bug; documented below | — |
| 3 | "still a diff on npm audit and ncm report" | Fixed (root cause shared with #1) | `lib/ncm-analyze-tree.js` |
| 4 | "we would need to investigate if our vulnDB has all the vulns" | Now meaningfully testable | see below |
Tests: **24 / 24** pass (`npm run test-only`).
---
## Root cause (issues 1 & 3)
`lib/ncm-analyze-tree.js` had been rewritten to use
[`dependency-tree`](https://www.npmjs.com/package/dependency-tree) — a
**source-code import analyzer** — instead of
[`universal-module-tree`](https://www.npmjs.com/package/universal-module-tree),
which walks the real npm dependency graph
(`package-lock.json` → `yarn.lock` → `node_modules`).
Three concrete defects in that code produced the payloads the user saw:
1. **Filename-as-package-name with placeholder version.** `dependency-tree`
returns resolved file paths. The replacement code turned each path
into a "package":
```js
const name = path.basename(dep, path.extname(dep));
const childNode = {
data: { name, version: '0.0.0' }, // hard-coded
children: []
};
```
So `jsonwebtoken.js` → `jwt @ 0.0.0`, `index.js` → `index @ 0.0.0`,
and similar. That is the exact `jwt version 0.0.0` entry the user
saw going out.
2. **Range-string fallback also produced `0.0.0`.** The auxiliary
`getNpmDependencies` helper only read direct deps from `package.json`
and did:
```js
const cleanVersion = version.replace(/[^\d.]/g, '') || version;
```
For `"latest"`, `"file:…"`, `"git+https://…"`, `"workspace:*"`, etc.
the strip leaves `''` and the `||` fallback is still non-semver —
so those deps were emitted as `name @ 0.0.0` too.
3. **Missing transitive deps.** `dependency-tree` only surfaces files
the entry point actually `require()`s, and the `package.json`
fallback only listed direct dependencies. The full transitive
graph was never walked. This is the direct mechanical cause of the
diff vs `npm audit`.
4. **`commands/report.js` had been patched to hide the fallout.** Any
`null` / `undefined` `version` returned by the NCM service was
coerced to `'0.0.0'` and still included in the rendered report,
so bogus rows showed up in the UI too.
## Fix
### `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines)
- Reverted to `universal-module-tree` (already in `dependencies`, and
the same library `commands/details.js` uses).
- Removed, in full:
- the `dependency-tree` invocation and entry-point discovery logic
(guessing `index.js`, `app.js`, `bin/*.js`, …),
- filename-as-package-name logic,
- `getNpmDependencies` and the duplicate `readPackagesFromPackageJson`
that stripped `^` / `~` and fell back to `'0.0.0'`,
- dead counters.
The new file gets a tree from `universalModuleTree(dir)`, walks it
keeping dedup + `paths` info, and POSTs `{ name, version }` pairs to
the GraphQL endpoint — same shape that already worked in
`commands/details.js`.
### `commands/report.js`
- Removed the `effectiveVersion = '0.0.0'` coercion. Packages the NCM
service returns with `version === null` (unpublished / unknown) are
now **skipped** instead of rendered as `@ 0.0.0`:
```js
if (version === null || version === undefined) continue;
```
- Removed unused `includedCount` / `skippedCount` counters and the
dead `getLicenseScore` helper.
### `package.json` / `package-lock.json`
- Removed the unused `dependency-tree` dependency
(`npm uninstall dependency-tree`). `universal-module-tree` was
already listed and is now the single source of truth for the
dependency graph.
---
## Verification
### Empirical: correct coordinates go out
```
$ node -e "const umt=require('universal-module-tree');
(async()=>{
const t=await umt('./test/fixtures/mock-project');
const f=umt.flatten(t);
console.log('total:', f.length);
console.log('zero-versions:', f.filter(p=>p.version==='0.0.0'));
console.log('filename-like:', f.filter(p=>/\\.|\\//.test(p.name)));
})()"
total: 37
zero-versions: []
filename-like: []
```
No `@ 0.0.0`, no `jwt`-style filename names — just real installed
packages from `package-lock.json`.
### Tests: `npm run test-only` → 24 / 24 pass
Notable guards:
- `report › report output matches snapshot` — asserts **"36 packages
checked"**, which only holds when the transitive graph is walked
correctly.
- `report › report with poisoned project` — asserts a mock package
returned with `version: null` is **skipped**, not rendered as
`@ 0.0.0`.
- All `--filter=*`, `--compliance`, `--security`, `--long` snapshot
tests pass, plus `details`, `whitelist`, `install`,
`github-actions`, and `proxied` suites.
---
## Issue 2: bare `ncm` "with no success"
`ncm` with no subcommand shows help and exits 0 — same as `git` or
`npm` without args. `bin/ncm-cli.js`:
```js
let [command = 'help', ...subargs] = argv._
if (!Object.keys(commands).includes(command)) command = 'help'
```
No API request is ever made, which is why the debug session looked
like "nothing happening". Verified post-fix: `node bin/ncm-cli.js`
prints help and exits 0. To actually exercise the analyzer against
the local API:
```sh
NCM_TOKEN=6c044113-c485-40cb-915b-cbc9d3a26730 \
NCM_API=http://localhost:3000 \
ncm report --dir=/path/to/some/project
```
---
## Issue 4: vulnDB coverage investigation
Previously impossible to do meaningfully — the payload was garbage
(issue 1) and incomplete (issue 3). Now that real coordinates for the
full transitive tree are sent, a direct diff is valid:
```sh
cd /path/to/project
npm install # ensure lockfile + node_modules
npm audit --json > /tmp/npm-audit.json # npm's view
NCM_TOKEN=… NCM_API=http://localhost:3000 \
ncm report --long --json > /tmp/ncm-report.json # ncm's view
```
Compare the `{name, version}` sets:
- advisories in `npm-audit` but not `ncm-report` → real gaps in the
vulnDB, feed back to the data team;
- advisories in `ncm-report` but not `npm-audit` → extra NCM coverage
(or a false positive to review).
---
## Files changed
- `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines).
- `commands/report.js` — `0.0.0` coercion and dead code removed.
- `package.json`, `package-lock.json` — `dependency-tree` removed.
* Fix: wrong package coordinates sent to NCM API
# Fix: wrong package coordinates sent to NCM API
## Summary
| # | Report | Status | Where |
|---|---|---|---|
| 1 | "sending odd packages versions to the API like this one jwt version 0.0.0" | Fixed | `lib/ncm-analyze-tree.js`, `commands/report.js` |
| 2 | `NCM_TOKEN=… NCM_API=… ncm` "with no success" | Not a bug; documented below | — |
| 3 | "still a diff on npm audit and ncm report" | Fixed (root cause shared with #1) | `lib/ncm-analyze-tree.js` |
| 4 | "we would need to investigate if our vulnDB has all the vulns" | Now meaningfully testable | see below |
Tests: **24 / 24** pass (`npm run test-only`).
---
## Root cause (issues 1 & 3)
`lib/ncm-analyze-tree.js` had been rewritten to use
[`dependency-tree`](https://www.npmjs.com/package/dependency-tree) — a
**source-code import analyzer** — instead of
[`universal-module-tree`](https://www.npmjs.com/package/universal-module-tree),
which walks the real npm dependency graph
(`package-lock.json` → `yarn.lock` → `node_modules`).
Three concrete defects in that code produced the payloads the user saw:
1. **Filename-as-package-name with placeholder version.** `dependency-tree`
returns resolved file paths. The replacement code turned each path
into a "package":
```js
const name = path.basename(dep, path.extname(dep));
const childNode = {
data: { name, version: '0.0.0' }, // hard-coded
children: []
};
```
So `jsonwebtoken.js` → `jwt @ 0.0.0`, `index.js` → `index @ 0.0.0`,
and similar. That is the exact `jwt version 0.0.0` entry the user
saw going out.
2. **Range-string fallback also produced `0.0.0`.** The auxiliary
`getNpmDependencies` helper only read direct deps from `package.json`
and did:
```js
const cleanVersion = version.replace(/[^\d.]/g, '') || version;
```
For `"latest"`, `"file:…"`, `"git+https://…"`, `"workspace:*"`, etc.
the strip leaves `''` and the `||` fallback is still non-semver —
so those deps were emitted as `name @ 0.0.0` too.
3. **Missing transitive deps.** `dependency-tree` only surfaces files
the entry point actually `require()`s, and the `package.json`
fallback only listed direct dependencies. The full transitive
graph was never walked. This is the direct mechanical cause of the
diff vs `npm audit`.
4. **`commands/report.js` had been patched to hide the fallout.** Any
`null` / `undefined` `version` returned by the NCM service was
coerced to `'0.0.0'` and still included in the rendered report,
so bogus rows showed up in the UI too.
## Fix
### `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines)
- Reverted to `universal-module-tree` (already in `dependencies`, and
the same library `commands/details.js` uses).
- Removed, in full:
- the `dependency-tree` invocation and entry-point discovery logic
(guessing `index.js`, `app.js`, `bin/*.js`, …),
- filename-as-package-name logic,
- `getNpmDependencies` and the duplicate `readPackagesFromPackageJson`
that stripped `^` / `~` and fell back to `'0.0.0'`,
- dead counters.
The new file gets a tree from `universalModuleTree(dir)`, walks it
keeping dedup + `paths` info, and POSTs `{ name, version }` pairs to
the GraphQL endpoint — same shape that already worked in
`commands/details.js`.
### `commands/report.js`
- Removed the `effectiveVersion = '0.0.0'` coercion. Packages the NCM
service returns with `version === null` (unpublished / unknown) are
now **skipped** instead of rendered as `@ 0.0.0`:
```js
if (version === null || version === undefined) continue;
```
- Removed unused `includedCount` / `skippedCount` counters and the
dead `getLicenseScore` helper.
### `package.json` / `package-lock.json`
- Removed the unused `dependency-tree` dependency
(`npm uninstall dependency-tree`). `universal-module-tree` was
already listed and is now the single source of truth for the
dependency graph.
---
## Verification
### Empirical: correct coordinates go out
```
$ node -e "const umt=require('universal-module-tree');
(async()=>{
const t=await umt('./test/fixtures/mock-project');
const f=umt.flatten(t);
console.log('total:', f.length);
console.log('zero-versions:', f.filter(p=>p.version==='0.0.0'));
console.log('filename-like:', f.filter(p=>/\\.|\\//.test(p.name)));
})()"
total: 37
zero-versions: []
filename-like: []
```
No `@ 0.0.0`, no `jwt`-style filename names — just real installed
packages from `package-lock.json`.
### Tests: `npm run test-only` → 24 / 24 pass
Notable guards:
- `report › report output matches snapshot` — asserts **"36 packages
checked"**, which only holds when the transitive graph is walked
correctly.
- `report › report with poisoned project` — asserts a mock package
returned with `version: null` is **skipped**, not rendered as
`@ 0.0.0`.
- All `--filter=*`, `--compliance`, `--security`, `--long` snapshot
tests pass, plus `details`, `whitelist`, `install`,
`github-actions`, and `proxied` suites.
---
## Issue 2: bare `ncm` "with no success"
`ncm` with no subcommand shows help and exits 0 — same as `git` or
`npm` without args. `bin/ncm-cli.js`:
```js
let [command = 'help', ...subargs] = argv._
if (!Object.keys(commands).includes(command)) command = 'help'
```
No API request is ever made, which is why the debug session looked
like "nothing happening". Verified post-fix: `node bin/ncm-cli.js`
prints help and exits 0. To actually exercise the analyzer against
the local API:
```sh
NCM_TOKEN=6c044113-c485-40cb-915b-cbc9d3a26730 \
NCM_API=http://localhost:3000 \
ncm report --dir=/path/to/some/project
```
---
## Issue 4: vulnDB coverage investigation
Previously impossible to do meaningfully — the payload was garbage
(issue 1) and incomplete (issue 3). Now that real coordinates for the
full transitive tree are sent, a direct diff is valid:
```sh
cd /path/to/project
npm install # ensure lockfile + node_modules
npm audit --json > /tmp/npm-audit.json # npm's view
NCM_TOKEN=… NCM_API=http://localhost:3000 \
ncm report --long --json > /tmp/ncm-report.json # ncm's view
```
Compare the `{name, version}` sets:
- advisories in `npm-audit` but not `ncm-report` → real gaps in the
vulnDB, feed back to the data team;
- advisories in `ncm-report` but not `npm-audit` → extra NCM coverage
(or a false positive to review).
---
## Files changed
- `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines).
- `commands/report.js` — `0.0.0` coercion and dead code removed.
- `package.json`, `package-lock.json` — `dependency-tree` removed.
* Fix: wrong package coordinates sent to NCM API
# Fix: wrong package coordinates sent to NCM API
## Summary
| # | Report | Status | Where |
|---|---|---|---|
| 1 | "sending odd packages versions to the API like this one jwt version 0.0.0" | Fixed | `lib/ncm-analyze-tree.js`, `commands/report.js` |
| 2 | `NCM_TOKEN=… NCM_API=… ncm` "with no success" | Not a bug; documented below | — |
| 3 | "still a diff on npm audit and ncm report" | Fixed (root cause shared with #1) | `lib/ncm-analyze-tree.js` |
| 4 | "we would need to investigate if our vulnDB has all the vulns" | Now meaningfully testable | see below |
Tests: **24 / 24** pass (`npm run test-only`).
---
## Root cause (issues 1 & 3)
`lib/ncm-analyze-tree.js` had been rewritten to use
[`dependency-tree`](https://www.npmjs.com/package/dependency-tree) — a
**source-code import analyzer** — instead of
[`universal-module-tree`](https://www.npmjs.com/package/universal-module-tree),
which walks the real npm dependency graph
(`package-lock.json` → `yarn.lock` → `node_modules`).
Three concrete defects in that code produced the payloads the user saw:
1. **Filename-as-package-name with placeholder version.** `dependency-tree`
returns resolved file paths. The replacement code turned each path
into a "package":
```js
const name = path.basename(dep, path.extname(dep));
const childNode = {
data: { name, version: '0.0.0' }, // hard-coded
children: []
};
```
So `jsonwebtoken.js` → `jwt @ 0.0.0`, `index.js` → `index @ 0.0.0`,
and similar. That is the exact `jwt version 0.0.0` entry the user
saw going out.
2. **Range-string fallback also produced `0.0.0`.** The auxiliary
`getNpmDependencies` helper only read direct deps from `package.json`
and did:
```js
const cleanVersion = version.replace(/[^\d.]/g, '') || version;
```
For `"latest"`, `"file:…"`, `"git+https://…"`, `"workspace:*"`, etc.
the strip leaves `''` and the `||` fallback is still non-semver —
so those deps were emitted as `name @ 0.0.0` too.
3. **Missing transitive deps.** `dependency-tree` only surfaces files
the entry point actually `require()`s, and the `package.json`
fallback only listed direct dependencies. The full transitive
graph was never walked. This is the direct mechanical cause of the
diff vs `npm audit`.
4. **`commands/report.js` had been patched to hide the fallout.** Any
`null` / `undefined` `version` returned by the NCM service was
coerced to `'0.0.0'` and still included in the rendered report,
so bogus rows showed up in the UI too.
## Fix
### `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines)
- Reverted to `universal-module-tree` (already in `dependencies`, and
the same library `commands/details.js` uses).
- Removed, in full:
- the `dependency-tree` invocation and entry-point discovery logic
(guessing `index.js`, `app.js`, `bin/*.js`, …),
- filename-as-package-name logic,
- `getNpmDependencies` and the duplicate `readPackagesFromPackageJson`
that stripped `^` / `~` and fell back to `'0.0.0'`,
- dead counters.
The new file gets a tree from `universalModuleTree(dir)`, walks it
keeping dedup + `paths` info, and POSTs `{ name, version }` pairs to
the GraphQL endpoint — same shape that already worked in
`commands/details.js`.
### `commands/report.js`
- Removed the `effectiveVersion = '0.0.0'` coercion. Packages the NCM
service returns with `version === null` (unpublished / unknown) are
now **skipped** instead of rendered as `@ 0.0.0`:
```js
if (version === null || version === undefined) continue;
```
- Removed unused `includedCount` / `skippedCount` counters and the
dead `getLicenseScore` helper.
### `package.json` / `package-lock.json`
- Removed the unused `dependency-tree` dependency
(`npm uninstall dependency-tree`). `universal-module-tree` was
already listed and is now the single source of truth for the
dependency graph.
---
## Verification
### Empirical: correct coordinates go out
```
$ node -e "const umt=require('universal-module-tree');
(async()=>{
const t=await umt('./test/fixtures/mock-project');
const f=umt.flatten(t);
console.log('total:', f.length);
console.log('zero-versions:', f.filter(p=>p.version==='0.0.0'));
console.log('filename-like:', f.filter(p=>/\\.|\\//.test(p.name)));
})()"
total: 37
zero-versions: []
filename-like: []
```
No `@ 0.0.0`, no `jwt`-style filename names — just real installed
packages from `package-lock.json`.
### Tests: `npm run test-only` → 24 / 24 pass
Notable guards:
- `report › report output matches snapshot` — asserts **"36 packages
checked"**, which only holds when the transitive graph is walked
correctly.
- `report › report with poisoned project` — asserts a mock package
returned with `version: null` is **skipped**, not rendered as
`@ 0.0.0`.
- All `--filter=*`, `--compliance`, `--security`, `--long` snapshot
tests pass, plus `details`, `whitelist`, `install`,
`github-actions`, and `proxied` suites.
---
## Issue 2: bare `ncm` "with no success"
`ncm` with no subcommand shows help and exits 0 — same as `git` or
`npm` without args. `bin/ncm-cli.js`:
```js
let [command = 'help', ...subargs] = argv._
if (!Object.keys(commands).includes(command)) command = 'help'
```
No API request is ever made, which is why the debug session looked
like "nothing happening". Verified post-fix: `node bin/ncm-cli.js`
prints help and exits 0. To actually exercise the analyzer against
the local API:
```sh
NCM_TOKEN=6c044113-c485-40cb-915b-cbc9d3a26730 \
NCM_API=http://localhost:3000 \
ncm report --dir=/path/to/some/project
```
---
## Issue 4: vulnDB coverage investigation
Previously impossible to do meaningfully — the payload was garbage
(issue 1) and incomplete (issue 3). Now that real coordinates for the
full transitive tree are sent, a direct diff is valid:
```sh
cd /path/to/project
npm install # ensure lockfile + node_modules
npm audit --json > /tmp/npm-audit.json # npm's view
NCM_TOKEN=… NCM_API=http://localhost:3000 \
ncm report --long --json > /tmp/ncm-report.json # ncm's view
```
Compare the `{name, version}` sets:
- advisories in `npm-audit` but not `ncm-report` → real gaps in the
vulnDB, feed back to the data team;
- advisories in `ncm-report` but not `npm-audit` → extra NCM coverage
(or a false positive to review).
---
## Files changed
- `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines).
- `commands/report.js` — `0.0.0` coercion and dead code removed.
- `package.json`, `package-lock.json` — `dependency-tree` removed.
* Fix: wrong package coordinates sent to NCM API
# Fix: wrong package coordinates sent to NCM API
## Summary
| # | Report | Status | Where |
|---|---|---|---|
| 1 | "sending odd packages versions to the API like this one jwt version 0.0.0" | Fixed | `lib/ncm-analyze-tree.js`, `commands/report.js` |
| 2 | `NCM_TOKEN=… NCM_API=… ncm` "with no success" | Not a bug; documented below | — |
| 3 | "still a diff on npm audit and ncm report" | Fixed (root cause shared with #1) | `lib/ncm-analyze-tree.js` |
| 4 | "we would need to investigate if our vulnDB has all the vulns" | Now meaningfully testable | see below |
Tests: **24 / 24** pass (`npm run test-only`).
---
## Root cause (issues 1 & 3)
`lib/ncm-analyze-tree.js` had been rewritten to use
[`dependency-tree`](https://www.npmjs.com/package/dependency-tree) — a
**source-code import analyzer** — instead of
[`universal-module-tree`](https://www.npmjs.com/package/universal-module-tree),
which walks the real npm dependency graph
(`package-lock.json` → `yarn.lock` → `node_modules`).
Three concrete defects in that code produced the payloads the user saw:
1. **Filename-as-package-name with placeholder version.** `dependency-tree`
returns resolved file paths. The replacement code turned each path
into a "package":
```js
const name = path.basename(dep, path.extname(dep));
const childNode = {
data: { name, version: '0.0.0' }, // hard-coded
children: []
};
```
So `jsonwebtoken.js` → `jwt @ 0.0.0`, `index.js` → `index @ 0.0.0`,
and similar. That is the exact `jwt version 0.0.0` entry the user
saw going out.
2. **Range-string fallback also produced `0.0.0`.** The auxiliary
`getNpmDependencies` helper only read direct deps from `package.json`
and did:
```js
const cleanVersion = version.replace(/[^\d.]/g, '') || version;
```
For `"latest"`, `"file:…"`, `"git+https://…"`, `"workspace:*"`, etc.
the strip leaves `''` and the `||` fallback is still non-semver —
so those deps were emitted as `name @ 0.0.0` too.
3. **Missing transitive deps.** `dependency-tree` only surfaces files
the entry point actually `require()`s, and the `package.json`
fallback only listed direct dependencies. The full transitive
graph was never walked. This is the direct mechanical cause of the
diff vs `npm audit`.
4. **`commands/report.js` had been patched to hide the fallout.** Any
`null` / `undefined` `version` returned by the NCM service was
coerced to `'0.0.0'` and still included in the rendered report,
so bogus rows showed up in the UI too.
## Fix
### `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines)
- Reverted to `universal-module-tree` (already in `dependencies`, and
the same library `commands/details.js` uses).
- Removed, in full:
- the `dependency-tree` invocation and entry-point discovery logic
(guessing `index.js`, `app.js`, `bin/*.js`, …),
- filename-as-package-name logic,
- `getNpmDependencies` and the duplicate `readPackagesFromPackageJson`
that stripped `^` / `~` and fell back to `'0.0.0'`,
- dead counters.
The new file gets a tree from `universalModuleTree(dir)`, walks it
keeping dedup + `paths` info, and POSTs `{ name, version }` pairs to
the GraphQL endpoint — same shape that already worked in
`commands/details.js`.
### `commands/report.js`
- Removed the `effectiveVersion = '0.0.0'` coercion. Packages the NCM
service returns with `version === null` (unpublished / unknown) are
now **skipped** instead of rendered as `@ 0.0.0`:
```js
if (version === null || version === undefined) continue;
```
- Removed unused `includedCount` / `skippedCount` counters and the
dead `getLicenseScore` helper.
### `package.json` / `package-lock.json`
- Removed the unused `dependency-tree` dependency
(`npm uninstall dependency-tree`). `universal-module-tree` was
already listed and is now the single source of truth for the
dependency graph.
---
## Verification
### Empirical: correct coordinates go out
```
$ node -e "const umt=require('universal-module-tree');
(async()=>{
const t=await umt('./test/fixtures/mock-project');
const f=umt.flatten(t);
console.log('total:', f.length);
console.log('zero-versions:', f.filter(p=>p.version==='0.0.0'));
console.log('filename-like:', f.filter(p=>/\\.|\\//.test(p.name)));
})()"
total: 37
zero-versions: []
filename-like: []
```
No `@ 0.0.0`, no `jwt`-style filename names — just real installed
packages from `package-lock.json`.
### Tests: `npm run test-only` → 24 / 24 pass
Notable guards:
- `report › report output matches snapshot` — asserts **"36 packages
checked"**, which only holds when the transitive graph is walked
correctly.
- `report › report with poisoned project` — asserts a mock package
returned with `version: null` is **skipped**, not rendered as
`@ 0.0.0`.
- All `--filter=*`, `--compliance`, `--security`, `--long` snapshot
tests pass, plus `details`, `whitelist`, `install`,
`github-actions`, and `proxied` suites.
---
## Issue 2: bare `ncm` "with no success"
`ncm` with no subcommand shows help and exits 0 — same as `git` or
`npm` without args. `bin/ncm-cli.js`:
```js
let [command = 'help', ...subargs] = argv._
if (!Object.keys(commands).includes(command)) command = 'help'
```
No API request is ever made, which is why the debug session looked
like "nothing happening". Verified post-fix: `node bin/ncm-cli.js`
prints help and exits 0. To actually exercise the analyzer against
the local API:
```sh
NCM_TOKEN=6c044113-c485-40cb-915b-cbc9d3a26730 \
NCM_API=http://localhost:3000 \
ncm report --dir=/path/to/some/project
```
---
## Issue 4: vulnDB coverage investigation
Previously impossible to do meaningfully — the payload was garbage
(issue 1) and incomplete (issue 3). Now that real coordinates for the
full transitive tree are sent, a direct diff is valid:
```sh
cd /path/to/project
npm install # ensure lockfile + node_modules
npm audit --json > /tmp/npm-audit.json # npm's view
NCM_TOKEN=… NCM_API=http://localhost:3000 \
ncm report --long --json > /tmp/ncm-report.json # ncm's view
```
Compare the `{name, version}` sets:
- advisories in `npm-audit` but not `ncm-report` → real gaps in the
vulnDB, feed back to the data team;
- advisories in `ncm-report` but not `npm-audit` → extra NCM coverage
(or a false positive to review).
---
## Files changed
- `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines).
- `commands/report.js` — `0.0.0` coercion and dead code removed.
- `package.json`, `package-lock.json` — `dependency-tree` removed.
* Fix: wrong package coordinates sent to NCM API
# Fix: wrong package coordinates sent to NCM API
## Summary
| # | Report | Status | Where |
|---|---|---|---|
| 1 | "sending odd packages versions to the API like this one jwt version 0.0.0" | Fixed | `lib/ncm-analyze-tree.js`, `commands/report.js` |
| 2 | `NCM_TOKEN=… NCM_API=… ncm` "with no success" | Not a bug; documented below | — |
| 3 | "still a diff on npm audit and ncm report" | Fixed (root cause shared with #1) | `lib/ncm-analyze-tree.js` |
| 4 | "we would need to investigate if our vulnDB has all the vulns" | Now meaningfully testable | see below |
Tests: **24 / 24** pass (`npm run test-only`).
---
## Root cause (issues 1 & 3)
`lib/ncm-analyze-tree.js` had been rewritten to use
[`dependency-tree`](https://www.npmjs.com/package/dependency-tree) — a
**source-code import analyzer** — instead of
[`universal-module-tree`](https://www.npmjs.com/package/universal-module-tree),
which walks the real npm dependency graph
(`package-lock.json` → `yarn.lock` → `node_modules`).
Three concrete defects in that code produced the payloads the user saw:
1. **Filename-as-package-name with placeholder version.** `dependency-tree`
returns resolved file paths. The replacement code turned each path
into a "package":
```js
const name = path.basename(dep, path.extname(dep));
const childNode = {
data: { name, version: '0.0.0' }, // hard-coded
children: []
};
```
So `jsonwebtoken.js` → `jwt @ 0.0.0`, `index.js` → `index @ 0.0.0`,
and similar. That is the exact `jwt version 0.0.0` entry the user
saw going out.
2. **Range-string fallback also produced `0.0.0`.** The auxiliary
`getNpmDependencies` helper only read direct deps from `package.json`
and did:
```js
const cleanVersion = version.replace(/[^\d.]/g, '') || version;
```
For `"latest"`, `"file:…"`, `"git+https://…"`, `"workspace:*"`, etc.
the strip leaves `''` and the `||` fallback is still non-semver —
so those deps were emitted as `name @ 0.0.0` too.
3. **Missing transitive deps.** `dependency-tree` only surfaces files
the entry point actually `require()`s, and the `package.json`
fallback only listed direct dependencies. The full transitive
graph was never walked. This is the direct mechanical cause of the
diff vs `npm audit`.
4. **`commands/report.js` had been patched to hide the fallout.** Any
`null` / `undefined` `version` returned by the NCM service was
coerced to `'0.0.0'` and still included in the rendered report,
so bogus rows showed up in the UI too.
## Fix
### `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines)
- Reverted to `universal-module-tree` (already in `dependencies`, and
the same library `commands/details.js` uses).
- Removed, in full:
- the `dependency-tree` invocation and entry-point discovery logic
(guessing `index.js`, `app.js`, `bin/*.js`, …),
- filename-as-package-name logic,
- `getNpmDependencies` and the duplicate `readPackagesFromPackageJson`
that stripped `^` / `~` and fell back to `'0.0.0'`,
- dead counters.
The new file gets a tree from `universalModuleTree(dir)`, walks it
keeping dedup + `paths` info, and POSTs `{ name, version }` pairs to
the GraphQL endpoint — same shape that already worked in
`commands/details.js`.
### `commands/report.js`
- Removed the `effectiveVersion = '0.0.0'` coercion. Packages the NCM
service returns with `version === null` (unpublished / unknown) are
now **skipped** instead of rendered as `@ 0.0.0`:
```js
if (version === null || version === undefined) continue;
```
- Removed unused `includedCount` / `skippedCount` counters and the
dead `getLicenseScore` helper.
### `package.json` / `package-lock.json`
- Removed the unused `dependency-tree` dependency
(`npm uninstall dependency-tree`). `universal-module-tree` was
already listed and is now the single source of truth for the
dependency graph.
---
## Verification
### Empirical: correct coordinates go out
```
$ node -e "const umt=require('universal-module-tree');
(async()=>{
const t=await umt('./test/fixtures/mock-project');
const f=umt.flatten(t);
console.log('total:', f.length);
console.log('zero-versions:', f.filter(p=>p.version==='0.0.0'));
console.log('filename-like:', f.filter(p=>/\\.|\\//.test(p.name)));
})()"
total: 37
zero-versions: []
filename-like: []
```
No `@ 0.0.0`, no `jwt`-style filename names — just real installed
packages from `package-lock.json`.
### Tests: `npm run test-only` → 24 / 24 pass
Notable guards:
- `report › report output matches snapshot` — asserts **"36 packages
checked"**, which only holds when the transitive graph is walked
correctly.
- `report › report with poisoned project` — asserts a mock package
returned with `version: null` is **skipped**, not rendered as
`@ 0.0.0`.
- All `--filter=*`, `--compliance`, `--security`, `--long` snapshot
tests pass, plus `details`, `whitelist`, `install`,
`github-actions`, and `proxied` suites.
---
## Issue 2: bare `ncm` "with no success"
`ncm` with no subcommand shows help and exits 0 — same as `git` or
`npm` without args. `bin/ncm-cli.js`:
```js
let [command = 'help', ...subargs] = argv._
if (!Object.keys(commands).includes(command)) command = 'help'
```
No API request is ever made, which is why the debug session looked
like "nothing happening". Verified post-fix: `node bin/ncm-cli.js`
prints help and exits 0. To actually exercise the analyzer against
the local API:
```sh
NCM_TOKEN=6c044113-c485-40cb-915b-cbc9d3a26730 \
NCM_API=http://localhost:3000 \
ncm report --dir=/path/to/some/project
```
---
## Issue 4: vulnDB coverage investigation
Previously impossible to do meaningfully — the payload was garbage
(issue 1) and incomplete (issue 3). Now that real coordinates for the
full transitive tree are sent, a direct diff is valid:
```sh
cd /path/to/project
npm install # ensure lockfile + node_modules
npm audit --json > /tmp/npm-audit.json # npm's view
NCM_TOKEN=… NCM_API=http://localhost:3000 \
ncm report --long --json > /tmp/ncm-report.json # ncm's view
```
Compare the `{name, version}` sets:
- advisories in `npm-audit` but not `ncm-report` → real gaps in the
vulnDB, feed back to the data team;
- advisories in `ncm-report` but not `npm-audit` → extra NCM coverage
(or a false positive to review).
---
## Files changed
- `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines).
- `commands/report.js` — `0.0.0` coercion and dead code removed.
- `package.json`, `package-lock.json` — `dependency-tree` removed.
* Fix: wrong package coordinates sent to NCM API
# Fix: wrong package coordinates sent to NCM API
## Summary
| # | Report | Status | Where |
|---|---|---|---|
| 1 | "sending odd packages versions to the API like this one jwt version 0.0.0" | Fixed | `lib/ncm-analyze-tree.js`, `commands/report.js` |
| 2 | `NCM_TOKEN=… NCM_API=… ncm` "with no success" | Not a bug; documented below | — |
| 3 | "still a diff on npm audit and ncm report" | Fixed (root cause shared with #1) | `lib/ncm-analyze-tree.js` |
| 4 | "we would need to investigate if our vulnDB has all the vulns" | Now meaningfully testable | see below |
Tests: **24 / 24** pass (`npm run test-only`).
---
## Root cause (issues 1 & 3)
`lib/ncm-analyze-tree.js` had been rewritten to use
[`dependency-tree`](https://www.npmjs.com/package/dependency-tree) — a
**source-code import analyzer** — instead of
[`universal-module-tree`](https://www.npmjs.com/package/universal-module-tree),
which walks the real npm dependency graph
(`package-lock.json` → `yarn.lock` → `node_modules`).
Three concrete defects in that code produced the payloads the user saw:
1. **Filename-as-package-name with placeholder version.** `dependency-tree`
returns resolved file paths. The replacement code turned each path
into a "package":
```js
const name = path.basename(dep, path.extname(dep));
const childNode = {
data: { name, version: '0.0.0' }, // hard-coded
children: []
};
```
So `jsonwebtoken.js` → `jwt @ 0.0.0`, `index.js` → `index @ 0.0.0`,
and similar. That is the exact `jwt version 0.0.0` entry the user
saw going out.
2. **Range-string fallback also produced `0.0.0`.** The auxiliary
`getNpmDependencies` helper only read direct deps from `package.json`
and did:
```js
const cleanVersion = version.replace(/[^\d.]/g, '') || version;
```
For `"latest"`, `"file:…"`, `"git+https://…"`, `"workspace:*"`, etc.
the strip leaves `''` and the `||` fallback is still non-semver —
so those deps were emitted as `name @ 0.0.0` too.
3. **Missing transitive deps.** `dependency-tree` only surfaces files
the entry point actually `require()`s, and the `package.json`
fallback only listed direct dependencies. The full transitive
graph was never walked. This is the direct mechanical cause of the
diff vs `npm audit`.
4. **`commands/report.js` had been patched to hide the fallout.** Any
`null` / `undefined` `version` returned by the NCM service was
coerced to `'0.0.0'` and still included in the rendered report,
so bogus rows showed up in the UI too.
## Fix
### `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines)
- Reverted to `universal-module-tree` (already in `dependencies`, and
the same library `commands/details.js` uses).
- Removed, in full:
- the `dependency-tree` invocation and entry-point discovery logic
(guessing `index.js`, `app.js`, `bin/*.js`, …),
- filename-as-package-name logic,
- `getNpmDependencies` and the duplicate `readPackagesFromPackageJson`
that stripped `^` / `~` and fell back to `'0.0.0'`,
- dead counters.
The new file gets a tree from `universalModuleTree(dir)`, walks it
keeping dedup + `paths` info, and POSTs `{ name, version }` pairs to
the GraphQL endpoint — same shape that already worked in
`commands/details.js`.
### `commands/report.js`
- Removed the `effectiveVersion = '0.0.0'` coercion. Packages the NCM
service returns with `version === null` (unpublished / unknown) are
now **skipped** instead of rendered as `@ 0.0.0`:
```js
if (version === null || version === undefined) continue;
```
- Removed unused `includedCount` / `skippedCount` counters and the
dead `getLicenseScore` helper.
### `package.json` / `package-lock.json`
- Removed the unused `dependency-tree` dependency
(`npm uninstall dependency-tree`). `universal-module-tree` was
already listed and is now the single source of truth for the
dependency graph.
---
## Verification
### Empirical: correct coordinates go out
```
$ node -e "const umt=require('universal-module-tree');
(async()=>{
const t=await umt('./test/fixtures/mock-project');
const f=umt.flatten(t);
console.log('total:', f.length);
console.log('zero-versions:', f.filter(p=>p.version==='0.0.0'));
console.log('filename-like:', f.filter(p=>/\\.|\\//.test(p.name)));
})()"
total: 37
zero-versions: []
filename-like: []
```
No `@ 0.0.0`, no `jwt`-style filename names — just real installed
packages from `package-lock.json`.
### Tests: `npm run test-only` → 24 / 24 pass
Notable guards:
- `report › report output matches snapshot` — asserts **"36 packages
checked"**, which only holds when the transitive graph is walked
correctly.
- `report › report with poisoned project` — asserts a mock package
returned with `version: null` is **skipped**, not rendered as
`@ 0.0.0`.
- All `--filter=*`, `--compliance`, `--security`, `--long` snapshot
tests pass, plus `details`, `whitelist`, `install`,
`github-actions`, and `proxied` suites.
---
## Issue 2: bare `ncm` "with no success"
`ncm` with no subcommand shows help and exits 0 — same as `git` or
`npm` without args. `bin/ncm-cli.js`:
```js
let [command = 'help', ...subargs] = argv._
if (!Object.keys(commands).includes(command)) command = 'help'
```
No API request is ever made, which is why the debug session looked
like "nothing happening". Verified post-fix: `node bin/ncm-cli.js`
prints help and exits 0. To actually exercise the analyzer against
the local API:
```sh
NCM_TOKEN=6c044113-c485-40cb-915b-cbc9d3a26730 \
NCM_API=http://localhost:3000 \
ncm report --dir=/path/to/some/project
```
---
## Issue 4: vulnDB coverage investigation
Previously impossible to do meaningfully — the payload was garbage
(issue 1) and incomplete (issue 3). Now that real coordinates for the
full transitive tree are sent, a direct diff is valid:
```sh
cd /path/to/project
npm install # ensure lockfile + node_modules
npm audit --json > /tmp/npm-audit.json # npm's view
NCM_TOKEN=… NCM_API=http://localhost:3000 \
ncm report --long --json > /tmp/ncm-report.json # ncm's view
```
Compare the `{name, version}` sets:
- advisories in `npm-audit` but not `ncm-report` → real gaps in the
vulnDB, feed back to the data team;
- advisories in `ncm-report` but not `npm-audit` → extra NCM coverage
(or a false positive to review).
---
## Files changed
- `lib/ncm-analyze-tree.js` — rewritten (550 → 131 lines).
- `commands/report.js` — `0.0.0` coercion and dead code removed.
- `package.json`, `package-lock.json` — `dependency-tree` removed.1 parent 1542f41 commit c46fb91
13 files changed
Lines changed: 2010 additions & 575 deletions
File tree
- commands
- lib
- report
- tap-snapshots/test
- test
- fixtures/mock-project
- lib
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
14 | 14 | | |
15 | 15 | | |
16 | 16 | | |
| 17 | + | |
17 | 18 | | |
18 | 19 | | |
19 | 20 | | |
| |||
139 | 140 | | |
140 | 141 | | |
141 | 142 | | |
142 | | - | |
143 | | - | |
144 | | - | |
145 | 143 | | |
146 | 144 | | |
147 | 145 | | |
| |||
170 | 168 | | |
171 | 169 | | |
172 | 170 | | |
173 | | - | |
174 | | - | |
175 | | - | |
176 | | - | |
177 | | - | |
178 | | - | |
179 | | - | |
180 | | - | |
181 | | - | |
182 | | - | |
183 | | - | |
184 | | - | |
185 | | - | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
186 | 204 | | |
187 | | - | |
188 | | - | |
189 | | - | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
190 | 208 | | |
191 | 209 | | |
192 | 210 | | |
193 | 211 | | |
194 | | - | |
195 | 212 | | |
196 | 213 | | |
197 | | - | |
| 214 | + | |
198 | 215 | | |
199 | 216 | | |
200 | 217 | | |
201 | 218 | | |
202 | 219 | | |
203 | 220 | | |
204 | | - | |
205 | | - | |
206 | 221 | | |
207 | | - | |
208 | | - | |
209 | 222 | | |
210 | 223 | | |
211 | 224 | | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
212 | 230 | | |
213 | 231 | | |
214 | 232 | | |
| |||
251 | 269 | | |
252 | 270 | | |
253 | 271 | | |
| 272 | + | |
254 | 273 | | |
255 | 274 | | |
256 | 275 | | |
| |||
266 | 285 | | |
267 | 286 | | |
268 | 287 | | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
269 | 311 | | |
270 | 312 | | |
271 | 313 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
191 | 192 | | |
192 | 193 | | |
193 | 194 | | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
194 | 217 | | |
195 | 218 | | |
196 | 219 | | |
| |||
0 commit comments