From 0a303f65a28106350b3392163c82987fd0abf511 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=C4=B0nan=C3=A7=20G=C3=BCm=C3=BC=C5=9F?= Date: Sat, 27 Jun 2026 12:31:09 -0400 Subject: [PATCH] Add OpenSpec change for extension usage telemetry --- .../add-extension-telemetry/.openspec.yaml | 2 + .../changes/add-extension-telemetry/design.md | 115 +++++++++++ .../add-extension-telemetry/proposal.md | 45 ++++ .../specs/extension-telemetry/spec.md | 194 ++++++++++++++++++ .../changes/add-extension-telemetry/tasks.md | 148 +++++++++++++ 5 files changed, 504 insertions(+) create mode 100644 openspec/changes/add-extension-telemetry/.openspec.yaml create mode 100644 openspec/changes/add-extension-telemetry/design.md create mode 100644 openspec/changes/add-extension-telemetry/proposal.md create mode 100644 openspec/changes/add-extension-telemetry/specs/extension-telemetry/spec.md create mode 100644 openspec/changes/add-extension-telemetry/tasks.md diff --git a/openspec/changes/add-extension-telemetry/.openspec.yaml b/openspec/changes/add-extension-telemetry/.openspec.yaml new file mode 100644 index 0000000000..f9be753a1e --- /dev/null +++ b/openspec/changes/add-extension-telemetry/.openspec.yaml @@ -0,0 +1,2 @@ +schema: spec-driven +created: 2026-06-27 diff --git a/openspec/changes/add-extension-telemetry/design.md b/openspec/changes/add-extension-telemetry/design.md new file mode 100644 index 0000000000..c84ca9af2f --- /dev/null +++ b/openspec/changes/add-extension-telemetry/design.md @@ -0,0 +1,115 @@ +## Context + +The usage report (`internal/cmd/report.go`) builds a `map[string]any` from a shared `*usage.Usage` collector plus fixed fields and POSTs it to `https://stats.grafana.org/k6-usage-report` after a `k6 run`, gated by `K6_NO_USAGE_REPORT`. Extensions are currently excluded at the collection point: `js/modules/resolution.go:81-87` guards module recording with `!strings.HasPrefix(name, "k6/x/")`. Extension version and kind are already available at runtime via `ext.GetAll()` (`ext/ext.go`), populated from build info. + +This change reports public-catalog extension usage for the importable, output, and subcommand surfaces, filtered against the public registry catalog by reusing the existing cached catalog fetch, as a single PR. The output path is built now; because the catalog lists no output extensions yet, it reports nothing in production until they are catalogued (tracked separately). + +## Goals / Non-Goals + +Goals: +- Report public-catalog extension usage for importable, output, and subcommand extensions. +- Never leak private or unlisted extensions. +- No new endpoint, no embedded catalog snapshot, no new opt-out; reuse the existing catalog fetch and cache. + +Non-Goals: +- Unique-installation identity (the denominator that makes counts interpretable); see the [telemetry issue](https://github.com/grafana/k6/issues/4077). +- Distinguishing xk6-built vs provisioning-built binaries (tracked in [grafana/k6#6111](https://github.com/grafana/k6/issues/6111)). +- Reporting unlisted or private extensions, even anonymously. +- Secret-source extensions (a non-user-facing `ExtensionType`). Of the four extension types, only the three user-facing ones (importable, output, subcommand) are in scope; this is why `kind` has exactly three values (`js`, `output`, `subcommand`). +- Populating the catalog with output extensions. The output code path ships now, but the public registry catalog lists none, so no output is reported in production until they are added; that work is tracked separately. Built-in output reporting stays unchanged, classified by output-registry membership so literal-registered built-ins like `web-dashboard` never reach the `extensions` bucket. + +## Decisions + +### Decision: Reuse the existing live catalog fetch, not an embedded snapshot +Filter against the public catalog by reusing the fetch that `k6 x` already performs (`fetchCatalog`, today in `internal/cmd/x_registry.go`): the same registry URL, the same on-disk cache and TTL, and the same `K6_PROVISION_CATALOG_URL` override. That fetch already sits in package `cmd` (`x_registry.go`, called from `subcommand.go`), so the run path only needs to call it too; there is no cross-package move. The groundwork that lands first is self-contained: add `catalogModulePaths` and, optionally, rename the file to a neutral name so it reads as shared rather than `k6 x`-specific, with no behaviour change. The catalog top-level keys are import specifiers (`k6/x/sql`) and subcommand tokens (`subcommand:docs`), not module paths; each entry carries a `module` field with its Go module path. The existing `catalogEntry` decoder ignores `module`, so `catalogModulePaths` acquires the catalog through the same cache-then-fetch path `k6 x` uses (`readCachedCatalog`, else `fetchCatalog` against the `K6_PROVISION_CATALOG_URL`-or-default URL, then `writeCachedCatalog`), then re-parses the raw bytes into a new struct that reads each entry's `module` field and collects those values, and `filterExtensions` filters against that set. One helper serves both the run report and the subcommand report, so the acquisition is not duplicated. The live cached fetch needs no build-time generator, `go:embed`, or new make target, and makes a newly catalogued extension reportable as soon as the cache refreshes rather than only at the next k6 release. It costs a per-run network call, mitigated by the existing cache, and bounded and non-fatal because it rides the same bounded report send. All three surfaces filter through this one set: an output extension is dropped today because the catalog lists none, and starts reporting automatically once the catalog lists it. + +### Decision: Record the resolved extension, filter at assembly +Record each used extension into its own `extensions` entry on the `usage.Usage` collector at the point of use: the resolution site for imports (`js/modules/resolution.go`) and the output-construction site for outputs (`internal/cmd/outputs.go`), leaving the existing `modules` and `outputs` lists untouched. What is recorded is the resolved `*ext.Extension` itself (the registry entry already in hand at the point of use), not its name. `usage.Usage` keeps a generic `Values(key, any)` method beside `Strings`/`Uint64` that appends an arbitrary value to a `[]any`; the recording sites call it, so `usage.Usage` stays type-agnostic and learns nothing about extensions. At report assembly, `resolveExtensions` reads that `[]any` bucket back out of the map, and `filterExtensions` keeps only the extensions whose module path is in the catalog, de-duplicates per (module, kind) so a module recorded more than once on a surface appears once, converts each survivor to a `{module, version, kind}` entry, and replaces the raw bucket with that list. Because de-duplication is per (module, kind), the same module used as both an import and an output yields two entries, one of each kind. + +Recording the resolved extension rather than its name means assembly never re-derives anything from the registry: the module path, version, and kind are read straight off the recorded `*ext.Extension`. Because extension entries never enter `modules`/`outputs`, those lists stay untouched. Privacy stays in one place (the catalog filter) and off the hot path. Fail closed: the raw bucket is dropped before the catalog is consulted, so an unfetchable catalog reports no extensions, never the unfiltered list. + +### Decision: One shared reporting mechanism, every surface binds to it +Both surfaces feed the single `usage.Usage` collector that already hangs off `GlobalState` (`cmd/state/state.go`), and one transport sends it. No surface builds its own sender. + +The hard invariant: a single invocation hits the report URL at most once. + +- Importable usage already flows through the `k6 run` report; it only needs recording. `js/modules/resolution.go` keeps the `modules` list exactly as today and records the resolved registry entry (`ext.Get(ext.JSExtension)[name]`, identified by registry membership rather than by the `k6/x/` name) into the dedicated `extensions` bucket instead of discarding it. `k6 run` sends its own report, exactly as today; this change does not touch that, so the run path needs no once-only handling. +- Output usage also flows through the `k6 run` report. `internal/cmd/outputs.go` records a selected `--out` into the `extensions` bucket when the output is an extension (membership in `ext.Get(ext.OutputExtension)`), and leaves the built-in `outputs` list and the literal-registered `web-dashboard` on their current path. No new send: it rides the run report like imports. +- Subcommands report from the one place that knows both which command ran and whether it succeeded: the root `execute` seam in `internal/cmd/root.go`. It runs the command with cobra's `ExecuteC`, which returns the executed leaf command and its error, then calls `reportSubcommandUsage` with that command. `reportSubcommandUsage` reports only when the executed command's name is a registered subcommand extension (`ext.Get(ext.SubcommandExtension)`), so `k6 run`, the `k6 x` help, and the provisioning stubs are skipped. The report fires whether the subcommand exited zero or non-zero, since telemetry counts the invocation regardless of outcome. A baked-in subcommand reports in-process. A provisioned subcommand reports from the child that actually runs it: the host never builds the command for an extension it lacks, so its `ExecuteC` never returns that command and it never reports, while the provisioned child reports exactly once. The launcher is not involved and needs no child suppression. The subcommand's own args and internals are out of scope; the name, version, and kind are enough. The report fires uniformly whether the command succeeded or failed and leaves the command's own run hook untouched. +- Opt-out: the `--no-usage-report` flag lives on the per-test `Config`, not `GlobalState`, so the subcommand send, which runs outside the run path, gates on the same option parsed from the environment with `readEnvConfig(gs.Env)` (honoring `K6_NO_USAGE_REPORT`), which is reachable everywhere and passes through to children. + +Minimal factoring: `internal/cmd/report.go` holds one shared transport, `reportUsage(ctx, gs, create)`, which builds the report via the passed `create` closure, sends it bounded, and debug-logs the outcome; both surfaces call it. `createReport` builds the run report (execution stats plus the extensions field) and `createSubcommandReport` builds the small subcommand report (just the identity fields plus the extensions field); the identity fields (`k6_version`, `goos`, `goarch`, `is_ci`) are stamped by a shared `addEnvironmentInfo` so they live in one place. `k6 run` runs `reportUsage` in the background as today; the `k6 x` path calls it directly. One transport, two builders, no abstraction layer. + +### Decision: Reuse relay and storage unchanged +The usage-stats backend stores the whole report JSON verbatim with no field allowlist (`is_ci` and `features` already ride the blob with no typed column). So `extensions` is stored and queryable (`JSON_EXTRACT_ARRAY(report, '$.extensions')`) with zero backend change. A typed column and a dbt staging column are optional later optimizations for cleaner dashboards. + +### Decision: One new env for the report endpoint, none for the catalog +The report endpoint is a hardcoded `const` sent via `http.DefaultClient`, so a test cannot capture the report today. Make it overridable with one new env var, `K6_USAGE_REPORT_URL` (defined in `cmd/state/state.go` beside `K6_PROVISION_CATALOG_URL`), defaulting to the production endpoint. This is the only new env var. The catalog needs no new seam: tests steer membership through the existing `K6_PROVISION_CATALOG_URL`, pointed at a stand-in catalog. No `//go:linkname` and no package-level test hook. + +### Decision: Keep the existing names, thread the new inputs through them +`createReport` and `reportUsage` keep their names, so the run path reads as it did. Their signatures do change, because the feature threads new inputs through them: `createReport` gains the catalog set it filters against, and `reportUsage` becomes the shared transport, taking a `create` closure so both surfaces build their own report and share one bounded send. The rest of the behaviour lands in new helpers, all in `internal/cmd/report.go` except the two catalog and root pieces: `addEnvironmentInfo` (the identity fields), `createSubcommandReport` (the subcommand's report), `resolveExtensions` and `filterExtensions` (read the recorded bucket, drop non-catalogued, de-duplicate, convert to entries), `postUsageReport` (marshal and POST), `catalogModulePaths` (the shared catalog set, in `x_registry.go`), and `reportSubcommandUsage` (the root `execute` seam that reports the invoked subcommand). The diff against master stays additive apart from the two threaded signatures and the recording sites. + +## Data shape + +```json +"extensions": [ + {"module": "github.com/grafana/xk6-sql", "version": "v0.4.0", "kind": "js"}, + {"module": "github.com/grafana/xk6-docs", "version": "v0.1.0", "kind": "subcommand"} +] +``` + +Field names align with the cloud `k6_dependencies` BigQuery column so cloud-executed and local-executed sources can be merged in analytics. This is a forward goal, not a join that exists today: that column covers cloud-executed runs only and is keyed by import name, while this field is keyed by module path. + +## Risks / Trade-offs + +- **Community sentiment on CLI telemetry**. Mitigated by public-only scope and the existing opt-out. No new data category beyond what already ships for built-in modules, and no new endpoint. +- **Per-run catalog fetch**: reusing the live fetch adds a network call, but it rides the existing catalog cache and TTL, runs inside the bounded report send, and is non-fatal: a fetch failure just reports no extensions. +- **Usefulness without unique-user counting**: real but orthogonal; this change still improves on the current zero-visibility baseline for local execution. + +## Testing + +Integration tests only, no unit tests, and table-driven: one test per command surface (`k6 run`, `k6 x`) iterating over a slice of cases, not a separate test function per behaviour. Every behaviour is verified by driving a real `k6 run` / `k6 x` through the command test harness (`internal/cmd/tests`, `NewGlobalTestState`) and asserting on the report actually sent. Follow the existing conventions in that package; do not change production code or comments beyond the one new env var below. + +The report can't be observed in a test today: the endpoint is a hardcoded `const` sent via `http.DefaultClient`, and the harness disables reporting by default (`K6_NO_USAGE_REPORT=true`). + +- Overridable report endpoint via the new `K6_USAGE_REPORT_URL` (`cmd/state/state.go`). A test enables reporting and points the endpoint at an `httptest.Server`, then asserts on the received JSON. +- Catalog membership via the existing `K6_PROVISION_CATALOG_URL` override (tests already point it at `http://127.0.0.1:1/unreachable`), aimed at a stand-in catalog whose entries carry `module` fields matching each test extension's resolved module path. Test extensions register through the normal `modules.Register` / `ext.Register` paths, so nothing is faked. + +Two tables, as input/output vectors. Input is the stand-in catalog data, the script and flags, and the command; output is the asserted usage report. `M_import`, `M_output`, `M_sub`, and `M_fork` denote the module paths the test extensions resolve to. + +`k6 run` surface (`TestRunReportsExtensions`): + +| Catalog | Script / flags | Command | Expected report | +|---|---|---|---| +| any | trivial script | `k6 run` | the existing report posts to the stand-in endpoint (harness baseline, not a spec scenario; proves the `K6_USAGE_REPORT_URL` seam) | +| lists `M_import` | imports `k6/x/testimport` | `k6 run` | `extensions` is one `{M_import, version, js}` | +| lists `M_import` | imports `k6/x/testimport` | `k6 run --vus 5 --iterations 5` | `extensions` is exactly one `{M_import, _, js}` | +| lists `M_import` and `M_import2` (distinct Go packages) | imports both `k6/x/testimport` and `k6/x/testimport2` | `k6 run` | `extensions` is one `{M_import, _, js}` and one `{M_import2, _, js}` | +| lists `M_import` | imports nothing | `k6 run` | no `extensions` key (not `[]`) | +| omits `M_import` | imports `k6/x/testimport` | `k6 run` | no entry for it | +| lists real `M_import`; fork resolves to `M_fork` | imports the fork's `k6/x/testimport` | `k6 run` | no entry (matched by module path) | +| lists `M_output` | trivial script | `k6 run --out testoutput` | `extensions` is one `{M_output, _, output}` | +| omits `M_output` | trivial script | `k6 run --out testoutput` | no entry for it | +| any | trivial script | `k6 run --out json` | no `output`-kind entry; `outputs` still lists `json` | +| unreachable, no cache | imports `k6/x/testimport` | `k6 run` | no `extensions`; exit 0 | +| lists `M_import` | imports `k6/x/testimport` | `k6 run` with `K6_NO_USAGE_REPORT=true` | stand-in server receives no request | + +`k6 x` surface (`TestSubcommandReportsUsage`): + +| Catalog | Command | Expected report | +|---|---|---| +| lists `M_sub` | `k6 x testsub` | exactly one request, `{M_sub, _, subcommand}` | +| omits `M_sub` | `k6 x testsub` | no entry for it | +| unreachable | `k6 x not-a-catalog-name` | no request (the name cannot be provisioned) | +| lists `M_sub`, report URL unreachable or slow | `k6 x testsub` | exit 0, no error, a debug-level log, bounded to the run-report timeout | +| lists `M_sub` | `k6 x testsub` with `K6_NO_USAGE_REPORT=true` | no request | + +Four cases are not integration-testable from a single binary and hold by construction, so they need no end-to-end row: +- Build-method independence: the reporting code reads the running binary, so xk6 and provisioning report identically. +- The exact `version` value: an in-tree test extension is absent from build deps, so `ext.Extension.Version` resolves to empty; tests assert `module` and `kind` and treat `version` as passed through verbatim. +- Provisioned-child once-only: provisioning runs a separately built binary the harness cannot produce; the baked-in once-only is tested end to end, and the provisioned child holds because the host never builds the command for an extension it lacks. +- Dual-purpose (import plus output) coexistence: an import resolves to its package path and an output to its constructor's function name, converging on one module path only for a real external dependency, so the in-tree harness cannot make a single module match both kinds. The (module, kind) de-duplication handles it by construction. + +## Migration + +Additive. No config migration. The only behavioural change is that `k6/x/*` usage, previously discarded, is now collected and (when catalogued) reported, covered by the existing opt-out. diff --git a/openspec/changes/add-extension-telemetry/proposal.md b/openspec/changes/add-extension-telemetry/proposal.md new file mode 100644 index 0000000000..647c319d72 --- /dev/null +++ b/openspec/changes/add-extension-telemetry/proposal.md @@ -0,0 +1,45 @@ +## Why + +k6 collects an anonymous usage report but **deliberately excludes extensions**: `js/modules/resolution.go` skips every `k6/x/*` import. The original reason was privacy: reporting extension names would leak private extensions, which the community disliked. + +Now that binary provisioning is GA and there is a public catalog of official and community extensions, we have no visibility into which extensions are actually used. We cannot tell which community extensions deserve promotion to official, where to spend maintenance effort, or how binary provisioning is being adopted. Cloud-executed runs are partly tracked via a BigQuery `k6_dependencies` column, but **local execution, most OSS usage, is invisible**. + +This change reports extension usage through the existing usage report, scoped to the **public catalog only** so private extensions are never reported. + +## What Changes + +- Add an `extensions` field to the usage report listing the extensions actually used in a run, filtered to the public catalog. +- Cover the importable JS (`k6/x/*`), output (`--out`), and subcommand (`k6 x `) surfaces. The output path ships now but reports nothing in production until the public catalog lists output extensions (tracked separately). +- Filter against the **public registry catalog**, reusing the catalog fetch that `k6 x` already performs (same cache, TTL, and override); no embedded snapshot and no build-time generator, and private/unlisted extensions are dropped. +- Report identically whether the binary was built by `xk6` or produced by binary provisioning (the running binary reports on itself). +- Reuse the existing relay, storage, and opt-out (`--no-usage-report` / `K6_NO_USAGE_REPORT`). + +## Capabilities + +### New Capabilities +- `extension-telemetry`: report public-catalog extension usage (importable, output, subcommand) through the existing usage report, filtered against the public registry catalog (reusing the existing cached fetch), respecting the existing opt-out. + +### Modified Capabilities + + +## Impact + +- **grafana/k6**, the only code change: + - `js/modules/resolution.go`: keep the `modules` list unchanged; record the resolved registry entry for Go extension imports (identified via the extension registry, not by the `k6/x/` name) into a dedicated `extensions` usage bucket instead of discarding them. + - `internal/usage/usage.go`: add a generic `Values(key, any)` method beside `Strings`/`Uint64`, so callers can record a resolved extension entry without `usage.Usage` knowing about extensions. + - `internal/cmd/outputs.go`: record a selected `--out` into the `extensions` bucket when it is an output extension (membership in `ext.Get(ext.OutputExtension)`). Built-in output recording stays exactly as today, so built-ins and the literal-registered `web-dashboard` never reach the bucket. + - `internal/cmd/root.go` and `internal/cmd/subcommand.go`: report subcommand usage once, from the root `execute` seam. `root.go` runs the command with cobra's `ExecuteC` (which returns the executed command and its error) and calls `reportSubcommandUsage`, which reports whenever the executed command is a registered subcommand extension, whether it succeeded or failed, through the shared mechanism, not a bespoke one. In-process when baked in, from the provisioned child otherwise. The launcher is not touched: the host never builds the command for an extension it lacks, so only the running command reports. + - `internal/cmd/report.go`: keep `createReport` and `reportUsage` (the shared transport now takes a `create` closure); assemble the `extensions` field by reading the recorded `*ext.Extension` entries back out of the bucket and filtering them by module path against the catalog fetched via the existing `internal/cmd/x_registry.go` fetch. No re-resolution: module path, version, and kind come straight off the recorded entry. + - `cmd/state/state.go`: add one env var, `K6_USAGE_REPORT_URL`, so the report endpoint is overridable for integration tests; the catalog reuses the existing `K6_PROVISION_CATALOG_URL`. No embedded file and no new generator. + - `release notes/`: add the entry documenting the new `extensions` report field. +- **Backend / analytics**: no change required to store or query. The usage-stats service stores the whole report JSON as a blob with no field allowlist (`is_ci` and `features` already ride it), so `extensions` is queryable via JSON extraction. A typed column or a dbt column are optional later optimizations; a dashboard panel is the only step needed to visualize. +- **grafana/k6-docs**: the [usage collection page](https://grafana.com/docs/k6/latest/set-up/usage-collection/) states "Only k6 built-in JavaScript modules and outputs are considered. Private modules and custom extensions are excluded." This change makes that sentence false: update the page to document the `extensions` field, its public-catalog-only scope, and that the existing opt-out covers it. +- **Scope**: single PR. The importable, output, and subcommand surfaces are in scope. The output path ships now; it reports nothing until output extensions are added to the public catalog, which is tracked separately. +- **Out of scope**: counting unique users or installations ([telemetry issue](https://github.com/grafana/k6/issues/4077)); distinguishing xk6-built from provisioning-built binaries ([grafana/k6#6111](https://github.com/grafana/k6/issues/6111)); reporting unlisted/private extensions; secret-source extensions; a delisted-but-buried `k6/x/*` extension still used on a hand-built binary, which the catalog filter drops (a shrinking cohort, since the modern path is already counted as a built-in once the extension is migrated into core). + +## References + +- [grafana/k6#6109](https://github.com/grafana/k6/issues/6109), the feature issue (assignee: inancgumus). +- [grafana/k6-cloud#3575](https://github.com/grafana/k6-cloud/issues/3575), original tracking issue. +- [grafana/k6#2952](https://github.com/grafana/k6/issues/2952), request to add extension versions to the usage report. +- Existing report mechanism PR [grafana/k6#3917](https://github.com/grafana/k6/pull/3917); current exclusion at `js/modules/resolution.go:81-87`. diff --git a/openspec/changes/add-extension-telemetry/specs/extension-telemetry/spec.md b/openspec/changes/add-extension-telemetry/specs/extension-telemetry/spec.md new file mode 100644 index 0000000000..cc8f018909 --- /dev/null +++ b/openspec/changes/add-extension-telemetry/specs/extension-telemetry/spec.md @@ -0,0 +1,194 @@ +## ADDED Requirements + +Every scenario below is written as a test vector: the **GIVEN** fixes the catalog data (and any relevant build state), the **WHEN** fixes the script and the k6 command, and the **THEN** fixes the expected usage report. These vectors become the rows of two table-driven integration tests, one per command surface (`k6 run`, `k6 x`). The tests use no Go-level mocks; the catalog and the report endpoint are stand-in local servers, so no report reaches the production endpoint (see the testing requirement at the end). + + + +### Requirement: Report used catalog extensions in the usage report + +The usage report MUST include an `extensions` field listing the extensions actually used during the run. An extension is "used" when the test exercises it, not merely when it is compiled into the binary: an importable extension MUST be reported only if one of its `k6/x/*` modules was resolved, an output extension MUST be reported only if it was selected with `--out`, and a subcommand extension MUST be reported only if it was invoked. + +The field MUST contain only extensions present in the public registry catalog. Extensions absent from the catalog MUST NOT appear. When no catalog extension was used, the field MUST be omitted, consistent with how `modules` behaves today: the key appears only once a value is recorded, so an empty `extensions` array is never emitted. + +#### Scenario: Used public import is reported +- **GIVEN** the stand-in catalog lists the module path that `k6/x/sql` resolves to +- **WHEN** `k6 run` executes a script that imports `k6/x/sql` +- **THEN** the report `extensions` contains one entry for that module path with kind `js` + +#### Scenario: Compiled-but-unused extension is not reported +- **GIVEN** a catalogued extension compiled into the binary, and a script that imports, selects, and invokes nothing +- **WHEN** `k6 run` executes that script +- **THEN** the report omits the `extensions` key rather than sending an empty array + +### Requirement: Importable extension usage is recorded + +Importable JS extensions (Go extensions registered in the binary, today imported under the `k6/x/` prefix) that are resolved during a run MUST be recorded for reporting. This reverses the current behaviour, where `k6/x/` modules are deliberately excluded from usage collection. An import MUST be identified as an extension by membership in the extension registry rather than by its name, so detection keeps working if extensions are ever imported under a name that is not prefixed with `k6/x/`. Recording MUST happen at the same cached resolution site as built-in modules, so each distinct extension module is recorded once regardless of how many VUs import it. The report-assembly step MUST de-duplicate extension entries per (module, kind), so a module recorded more than once on one surface appears once. + +#### Scenario: A module imported by many VUs is recorded once +- **GIVEN** the stand-in catalog lists the module path behind `k6/x/sql` +- **WHEN** `k6 run --vus 5 --iterations 5` executes a script that imports `k6/x/sql` +- **THEN** the report `extensions` contains exactly one `js` entry for that module path + +#### Scenario: Multiple imported extensions are each reported +- **GIVEN** the stand-in catalog lists the module paths behind `k6/x/sql` and `k6/x/redis` +- **WHEN** `k6 run` executes a script that imports both `k6/x/sql` and `k6/x/redis` +- **THEN** the report `extensions` contains one `js` entry for each + +### Requirement: Extension record shape + +Each entry in the `extensions` field MUST identify the extension by its Go module path and MUST include its version and its kind. The module path and the version MUST come from the binary build information, the same source as `k6 version`. The kind MUST be one of `js`, `output`, or `subcommand`. + +#### Scenario: Entry carries module, version, and kind +- **GIVEN** the stand-in catalog lists the module path behind an imported `k6/x/sql` +- **WHEN** `k6 run` builds the report +- **THEN** the entry reports that module path, the version taken verbatim from build info (empty for an in-tree test extension, which carries no build dependency), and kind `js` + + + +### Requirement: Public-catalog access is shared between run and x + +The catalog fetch (`fetchCatalog`) already lives in the `cmd` package (in `internal/cmd/x_registry.go`, used by `k6 x`), so both the run path (imports and outputs) and the subcommand path can call it without a second fetch. The run path MUST reuse that same fetch rather than add its own, and a shared helper `catalogModulePaths` MUST read the per-entry `module` fields that the existing `catalogEntry` decoder drops. The file MAY be renamed to a neutral name (for example `catalog.go`) to show it is shared rather than `k6 x`-specific; that rename is a pure move with no behaviour change. `k6 x` behaviour MUST NOT change: the same registry URL, on-disk cache, TTL, and `K6_PROVISION_CATALOG_URL` override. This groundwork is self-contained and can land before any recording or filtering work. + +#### Scenario: Shared catalog access leaves k6 x unchanged +- **GIVEN** `K6_PROVISION_CATALOG_URL` points at a stand-in catalog, after `catalogModulePaths` is added beside the existing `fetchCatalog` +- **WHEN** `k6 x` resolves catalog membership for a name +- **THEN** it resolves identically to before the change (same URL, cache, TTL, override), guarded by the existing `k6 x` catalog tests + +### Requirement: Public-catalog filter via the registry catalog + +The system MUST decide whether an extension is public by comparing its Go module path against the public registry catalog. It MUST reuse the shared catalog fetch described in the previous requirement: the same registry URL, the same on-disk cache and TTL, and the same `K6_PROVISION_CATALOG_URL` override. Each catalog entry carries a `module` field with its Go module path; the catalog's top-level keys are import specifiers and subcommand tokens, not module paths. The system MUST build the public module-path set from those `module` fields, and MUST NOT report any extension whose module path is absent from that set. + +Each surface records the resolved extension at the point of use, not its name: the registry entry it records already carries the Go module path and version (read from build info), so assembly performs no re-resolution. Matching MUST be by module path, never by the recorded name, so a private fork that reuses a public import name resolves at its recording site to its own module path and is dropped. + +The catalog is fetched, or read from the existing cache, as part of the bounded, non-fatal usage-report send, never on the run's hot path. A newly catalogued extension becomes reportable as soon as the cache refreshes, with no new k6 release required. + +#### Scenario: Extension not in the catalog is filtered out +- **GIVEN** the stand-in catalog omits the module path of a used extension +- **WHEN** `k6 run` executes a script that imports it and reports usage +- **THEN** the report omits that extension + +#### Scenario: Private fork reusing a public import name is dropped +- **GIVEN** a fork registered under the import name `k6/x/sql` that resolves to module path `github.com/acme/xk6-sql-fork`, while the stand-in catalog lists only the real `github.com/grafana/xk6-sql` +- **WHEN** `k6 run` executes a script that imports `k6/x/sql` and reports usage +- **THEN** the report omits it, because matching is by module path and the fork resolves to its own path + + + +### Requirement: No errors when the catalog is unavailable + +If the registry catalog cannot be fetched or read from cache, or is empty or cannot be parsed, the system MUST report no extensions and MUST NOT fail or interrupt the run. Reporting nothing is the privacy-safe default; the system MUST NEVER emit unfiltered extension data as a fallback. + +#### Scenario: Unavailable catalog reports nothing +- **GIVEN** `K6_PROVISION_CATALOG_URL` points at an unreachable server and no cached copy exists +- **WHEN** `k6 run` executes a script that imports a used extension +- **THEN** the report contains no extensions +- **AND** the run exits 0 + +### Requirement: Extension telemetry respects the existing opt-out + +Extension telemetry MUST NOT introduce a separate opt-out. It MUST be suppressed whenever the existing usage report is disabled, via `--no-usage-report` or `K6_NO_USAGE_REPORT`, across the run path and the subcommand path. When usage reporting is enabled, extension data is included by the same mechanism that sends the rest of the report. + +#### Scenario: Opt-out suppresses extension data +- **GIVEN** `K6_NO_USAGE_REPORT=true`, `K6_USAGE_REPORT_URL` pointed at a stand-in server, and the catalog listing the extension +- **WHEN** `k6 run` executes a script that imports it, or `k6 x ` runs a catalogued subcommand +- **THEN** the stand-in server receives no request + +#### Scenario: No separate flag is required +- **GIVEN** usage reporting enabled (the default) and the stand-in catalog listing the extension +- **WHEN** `k6 run` executes a script that imports it +- **THEN** the extension is reported with no additional flag or environment variable + + + +### Requirement: Subcommand extension usage is reported + +Subcommand extensions (`k6 x `) MUST report their usage through the same shared usage-report mechanism every command uses: the same collector, transport, opt-out, and bounded, debug-logged, never-failing send, not a bespoke path. The report is sent from the seam that runs the command and thus learns which command actually ran, reporting only when that command is a registered subcommand extension: a baked-in subcommand reports in-process; a provisioned subcommand reports from the provisioned child that runs it, because the host never builds the command for an extension it lacks and so never reports. The report MUST be sent whether the subcommand exited zero or non-zero, since telemetry counts the invocation regardless of outcome, and MUST NOT alter the command's own run hook. Only subcommands present in the registry catalog MUST be reported. + +Reporting the subcommand name, version, and kind is sufficient. The subcommand's own arguments or internal behaviour are out of scope. The subcommand report carries the identifying fields the run report already sends that do not depend on a run (`k6_version`, `is_ci`, `goos`, `goarch`) alongside the `extensions` entry, and omits the run-only execution stats (`duration`, `iterations`, `vus_max`, `executors`) that require a run scheduler. + +#### Scenario: Catalogued subcommand run is reported +- **GIVEN** the stand-in catalog lists the module path behind a baked-in subcommand `testsub` +- **WHEN** the user runs `k6 x testsub` +- **THEN** the stand-in server receives exactly one report listing that module path with kind `subcommand` + +#### Scenario: Unknown subcommand is not reported +- **GIVEN** `K6_PROVISION_CATALOG_URL` points at an unreachable server +- **WHEN** the user runs `k6 x not-a-catalog-name`, an unregistered name +- **THEN** the stand-in server receives no request, because the name cannot be provisioned and the host builds no command for it + +#### Scenario: Registered private subcommand is dropped +- **GIVEN** a baked-in subcommand whose module path the stand-in catalog omits +- **WHEN** the user runs `k6 x ` to completion +- **THEN** the report contains no entry for it + +#### Scenario: Slow endpoint does not delay or fail the command +- **GIVEN** `K6_USAGE_REPORT_URL` points at an unreachable or slow server and the catalog lists `testsub` +- **WHEN** the user runs `k6 x testsub` +- **THEN** the command exits 0 with no error surfaced +- **AND** the send is abandoned within the same bounded timeout the run report uses +- **AND** only a debug-level log records the failure + + + +### Requirement: Output extension usage is reported + +Output extensions (a third-party `--out` provided by an `xk6-output-*` extension) that a run selects MUST be reported with kind `output`, filtered against the public catalog by module path like every other surface. An output MUST be identified as an extension by membership in the output registry (`ext.Get(ext.OutputExtension)`), not by the built-in output enum, so built-in outputs, including the literal-registered `web-dashboard`, keep their current reporting and never reach the `extensions` bucket. Usage MUST be recorded at the point the output is selected and constructed (`internal/cmd/outputs.go`), leaving the existing built-in `outputs` list untouched. + +The public registry catalog lists no output extensions today, so in production no output is reported until the catalog lists them. The code path is built now so it activates automatically once they are catalogued; adding output extensions to the catalog is tracked separately. + +#### Scenario: Catalogued output extension is reported +- **GIVEN** the stand-in catalog lists the module path that an output extension `testoutput` resolves to +- **WHEN** `k6 run --out testoutput` executes a script +- **THEN** the report `extensions` contains one entry for that module path with kind `output` + +#### Scenario: Non-catalog output extension is dropped +- **GIVEN** the stand-in catalog omits the module path behind `testoutput` +- **WHEN** `k6 run --out testoutput` executes a script +- **THEN** the report contains no entry for it + +#### Scenario: Built-in output is unchanged +- **GIVEN** a built-in output such as `json` +- **WHEN** `k6 run --out json` executes a script and reports usage +- **THEN** the report contains no `extensions` entry of kind `output` +- **AND** the `outputs` field still lists `json` + + + +### Requirement: At most one usage report per invocation + +A single k6 invocation MUST hit the report URL at most once. `k6 run` sends its own report, as it does today, and nothing in this change alters that. For the subcommand path, exactly one process reports: a baked-in subcommand reports in-process, and a provisioned subcommand reports only from the child that runs it, because the host never builds the command for an extension it lacks. The launcher does not report, so no child suppression is needed. + +#### Scenario: Provisioned subcommand reports once, from the child +- **GIVEN** a `k6 x docs` that triggers binary provisioning +- **WHEN** the command completes +- **THEN** exactly one usage report is sent, by the provisioned child that runs the command +- **AND** the host process sends no report + +### Requirement: Coverage is independent of build method + +Extension usage MUST be reported identically whether the running binary was produced by `xk6` or by k6 binary provisioning. The reporting code lives in k6 core and reads what is compiled into the running binary, so it MUST NOT depend on how the binary was built. + +#### Scenario: Provisioned binary reports like an xk6 binary +- **GIVEN** two binaries containing the same catalog extension, one built by xk6 and one produced by binary provisioning +- **WHEN** each runs the same test that uses the extension +- **THEN** both emit the same `extensions` entry + + + +### Requirement: Behaviour is verified by table-driven integration tests + +Every behaviour in this capability MUST be verified by integration tests that drive real k6 commands through the standard command test harness (`internal/cmd/tests`, `NewGlobalTestState`) and assert on the usage report that is actually sent, not on internal functions. Unit tests of report internals MUST NOT be the means of verifying these behaviours. The tests MUST be table-driven, with exactly two tables, one per command surface (`k6 run` and `k6 x`), each iterating over the scenario vectors above, rather than a separate verbose test function per scenario. Nothing may be mocked: extensions register through the normal `modules.Register` / `output.RegisterExtension` / `ext.Register` paths, and both the catalog and the report endpoint are real local HTTP servers, so no report leaves the test process. + +Each scenario above is an acceptance criterion and MUST be exercised end to end through a real `k6 run` or `k6 x` invocation, except four cases the single-binary harness cannot drive, which hold by construction: the exact `version` value in the record-shape scenario (a test extension is absent from build deps, so its version resolves to empty; the test asserts `module` and `kind` and treats `version` as the verbatim `ext.Extension.Version`), the provisioned-child once-only scenario (provisioning runs a separately built binary the harness cannot produce; the baked-in once-only is tested end to end and the provisioned child holds because the host never builds the command for an extension it lacks), build-method independence (the reporting code reads the running binary, so xk6 and provisioning report identically), and a same-module extension used as both an import and an output (an import resolves to its package path and an output to its constructor's function name, converging only for a real external dependency, so the harness cannot make one in-tree module match both; the (module, kind) de-duplication handles it by construction, and there is no separate scenario for it). The shared-catalog groundwork scenario is a behaviour-preserving refactor verified by the existing `k6 x` catalog tests, separate from the two telemetry tables. + +To make the sent report observable, a test MUST enable reporting (the harness disables it by default) and point the report endpoint at a local stand-in server via the new `K6_USAGE_REPORT_URL` env var, then assert on the received JSON. Catalog membership MUST be controlled through the existing `K6_PROVISION_CATALOG_URL` override, pointed at a stand-in catalog whose `module` fields match each test extension's resolved module path. For the private-fork case, the stand-in catalog MUST map the public import name to the real catalogued module path while the in-tree fork resolves to a different module path, so the drop is exercised by the module-path mismatch rather than by a missing catalog entry. The single new env var (`K6_USAGE_REPORT_URL`, the overridable report endpoint) is the only production addition the tests require beyond the feature itself; the catalog reuses the existing override, and no other production code or comments may change for testing. + +#### Scenario: A reported behaviour is exercised end to end +- **GIVEN** the command test harness with reporting enabled, the report endpoint pointed at a local server, and the stand-in catalog listing a used extension +- **WHEN** `k6 run` executes a script that imports that extension +- **THEN** the test asserts the received report's `extensions` contains it with its module, version, and kind + +#### Scenario: Opt-out is verified by the absence of a request +- **GIVEN** `K6_NO_USAGE_REPORT=true` and the endpoint pointed at a local server +- **WHEN** `k6 run` executes a script that imports a catalogued extension +- **THEN** the local server receives no request diff --git a/openspec/changes/add-extension-telemetry/tasks.md b/openspec/changes/add-extension-telemetry/tasks.md new file mode 100644 index 0000000000..274f51c6cb --- /dev/null +++ b/openspec/changes/add-extension-telemetry/tasks.md @@ -0,0 +1,148 @@ +## How to work these tasks + +This file is the implementation plan: none of the code or tests below exists yet, and each task tells the implementer what to build next. The capability lands as two table-driven integration tests on the real command harness (`internal/cmd/tests`, `NewGlobalTestState`), one per surface: `TestRunReportsExtensions` for `k6 run` and `TestSubcommandReportsUsage` for `k6 x`. The tests use no Go-level mocks: extensions register through the normal paths, and the catalog and the report endpoint are stand-in `httptest` servers, so no report ever reaches the production endpoint. + +Each task is one increment and is self-contained: a fresh implementer needs only this task, the spec (`specs/extension-telemetry/spec.md`), the design (`design.md`), and the codebase. Every code task follows the same shape: + +- **Prereq**: what earlier tasks already put in place, so you know the starting state. +- **Row**: the scenario vector to add as one row of an existing table. Do not add a new test function; extend the table to avoid test churn. +- **Red or guard**: a red task drives new production code (write the row, watch it fail on its assertion before touching production code). A guard row passes the moment it is added because an earlier task already satisfied it; it exists to fail if someone regresses. If a guard row is red on arrival, an earlier task is broken. +- **Implement**: the exact production change. +- **Verify**: run the named test and confirm the expected red-to-green (or green-on-arrival), then `make lint` clean. The tree stays green at the end of every task, so the next task starts from a working state. + +Tasks are ordered so each builds on the last. The two groundwork tasks have no dependency on recording and come first. Module-path placeholders (`M_import`, `M_import2`, `M_output`, `M_sub`, `M_fork`) match the design's testing tables. + +## Groundwork + +- [ ] 1. **Share catalog access between run and x** (spec: "Public-catalog access is shared between run and x") + - Prereq: none. This is the recommended first commit. + - `fetchCatalog` (`internal/cmd/x_registry.go:110`), `readCachedCatalog` (`:81`), and `writeCachedCatalog` (`:141`) already live in package `cmd`, called from `subcommand.go`, so there is no cross-package move. Add `catalogModulePaths(ctx, gs)` beside them. It returns the set of Go module paths in the catalog, acquired exactly the way `k6 x` does so it shares the same cache file, TTL, and `K6_PROVISION_CATALOG_URL` override: read the on-disk cache (`writeCachedCatalog` stores raw JSON); on a miss, `fetchCatalog(ctx, cmp.Or(gs.Env[state.ProvisionCatalogURL], defaultCatalogURL()))` and `writeCachedCatalog` the fresh bytes, with `ctx` bounded by the report send so the fetch is non-fatal and never on the run's hot path. Parse those raw bytes with a new struct that reads each entry's `module` field, which the existing `catalogEntry`/`registrySubcommands` decoder drops. Both the run report (task 7) and the subcommand report (task 14) call this one helper, so the acquisition is not duplicated. Optionally rename the file to a neutral name (for example `catalog.go`) as a pure move with no logic change. + - Test: this is a behaviour-preserving refactor of the shared fetch, so the "test" is that `k6 x` catalog resolution is unchanged. Keep the existing `k6 x` catalog tests green; if none exercises membership through `K6_PROVISION_CATALOG_URL`, add one characterization test that a name resolves identically before and after. Do not add a bespoke unit test for `catalogModulePaths`; it gets behavioural coverage from the filter task (7) via the run table, so a separate unit test would be inflation. Every catalog fixture from here on MUST carry a `module` field per entry (production shape: top-level keys are import specifiers and `subcommand:` tokens, each entry has a `module`); the legacy `testCatalogJSON` keys by module path with no `module` field, so reusing it verbatim yields an empty set. + - Verify: `go test -race ./internal/cmd/ -run 'TestX|TestExtensionSubcommands'` stays green; `make lint` clean. + +- [ ] 2. **Make the report endpoint observable** (spec: harness baseline for "Behaviour is verified by table-driven integration tests") + - Prereq: none. + - Row: create `TestRunReportsExtensions` with its first row. GIVEN any catalog, WHEN a trivial script runs under `k6 run` with reporting enabled and `K6_USAGE_REPORT_URL` pointed at an `httptest.Server`, THEN that server receives the existing usage report. This row is the harness baseline, not a spec scenario; it proves the seam every later run-surface row reuses. + - Red: it fails because the report endpoint is a hardcoded `const`. + - Implement: add `K6_USAGE_REPORT_URL` to `cmd/state/state.go` (defaulting to the production endpoint) and read it at the send site in `internal/cmd/report.go`. No other behaviour change. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` red then green; `make lint` clean. + +## Importable extension telemetry + +- [ ] 3. **Record and report an imported extension** (spec: "Report used catalog extensions" / "Used public import is reported", "Importable extension usage is recorded", "Extension record shape" / "Entry carries module, version, and kind", "Extension telemetry respects the existing opt-out" / "No separate flag is required") + - Prereq: task 2's `K6_USAGE_REPORT_URL` seam and the `TestRunReportsExtensions` table. + - Row: GIVEN the stand-in catalog lists `M_import`, WHEN `k6 run` runs a script importing `k6/x/testimport`, THEN the report's `extensions` holds one entry `{M_import, version, kind: "js"}`. + - Test setup: register the in-tree `k6/x/testimport` extension once through a `sync.Once` (mirror `registerTestSubcommandExtensions` in `internal/cmd/subcommand_test.go:416`), because `ext.Register` panics on a duplicate name/type and the registry is process-global; every later run-table extension (tasks 5, 11) registers through that same once. + - Red: it fails because `js/modules/resolution.go` deliberately drops `k6/x/*` imports and no code assembles `extensions`. + - Implement: in `js/modules/resolution.go`, at the cached resolution site, record the resolved registry entry (`ext.Get(ext.JSExtension)[name]`, identified by registry membership, not by the `k6/x/` name) into a new `extensions` bucket on the `usage.Usage` collector via a generic `Values(key, any)` method added beside `Strings`/`Uint64`; leave the `modules` list unchanged. In `internal/cmd/report.go`, keep `createReport` and turn `reportUsage` into the shared transport (it takes a `create` closure, sends bounded, and debug-logs the outcome) with `postUsageReport` as the inner POST; add `resolveExtensions`/`filterExtensions` to read the recorded `*ext.Extension` entries back out of the bucket and emit `{module, version, kind}` straight off each entry, with no re-resolution. + - By construction (no separate row): `version` is empty for an in-tree test extension because it carries no build dependency, so assert `module` and `kind` and treat `version` as the verbatim `ext.Extension.Version`. Because the reporting code reads the running binary, an xk6 build and a provisioning build report identically, so build-method independence needs no row. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` red then green; `make lint` clean. + +- [ ] 4. **De-duplicate a module imported by many VUs** (spec: "A module imported by many VUs is recorded once") + - Prereq: task 3's recording and `filterExtensions`. + - Row: GIVEN the catalog lists `M_import`, WHEN `k6 run --vus 5 --iterations 5` runs a script importing `k6/x/testimport`, THEN `extensions` holds exactly one `js` entry for `M_import`. + - Red: it fails if recording happens per import rather than once per module. + - Implement: de-duplicate per (module, kind) inside `filterExtensions`. Recording at the cached resolution site already collapses repeated VU imports; this makes the guarantee explicit at assembly. + - By construction (no separate row): because dedup is per (module, kind), a single module used as both an import and an output would yield two entries, one per kind. The in-tree harness cannot make one module resolve to both an import path and an output constructor name, so dual-purpose coexistence needs no row. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` red then green; `make lint` clean. + +- [ ] 5. **Report each of several imported extensions** (spec: "Multiple imported extensions are each reported") + - Prereq: task 3 (appends each resolved import) and task 4 (dedup per module). + - Row: GIVEN the catalog lists `M_import` and `M_import2`, WHEN `k6 run` runs a script importing both `k6/x/testimport` and `k6/x/testimport2`, THEN `extensions` holds one `js` entry for each. + - Guard: passes once tasks 3 and 4 land; it exists to fail if recording overwrites a single slot or dedup collapses distinct modules. Register the two in-tree test extensions in two distinct Go packages so they resolve to distinct module paths (a same-package pair resolves to one path and dedup would collapse them), and list both paths in the stand-in catalog. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` green on arrival; `make lint` clean. + +- [ ] 6. **Omit the key when nothing was used** (spec: "Compiled-but-unused extension is not reported") + - Prereq: task 3's recording. + - Row: GIVEN a catalogued extension compiled into the binary, WHEN `k6 run` runs a script that imports, selects, and invokes nothing, THEN the report omits the `extensions` key entirely (never `[]`). + - Red for the omission half: the empty bucket serializes as `[]`. Make it pass by omitting the key when the bucket is empty, consistent with how `modules` behaves. The compiled-but-unused half is a guard: it already holds because task 3 records only at the import site, so an unused extension is never recorded. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` red then green; `make lint` clean. + +## Filtering + +- [ ] 7. **Filter reported extensions against the catalog** (spec: "Public-catalog filter via the registry catalog" / "Extension not in the catalog is filtered out") + - Prereq: task 1's `catalogModulePaths`, task 3's `resolveExtensions`/`filterExtensions`. + - Row: GIVEN the stand-in catalog omits `M_import`, WHEN `k6 run` runs a script importing `k6/x/testimport`, THEN the report holds no entry for it. + - Red: it fails because task 3 reports everything recorded. Make it pass inside `filterExtensions`: keep only recorded extensions whose module path (read off the recorded `*ext.Extension`) is in the set returned by `catalogModulePaths(ctx, gs)` (task 1), which reads the catalog through the same cache-then-fetch path as `k6 x`. Call it within the bounded, non-fatal report send, never on the run's hot path. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` red then green; `make lint` clean. + +- [ ] 8. **Match by module path, not import name** (spec: "Private fork reusing a public import name is dropped") + - Prereq: task 7's filter. + - Row: GIVEN the catalog lists the real `M_import` for import name `k6/x/testimport` while an in-tree fork registered under the same import name resolves to `M_fork`, WHEN `k6 run` runs a script importing that name, THEN the report holds no entry, because matching is by module path. + - Guard: passes as soon as task 7 lands; it exists to fail if the match ever compares import names instead of module paths. No production change beyond task 7. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` green on arrival; `make lint` clean. + +## Privacy + +- [ ] 9. **Fail closed when the catalog is unavailable** (spec: "No errors when the catalog is unavailable" / "Unavailable catalog reports nothing") + - Prereq: task 7's filter. + - Row: GIVEN `K6_PROVISION_CATALOG_URL` points at an unreachable server with no cache, WHEN `k6 run` runs a script importing a used extension, THEN the report holds no extensions AND the run exits 0. + - Red: it fails because task 7's filter has no failure handling, so a fetch error either aborts the run or leaks the unfiltered bucket. Make it pass by dropping the bucket before the fetch and reporting nothing when the catalog cannot be read, keeping the fetch and send bounded and non-fatal. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` red then green; `make lint` clean. + +- [ ] 10. **Opt-out suppresses everything on the run path** (spec: "Extension telemetry respects the existing opt-out" / "Opt-out suppresses extension data") + - Prereq: tasks 2, 7, 9 (the send and the catalog fetch now sit behind report assembly). + - Row: GIVEN `K6_NO_USAGE_REPORT=true` with both stand-in servers wired, WHEN `k6 run` runs a script importing a catalogued extension, THEN neither the report endpoint nor the catalog server receives a request. + - Guard: passes as long as the existing opt-out gate stays ahead of report assembly; it exists to fail if extracting `postUsageReport` (task 3) or adding the catalog fetch (task 7) moved work in front of that gate. No production change if the gate already sits ahead of report assembly. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` green on arrival; `make lint` clean. + +## Output extension telemetry + +- [ ] 11. **Report a selected output extension** (spec: "Output extension usage is reported" / "Catalogued output extension is reported") + - Prereq: task 3's `resolveExtensions`/`filterExtensions`, task 7's filter (output rides the same bucket and filter). + - Row: GIVEN the catalog lists `M_output`, WHEN `k6 run --out testoutput` runs a script, THEN `extensions` holds one entry `{M_output, version, kind: "output"}`. Register an in-tree output extension via `output.RegisterExtension` and list its resolved path in the stand-in catalog. + - Red: it fails because output extensions are never recorded. Make it pass by recording the resolved output extension entry (`ext.Get(ext.OutputExtension)[name]`) into the `extensions` bucket via `usage.Values` in `internal/cmd/outputs.go` when the output is an extension; `resolveExtensions`/`filterExtensions` then handle it uniformly, and its kind (`output`) comes straight off the recorded entry. Leave the built-in `outputs` recording untouched. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` red then green; `make lint` clean. + +- [ ] 12. **Drop a non-catalog output extension** (spec: "Non-catalog output extension is dropped") + - Prereq: task 11's output recording, task 7's filter. + - Row: GIVEN the catalog omits `M_output`, WHEN `k6 run --out testoutput` runs a script, THEN the report holds no entry for it. + - Guard: passes because task 7's filter already covers the output bucket; it exists to fail if the output path bypasses the filter. No new production change if the filter already covers this bucket. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` green on arrival; `make lint` clean. + +- [ ] 13. **Leave built-in outputs unchanged** (spec: "Built-in output is unchanged") + - Prereq: task 11's output recording. + - Row: GIVEN a built-in output such as `json`, WHEN `k6 run --out json` runs a script, THEN the report holds no `kind: "output"` entry AND `outputs` still lists `json`. + - Guard: passes because built-ins are classified by the built-in output enum and never enter the extensions bucket; it exists to fail if output recording ever broadens to all outputs instead of output-registry members. This is what keeps the literal-registered `web-dashboard` out of the bucket. Keep the built-in path in `internal/cmd/outputs.go` unchanged. + - Verify: `go test -race ./internal/cmd/tests/ -run TestRunReportsExtensions` green on arrival; `make lint` clean. + +## Subcommand extension telemetry + +- [ ] 14. **Report a baked-in subcommand run** (spec: "Subcommand extension usage is reported" / "Catalogued subcommand run is reported", "At most one usage report per invocation") + - Prereq: task 2's `K6_USAGE_REPORT_URL` seam and `postUsageReport`, task 1's `catalogModulePaths`. + - Row: create `TestSubcommandReportsUsage` with its first row, mirroring task 2's setup. GIVEN the catalog lists `M_sub` for a baked-in subcommand `testsub`, WHEN the user runs `k6 x testsub`, THEN the stand-in server receives exactly one report `{M_sub, version, kind: "subcommand"}`. + - Red: it fails because no subcommand reports. Make it pass at the root `execute` seam in `internal/cmd/root.go`: run the command with cobra's `ExecuteC` (which returns the executed leaf command and its error) and call a new `reportSubcommandUsage(gs, executed)` in `internal/cmd/subcommand.go`. It reports only when `executed.Name()` is a registered subcommand extension (`ext.Get(ext.SubcommandExtension)`), whether the command exited zero or non-zero, without touching the command's run hook. It builds the report with a `createSubcommandReport` (identity fields `k6_version`, `is_ci`, `goos`, `goarch` via the shared `addEnvironmentInfo`, plus the entry) and sends it through the shared `reportUsage`, keeping the entry only if its module path is in the set returned by `catalogModulePaths(ctx, gs)` (task 1, the same shared cache-then-fetch helper the run path uses), with `ctx` bounded like the run report. + - By construction (no separate row): the baked-in report is tested end to end here; a provisioned subcommand reports once from the child that runs it, because the host never builds the command for an extension it lacks, so the provisioned-child once-only case holds without a row the single-binary harness cannot drive. + - Verify: `go test -race ./internal/cmd/tests/ -run TestSubcommandReportsUsage` red then green; `make lint` clean. + +- [ ] 15. **Drop a private subcommand** (spec: "Registered private subcommand is dropped") + - Prereq: task 14's subcommand report, task 1's `catalogModulePaths`. + - Row: GIVEN a baked-in subcommand whose module path the catalog omits, WHEN the user runs `k6 x ` to completion, THEN the report holds no entry for it. + - Red: it fails because task 14 reports every subcommand that runs. Make it pass by reusing the module-path filter before sending. + - Verify: `go test -race ./internal/cmd/tests/ -run TestSubcommandReportsUsage` red then green; `make lint` clean. + +- [ ] 16. **Never report an unknown subcommand** (spec: "Unknown subcommand is not reported") + - Prereq: task 14's subcommand report. + - Row: GIVEN `K6_PROVISION_CATALOG_URL` points at an unreachable server, WHEN the user runs `k6 x not-a-catalog-name`, THEN neither stand-in server receives a request, because the name cannot be provisioned and the host builds no command for it. + - Guard: passes because the host never builds a command for a name it lacks, so `ExecuteC` never returns it as a registered subcommand extension and `reportSubcommandUsage` finds nothing to report; it exists to fail if the report ever fires for a command the binary did not actually run. + - Verify: `go test -race ./internal/cmd/tests/ -run TestSubcommandReportsUsage` green on arrival; `make lint` clean. + +- [ ] 17. **Never delay or fail the command on a slow endpoint** (spec: "Slow endpoint does not delay or fail the command") + - Prereq: task 14's `reportSubcommandUsage` and `postUsageReport`. + - Row: GIVEN `K6_USAGE_REPORT_URL` points at an unreachable or slow server and the catalog lists `testsub`, WHEN the user runs `k6 x testsub`, THEN the command exits 0 with no error surfaced, the send is abandoned within the same bounded timeout the run report uses, and only a debug-level log records the failure (asserted through the harness `LoggerHook`). + - Red: it fails because a naive send blocks or propagates the error. Make it pass by routing the subcommand send through the shared bounded, non-fatal `reportUsage`. + - Verify: `go test -race ./internal/cmd/tests/ -run TestSubcommandReportsUsage` red then green; `make lint` clean. + +- [ ] 18. **Opt-out suppresses the subcommand report** (spec: "Extension telemetry respects the existing opt-out" / "Opt-out suppresses extension data") + - Prereq: task 14's `reportSubcommandUsage`. + - Row: GIVEN `K6_NO_USAGE_REPORT=true` with both stand-in servers wired, WHEN the user runs `k6 x testsub`, THEN neither server receives a request. + - Red: it fails because `reportSubcommandUsage` is a new send with no opt-out check (the `--no-usage-report` flag lives on the per-test `Config`, not `GlobalState`). Make it pass by gating the send on the opt-out parsed from the environment via `readEnvConfig(gs.Env)` (honoring `K6_NO_USAGE_REPORT`), which reaches a provisioned child through `buildSubprocessEnv`. + - Verify: `go test -race ./internal/cmd/tests/ -run TestSubcommandReportsUsage` red then green; `make lint` clean. + +## Docs + +- [ ] 19. **Document the new field** (spec: "Report used catalog extensions in the usage report") + - Prereq: tasks 1-18 (document the shipped behaviour). Docs, not code: no test row. + - The public [usage collection page](https://grafana.com/docs/k6/latest/set-up/usage-collection/) states "Only k6 built-in JavaScript modules and outputs are considered. Private modules and custom extensions are excluded.", which this change makes false. + - Implement: update that page in grafana/k6-docs to document the `extensions` field (`{module, version, kind}`), its public-catalog-only scope, and that the existing opt-out covers it. Add the implementation PR's entry under `release notes/`. + - Verify: the k6-docs PR is open and linked from the implementation PR; the release-notes entry rides the implementation PR.