Add extension usage telemetry#6126
Open
inancgumus wants to merge 16 commits into
Open
Conversation
The usage report endpoint was a hardcoded const posting to stats.grafana.org, so no test could observe the anonymous usage report. Reading the endpoint from K6_USAGE_REPORT_URL (defaulting to the production endpoint) lets integration tests point it at a local server and assert the report is sent.
The usage report tracked only built-in modules and dropped every k6/x/ import, so extension adoption was invisible. Resolved Go extensions are now recorded in a dedicated extensions bucket at the cached resolution site, identified by registry membership rather than the k6/x/ name, and reported with their Go module path, version, and kind, de-duplicated per module and kind. Only extensions whose module path appears in the public registry catalog are reported, so private or unlisted extensions never leak. The catalog is read through the same cache file, TTL, and K6_PROVISION_CATALOG_URL override the k6 x subcommand path uses, fetched inside the bounded report send off the run's hot path, and an unreachable catalog fail-closed reports nothing. When nothing catalogued was used, the extensions key is omitted rather than sent as an empty array, consistent with how modules behaves.
The usage report must not list the same extension multiple times when many VUs import it. This adds a --vus 5 --iterations 5 acceptance row asserting a module imported across VUs yields exactly one js entry, locking in the per-(module,kind) de-duplication done at report assembly.
The usage report must list every imported extension, not collapse distinct modules into one slot. This guards run's reporting so that two extensions in separate Go packages, which resolve to distinct module paths, each surface as their own js entry and survive de-duplication.
An empty or fully filtered extensions result omits the key entirely, consistent with how modules behaves, so key presence always means at least one catalogued extension was used. Nothing pinned that. This adds a row asserting a run that uses no extension produces no extensions key rather than an empty array.
The report filter keys on Go module path, but nothing pinned that a private fork reusing a public import name resolves to its own path and is dropped. A guard row now imports an in-tree fork registered under a catalogued import name while the catalog advertises only the real extension's path; the fork's distinct module path keeps it out of the report, so the row fails if matching ever regresses to import names.
catalogModulePaths returns nothing when the catalog cannot be fetched and resolveExtensions drops the raw bucket before consulting it, so an unreachable catalog with no cache reports no extensions instead of leaking the unfiltered list. Nothing pinned that. This adds a row importing a used extension against an unreachable catalog, asserting the report carries no extensions and the run still exits 0.
The usage report gained a catalog fetch and extension enrichment. This guards that the existing K6_NO_USAGE_REPORT opt-out still gates all of it: with the opt-out set, neither the report endpoint nor the catalog server is contacted, so no new opt-out is needed and none of the added work runs ahead of the gate.
Output extensions chosen via --out were tracked only in the built-in outputs list, so catalogued xk6 output extensions never appeared in the usage report. A selected output that is a member of the output registry is now recorded as a resolved registry entry in the shared extensions bucket, so the report filters it by module path like every other surface. Built-in outputs stay out of the bucket.
The output extension bucket shared the same catalog filter as JS imports, but nothing pinned that behavior. This adds a test row proving a used output extension absent from the catalog is dropped from the usage report, so a future change that routes outputs around the filter fails loudly.
Extension usage recording keys the extensions bucket on output-registry membership, so built-in outputs like json were never reported as extensions. Without a test, a future change could broaden recording to all outputs and start leaking built-ins. This guards that json stays listed under outputs and never produces an output-kind extension entry.
Baked-in subcommands invoked via `k6 x` sent no usage telemetry, so the extension registry had no signal about their use. The root execute seam now reports once after the command runs, using cobra's ExecuteC to recover the command that ran and its error, so a registered subcommand extension is reported whether it succeeds or fails, through the shared usage-report transport. It carries only the run-independent identifying fields and the catalogued extension entry, bounded and non-fatal so it never fails or delays the command, and the command's own hook is left untouched.
createSubcommandReport keeps a subcommand's entry only when its module path appears in the public catalog, matching the run path's filter. Nothing pinned that. This adds a row running a baked-in subcommand whose module path the catalog omits, asserting the report carries no entry for it.
Before, nothing guarded against a usage report firing for a subcommand name the binary does not provide. This adds a guard: running k6 x with an unregistered name against an unreachable catalog builds no command and sends no report, so an early or stray report call would now be caught.
The subcommand report rides the shared bounded, non-fatal send, so a slow or unreachable endpoint cannot stall or fail k6 x. Nothing pinned that. This adds a row proving the send is abandoned within the bounded run-report timeout, the command still exits 0 with no surfaced error, and the failure surfaces only as a debug-level log.
Nothing exercised the K6_NO_USAGE_REPORT gate on the k6 x report path, so a regression could re-enable telemetry unnoticed. This adds a table row that runs k6 x under the opt-out and asserts neither the report endpoint nor the catalog is consulted.
inancgumus
requested review from
ankur22,
mstoykov and
szkiba
and removed request for
a team and
mstoykov
inancgumus
temporarily deployed
to
azure-trusted-signing
GitHub Actions
Inactive
inancgumus
temporarily deployed
to
azure-trusted-signing
GitHub Actions
Inactive
inancgumus
force-pushed
the
add/ext-telemetry-spec
branch
from
47cb6c9 to
0a303f6
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What?
Adds anonymous extension usage to the k6 usage report, covering the three user-facing surfaces:
k6/x/module) is reported when a run resolves it, not when it is merely compiled into the binary.--out; built-in outputs are unaffected.k6 x, whether it exits zero or non-zero.Each entry carries the extension's Go module path, version, and kind. Only extensions listed in the public registry catalog are reported: matching is by module path, so a private fork reusing a public import name is dropped, and an unreachable catalog reports nothing rather than falling back to unfiltered data. When no catalogued extension was used, the report omits the field entirely. The existing
--no-usage-report/K6_NO_USAGE_REPORTopt-out gates all of it; no new opt-out, no new endpoint, and the send stays bounded and non-fatal, so a slow or unreachable endpoint never delays or fails a command.Why?
Extension adoption is a blind spot: the usage report tracks built-in modules but deliberately drops every extension, so decisions about which extensions to promote, maintain, or retire run on guesswork. Cloud-executed runs are partly tracked, while local execution, most OSS usage, stays invisible. Reporting catalog extensions through the same anonymous report k6 already sends closes that gap without a new data category: private and unlisted extensions never appear, xk6-built and provisioned binaries report identically, and a newly catalogued extension becomes reportable as soon as the catalog cache refreshes, with no new k6 release.
Note
K6_USAGE_REPORT_URL, makes the report endpoint overridable so the integration tests can assert on the actually-sent report; catalog membership reuses the existingK6_PROVISION_CATALOG_URLoverride.Checklist
make check) and all pass.Related PR(s)/Issue(s)