Skip to content

Commit d6ed5e0

Browse files
LeoRoccoBreedtclaudeDouglas Blank
authored
feat(admin): add growth-report — cross-platform use-case growth & rates (HTML) (#25)
* feat(admin): add growth-report action skeleton (cross-platform growth HTML) * feat(admin): core creation-event, usage-metric, and window models for growth-report * refactor(admin): relocate time-bucket helpers to utils; add growth rate layer * feat(admin): EM collector — project use cases (proxied creation) + experiment/registry usage metrics * fix(admin): fetch EM experiments once per project (avoid duplicate API calls) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(admin): Opik collector — project use cases + span-count usage metrics * feat(admin): MPM collector — monitored-model use cases (probe/proxy) + prediction-volume usage * fix(admin): guard MPM enumeration against malformed workspace/model elements Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(admin): unified use-case/department roll-up + usage grouping + workspace proxy * feat(admin): self-contained HTML growth dashboard renderer * test(admin): make growth-report secret-leak test meaningful; drop unused imports/noqa Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(admin): growth-report orchestrator + workspace resolution + docs Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * refactor(admin): drop unused growth-report helpers; add end-to-end render test; polish Removes bucket_events/workspace_creation_events/usage_by_workspace (verified unused by the pipeline), fixes _open() to resolve relative --output paths, drops a dead noqa line, and adds a full-chain generate_growth_report -> build() -> build_html() integration test. * fix(admin): filter EM projects to the requested workspace The Comet EM projects endpoint returns a fallback set from another workspace (ignoring workspaceName) when the API key isn't a member of the requested workspace. That caused those foreign projects to be mis-attributed to every inaccessible workspace, showing as duplicate project names in the EM adoption table and inflating EM growth counts. Keep only projects whose own workspaceName matches the requested one; warn when projects are dropped, and still report the workspace at 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(admin): make growth-report tables collapsible (collapsed by default) Wrap every report table in a native <details>/<summary> disclosure (no JS, stays self-contained). Tables start collapsed with the title + row count in the summary, so large tables don't dominate the page as data grows. Also give the per-product growth breakdown tables real titles so their summaries read meaningfully instead of a generic label. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(admin): growth-report render improvements - Rename 'Departments' -> 'Workspaces' everywhere, with a 'proxy for teams / departments' clarification (label was misleading; these are workspaces used as a department/group proxy). - Cumulative (area) charts: add per-point hover tooltips, x-axis time labels on both edges (+ intervals), a max-value gridline label, and the final cumulative value — previously they showed no numbers. - Adoption/usage sections: add usage growth KPIs (New / Growth% over the window) and a fastest-growing-project KPI, not just the total. - Adoption/usage sections: add a cumulative area chart alongside the per-period bars. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(admin): real chart hover tooltips; cumulative edge-only dates; always show fastest-growing project 1. Replace flaky native SVG <title> tooltips with a real interactive tooltip (positioned div + vertical guide line) driven by per-column hit rects; shared across bar, area, and stacked charts. 2. Cumulative (area) charts now label only the earliest and latest dates on the x-axis (instead of every interval), per request. 3. Always surface the fastest-growing-project KPI for every product (incl. EM = most new experiments in the analysis window), showing '—' when nothing grew in-window so it never appears missing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(admin): skip optional-extra tests when opik/comet_mpm absent (fix CI) CI installs only requirements.txt (opik present, comet_mpm not), but the growth-report tests patched comet_mpm/opik unconditionally, erroring at setup with ModuleNotFoundError. Guard the SDK-patching tests with skipif(find_spec(...) is None), and make the two cross-platform assembly tests env-independent by forcing the resolved platform set instead of relying on which optional extras happen to be installed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(admin): coerce datapoint values to float before count aggregation Non-numeric or mixed-type value/y from Opik SPAN_COUNT and MPM prediction responses could raise TypeError on `counts[...] += ...` and abort collection. Add _as_float() to coerce datapoints (None/blank/non-numeric -> 0.0). Addresses baz-reviewer Type Inconsistency findings. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * refactor(admin): extract shared workspace-summary UsageMetric helper EM/Opik/MPM collectors each appended a near-identical workspace-level (project=None) UsageMetric block, so the summary shape could drift. Route all three through _workspace_usage_metric(). Addresses baz-reviewer Code Dedup finding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(admin): derive EM experiment KPI total from bucketed counts, not metadata The EM EXPERIMENT_COUNT total (and its workspace roll-up) was sourced from `numberOfExperiments` metadata while the chart series was built from experiments bucketed by start_server_timestamp. When those disagree, the adoption KPI total didn't equal the cumulative chart sum. Derive `total` from the same counts map that feeds the series so they always match. Add a regression test covering the metadata-disagrees case. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Douglas Blank <doug@comet.com>
1 parent 647b1b4 commit d6ed5e0

8 files changed

Lines changed: 4026 additions & 127 deletions

File tree

README-ADMIN.md

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,3 +122,76 @@ When using the `--app` flag, an interactive web interface is launched where you
122122
- Select workspace and project from dropdowns
123123
- View statistics and charts interactively
124124
- Change time units and regenerate reports
125+
126+
## growth-report
127+
128+
Generate a cross-platform use-case growth & adoption report — Opik projects, EM projects,
129+
and MPM monitored models — as a single self-contained HTML page. This is distinct from
130+
`usage-report` (an experiment-count PDF): `growth-report` tracks *how many use cases exist
131+
and how fast they're being created*, per workspace/department, across all three products.
132+
133+
### Basic Usage
134+
135+
```shell
136+
cometx admin growth-report
137+
cometx admin growth-report my-workspace
138+
cometx admin growth-report my-workspace another-workspace
139+
```
140+
141+
If no workspace is given, all workspaces visible to the current API key are used.
142+
143+
### Options
144+
145+
- **`WORKSPACE ...`**: Zero or more workspaces to include. If omitted, resolved via `get_workspaces()`.
146+
- **`--units {month,week,day,hour}`**: Chart bucket granularity (default: `month`). Charts always render **all-time** history at this granularity — this is a separate concept from `--window`, below.
147+
- **`--window WINDOW`**: Relative analysis window for the KPI numbers, e.g. `7d`, `14d`, `30d`, `90d`, `2w`, `6m`, `1y` (default: `7d`). Format is `\d+[dwmy]`: `d`=days, `w`=weeks (×7 days), `m`=months (approximated as 30 days), `y`=years (approximated as 365 days). The month/year approximation is intentional — the window is only used to compute an "installed base before window" cutoff, not calendar-exact arithmetic.
148+
- **`--platforms PLATFORMS`**: Comma-separated platforms to include (default: `em,opik,mpm`).
149+
- **`--output PATH`**: Output HTML file path (default: `growth_report.html`).
150+
- **`--limit N`**: Limit the number of workspaces processed — useful for a fast smoke-test run.
151+
- **`--no-open`**: Don't automatically open the generated HTML file after generation.
152+
153+
### The two time concepts
154+
155+
- **`--units`** is the *chart granularity*: every chart shows the complete all-time history bucketed at this resolution.
156+
- **`--window`** is the *KPI analysis window*: Total/New/% Growth numbers compare "in the last `--window`" against "before the window." On the charts, the window is drawn as a shaded band overlaid on the all-time series, rather than filtering the data.
157+
158+
### Growth rates
159+
160+
Growth and adoption rates are computed directly from use-case **creation timestamps**:
161+
`pct_growth = new_in_window / count_before_window`, where `count_before_window` is the
162+
number of use cases created before `window.start` and `new_in_window` is the number
163+
created inside `[window.start, window.end]`.
164+
165+
### Examples
166+
167+
```shell
168+
# Full cross-platform report for all workspaces, default 7-day window
169+
cometx admin growth-report
170+
171+
# Just Opik + EM, 30-day window, for two workspaces
172+
cometx admin growth-report --platforms em,opik --window 30d my-workspace another-workspace
173+
174+
# Fast smoke-test run against a single workspace, don't auto-open
175+
cometx admin growth-report --limit 1 --no-open --output growth.html my-workspace
176+
```
177+
178+
### Output
179+
180+
The growth report generates a single self-contained HTML file containing:
181+
182+
- A **unified section** ("Use cases across all platforms") with department/use-case KPIs, a stacked created-per-period chart broken down by kind (Opik/EM/MPM), a use-cases-by-department chart, and a breakdown table.
183+
- Per-product **growth** sections (Opik / EM / MPM) with Total / New / % Growth KPIs, a new-use-cases bar chart, a cumulative area chart (both carrying the analysis window as a shaded band), and a breakdown table.
184+
- Per-product **adoption** sections with secondary usage metrics — Opik span counts, EM experiment counts, MPM prediction volume — each broken down by project.
185+
186+
The breakdown tables follow one rule throughout: when more than one workspace is included,
187+
tables break down **by workspace/department**; when only a single workspace is included,
188+
tables break down **by individual use case** instead (a by-workspace table with one row
189+
would be uninformative).
190+
191+
### Caveats
192+
193+
- **EM "created" is a proxy.** The Comet API has no true EM project-creation timestamp, so an EM project's creation time is approximated as the earliest experiment start time in that project, falling back to the project's `lastUpdated` time if it has no experiments.
194+
- **Workspace/department "created" is also a proxy** — the earliest use-case creation timestamp seen in that workspace, across all platforms.
195+
- **The EM model-registry engagement panel is a snapshot**, not an over-time series ("Registered models" / "Model versions" per workspace) — it is kept in its own labeled panel and is **never** combined with MPM's monitored-model counts, even though both are "model" concepts.
196+
- **MPM requires provisioning.** If the account/workspace has no MPM model-monitoring provisioned, the MPM collector degrades gracefully — an empty MPM section and `collectors.mpm: false` in the underlying report data — rather than failing the whole report.
197+
- Opik and MPM collection require their optional SDKs (`opik`, `comet_mpm`). Install them with `pip install 'cometx[all]'` if not already available; any platform whose SDK can't be imported is silently dropped from `--platforms` (a note is printed to the console) so the report still succeeds with the remaining platforms.

README.md

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -578,6 +578,55 @@ cometx admin usage-report workspace --units day --no-open
578578
cometx admin usage-report --app
579579
```
580580

581+
#### growth-report
582+
583+
Generate a cross-platform use-case growth & adoption report (Opik projects, EM projects,
584+
MPM monitored models) as a single self-contained HTML page. Distinct from `usage-report`
585+
(experiment-count PDF): `growth-report` tracks *how many use cases exist and how fast
586+
they're being created*, per workspace/department, across all three products.
587+
588+
```
589+
cometx admin growth-report [WORKSPACE ...]
590+
```
591+
592+
**Arguments:**
593+
* `WORKSPACE` (optional, zero or more) - Workspaces to include. If omitted, all workspaces visible to the current API key are used (via `get_workspaces()`).
594+
595+
**Options:**
596+
* `--units {month,week,day,hour}` - Chart bucket granularity (default: month). Charts always render **all-time** history at this granularity; this is unrelated to `--window` below.
597+
* `--window WINDOW` - Relative analysis window for the KPIs, e.g. `7d`, `14d`, `30d`, `90d`, `2w`, `6m`, `1y` (default: `7d`). `d`=days, `w`=weeks (×7d), `m`=months (approximated as 30 days), `y`=years (approximated as 365 days) — the approximation is intentional, since the window only needs an "installed base before window" cutoff, not calendar-exact arithmetic.
598+
* `--platforms PLATFORMS` - Comma-separated list of platforms to include (default: `em,opik,mpm`). Any of `em`, `opik`, `mpm`.
599+
* `--output PATH` - Output HTML file path (default: `growth_report.html`).
600+
* `--limit N` - Limit the number of workspaces processed (useful for a fast smoke-test run).
601+
* `--no-open` - Don't automatically open the generated HTML file after generation.
602+
603+
**Two time concepts:**
604+
* `--units` controls chart *granularity* — every chart shows the full all-time history bucketed at this resolution.
605+
* `--window` controls the *analysis window* used for the KPI numbers (Total / New / % growth) — it's drawn as a shaded band on top of the all-time charts rather than filtering them out.
606+
607+
**Growth rates:** growth/adoption rates are computed from use-case **creation timestamps**
608+
(`--window` boundary vs. each event's `created` time), not from a separate "installed base"
609+
snapshot, so `pct_growth` is always `new_in_window / count_before_window`.
610+
611+
**Caveats:**
612+
* **EM "created" is a proxy** — the Comet API has no true EM project-creation timestamp, so EM project creation is approximated as the earliest experiment start time in that project (falling back to the project's `lastUpdated` time if no experiments exist).
613+
* **Workspace/department "created" is also a proxy** — it's the earliest use-case creation timestamp seen in that workspace, across all platforms.
614+
* **MPM requires provisioning** — if the account/workspace has no MPM model-monitoring provisioned, the MPM collector degrades gracefully (an empty MPM section, `collectors.mpm: false`) instead of failing the whole report.
615+
616+
Opik and MPM collection require their optional SDKs; install them with `pip install 'cometx[all]'` if they aren't already available. Platforms whose SDK can't be imported are silently dropped from `--platforms` (a note is printed) so the report still succeeds with the remaining platforms.
617+
618+
**Examples:**
619+
```
620+
# Full cross-platform report for all workspaces, default 7-day window
621+
cometx admin growth-report
622+
623+
# Just Opik + EM, 30-day window, monthly chart granularity, for two workspaces
624+
cometx admin growth-report --platforms em,opik --window 30d my-workspace another-workspace
625+
626+
# Fast smoke-test run against a single workspace, don't auto-open
627+
cometx admin growth-report --limit 1 --no-open --output growth.html my-workspace
628+
```
629+
581630
#### gpu-report
582631

583632
Generate a GPU usage report for one or more workspaces/projects with detailed GPU metrics analysis.

cometx/cli/admin.py

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,7 @@
154154
from comet_ml import API
155155

156156
from .admin_gpu_report import main as gpu_report_main
157+
from .admin_growth_report import generate_growth_report
157158
from .admin_optimizer_report import generate_json_report
158159
from .admin_usage_report import generate_usage_report
159160

@@ -461,6 +462,79 @@ def get_parser_arguments(parser):
461462
action="store_true",
462463
)
463464

465+
# growth-report subcommand
466+
growth_report_description = """Generate a cross-platform use-case growth report for one or more workspaces.
467+
468+
Arguments:
469+
WORKSPACE (optional, one or more)
470+
One or more workspaces to run the growth report for.
471+
If not provided, all workspaces are included.
472+
473+
Output:
474+
Generates a self-contained HTML page containing cross-platform (Opik + EM + MPM)
475+
use-case growth and rate charts, broken down by workspace/department.
476+
477+
Examples:
478+
cometx admin growth-report
479+
cometx admin growth-report my-workspace
480+
cometx admin growth-report workspace1 workspace2 --units week
481+
cometx admin growth-report my-workspace --window 30d
482+
cometx admin growth-report my-workspace --platforms em,opik
483+
cometx admin growth-report my-workspace --output report.html --no-open
484+
"""
485+
growth_parser = subparsers.add_parser(
486+
"growth-report",
487+
help="Generate a cross-platform use-case growth report for one or more workspaces",
488+
description=growth_report_description,
489+
formatter_class=argparse.RawDescriptionHelpFormatter,
490+
)
491+
# Add global arguments to subparser so they show in help
492+
add_global_arguments(growth_parser)
493+
growth_parser.add_argument(
494+
"WORKSPACE",
495+
nargs="*",
496+
help="One or more workspaces to run the growth report for (empty = all)",
497+
metavar="WORKSPACE",
498+
type=str,
499+
)
500+
growth_parser.add_argument(
501+
"--units",
502+
help="Time unit for chart bucket granularity (default: month)",
503+
choices=["month", "week", "day", "hour"],
504+
default="month",
505+
type=str,
506+
)
507+
growth_parser.add_argument(
508+
"--window",
509+
help="Relative analysis window for KPIs, e.g. 7d/14d/30d/90d (default: 7d)",
510+
default="7d",
511+
type=str,
512+
)
513+
growth_parser.add_argument(
514+
"--platforms",
515+
help="Comma-separated list of platforms to include (default: em,opik,mpm)",
516+
default="em,opik,mpm",
517+
type=str,
518+
)
519+
growth_parser.add_argument(
520+
"--output",
521+
help="Output HTML file path (default: growth_report.html)",
522+
default="growth_report.html",
523+
type=str,
524+
)
525+
growth_parser.add_argument(
526+
"--limit",
527+
help="Optional limit on the number of items collected per platform",
528+
type=int,
529+
default=None,
530+
)
531+
growth_parser.add_argument(
532+
"--no-open",
533+
help="Don't automatically open the generated HTML file",
534+
default=False,
535+
action="store_true",
536+
)
537+
464538

465539
def admin(parsed_args, remaining=None):
466540
# Called via `cometx admin ...`
@@ -746,6 +820,25 @@ def admin(parsed_args, remaining=None):
746820

747821
traceback.print_exc()
748822
return
823+
elif parsed_args.ACTION == "growth-report":
824+
try:
825+
generate_growth_report(
826+
api,
827+
parsed_args.WORKSPACE,
828+
window=parsed_args.window,
829+
units=parsed_args.units,
830+
platforms=parsed_args.platforms,
831+
output=parsed_args.output,
832+
no_open=parsed_args.no_open,
833+
limit=parsed_args.limit,
834+
)
835+
except Exception as e:
836+
print("ERROR: " + str(e))
837+
if parsed_args.debug:
838+
import traceback
839+
840+
traceback.print_exc()
841+
return
749842

750843
except KeyboardInterrupt:
751844
if parsed_args.debug:

0 commit comments

Comments
 (0)