[NA] [DOCS] docs: address Cost Intelligence docs review feedback - #7920
[NA] [DOCS] docs: address Cost Intelligence docs review feedback#7920jverre wants to merge 1 commit into
Conversation
- Pilot stage: scoping comes from install footprint, not group-scoped policy - Validation corpus: internal usage and opted-in customers only - Denied MCP servers stay off; only skills re-enable by name - macOS app touches agent certificate trust; drop 'agent untouched' - Scope the overview flow to Claude Code on the plugin path - Update the stale fallback framing on the MDM page's macOS app card Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
github-actions
Bot
added
the
documentation
⏱️ pre-commit per-hook timingNo linted files changed — nothing to run. ⏭️ 43 skipped (no matching files changed)
|
|
🌿 Preview your docs: https://opik-preview-01a01ad0-6d4c-7155-ac2b-b3397f7cc928.docs.buildwithfern.com/docs/opik No broken links found Unverified links (timeout / rate-limited / server error — not failing the check)• https://aistudio.google.com/apikey (401) 📌 Results for commit 8803d16 |
![baz-reviewer[bot]](https://avatars.githubusercontent.com/in/933528?s=60&v=4){kind=small}
| ## How it works | ||
|
|
||
| Each developer machine runs its own local `opik-cipx` daemon. The coding agent talks to it over the loopback interface, and the daemon forwards every call to the provider unchanged — there is no shared collector, and none of your traffic routes through Comet. What ships to your Opik workspace is a separate, asynchronous stream of metadata-only spans: token counts, costs, and structure, [never content](/cost-intelligence/data-privacy-security). | ||
| Each developer machine runs its own local `opik-cipx` daemon. The coding agent talks to it over the loopback interface, and the daemon forwards every call to the provider unchanged — there is no shared collector, and none of your traffic routes through Comet. What ships to your Opik workspace is a separate, asynchronous stream of metadata-only spans: token counts, costs, and structure, [never content](/cost-intelligence/data-privacy-security). The diagram shows Claude Code on the plugin path, the most common setup; see [Installation](/cost-intelligence/install/overview) for the other agents and rollout paths. |
There was a problem hiding this comment.
Supported agents lack valid setup paths
The supported-agent pointer sends Codex and Cursor users to Installation, which documents only the Claude Code plugin flow, so they may apply the wrong setup — should we link distinct Codex/Cursor instructions or narrow the supported-agent claim until those paths are documented?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/overview.mdx around
lines 25-25, revise the “How it works” installation pointer so it does not send
Codex and Cursor users to Claude Code-only instructions. Link to distinct Codex and
Cursor Cost Intelligence setup guides, including their metadata/export configuration, or
narrow the supported-agent wording at lines 81-85 until those guides exist; preserve the
Claude Code installation link for Claude users.
| href="/cost-intelligence/install/macos-app" | ||
| > | ||
| Transparent capture when no client configuration can be delivered. | ||
| Transparent capture of every user, including the Claude Code desktop app. |
There was a problem hiding this comment.
MDM capture coverage is overstated
The card promises “every user,” but capture is limited to users on targeted Macs with ~/.opik-cipx/config.toml, so non-targeted or non-Mac users and unprovisioned shared-device accounts remain uncaptured — should we qualify it as “every user on each targeted Mac” or document a supported system-wide credential path?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/install/mdm.mdx`
around lines 177-177, revise the MDM card description for macOS capture because “every
user” incorrectly implies coverage across all users and platforms. Qualify it to state
that it covers users on each targeted Mac who have the required per-user
`~/.opik-cipx/config.toml` configuration, unless a supported system-wide credential path
is documented.
| **Disabling a skill doesn't take it away.** The most common recommendation — | ||
| turning off unused skills and MCP servers — only keeps them out of the | ||
| turning off unused skills and MCP servers — removes their weight from the | ||
| context that ships with every request. Nothing is uninstalled, and a disabled | ||
| skill stays callable: when a developer asks for it by name, Claude Code | ||
| re-enables it and uses it. The saving comes from the thousands of requests | ||
| that never needed it, not from taking capability away. | ||
| re-enables it and uses it. A denied MCP server does stay off until the policy |
There was a problem hiding this comment.
Disabled skills remain unusable
The note claims that skillOverrides: "off" re-enables a policy-disabled skill when invoked by name, but Claude Code’s skillOverrides: {"name": "off"} behavior hides it and returns a skillOverrides error. disable-model-invocation: true is a separate control that still allows /skill-name; should we document these semantics accurately, retain the deniedMcpServers distinction, or change the implementation if reactivation is intended?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/roll-out-cost-policies.mdx
around lines 31-35, update the Note to remove the claim that Claude Code re-enables
policy-disabled skills when invoked by name. Document that skillOverrides set to off
hides the skill and causes full-name invocation to return a skillOverrides error, and
distinguish this from disable-model-invocation: true, which prevents automatic loading
but still permits explicit user invocation. Keep the separate statement that denied MCP
servers remain blocked until policy changes.
|
|
||
| 1. **Observe first.** Roll out to a pilot group and collect data only — no policies yet. You learn where the money goes before anything changes. | ||
| 2. **Apply to the pilot.** Review the recommendations priced from the pilot's own traffic and apply the ones you approve, to that group only. | ||
| 2. **Apply to the pilot.** Review the recommendations priced from the pilot's own traffic and apply the ones you approve. Policies only reach machines where Cost Intelligence is installed, so while only the pilot is rolled out, the changes land on the pilot's machines alone. |
There was a problem hiding this comment.
Pilot policy reaches non-pilot users
The pilot is described as applying only to pilot machines, but shared-device MDM settings reach every signed-in user, so users outside the pilot are included and pilot-priced recommendations are not pilot-specific — should we state the device scope and shared-machine impact explicitly, or define it as device-targeted?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/reduce-agent-spend.mdx
around line 25, revise the “Apply to the pilot” rollout step to accurately describe
device-level policy scope. State that policies apply to every user who signs in on a
targeted machine, warn that shared pilot devices can affect users outside the pilot, and
frame the pilot as device-targeted rather than implying recommendations or effects are
limited to pilot users.
| **Against benchmarks.** We evaluate agent performance on open and private benchmarks — TerminalBench among them — before and after each change, so a method that degrades what the agent can do never becomes a recommendation. | ||
|
|
||
| **Against real sessions.** We maintain a body of tens of thousands of real, labeled coding-agent sessions that are replayable inside Opik. Every method is tuned against it offline, in large experiments, and monitored online with LLM-as-a-judge evaluation of real outcomes. As agents and usage patterns evolve, the same loop keeps the recommendations current. | ||
| **Against real sessions.** We maintain a body of tens of thousands of real, labeled coding-agent sessions — drawn from our own internal usage and from customers who explicitly opt in — that are replayable inside Opik. Every method is tuned against it offline, in large experiments, and monitored online with LLM-as-a-judge evaluation of real outcomes. As agents and usage patterns evolve, the same loop keeps the recommendations current. |
There was a problem hiding this comment.
Customer session privacy boundaries undocumented
The new claim promises opted-in, replayable, outcome-judged customer sessions, but the docs cover only CIPX_CAPTURE_CONTENT while the visible ingestion path is metadata-only and documents no consent or access, retention, or deletion lifecycle, so readers may infer an approved reuse flow that cannot replay or evaluate sessions without exposing prompt, response, tool, and repository content. Should we qualify this as an internal/unshipped process or document consent, workspace segregation, evaluator access, retention, revocation, and the captured fields and protections?
Want Baz to fix this for you? Activate Fixer
Other fix methods
Prompt for AI Agents
Before applying, verify this suggestion against the current code. In
`apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/reduce-agent-spend.mdx`
around lines 91-91, revise the “Against real sessions” paragraph so it does not
present customer-session replay, content capture, or LLM judging as an established
customer-facing capability. Qualify the corpus as an internal or unshipped process that
does not use customer content, or explicitly document the required opt-in/consent
lifecycle, workspace isolation, captured fields, evaluator access, retention, deletion,
and revocation controls before making these claims.
1 participant
Details
Follow-up to #7904 (merged), addressing the review comments left on it. Prose-only accuracy fixes across the Cost Intelligence docs:
Review comments on the SVG diagrams were intentionally not addressed (graphics excluded from this pass).
Change checklist
Issues
AI-WATERMARK
AI-WATERMARK: yes
Testing
Documentation
apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/overview.mdxapps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/reduce-agent-spend.mdxapps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/roll-out-cost-policies.mdxapps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/install/overview.mdxapps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/install/macos-app.mdxapps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/install/mdm.mdx🤖 Generated with Claude Code