Skip to content

[NA] [DOCS] docs: address Cost Intelligence docs review feedback #7920

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
jverre wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

[NA] [DOCS] docs: address Cost Intelligence docs review feedback #7920

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...ocumentation/documentation/fern/docs-v2/cost_intelligence/install/macos-app.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ The data landing in Opik is the same as the plugin produces. What differs is how
| --- | --- | --- |
| How traffic arrives | Agent is pointed at a local port | The OS diverts the flows |
| Captures | Claude Code CLI | CLI **and** the Claude Code desktop app |
| Configuration touches | The agent's settings | The app only — the agent is untouched |
| Configuration touches | The agent's settings | Certificate trust only, set up by the app |
| Delivered via | MDM or Claude managed settings | MDM |
| TLS | Not intercepted | Terminated locally |
| Platforms | macOS, Linux, Windows | macOS only |
Expand Down
2 changes: 1 addition & 1 deletion ...opik-documentation/documentation/fern/docs-v2/cost_intelligence/install/mdm.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,6 +174,6 @@ Then, on each device, `opik-cipx uninstall` removes the hook, stops the daemon,
icon="fa-regular fa-shield-halved"
href="/cost-intelligence/install/macos-app"
>
Transparent capture when no client configuration can be delivered.
Transparent capture of every user, including the Claude Code desktop app.

@baz-reviewer baz-reviewer Bot Aug 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MDM capture coverage is overstated

The card promises “every user,” but capture is limited to users on targeted Macs with ~/.opik-cipx/config.toml, so non-targeted or non-Mac users and unprovisioned shared-device accounts remain uncaptured — should we qualify it as “every user on each targeted Mac” or document a supported system-wide credential path?

Severity

Want Baz to fix this for you? Activate Fixer

Other fix methods

Fix in Cursor

Prompt for AI Agents 
Before applying, verify this suggestion against the current code. In
`apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/install/mdm.mdx`
around lines 177-177, revise the MDM card description for macOS capture because “every
user” incorrectly implies coverage across all users and platforms. Qualify it to state
that it covers users on each targeted Mac who have the required per-user
`~/.opik-cipx/config.toml` configuration, unless a supported system-wide credential path
is documented.

</Card>
</CardGroup>
2 changes: 1 addition & 1 deletion ...documentation/documentation/fern/docs-v2/cost_intelligence/install/overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Which one fits:
| **Platforms** | macOS, Linux, Windows | macOS, Linux, Windows | macOS only |
| **Who receives it** | Whichever devices or groups you target | Every authenticated user in the org | Targeted devices |
| **Staged / pilot rollout** | Yes | No, all users at once | Yes |
| **Agent config needed** | Yes (delivered for you) | Yes (delivered for you) | None — the agent is untouched |
| **Agent config needed** | Yes (delivered for you) | Yes (delivered for you) | None to deliver — cert trust set up on-device |
| **Intercepts TLS** | No | No | Yes, locally |
| **User can disable it** | No, enforced | No, enforced | No, enforced |
| **Effort** | Low | Lowest | Highest |
Expand Down
2 changes: 1 addition & 1 deletion apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/overview.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Cost Intelligence closes that gap. It captures every coding-agent API call on th

## How it works

Each developer machine runs its own local `opik-cipx` daemon. The coding agent talks to it over the loopback interface, and the daemon forwards every call to the provider unchanged — there is no shared collector, and none of your traffic routes through Comet. What ships to your Opik workspace is a separate, asynchronous stream of metadata-only spans: token counts, costs, and structure, [never content](/cost-intelligence/data-privacy-security).
Each developer machine runs its own local `opik-cipx` daemon. The coding agent talks to it over the loopback interface, and the daemon forwards every call to the provider unchanged — there is no shared collector, and none of your traffic routes through Comet. What ships to your Opik workspace is a separate, asynchronous stream of metadata-only spans: token counts, costs, and structure, [never content](/cost-intelligence/data-privacy-security). The diagram shows Claude Code on the plugin path, the most common setup; see [Installation](/cost-intelligence/install/overview) for the other agents and rollout paths.

@baz-reviewer baz-reviewer Bot Aug 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Supported agents lack valid setup paths

The supported-agent pointer sends Codex and Cursor users to Installation, which documents only the Claude Code plugin flow, so they may apply the wrong setup — should we link distinct Codex/Cursor instructions or narrow the supported-agent claim until those paths are documented?

Severity

Want Baz to fix this for you? Activate Fixer

Other fix methods

Fix in Cursor

Prompt for AI Agents 
Before applying, verify this suggestion against the current code. In
apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/overview.mdx around
lines 25-25, revise the “How it works” installation pointer so it does not send
Codex and Cursor users to Claude Code-only instructions. Link to distinct Codex and
Cursor Cost Intelligence setup guides, including their metadata/export configuration, or
narrow the supported-agent wording at lines 81-85 until those guides exist; preserve the
Claude Code installation link for Claude users.


<Frame>
<img src="/img/v2/cost-intelligence/architecture.svg" alt="Cost Intelligence architecture: a local opik-cipx daemon on each developer machine forwards Claude Code traffic unchanged to Anthropic and ships metadata-only spans to Opik" />
Expand Down
4 changes: 2 additions & 2 deletions ...cumentation/documentation/fern/docs-v2/cost_intelligence/reduce-agent-spend.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ Cost Intelligence closes that loop. It shows you where the money went, prices wh
The loop below works at any scale, but you don't have to run it fleet-wide on day one. The lowest-risk way in is staged:

1. **Observe first.** Roll out to a pilot group and collect data only — no policies yet. You learn where the money goes before anything changes.
2. **Apply to the pilot.** Review the recommendations priced from the pilot's own traffic and apply the ones you approve, to that group only.
2. **Apply to the pilot.** Review the recommendations priced from the pilot's own traffic and apply the ones you approve. Policies only reach machines where Cost Intelligence is installed, so while only the pilot is rolled out, the changes land on the pilot's machines alone.

@baz-reviewer baz-reviewer Bot Aug 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pilot policy reaches non-pilot users

The pilot is described as applying only to pilot machines, but shared-device MDM settings reach every signed-in user, so users outside the pilot are included and pilot-priced recommendations are not pilot-specific — should we state the device scope and shared-machine impact explicitly, or define it as device-targeted?

Severity

Want Baz to fix this for you? Activate Fixer

Other fix methods

Fix in Cursor

Prompt for AI Agents 
Before applying, verify this suggestion against the current code. In
apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/reduce-agent-spend.mdx
around line 25, revise the “Apply to the pilot” rollout step to accurately describe
device-level policy scope. State that policies apply to every user who signs in on a
targeted machine, warn that shared pilot devices can affect users outside the pilot, and
frame the pilot as device-targeted rather than implying recommendations or effects are
limited to pilot users.

3. **Widen.** Expand observability and the approved policies to the whole organization.
4. **Tune per user.** Use [user policies](/cost-intelligence/roll-out-cost-policies) to handle the heaviest spenders and the teams that genuinely need different settings.

Expand Down Expand Up @@ -88,7 +88,7 @@ A recommendation that cuts spend by making the agent worse is not a saving, so e

**Against benchmarks.** We evaluate agent performance on open and private benchmarks — TerminalBench among them — before and after each change, so a method that degrades what the agent can do never becomes a recommendation.

**Against real sessions.** We maintain a body of tens of thousands of real, labeled coding-agent sessions that are replayable inside Opik. Every method is tuned against it offline, in large experiments, and monitored online with LLM-as-a-judge evaluation of real outcomes. As agents and usage patterns evolve, the same loop keeps the recommendations current.
**Against real sessions.** We maintain a body of tens of thousands of real, labeled coding-agent sessions — drawn from our own internal usage and from customers who explicitly opt in — that are replayable inside Opik. Every method is tuned against it offline, in large experiments, and monitored online with LLM-as-a-judge evaluation of real outcomes. As agents and usage patterns evolve, the same loop keeps the recommendations current.

@baz-reviewer baz-reviewer Bot Aug 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Customer session privacy boundaries undocumented

The new claim promises opted-in, replayable, outcome-judged customer sessions, but the docs cover only CIPX_CAPTURE_CONTENT while the visible ingestion path is metadata-only and documents no consent or access, retention, or deletion lifecycle, so readers may infer an approved reuse flow that cannot replay or evaluate sessions without exposing prompt, response, tool, and repository content. Should we qualify this as an internal/unshipped process or document consent, workspace segregation, evaluator access, retention, revocation, and the captured fields and protections?

Severity

Want Baz to fix this for you? Activate Fixer

Other fix methods

Fix in Cursor

Prompt for AI Agents 
Before applying, verify this suggestion against the current code. In
`apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/reduce-agent-spend.mdx`
around lines 91-91, revise the “Against real sessions” paragraph so it does not
present customer-session replay, content capture, or LLM judging as an established
customer-facing capability. Qualify the corpus as an internal or unshipped process that
does not use customer content, or explicitly document the required opt-in/consent
lifecycle, workspace isolation, captured fields, evaluator access, retention, deletion,
and revocation controls before making these claims.


## Next steps

Expand Down
8 changes: 5 additions & 3 deletions ...ntation/documentation/fern/docs-v2/cost_intelligence/roll-out-cost-policies.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,11 +29,13 @@ Delivery is automatic once you are set up. Bootstrap configuration (workspace, A

<Note>
**Disabling a skill doesn't take it away.** The most common recommendation —
turning off unused skills and MCP servers — only keeps them out of the
turning off unused skills and MCP servers — removes their weight from the
context that ships with every request. Nothing is uninstalled, and a disabled
skill stays callable: when a developer asks for it by name, Claude Code
re-enables it and uses it. The saving comes from the thousands of requests
that never needed it, not from taking capability away.
re-enables it and uses it. A denied MCP server does stay off until the policy
Comment on lines 31 to +35

@baz-reviewer baz-reviewer Bot Aug 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Disabled skills remain unusable

The note claims that skillOverrides: "off" re-enables a policy-disabled skill when invoked by name, but Claude Code’s skillOverrides: {"name": "off"} behavior hides it and returns a skillOverrides error. disable-model-invocation: true is a separate control that still allows /skill-name; should we document these semantics accurately, retain the deniedMcpServers distinction, or change the implementation if reactivation is intended?

Severity web_search

Want Baz to fix this for you? Activate Fixer

Other fix methods

Fix in Cursor

Prompt for AI Agents 
Before applying, verify this suggestion against the current code. In
apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/roll-out-cost-policies.mdx
around lines 31-35, update the Note to remove the claim that Claude Code re-enables
policy-disabled skills when invoked by name. Document that skillOverrides set to off
hides the skill and causes full-name invocation to return a skillOverrides error, and
distinguish this from disable-model-invocation: true, which prevents automatic loading
but still permits explicit user invocation. Keep the separate statement that denied MCP
servers remain blocked until policy changes.

changes, which is why the recommendation only targets servers nobody has been
using. The saving comes from the thousands of requests that never needed
them.
</Note>

## Grant exceptions per user
Expand Down
Loading