Skip to content

Commit 196889c

Browse files
jverreclaude
andcommitted
docs(cost-intelligence): address review feedback on accuracy claims
- Pilot stage: scoping comes from install footprint, not group-scoped policy - Validation corpus: internal usage and opted-in customers only - Denied MCP servers stay off; only skills re-enable by name - macOS app touches agent certificate trust; drop 'agent untouched' - Scope the overview flow to Claude Code on the plugin path - Update the stale fallback framing on the MDM page's macOS app card Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
1 parent b957a9a commit 196889c

6 files changed

Lines changed: 11 additions & 9 deletions

File tree

apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/install/macos-app.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ The data landing in Opik is the same as the plugin produces. What differs is how
2424
| --- | --- | --- |
2525
| How traffic arrives | Agent is pointed at a local port | The OS diverts the flows |
2626
| Captures | Claude Code CLI | CLI **and** the Claude Code desktop app |
27-
| Configuration touches | The agent's settings | The app only — the agent is untouched |
27+
| Configuration touches | The agent's settings | Certificate trust only, set up by the app |
2828
| Delivered via | MDM or Claude managed settings | MDM |
2929
| TLS | Not intercepted | Terminated locally |
3030
| Platforms | macOS, Linux, Windows | macOS only |

apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/install/mdm.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -174,6 +174,6 @@ Then, on each device, `opik-cipx uninstall` removes the hook, stops the daemon,
174174
icon="fa-regular fa-shield-halved"
175175
href="/cost-intelligence/install/macos-app"
176176
>
177-
Transparent capture when no client configuration can be delivered.
177+
Transparent capture of every user, including the Claude Code desktop app.
178178
</Card>
179179
</CardGroup>

apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/install/overview.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ Which one fits:
5050
| **Platforms** | macOS, Linux, Windows | macOS, Linux, Windows | macOS only |
5151
| **Who receives it** | Whichever devices or groups you target | Every authenticated user in the org | Targeted devices |
5252
| **Staged / pilot rollout** | Yes | No, all users at once | Yes |
53-
| **Agent config needed** | Yes (delivered for you) | Yes (delivered for you) | None — the agent is untouched |
53+
| **Agent config needed** | Yes (delivered for you) | Yes (delivered for you) | None to deliver — cert trust set up on-device |
5454
| **Intercepts TLS** | No | No | Yes, locally |
5555
| **User can disable it** | No, enforced | No, enforced | No, enforced |
5656
| **Effort** | Low | Lowest | Highest |

apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/overview.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ Cost Intelligence closes that gap. It captures every coding-agent API call on th
2222

2323
## How it works
2424

25-
Each developer machine runs its own local `opik-cipx` daemon. The coding agent talks to it over the loopback interface, and the daemon forwards every call to the provider unchanged — there is no shared collector, and none of your traffic routes through Comet. What ships to your Opik workspace is a separate, asynchronous stream of metadata-only spans: token counts, costs, and structure, [never content](/cost-intelligence/data-privacy-security).
25+
Each developer machine runs its own local `opik-cipx` daemon. The coding agent talks to it over the loopback interface, and the daemon forwards every call to the provider unchanged — there is no shared collector, and none of your traffic routes through Comet. What ships to your Opik workspace is a separate, asynchronous stream of metadata-only spans: token counts, costs, and structure, [never content](/cost-intelligence/data-privacy-security). The diagram shows Claude Code on the plugin path, the most common setup; see [Installation](/cost-intelligence/install/overview) for the other agents and rollout paths.
2626

2727
<Frame>
2828
<img src="/img/v2/cost-intelligence/architecture.svg" alt="Cost Intelligence architecture: a local opik-cipx daemon on each developer machine forwards Claude Code traffic unchanged to Anthropic and ships metadata-only spans to Opik" />

apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/reduce-agent-spend.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,7 @@ Cost Intelligence closes that loop. It shows you where the money went, prices wh
2222
The loop below works at any scale, but you don't have to run it fleet-wide on day one. The lowest-risk way in is staged:
2323

2424
1. **Observe first.** Roll out to a pilot group and collect data only — no policies yet. You learn where the money goes before anything changes.
25-
2. **Apply to the pilot.** Review the recommendations priced from the pilot's own traffic and apply the ones you approve, to that group only.
25+
2. **Apply to the pilot.** Review the recommendations priced from the pilot's own traffic and apply the ones you approve. Policies only reach machines where Cost Intelligence is installed, so while only the pilot is rolled out, the changes land on the pilot's machines alone.
2626
3. **Widen.** Expand observability and the approved policies to the whole organization.
2727
4. **Tune per user.** Use [user policies](/cost-intelligence/roll-out-cost-policies) to handle the heaviest spenders and the teams that genuinely need different settings.
2828

@@ -88,7 +88,7 @@ A recommendation that cuts spend by making the agent worse is not a saving, so e
8888

8989
**Against benchmarks.** We evaluate agent performance on open and private benchmarks — TerminalBench among them — before and after each change, so a method that degrades what the agent can do never becomes a recommendation.
9090

91-
**Against real sessions.** We maintain a body of tens of thousands of real, labeled coding-agent sessions that are replayable inside Opik. Every method is tuned against it offline, in large experiments, and monitored online with LLM-as-a-judge evaluation of real outcomes. As agents and usage patterns evolve, the same loop keeps the recommendations current.
91+
**Against real sessions.** We maintain a body of tens of thousands of real, labeled coding-agent sessions — drawn from our own internal usage and from customers who explicitly opt in — that are replayable inside Opik. Every method is tuned against it offline, in large experiments, and monitored online with LLM-as-a-judge evaluation of real outcomes. As agents and usage patterns evolve, the same loop keeps the recommendations current.
9292

9393
## Next steps
9494

apps/opik-documentation/documentation/fern/docs-v2/cost_intelligence/roll-out-cost-policies.mdx

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,13 @@ Delivery is automatic once you are set up. Bootstrap configuration (workspace, A
2929

3030
<Note>
3131
**Disabling a skill doesn't take it away.** The most common recommendation —
32-
turning off unused skills and MCP servers — only keeps them out of the
32+
turning off unused skills and MCP servers — removes their weight from the
3333
context that ships with every request. Nothing is uninstalled, and a disabled
3434
skill stays callable: when a developer asks for it by name, Claude Code
35-
re-enables it and uses it. The saving comes from the thousands of requests
36-
that never needed it, not from taking capability away.
35+
re-enables it and uses it. A denied MCP server does stay off until the policy
36+
changes, which is why the recommendation only targets servers nobody has been
37+
using. The saving comes from the thousands of requests that never needed
38+
them.
3739
</Note>
3840

3941
## Grant exceptions per user

0 commit comments

Comments
 (0)