Skip to content

Feat/cf dev cutover#234

Draft
breardon2011 wants to merge 4 commits intomainfrom
feat/cf-dev-cutover
Draft

Feat/cf dev cutover#234
breardon2011 wants to merge 4 commits intomainfrom
feat/cf-dev-cutover

Conversation

@breardon2011
Copy link
Copy Markdown
Contributor

No description provided.

@mintlify
Copy link
Copy Markdown

mintlify Bot commented May 7, 2026

Preview deployment for your docs. Learn more about Mintlify Previews.

Project Status Preview Updated (UTC)
opencomputer 🟢 Ready View Preview May 7, 2026, 9:17 PM

💡 Tip: Enable Workflows to automatically generate PRs for you.

@2027-evals
Copy link
Copy Markdown

2027-evals Bot commented May 7, 2026

✅ Eval complete for commit ce98ab3

URL Mapping
digger.dev opensandbox-feat-cf-dev-cutover.mintlify.app
opencomputer.dev opensandbox-feat-cf-dev-cutover.mintlify.app

Ran evals with prompts:

📉 -10.8 pts · complete the getting started guide at https://opencomputer.d — C (65.4/100) from B (76.2) · View metrics

Prompt text:

complete the getting started guide at https://opencomputer.dev

Verdict:

OpenComputer has solid documentation and a well-typed SDK, but an AI-opaque SPA homepage and missing redirects create significant discovery friction that inflates agent tool-call counts well beyond what the task requires.

Friction points:

  • 🔴 Main marketing site (opencomputer — .dev) is a JavaScript SPA with no server-rendered content — AI agents and crawlers receive only a one-line title, making the docs completely undiscoverable from the primary domain.
  • 🔴 The URL path /getting-started returns a 404; the actual quickstart lives at /quickstart with no redirect, causing agent detours and wasted tool calls — .
  • 🔴 The quickstart code example uses sandbox — .commands.run() which is marked @deprecated in the TypeScript types in favor of sandbox.exec — the guide should use the current API surface.
  • 🟡 No OpenAPI / Swagger spec is published, limiting automated client generation and making it harder for AI agents to reason about the full REST API surface — .
  • 🟡 The docs are not indexed on Context7, reducing the SDK's presence in AI training data and retrieval-augmented documentation toolchains — .

Result: C (65.4/100)

delta vs baseline: -10.8 pts

Dimension Baseline This PR
Setup Friction 86 86
Speed 75 67
Efficiency 54 23
Error Recovery 100 93
Doc Quality 70 60

Stats: 3m 40s · 31 tool calls · 1 error · 1 interruption · $1.80

View report → · View trace →


Evaluating agent experience using 2027.dev · View dashboard

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant