fix(bot): classify comments with one AI call - #2528
Conversation
|
github-actions
Bot
added
size/L
review/needs-review
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-playground | 7d00830 | Aug 17 2026, 12:03 PM |
@emdash-cms/admin
@emdash-cms/auth
@emdash-cms/auth-atproto
@emdash-cms/blocks
@emdash-cms/cloudflare
@emdash-cms/contentful-to-portable-text
emdash
create-emdash
@emdash-cms/gutenberg-to-portable-text
@emdash-cms/plugin-cli
@emdash-cms/plugin-types
@emdash-cms/registry-client
@emdash-cms/registry-lexicons
@emdash-cms/registry-verification
@emdash-cms/sandbox-workerd
@emdash-cms/x402
@emdash-cms/plugin-ai-moderation
@emdash-cms/plugin-atproto
@emdash-cms/plugin-audit-log
@emdash-cms/plugin-color
@emdash-cms/plugin-embeds
@emdash-cms/plugin-field-kit
@emdash-cms/plugin-forms
@emdash-cms/plugin-webhook-notifier
commit: |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-do | 7d00830 | Aug 17 2026, 12:02 PM |
Deploying with
|
| Status | Name | Latest Commit | Updated (UTC) |
|---|---|---|---|
| ✅ Deployment successful! View logs |
emdash-demo-cache | 7d00830 | Aug 17 2026, 12:03 PM |
There was a problem hiding this comment.
Pull request overview
This PR updates the @emdashbot comment classifier flow to avoid silently dropping free-text commands by performing a single typed Workers AI function-call request and reading the chosen command directly, rather than relying on Flue tool-result continuation. It also makes workers-pool integration tests deterministic by overriding classification in the test entrypoint and removing the need to bind to remote AI during those tests.
Changes:
- Replaced Flue dispatch/read-based classification with one Workers AI
run()call that returns aselect_commandtool call payload. - Added an overridable
requestClassification()hook onOrchestratorDO, and used it in the workers-pool test entrypoint to force deterministic classifier errors without remote inference. - Updated unit/integration tests and test wrangler config to align with the new classifier request path.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| infra/emdash-bot/wrangler.test.jsonc | Removes the AI binding from the workers-pool test config now that classification is overridden in the test entrypoint. |
| infra/emdash-bot/tests/unit/classifier-client.test.ts | Reworks unit tests to mock Workers AI tool-call outputs and validate new error modes. |
| infra/emdash-bot/tests/integration/webhook.test.ts | Updates integration test commentary to reflect deterministic classifier failure behavior. |
| infra/emdash-bot/tests/integration/_entry.ts | Overrides OrchestratorDO.requestClassification() to avoid remote inference and keep workers-pool tests deterministic. |
| infra/emdash-bot/.flue/lib/orchestrator.ts | Routes classification through a new requestClassification() method (defaulting to Workers AI) to enable test overrides. |
| infra/emdash-bot/.flue/lib/classifier-client.ts | Implements single-call Workers AI tool-call classification and parses select_command tool arguments into ClassifyResult. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| function selectCommandArguments(response: Ai_Cf_Qwen_Qwen3_30B_A3B_Fp8_Output): string | null { | ||
| if (typeof response !== "object" || response === null || !("choices" in response)) return null; | ||
| const choice = response.choices?.[0]; | ||
| if (!choice || !("message" in choice)) return null; | ||
| const toolCalls = choice.message?.tool_calls; | ||
| return ( | ||
| toolCalls?.find((call) => call.function.name === "select_command")?.function.arguments ?? null | ||
| ); | ||
| } |
![emdashbot[bot]](https://avatars.githubusercontent.com/in/3255304?s=60&v=4){kind=small}
There was a problem hiding this comment.
This is the right fix for the reported symptom. Routing free-text comments through a full Flue 2 agent round-trip introduced a brittle persistence/read step that could silently drop the selected command. Doing a single typed Workers AI function call inline is simpler, stateless where it needs to be, and lets the orchestrator read the result directly. I checked the changed classifier client, orchestrator integration, test harness, worker type definitions, and machine states; the implementation is internally consistent and the test override keeps workers-pool tests hermetic.
The only real concern is that the new classifier prompt/tool definition is looser than the old Flue agent's. The old agent listed none explicitly in the action list and constrained event to the known choices plus none in the tool input schema (and enforced reasoning length). The new code validates those things only after the model returns, which makes it easier for the model to emit responses that then turn into retry-causing errors. I left one suggestion to tighten the tool JSON schema and add the explicit none action to the prompt.
Aside from that, the code looks correct: the direct env.AI.run() call matches the generated Workers AI overloads, selectCommandArguments correctly extracts the function arguments, resolveClassification reuses the existing valibot schema, and the workers-pool entry overrides requestClassification so integration tests don't reach remote inference.
| const actionList = commands | ||
| .map( | ||
| (command) => | ||
| `- ${command.event}: ${command.description}${command.arg ? ` Set arg to the ${command.arg}.` : ""}`, | ||
| ) | ||
| .join("\n"); | ||
| return { | ||
| messages: [ | ||
| { | ||
| role: "system", | ||
| content: [ | ||
| "Route the comment to exactly one available action.", | ||
| "The state only limits the available list; every listed action is valid.", | ||
| "Call select_command exactly once. Prefer none over guessing. Do not answer with prose.", | ||
| ].join(" "), | ||
| }, | ||
| { | ||
| role: "user", | ||
| content: [ | ||
| `Issue: ${input.issueNumber}`, | ||
| `State: ${input.state ?? "unmanaged"}`, | ||
| "Available actions:", | ||
| actionList, | ||
| "Bot's last message:", | ||
| input.botContext?.trim() || "(none)", | ||
| "Comment:", | ||
| input.comment, | ||
| ].join("\n"), | ||
| }, | ||
| ], | ||
| tools: [ | ||
| { | ||
| type: "function", | ||
| function: { | ||
| name: "select_command", | ||
| description: "Return the single command intended by the comment, or none.", | ||
| parameters: { | ||
| type: "object", | ||
| properties: { | ||
| event: { type: "string", description: "The selected action or none" }, | ||
| arg: { type: "string", description: "The directive for the action, if any" }, | ||
| reasoning: { | ||
| type: "string", | ||
| description: "A short reason quoting the decisive phrase", | ||
| }, | ||
| }, | ||
| required: ["event", "reasoning"], | ||
| }, | ||
| }, | ||
| }, |
There was a problem hiding this comment.
[suggestion] The new classifier prompt and tool schema are weaker than the old Flue agent's, which can undermine the reliability this PR is trying to restore.
The old ClassifyCommand agent explicitly listed - \none`: no actionable intent matches the available actionsin the prompt and constrained the tool inputeventto the known choices plusnone; it also enforced reasoninglength. The new code only validates those constraints after the model returns viaresolveClassification`, so the model is more likely to return invalid events or out-of-range reasoning that become retries/errors.
Tighten the tool schema and prompt to match the old guidance:
| const actionList = commands | |
| .map( | |
| (command) => | |
| `- ${command.event}: ${command.description}${command.arg ? ` Set arg to the ${command.arg}.` : ""}`, | |
| ) | |
| .join("\n"); | |
| return { | |
| messages: [ | |
| { | |
| role: "system", | |
| content: [ | |
| "Route the comment to exactly one available action.", | |
| "The state only limits the available list; every listed action is valid.", | |
| "Call select_command exactly once. Prefer none over guessing. Do not answer with prose.", | |
| ].join(" "), | |
| }, | |
| { | |
| role: "user", | |
| content: [ | |
| `Issue: ${input.issueNumber}`, | |
| `State: ${input.state ?? "unmanaged"}`, | |
| "Available actions:", | |
| actionList, | |
| "Bot's last message:", | |
| input.botContext?.trim() || "(none)", | |
| "Comment:", | |
| input.comment, | |
| ].join("\n"), | |
| }, | |
| ], | |
| tools: [ | |
| { | |
| type: "function", | |
| function: { | |
| name: "select_command", | |
| description: "Return the single command intended by the comment, or none.", | |
| parameters: { | |
| type: "object", | |
| properties: { | |
| event: { type: "string", description: "The selected action or none" }, | |
| arg: { type: "string", description: "The directive for the action, if any" }, | |
| reasoning: { | |
| type: "string", | |
| description: "A short reason quoting the decisive phrase", | |
| }, | |
| }, | |
| required: ["event", "reasoning"], | |
| }, | |
| }, | |
| }, | |
| const actionList = [ | |
| ...commands.map( | |
| (command) => | |
| `- ${command.event}: ${command.description}${command.arg ? ` Set arg to the ${command.arg}.` : ""}`, | |
| ), | |
| "- none: no actionable intent matches the available actions", | |
| ].join("\n"); |
| const actionList = commands | |
| .map( | |
| (command) => | |
| `- ${command.event}: ${command.description}${command.arg ? ` Set arg to the ${command.arg}.` : ""}`, | |
| ) | |
| .join("\n"); | |
| return { | |
| messages: [ | |
| { | |
| role: "system", | |
| content: [ | |
| "Route the comment to exactly one available action.", | |
| "The state only limits the available list; every listed action is valid.", | |
| "Call select_command exactly once. Prefer none over guessing. Do not answer with prose.", | |
| ].join(" "), | |
| }, | |
| { | |
| role: "user", | |
| content: [ | |
| `Issue: ${input.issueNumber}`, | |
| `State: ${input.state ?? "unmanaged"}`, | |
| "Available actions:", | |
| actionList, | |
| "Bot's last message:", | |
| input.botContext?.trim() || "(none)", | |
| "Comment:", | |
| input.comment, | |
| ].join("\n"), | |
| }, | |
| ], | |
| tools: [ | |
| { | |
| type: "function", | |
| function: { | |
| name: "select_command", | |
| description: "Return the single command intended by the comment, or none.", | |
| parameters: { | |
| type: "object", | |
| properties: { | |
| event: { type: "string", description: "The selected action or none" }, | |
| arg: { type: "string", description: "The directive for the action, if any" }, | |
| reasoning: { | |
| type: "string", | |
| description: "A short reason quoting the decisive phrase", | |
| }, | |
| }, | |
| required: ["event", "reasoning"], | |
| }, | |
| }, | |
| }, | |
| properties: { | |
| event: { | |
| type: "string", | |
| enum: [...commands.map((command) => command.event), "none"], | |
| description: "The selected action or none", | |
| }, | |
| arg: { type: "string", description: "The directive for the action, if any" }, | |
| reasoning: { | |
| type: "string", | |
| minLength: 3, | |
| maxLength: 400, | |
| description: "A short reason quoting the decisive phrase", | |
| }, | |
| }, |
github-actions
Bot
added
review/awaiting-author
What does this PR do?
Fixes free-text
@emdashbotcommands being silently dropped after the classifier selected the correct action. The classifier now makes one typed Workers AI function-call request and reads the selected command directly, avoiding the failing Flue tool-result continuation.The workers-pool harness now overrides classification deterministically and no longer connects to the remote AI binding during integration tests. The existing classifier Durable Object remains in the production build for migration compatibility.
Related to #1623, where the failed bot command was observed. This PR does not close that issue.
Type of change
Checklist
pnpm typecheckpassespnpm lintpassespnpm testpasses (or targeted tests for my change)pnpm formathas been runmessages.pochanges except in translation PRs — a workflow extracts catalogs on merge tomain.Typecheck note: the bot package's existing
wrangler typesoutput lacks the generated Flue Durable Object and service-binding RPC types, so the full package typecheck fails across the existing DO tests and routes. It reports no errors in the changed classifier files.i18n is not applicable because this does not change admin UI. A changeset is not applicable because
infra/emdash-botis private infrastructure. A Discussion is not required for this bug fix.AI-generated code disclosure
Screenshots / test output
Not visual.
pnpm buildpnpm -s lint:json | jq '.diagnostics | length'→0pnpm --filter @emdash-cms/emdash-bot test:unit→ 220 passedpnpm --filter @emdash-cms/emdash-bot test:workers→ 46 passedpnpm --filter @emdash-cms/emdash-bot buildwrangler.test.jsoncTry this PR
Open a fresh playground →
A full working EmDash site, deployed from this branch. Each visit gets its own session-scoped sandbox: no login needed and no shared state. Try the admin, edit content, hit the public site.
Tracks
fix/bot-direct-classifier. Updated automatically when the playground redeploys.