fix: connector-delivery reconciler overwrites real responses from agents using non-connector tools - #99
Closed
siglimumuni wants to merge 1 commit into
Conversation
…delivery
reconcileConnectorDeliveryText was overwriting an agent's actual response
with a false-negative whenever:
1. the response language matched "positive delivery" patterns
(sent/delivered/scheduled + recipient/message id/connector context),
2. no connector_message_tool event was recorded for the turn,
3. **regardless of whether other tool events happened.**
Effect: any agent that sends email/SMS/etc. via the `execute` tool
(nodemailer in a shell, curl POST to a webhook, gh issue create, etc.)
loses their real response — overwritten with "I couldn't confirm that
the configured connector actually sent anything." This is wrong twice:
the agent did do real work, and the response the user sees is now a
lie in the other direction.
Concrete repro: configure an agent with a Proton-Bridge SMTP credential
injected as PROTON_SMTP_JSON. Agent writes a draft, runs nodemailer
from the `execute` shell, gets a real messageId back, posts a
confirmation in #feedback. SwarmClaw silently rewrites the confirmation
to the canned false-negative. Email actually arrived. User sees the
overwrite, thinks no email was sent, tries again, ends up with
duplicates and confusion.
Fix: when no connector_message_tool event was recorded, only emit the
false-negative if NO tool events ran at all for the turn. Any tool
event present is evidence the agent did some real work and the
reconciler shouldn't blindly overwrite. The connector-failure path
(connector events present but none succeeded) is unchanged — that one
has genuine evidence of failure to surface.
This narrows the reconciler's scope to its documented purpose: catch
agents that hallucinate sends without calling any tool at all. Agents
that legitimately use non-connector tools are no longer collateral
damage.
Files:
- src/lib/server/chat-execution/chat-execution-connector-delivery.ts
Member
|
Cherry-picked and shipped in v1.9.33 with connector sanitization coverage and non-connector delivery preservation. Thanks for the fix. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
`reconcileConnectorDeliveryText` is meant to catch agents that hallucinate "I sent your message" without actually invoking the connector tool. As written, it overwrites the agent's response with a canned false-negative whenever:
sent/delivered/scheduled+ recipient / message id / connector context), ANDEffect: any agent that sends through a non-connector path — `execute` running `nodemailer` against an SMTP server, `curl` against a webhook endpoint, `gh issue create` for GitHub triage, future connectors that don't route through `connector_message_tool` — loses its real response, replaced with
"I couldn't confirm that the configured connector actually sent anything."The agent did the work; the user sees a lie in the other direction.Concrete repro
In our deployment we wired a customer-service agent to send outbound email via Proton Bridge using `nodemailer` from her shell (credentials injected as `PROTON_SMTP_JSON` env). The agent writes a draft, runs nodemailer from `execute`, gets a real `messageId` back, posts a confirmation thread reply:
"✓ Reply sent to gmail.com, messageId: <id>". Reconciler sees `sent` + `recipient`-adjacent context, no `connector_message_tool` event, overwrites the entire response with the canned false-negative. The email arrived. The user sees the canned line and resends, producing duplicates.Fix
When no `connector_message_tool` event was recorded for the turn, only emit the false-negative when no tool events ran at all. Any tool event present is evidence the agent did real work; the reconciler shouldn't blindly overwrite. The connector-failure branch (connector events present but none succeeded) is unchanged — that path has genuine evidence to surface.
This narrows the reconciler's scope to its actual purpose: catch agents that hallucinate sends without invoking any tool. Agents that use non-connector tools to deliver are no longer collateral damage.
```diff
if (connectorEvents.length === 0) {
return `I couldn't confirm that the configured connector actually sent anything. No connector delivery tool call was recorded for this response.`
}
```
Workaround we shipped meanwhile
Updated the agent's system prompt to use language that doesn't trip the regex (`Reply queued via Proton bridge` instead of `Reply sent to `). Works as a prompt-discipline tax but the reconciler shouldn't be forcing it on every agent that uses an alternate send path.
Test plan
Files
🤖 Generated with Claude Code