Skip to content

fix: connector-delivery reconciler overwrites real responses from agents using non-connector tools - #99

Closed
siglimumuni wants to merge 1 commit into
swarmclawai:mainfrom
siglimumuni:fix/connector-delivery-reconciler-execute-tool
Closed

fix: connector-delivery reconciler overwrites real responses from agents using non-connector tools#99
siglimumuni wants to merge 1 commit into
swarmclawai:mainfrom
siglimumuni:fix/connector-delivery-reconciler-execute-tool

Conversation

@siglimumuni

Copy link
Copy Markdown
Contributor

Summary

`reconcileConnectorDeliveryText` is meant to catch agents that hallucinate "I sent your message" without actually invoking the connector tool. As written, it overwrites the agent's response with a canned false-negative whenever:

  1. The response language matches its positive-delivery regex (sent / delivered / scheduled + recipient / message id / connector context), AND
  2. No `connector_message_tool` event was recorded for the turn — regardless of whether other tool events happened.

Effect: any agent that sends through a non-connector path — `execute` running `nodemailer` against an SMTP server, `curl` against a webhook endpoint, `gh issue create` for GitHub triage, future connectors that don't route through `connector_message_tool` — loses its real response, replaced with "I couldn't confirm that the configured connector actually sent anything." The agent did the work; the user sees a lie in the other direction.

Concrete repro

In our deployment we wired a customer-service agent to send outbound email via Proton Bridge using `nodemailer` from her shell (credentials injected as `PROTON_SMTP_JSON` env). The agent writes a draft, runs nodemailer from `execute`, gets a real `messageId` back, posts a confirmation thread reply: "✓ Reply sent to gmail.com, messageId: <id>". Reconciler sees `sent` + `recipient`-adjacent context, no `connector_message_tool` event, overwrites the entire response with the canned false-negative. The email arrived. The user sees the canned line and resends, producing duplicates.

Fix

When no `connector_message_tool` event was recorded for the turn, only emit the false-negative when no tool events ran at all. Any tool event present is evidence the agent did real work; the reconciler shouldn't blindly overwrite. The connector-failure branch (connector events present but none succeeded) is unchanged — that path has genuine evidence to surface.

This narrows the reconciler's scope to its actual purpose: catch agents that hallucinate sends without invoking any tool. Agents that use non-connector tools to deliver are no longer collateral damage.

```diff
if (connectorEvents.length === 0) {

  • // No connector_message_tool event was recorded for this turn, but the
  • // agent may have legitimately delivered the message through another tool
  • // — typically execute running nodemailer / smtp / curl, or a future
  • // connector that doesn't route through connector_message_tool. If any
  • // tool ran this turn, the agent did real work; trust them rather than
  • // overwriting with a false-negative. Only reconcile when there is
  • // genuine evidence of hallucination (no tools called at all).
  • if (dedupedEvents.length > 0) return text
    return `I couldn't confirm that the configured connector actually sent anything. No connector delivery tool call was recorded for this response.`
    }
    ```

Workaround we shipped meanwhile

Updated the agent's system prompt to use language that doesn't trip the regex (`Reply queued via Proton bridge` instead of `Reply sent to `). Works as a prompt-discipline tax but the reconciler shouldn't be forcing it on every agent that uses an alternate send path.

Test plan

  • Reviewed by inspection.
  • Applied locally; agent's nodemailer-based send no longer gets its confirmation overwritten.
  • Once merged: revert the prompt workaround on our customer-service agent and confirm reconciler stays quiet.

Files

  • `src/lib/server/chat-execution/chat-execution-connector-delivery.ts` (12 lines, one branch added)

🤖 Generated with Claude Code

…delivery

reconcileConnectorDeliveryText was overwriting an agent's actual response
with a false-negative whenever:
  1. the response language matched "positive delivery" patterns
     (sent/delivered/scheduled + recipient/message id/connector context),
  2. no connector_message_tool event was recorded for the turn,
  3. **regardless of whether other tool events happened.**

Effect: any agent that sends email/SMS/etc. via the `execute` tool
(nodemailer in a shell, curl POST to a webhook, gh issue create, etc.)
loses their real response — overwritten with "I couldn't confirm that
the configured connector actually sent anything." This is wrong twice:
the agent did do real work, and the response the user sees is now a
lie in the other direction.

Concrete repro: configure an agent with a Proton-Bridge SMTP credential
injected as PROTON_SMTP_JSON. Agent writes a draft, runs nodemailer
from the `execute` shell, gets a real messageId back, posts a
confirmation in #feedback. SwarmClaw silently rewrites the confirmation
to the canned false-negative. Email actually arrived. User sees the
overwrite, thinks no email was sent, tries again, ends up with
duplicates and confusion.

Fix: when no connector_message_tool event was recorded, only emit the
false-negative if NO tool events ran at all for the turn. Any tool
event present is evidence the agent did some real work and the
reconciler shouldn't blindly overwrite. The connector-failure path
(connector events present but none succeeded) is unchanged — that one
has genuine evidence of failure to surface.

This narrows the reconciler's scope to its documented purpose: catch
agents that hallucinate sends without calling any tool at all. Agents
that legitimately use non-connector tools are no longer collateral
damage.

Files:
  - src/lib/server/chat-execution/chat-execution-connector-delivery.ts
@waydelyle

Copy link
Copy Markdown
Member

Cherry-picked and shipped in v1.9.33 with connector sanitization coverage and non-connector delivery preservation. Thanks for the fix.

@waydelyle waydelyle closed this May 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants