feat(gateway): persist inbound turns and outbound delivery across gateway restarts

[louiseboo]

Summary

Raven's channel intake and Spine scheduler currently provide in-process concurrency and per-conversation serialization, but they do not establish a durable acceptance boundary for inbound turns or a durable outbox for completed replies.

If the gateway restarts after a channel has received a message but before the turn finishes, the accepted turn cannot be recovered from persistent state. If outbound delivery fails after execution succeeds, there is no channel-neutral durable outbox that guarantees redelivery after recovery.

This proposal is about failure containment and recoverability. It does not assume that a specific channel, model provider, or network is at fault.

Version inspected

• Raven 0.1.7
• main at c3f3678a24d4cbda6204c5142c681235f5df15c4

Current behavior

The current Weixin adapter illustrates the gap:

1. It keeps recently seen message IDs in the process-local _seen collection.
2. It calls and awaits intake.publish(...) after parsing the inbound message.
3. Intake.publish(...) submits the TurnRequest to the in-process Spine scheduler.
4. Scheduler lanes and queued futures live in the gateway process and are intentionally drained or cancelled during shutdown.

This is a sound in-process scheduling model, but it does not persist the fact that Raven accepted a user request. A process restart therefore loses unstarted or unfinished in-memory work. The same gap exists on the outbound side: delivery success/failure is not represented by a durable, retryable outbox shared across channels.

Minimal reproduction

The issue is channel-neutral and can be reproduced with a synthetic slow runner:

1. Start the gateway with a channel adapter and a runner that blocks long enough to observe restart behavior.
2. Send one user turn and confirm that the adapter has parsed and submitted it.
3. Restart the gateway before the turn produces an outbound reply.
4. Observe that there is no persistent accepted-task record from which the turn can be resumed or definitively reported as failed.
5. Separately, make a completed turn's outbound send fail, restart the gateway, and observe that there is no durable outbox item guaranteeing later redelivery.

Provider timeouts such as #150 reduce the duration of one failure mode, and structured termination such as #149 improves diagnosis, but neither creates a durable channel-to-run-to-delivery contract.

Expected behavior

Once Raven acknowledges a turn as accepted:

• the inbound request should survive gateway restart;
• a stable idempotency key should prevent duplicate execution after channel redelivery;
• processing should be claimable with a lease and safely recoverable after worker death;
• task completion and outbound delivery should be recorded as separate states;
• a failed outbound send should remain pending and retry after channel recovery;
• users should be able to query an accepted task's state even if the original run failed.

Proposed direction

Introduce channel-neutral persistence interfaces rather than implementing reliability independently in every adapter:

1. Durable inbox

   • Persist TurnRequest, conversation ID, origin, channel message ID, and timestamps transactionally.
   • Enforce a unique idempotency key per channel message.

2. Recoverable dispatch

   • Claim accepted turns with expiring leases.
   • On restart, inspect expired claims before retrying so non-idempotent tool effects are not repeated blindly.

3. Durable outbox

   • Persist final replies and delivery attempts separately from run completion.
   • Retry transient channel failures and resume pending delivery after restart.

4. Acknowledgement and status hooks

   • Allow a channel to acknowledge only after durable acceptance.
   • Expose accepted, running, awaiting confirmation, completed, failed, cancelled, and delivery-pending states.

SQLite WAL could be a practical default local backend, but the core contract should remain storage-neutral. Whether the worker stays in the gateway process or moves to a separate service can remain an implementation choice as long as accepted work and delivery state survive process failure.

Suggested tests

• duplicate channel delivery produces one accepted turn;
• restart after acceptance but before execution resumes or reports the same turn;
• restart during execution safely handles an expired lease;
• successful execution plus failed send leaves a pending outbox item;
• channel recovery drains the outbox exactly once;
• scheduler shutdown cannot silently discard a previously acknowledged turn;
• no private message content or channel credentials appear in diagnostic status output.

I am happy to help refine the design and contribute a tested implementation once maintainers confirm the preferred persistence and process boundaries.