[Design Question] IPC timeout behaviour undefined - transport has no socket timeout and fail-open/fail-closed is unresolved per hook

[M-Masood4]

Summary

While reviewing the Phase 1 implementation I found that the Python transport
has no socket timeout set on the UDS path. A stalled or slow sidecar will
block the agent thread indefinitely. This also surfaces a broader design
question that was raised in the project Slack but never formally resolved:
what is the intended fail behaviour per hook when IPC fails or times out?

The concrete issue in transport.py

In _connect_and_send_uds (transport.py:75):

with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as sock:
    sock.connect(self.socket_path)
    sock.sendall(frame_bytes)
    return self._read_response(sock)

No sock.settimeout() is called. If the sidecar stalls mid-response,
_read_response blocks on sock.recv() forever.

Additionally, the retry logic in send() only catches
ConnectionRefusedError and FileNotFoundError. A socket.timeout
exception would not be retried and would propagate as an unhandled error.

Why this matters for a security layer

The architecture doc specifies a 4-8ms typical latency budget and ~10ms
worst-case. But there is currently no enforcement of that budget on the
SDK side. An attacker who can induce load on the sidecar process can
stall every agent call that goes through the firewall.

The consequence depends on the fail behaviour, which is currently
undefined:

• Fail-open (let the call proceed if firewall is unreachable):
  the attacker has bypassed the enforcement layer entirely
• Fail-closed (block the call if firewall is unreachable):
  the attacker has created a denial of service

Neither is acceptable as a silent default. This needs to be an explicit,
configurable decision.

The design question

Different hooks have different risk profiles and the right fail mode
probably differs per hook:

Hook	Suggested default	Reasoning
on_prompt	fail-closed	blocking a turn is recoverable
on_tool_call	fail-closed	tool execution without inspection is unsafe
on_context	configurable	degraded RAG is acceptable in some deployments
on_memory	fail-closed	a poisoned write that bypasses inspection persists

This could be expressed in sidecar.yaml under each hook definition,
similar to how on_ipc_timeout was proposed in the policy taxonomy
discussion.

Questions for the mentor

1. What is the intended fail behaviour when the sidecar is unreachable
   or times out: fail-open or fail-closed?
2. Should this be configurable per hook, or a single global setting for v1?
3. Should the SDK enforce the latency budget with a configurable timeout
   (e.g. ACF_TIMEOUT_MS env var), or is that the sidecar's responsibility?
4. Should socket.timeout be included in the retry set, or should it
   propagate immediately as a hard failure?

Once the direction is decided, I can put together a PR implementing it.

[VibhorGautam]

been digging into this while scoping phase 2 observability. the ipc-timeout bug and the fail-behaviour question are tangled together and i think both need to land in the same patch or the defaults get reasoned about in isolation

on the concrete bug in transport.py

three things need to line up to actually fix it:

1. sock.settimeout(cfg.ipc_timeout_ms / 1000) on the socket right after connect. missing that is the primary bug
2. the except ladder in send() has to grow to cover socket.timeout and OSError (which is what recv actually raises on some kernels when the peer drops mid-response). currently only ConnectionRefusedError and FileNotFoundError are handled, so a half-dead sidecar surfaces as an unhandled exception on the caller's stack
3. retries for timeouts specifically need a cap (say 2 retries max with tiny backoff). unbounded retry on a stalled peer is worse than fail-closed because the caller can't tell the call is still in flight

on Windows the named-pipe path in _connect_and_send_pipe has the same shape and needs parallel changes. the current code passes FILE_FLAG_OVERLAPPED = 0, which explicitly disables overlapped i/o rather than leaving it unwired. enabling it means passing 0x40000000 and adding a WaitForSingleObject with the same configured timeout as the uds path. without that, a stalled pipe is no more timeoutable on windows than uds is today

on the per-hook fail-mode matrix

the matrix in the issue body is basically correct. i would push it a bit further and tie each decision to a concrete threat model, because "fail-closed is a DoS" is only sometimes true:

hook	default	why	fail-open bypass risk	fail-closed DoS risk
on_prompt	fail-closed	blocking a turn shows up as a chat error the user sees; the agent can retry	prompt injection to the model	low, user just sees a retryable error
on_tool_call	fail-closed	tool execution with unscanned params is worse than a tool failure	arbitrary tool abuse, including exfil tools	medium, agent's tool loop breaks
on_memory	fail-closed	memory poisoning is persistent; a transient DoS on memory read is annoying but bounded	persistent memory poisoning that survives the session	low-medium
on_context	fail-open with sanitise	rag retrieval failures can't gracefully fail closed in most agent loops; they cascade into empty answers. better: return a synthetic "context unavailable" chunk and flag it	indirect injection slips through	high, every rag-heavy agent turn breaks

the on_context case is the one i want to flag specifically. hard fail-closed on context retrieval will kill most real agent runtimes because they don't have a graceful "degraded answer" mode. a sanitised-with-flag outcome is closer to what production rag stacks actually want

proposal

one firewall.yaml block, per-hook keys, three knobs each:

fail_behaviour:
  on_prompt:
    mode: closed         # open | closed | sanitise
    timeout_ms: 50
    max_retries: 2
  on_tool_call:
    mode: closed
    timeout_ms: 50
    max_retries: 1
  on_memory:
    mode: closed
    timeout_ms: 50
    max_retries: 2
  on_context:
    mode: sanitise       # returns synthetic "chunk-unavailable" payload with a signal
    timeout_ms: 50
    max_retries: 1

that keeps the defaults explicit and lets operators tune for their agent runtime without editing code. the same block belongs in the python sdk config and in the go sidecar's wire contract so both sides agree on what "timeout" means

visibility

worth mentioning because the observability scaffolding just got filled in: a fail-closed or fail-open event is a decision the audit layer should record, not just the application log. the audit schema already has decision and signals; adding a fail_closed, fail_open, or sanitise_on_timeout signal makes it trivial to count these after-the-fact. otherwise an operator won't notice when the sidecar is silently flapping

happy to open a draft pr against phase02 for the transport.py and sdk-side changes if the hook-level defaults look right. the sidecar side (recording the fail events on the audit line) is a small follow-up to the phase-2 telemetry work already in flight