TESTING.md

Testing

This repository tests the adapter scaffold, the subprocess-backed ACP runtime bridge, and the native Codex CLI command bridge Hecate uses today. Hecate's embedded path is covered by its agent-adapter integration tests and opt-in authenticated embedded smoke; this adapter repo still tracks deeper vendor-native parity before calling the adapter production-grade.

Covered Today

• CLI version output
• ACP initialize response shape
• custom ACP initialize handlers receiving raw client params before returning dynamic initialize results
• runtime-backed initialize passthrough so child agent info, capabilities, and auth methods reach the ACP client
• runtime-backed initialize result wrappers preserve runtime-provided extension fields through the runtime host
• runtime initialize forwarding for client terminal-auth capability
• runtime ACP auth calls: authenticate method-id forwarding and logout request forwarding
• request ID preservation
• server-to-client JSON-RPC requests from handlers, including successful client responses and client RPC errors
• in-flight client notification and concurrent cancel-request dispatch while a request handler is running, including prompt cancellation through the runtime bridge
• protocol-level $/cancel_request cancellation for in-flight method handlers waiting on server-to-client requests
• runtime bridge request context propagation, so protocol cancellation reaches context-aware runtime calls
• ordered ACP method execution without blocking notification dispatch behind a burst of queued method requests
• malformed JSON errors without stopping later requests
• invalid JSON-RPC version errors
• notification dispatch without responses
• fake runtime method dispatch through the stdio transport
• fake runtime error propagation
• fake runtime session creation
• fake prompt lifecycle with assistant chunk, tool call, and tool update notifications emitted before the prompt response
• cancel notification and cancel request behavior
• session close and post-close not-found behavior
• scaffold session/prompt not-implemented errors
• scaffold known ACP methods return structured not-implemented errors instead of method-not-found when no runtime binary is configured, including unstable SDK-known document, provider, and NES methods
• 1 MiB inbound message cap
• hardened process runner behavior: fixed argv, shell rejection, absolute cwd enforcement, env allowlists, redaction, output caps, missing-binary errors, non-zero exits, and context cancellation
• long-lived process start behavior: stdin/stdout pipes, bounded stderr capture, process IDs, exit errors, shell rejection, and cancellation
• doctor runtime-boundary probe success, missing-binary, shell rejection, failed version-probe, secret redaction, and invalid-workdir behavior
• runtime launcher defaults, env/argv merging, bounded stderr, shell rejection, missing-workdir validation, exit errors, and cancellation
• runtime JSON-RPC client request/response matching, notifications, child events, error responses, malformed stdout failure, and request timeout cleanup
• runtime JSON-RPC request cancellation sends $/cancel_request to the child runtime while ignoring the later abandoned response
• runtime JSON-RPC child request responses, including successful result replies and error replies written back to the child runtime
• ACP initialize negotiation over the runtime JSON-RPC client, including client info/capabilities, agent capability parsing, protocol-version mismatch, and runtime RPC error propagation
• session capability preservation in runtime initialize results
• MCP acp, http, and sse capability preservation in runtime initialize results
• ACP session lifecycle calls over the runtime JSON-RPC client: session/new, session/fork, session/prompt, session/cancel, session/close, session updates, stop reasons, MCP server payloads, and RPC error propagation
• session/new result preservation for runtime-provided configOptions and legacy modes, so model/reasoning/mode selectors survive the bridge
• runtime ACP config/mode setters: session/set_config_option and session/set_mode raw result forwarding, including session updates emitted before setter responses
• runtime ACP raw lifecycle/config helpers preserve validated-but-unknown extension fields when the bridge forwards params to subprocess runtimes
• runtime ACP prompt and session-list result wrappers preserve runtime-provided extension fields when the bridge returns responses to ACP clients
• ACP session load/resume/fork/list/delete protocol calls, including replay updates, raw resume/fork result preservation, listed session metadata, cursor parsing, and delete request forwarding
• ACP bridge forwarding for resume-time replay updates emitted before session/resume responds
• ACP-transport MCP server request payloads preserve their opaque id and _meta data when forwarded to subprocess runtimes
• unstable MCP-over-ACP mcp/message pass-through, including raw inner MCP response preservation
• unstable MCP-over-ACP mcp/message notification forwarding from ACP clients to subprocess runtimes
• ACP server-to-runtime bridge behavior: handler param validation, session method proxying, prompt update forwarding, cancel notifications, close requests, and runtime RPC error mapping
• ACP bridge forwarding for setup-time session/update notifications emitted before session/new responds
• ACP bridge forwarding for close-time session/update notifications emitted before session/close responds
• ACP server-to-runtime bridge coverage for session load, resume, list, and delete methods
• ACP server-to-runtime bridge coverage for session fork and MCP message forwarding
• ACP bridge forwarding for dynamic session/update payloads such as available_commands_update, config_option_update, and session_info_update
• ACP bridge forwarding for runtime child requests that require client responses, including returning the client result to the child runtime
• ACP bridge forwarding for runtime MCP child requests, including mcp/message params and client response delivery back to the child runtime
• cancellation while awaiting a runtime child request, so a prompt can settle as cancelled even when the client never answers the child request
• ACP bridge forwarding for authenticate and logout
• ACP bridge forwarding for session/set_config_option and legacy session/set_mode
• runtime host composition: subprocess launch, ACP initialize handshake, initialize result retention, bridge option exposure, prompt update forwarding, and protocol-version mismatch cleanup
• root ACP scaffold initialize metadata and capabilities for this adapter
• root doctor command Codex binary default, JSON report shape, and Codex environment status list
• root ACP runtime flags: opt-in subprocess-backed serving, deferred runtime startup with forwarded client initialize capabilities, required absolute runtime workdir, runtime argv passthrough, Codex-specific environment allowlist inheritance, and runtime-flag precedence over the native command bridge
• root ACP native command bridge: session creation with Codex model/reasoning config options, sandbox config option, approval-policy config option, web-search config option, config updates, current Codex search flag plus isolated codex exec --ignore-user-config --json argv construction, explicit native command environment allowlists, /review advertisement plus codex review --uncommitted argv construction, /init advertisement through the normal codex exec prompt path, additional workspace directories, ACP HTTP/stdio MCP server config propagation into Codex -c mcp_servers.<name>=... overrides, streamed JSONL parsing into ACP assistant text, reasoning, usage, terminal stop reasons, and provider-native tool classifications for shell, file, patch, web, MCP, image, plan, TODO, goal, and review updates, carried tool metadata for sparse completion events, failed status mapping for rejected/denied/blocked or timed-out provider tool results, generic command tool_call activity for the outer Codex process, in-memory load/resume/fork capability, bounded transcript replay for later command prompts, and prompt completion
• source-shaped Codex stream fixtures for permission requests, including MCP server/tool permission labels, option alias/default handling, shell/tool lifecycle updates, reasoning chunks, usage, terminal stop reasons, and live ACP client coverage for rejected and cancelled permission outcomes
• auth-required error classification for native Codex failures, including the HTTP 401 "missing bearer or basic authentication" shape
• opt-in real Codex CLI smoke coverage that requires an authenticated local codex binary and proves session list/load, a real prompt completion, a real tool/file update flow with permission auto-approval, a stdio MCP tool call through a local echo fixture, and cancellation with no double-settle through the ACP command bridge
• shared adapter conformance checks for the Hecate-facing ACP initialize contract, advertised auth/logout capabilities, session config selectors, and available slash-command names
• portable upstream parity checks adapted from the previous adapter suites: unsupported auth methods are rejected, session/new emits commands before returning config selectors, session/set_config_option updates and rejects invalid values, session/list/session/load expose live session state, unknown Codex sessions are not silently adopted by the command-backed bridge, and idle cancel/close/delete retain their ACP response shapes
• Coder ACP SDK compatibility guardrails for the adopted protocol primitives: JSON-RPC error shape, default initialize protocol version, and selected runtime ACP request JSON shapes
• release packaging gate through make release-check: unit tests, race tests, vet, and a version-stamped local binary build

Not Covered Yet

These are not blockers for Hecate's current Go-adapter replacement path, but they must be covered before treating the adapter as production-grade vendor-native Codex parity:

• vendor-specific durable persistent session storage and restore semantics across adapter process restarts
• wedged-runtime forced cancellation behavior
• model/config option discovery beyond the initial static command-backed selectors
• provider-native permission response edge cases beyond parsed request/result status mapping and the selected Codex sandbox / approval policy
• deeper provider-native review semantics beyond classifying review tool updates
• late permission responses after cancellation
• deeper vendor MCP connection lifecycle semantics and MCP tool approval elicitations beyond the local stdio echo-tool smoke
• unresolved stable-release parity gates listed in docs/STABLE_READINESS.md

Test Strategy

Use acp-adapter-kit for provider-neutral protocol/runtime/process tests. The kit owns ACP transport conformance, subprocess safety, JSON-RPC request/cancel behavior, runtime ACP DTO parity, runtime bridge forwarding, runtime host composition, fake-runtime fixtures, generic doctor-runner behavior, and portable upstream parity assertions that can run unchanged against both Go adapter implementations.

Keep this repository's tests focused on Codex-specific adapter behavior:

• CLI version and no-argument ACP stdio behavior;
• scaffold initialize metadata and Codex capability flags;
• doctor command defaults for the Codex binary and Codex environment list;
• runtime flag wiring from Cobra into the shared runtime host;
• command-backed codex exec argv construction and config-option mapping;
• command-backed ACP authenticate mapping to the native codex login command;
• command-backed ACP logout mapping to the native codex logout command;
• Codex-specific prompt, tool, permission, config, model, MCP, auth, and session mapping as those features land.

Do not recreate kit packages under internal/. Add reusable protocol/runtime coverage to acp-adapter-kit, then update this adapter to the new kit version and add only Codex-specific integration assertions here.

Real CLI Smoke

The real Codex CLI smoke is intentionally excluded from the default test suite because it requires local authentication, network access, and may use provider quota. Run it only on a prepared developer machine:

make real-cli-smoke

The target sets ACP_ADAPTER_REAL_CLI_SMOKE=1 and runs the real_cli build-tag test. It creates temporary workspaces, opens ACP sessions, verifies list/load, sends one minimal prompt through the native codex command bridge, runs a tool/file update prompt with permission requests auto-approved by the test client, passes a local stdio MCP echo server through mcpServers and asserts the provider calls its echo tool with approval_policy=bypass for that temporary MCP session, and cancels one long-running prompt to assert a single cancelled prompt settlement.