Epic: OpenAI background_jobs

[leseb]

Purpose

Implement OpenAI Responses background mode for locally orchestrated Responses requests: asynchronous execution, bounded scheduling, polling, cancellation, timeout handling, and resumable streaming.

This filter is opt-in. When it is not configured, backends that natively support background mode continue to receive the request unchanged.

Current OpenAI contract

The implementation must follow the current OpenAI background mode documentation:

• background=true starts asynchronous processing.
• Clients poll GET /v1/responses/{id} while status is queued or in_progress.
• POST /v1/responses/{id}/cancel cancels an in-flight response; repeated cancellation is idempotent.
• background=true is valid with store=false. Temporary operational state must still be retained long enough for polling and cancellation, without turning store=false into durable application storage.
• background=true is valid with stream=true. A client can resume the stream with GET /v1/responses/{id}?stream=true&starting_after={sequence_number} when the response was originally created with streaming enabled.

The existing stream + background and background + store=false rejection rules are obsolete and must be removed as part of this work or a prerequisite conformance change.

Architecture boundary

The filter schedules and exposes the OpenAI lifecycle; Praxis core owns background task lifecycle and execution through the configured proxy machinery.

A worker must run the request through the applicable routing, load balancing, upstream exchange, response filters, persistence, guardrails, and observability path. Do not implement this as:

• a hidden direct backend call from the filter;
• a loopback HTTP request into the public listener; or
• a second proxy implementation embedded in background_jobs.

The required Praxis jobs/task lifecycle work is tracked by praxis-proxy/praxis#807. The AI implementation must be based on the resulting core contract rather than inventing an incompatible local runtime.

Lifecycle

Enqueue

For a locally handled POST /v1/responses request with background=true:

1. Generate the stable response ID.
2. Reserve bounded queue capacity before persisting queued.
3. Persist the initial response object with status=queued.
4. Return the queued response immediately for non-streaming requests.
5. The worker atomically claims queued -> in_progress.
6. The worker executes the configured Responses path.
7. Completion atomically transitions in_progress to completed, incomplete, or failed.

The locally generated response ID must remain the public and stored ID even if the inference backend generates a different provider response ID.

Queue exhaustion returns a structured 429 response, preferably with Retry-After, and must not leave an orphaned stored response.

State transitions

Storage updates must use compare-and-set semantics:

queued -> in_progress -> completed | incomplete | failed
queued -> cancelled
in_progress -> cancelled

A worker may write a terminal result only while the stored status is still in_progress. Cancellation must not be overwritten by a concurrently completing worker.

A dequeued job that can no longer claim queued -> in_progress must be skipped.

Timeout and failure

• Per-request execution timeout is configurable; default 5 minutes.
• Timeout, worker failure, or task panic must leave a terminal stored response rather than queued or in_progress.
• Worker failure must not terminate the worker pool.
• Shutdown and dynamic configuration reload behavior must be explicit: jobs are either drained, transferred, or moved to a documented terminal state.

Polling

GET /v1/responses/{id} returns the latest stored response object, including queued, in_progress, and terminal states.

Tenant isolation and the existing not-found/access-denied behavior must apply to polling, cancellation, and resumed streams.

Cancellation

POST /v1/responses/{id}/cancel:

• queued: atomically update to cancelled; the worker skips it if later dequeued.
• in_progress: request task cancellation and atomically update to cancelled.
• cancelled: return the stored response unchanged (idempotent).
• other terminal states: return the documented structured error without changing the response.

The cancel route must bypass/defer request-body format validation so an empty POST body is not rejected before the handler runs.

Cancellation and task ownership must remain correct when polling/cancellation traffic reaches another worker or process. If the first implementation is intentionally single-process, document and enforce that limitation rather than silently exposing unsafe multi-process behavior.

Streaming and resumption

For background=true, stream=true:

• Begin returning Responses SSE events while processing continues independently of the client connection.
• Persist or otherwise retain events with their sequence_number.
• Client disconnect must not cancel the job.
• Resume requests use starting_after and return only later events.
• A new stream can only be started for a response originally created with stream=true.
• Event retention must be bounded by bytes and time and must preserve terminal events.

This is a separate implementation phase from the non-streaming queue/polling foundation, but it remains part of the umbrella compatibility contract.

Resource bounds

• Queue capacity: configurable, default 100 jobs.
• Workers: configurable, default 10.
• Timeout: configurable, default 300 seconds.
• Retained queued request bodies and resumable event logs must be bounded by bytes as well as item count.
• Do not clone request bodies, provider payloads, or streaming chunks merely to pass them between the request and worker paths. Move owned data at the background-job boundary or store only the data needed to resume processing.

Config

filter: background_jobs
queue_size: 100
num_workers: 10
timeout_seconds: 300

Additional byte and retention limits for queued payloads and resumable events must be defined before implementation is considered complete.

Dependencies

• Spike: Background Tasks + Probes praxis#807 — background task/job lifecycle and core execution contract
• Response store with atomic status transitions
• Tokio runtime
• Event retention/replay support for resumable streaming

Sub-issues

• background_jobs: queue, workers, and enqueue path #44 — queue, workers, and enqueue path; closed as a duplicate of the core work in Spike: Background Tasks + Probes praxis#807
• background_jobs: cancellation endpoint #45 — cancellation endpoint
• openai_responses_validate: allow current background parameter combinations #461 — current background parameter conformance cleanup
• response_store: add atomic background status transitions #462 — atomic response-store status transitions

Resumable background streaming should be tracked as a separately reviewable change under this umbrella issue.

Required tests

1. Unit tests for config, queue/byte bounds, status transitions, timeout, cancellation races, and worker survival.
2. Integration tests with a delayed backend for enqueue, polling, queue exhaustion, timeout, queued cancellation, in-progress cancellation, completion-vs-cancel races, reload, and shutdown.
3. Streaming integration tests for disconnect, replay from starting_after, retention limits, and terminal events.
4. Example config in examples/configs/.
5. Functional integration test for the example config.
6. Live backend proof that a worker traverses the real configured Responses path rather than only a recording or no-op stub.

Reference

• OpenAI background mode
• Proposal: Responses API Filters — Filter 13; update its obsolete validation and streaming statements as part of this work
• Part of Epic: Responses API and Agentic Loop Orchestration #93

[leseb]

Implementation note for the cancel handler: make sure POST /v1/responses/{id}/cancel is not rejected by responses_format before background_jobs sees it. If responses_format is earlier in the same chain with on_invalid: reject, an empty POST body can be classified as non_json; the background filter should either run before that classifier for cancel paths or explicitly bypass/defer body-format validation for the cancel endpoint.

[leseb]

Sub-issues

This issue has been decomposed into two scoped sub-issues:

1. feat(openai-files): implement S3 blob storage and crash recovery #503 — Queue, workers, and enqueue path (foundation)
2. feat(openai-files): compose and operate the PostgreSQL/S3 backend #504 — Cancellation endpoint (depends on background_jobs: queue, workers, and enqueue path #44)