Add provider-neutral per-run inputs for images, files, and audio

[Beckers26]

Summary

Add provider-neutral, per-invocation inputs for images, files, and audio while keeping Run and Stream as the only execution verbs.

The current public API accepts only prompt string, and driver.Request carries only Prompt string. An embedding host therefore cannot pass an image, PDF, or audio resource without converting it into an undocumented provider-specific prompt/path convention. That loses modality, prevents truthful capability negotiation, makes resource lifetime ambiguous, and encourages each host to add its own unsafe URL/file handling.

This issue proposes a backward-compatible root API, a resolved Driver SPI, explicit capability modes, typed pre-launch errors, and deterministic cleanup/retry semantics.

Motivation

Our embedding host is a long-running Go process that uses agent-adaptor to expose multiple local Agent CLIs behind one product protocol. The server remains the message/resource authority. Before a run, the host downloads an authorized canonical resource, verifies its MIME type, size and SHA-256, and stages it locally. We then need to give that resource to the configured Driver without:

• importing Codex/Claude/Cursor-specific request types into the host;
• interpolating a local path into the user's text prompt;
• claiming native vision/audio support when a Driver only mentions a workspace path;
• passing a remote URL to the SDK and expanding the SDK's SSRF/credential scope;
• keeping temporary resources alive forever or deleting them while a Driver still reads them;
• changing the Thread compatibility fingerprint for every turn attachment.

This is not only a media feature. A provider-neutral input contract is the boundary that lets hosts apply authorization and resource policy once while Drivers remain responsible for provider protocol encoding.

Relationship to #20

This proposal follows the immutable-Agent direction in #20:

• inputs are invocation content, not stable Agent topology;
• WithInputs is call-only and cannot install Agent defaults;
• individual input IDs, paths, sizes, and digests do not enter Thread/session compatibility fingerprints;
• Run and Stream remain the only execution verbs;
• Inspect may report dynamic account/model input limits, while Descriptor retains static transport capabilities;
• Prepare does not materialize per-run inputs because it has no invocation content.

If #20 lands as a coordinated breaking v1 revision, the exact DTO names can be frozen there, but the content/lifecycle rules below should remain the same.

Proposed root API

Keep existing text-only calls source-compatible and add a call-only option:

stream := agent.Stream(
    ctx,
    "Compare the diagram with the specification.",
    adaptor.WithInputs(
        input.FromPath(input.KindImage, stagedPNG,
            input.WithName("diagram.png"),
            input.WithMediaType("image/png"),
            input.WithSize(pngSize),
            input.WithSHA256(pngDigest),
            input.RequireMode(input.ModeNative),
        ),
        input.FromOpener(input.KindFile, "spec.pdf", openPDF,
            input.WithMediaType("application/pdf"),
            input.WithSize(pdfSize),
            input.WithSHA256(pdfDigest),
        ),
    ),
)

Suggested package and constructors:

package input

type Kind string

const (
    KindImage Kind = "image"
    KindFile  Kind = "file"
    KindAudio Kind = "audio"
)

type Mode string

const (
    ModeNative             Mode = "native"
    ModeWorkspaceReference Mode = "workspace_reference"
)

// Ref is an immutable, detached input declaration. Its source fields are
// intentionally hidden so invalid combinations cannot be constructed.
type Ref struct { /* unexported */ }

type OpenFunc func(context.Context) (io.ReadCloser, error)

func FromPath(kind Kind, path string, opts ...Option) Ref
func FromFS(kind Kind, fsys fs.FS, path string, opts ...Option) Ref
func FromOpener(kind Kind, name string, open OpenFunc, opts ...Option) Ref

func WithName(name string) Option
func WithMediaType(mediaType string) Option
func WithSize(size int64) Option
func WithSHA256(sum [32]byte) Option
func RequireMode(mode Mode) Option

Root option:

// WithInputs appends per-invocation non-text inputs in argument order.
// It implements CallOption only.
func WithInputs(refs ...input.Ref) CallOption

Option semantics

• The existing prompt string is the first text part. Inputs are ordered attachments after that text part.
• Multiple WithInputs options append in option order. An empty call is a no-op.
• There is no Agent-level default input. Reusing user content implicitly across calls would be surprising and unsafe.
• Ref metadata is detached when the run begins. Caller mutation after Run/Stream starts cannot change the request.
• FromOpener is called at most once for one logical invocation. A safe provider-process retry before prompt delivery reuses the already materialized file.
• v1 has no built-in URL source. A host that owns HTTP authorization/fetch policy exposes the verified body through FromOpener or a local staged path.
• WithInputs should apply equally to Agent.Run, Agent.Stream, Thread.Run, and Thread.Stream through the existing effective run settings.

The API deliberately does not add RunWithFiles, RunMultimodal, Upload, or a parallel execution handle.

Materialization contract

Core resolves all refs before launching the provider process or allowing the prompt to be delivered:

1. Validate kind, name, media type, declared size/digest, and requested mode.
2. Select a delivery mode from the Driver capability. Prefer native; use workspace_reference only when it is declared and the caller did not require native.
3. Open each source once and stream it into a run-owned directory.
4. Compute actual size and SHA-256 while copying; compare them with optional expected values.
5. Pass only resolved, immutable local metadata to the Driver.
6. Keep the directory alive until Driver completion and all Driver-owned event sources have stopped.
7. Remove it on success, cancellation, Driver error, stream close, and Agent close.

Suggested construction-only limits:

type Limits struct {
    MaxCount      int
    MaxInputBytes int64
    MaxTotalBytes int64
}

func WithInputLimits(limits input.Limits) Option

Zero values should select documented safe defaults, not mean unlimited. Hosts that need larger files can opt in explicitly.

Core should copy path/FS sources rather than let a long-running Driver reopen an arbitrary caller path later. That produces one lifetime model, detects mutation while hashing, and avoids making every Driver implement file safety. Platform-specific exact-inode/symlink/junction hardening can remain an internal materializer concern.

Static capability contract

Extend driver.Descriptor with static transport capabilities:

type InputMode string

const (
    InputModeNative             InputMode = "native"
    InputModeWorkspaceReference InputMode = "workspace_reference"
)

type InputKindCapability struct {
    Modes      []InputMode
    MediaTypes []string
    MaxCount   *int
    MaxBytes   *int64
}

type InputCapabilities struct {
    Image InputKindCapability
    File  InputKindCapability
    Audio InputKindCapability
}

type Descriptor struct {
    // existing fields...
    Inputs InputCapabilities
}

Invariants:

• Empty Modes means unsupported.
• native means the Driver uses a provider-native typed content/file mechanism and preserves the input modality.
• workspace_reference means the Driver intentionally places/references a verified local file in provider-visible workspace context. It is a declared fallback, not native multimodal support.
• MediaTypes uses normalized MIME types/patterns. An empty list means the static transport cannot provide a narrower guarantee; the Driver must still validate before provider launch.
• Nil limits mean unknown, not unlimited.
• Descriptor data is a static upper bound. Current CLI version, account, model, and quota limits belong in live inspection after Simplify live inspection and preparation for immutable Agents #20. Effective support is the intersection.
• A Driver must not advertise native merely because its implementation can append Please inspect /path to a prompt.

The host can use this contract to hide unsupported UI, but root validation remains mandatory because UI capability snapshots can become stale.

Driver SPI

Do not pass raw bytes, remote URLs, fs.FS, or host callbacks through the Driver boundary. Core passes resolved local inputs through the existing sole execution SPI:

type InputKind string
type InputMode string

type ResolvedInput struct {
    ID           string
    Kind         InputKind
    Name         string
    MediaType    string
    LocalPath    string
    Size         int64
    SHA256       [32]byte
    DeliveryMode InputMode
}

type Request struct {
    // existing fields...
    Prompt string
    Inputs []ResolvedInput
}

SPI invariants:

• LocalPath is absolute, run-owned, read-only input and remains valid until Driver.Run and all event sources return.
• Name is a sanitized display basename, never an alternate path.
• Size and SHA256 are always the observed values, even when the caller did not declare expectations.
• DeliveryMode is selected and validated by core; the Driver must implement exactly that semantic mode or return a contract error before prompt delivery.
• Drivers must not mutate, retain, or log LocalPath after the run ends.
• Request.Inputs is detached; Driver mutation cannot change root settings or another run.

Driver encoding examples are intentionally provider-specific:

• a native Driver maps image/audio/file entries to official typed protocol blocks;
• a workspace-reference Driver arranges an accessible workspace path and explicitly references it according to its documented transport;
• an unsupported Driver fails before launching the provider process.

No new Driver execution method is required; Driver.Run(ctx, req, sink) remains the only provider execution SPI.

Typed errors

Expose sentinels plus a typed error usable with errors.Is and errors.As:

var (
    ErrInputInvalid               = errors.New("invalid input")
    ErrInputKindUnsupported       = errors.New("input kind unsupported")
    ErrInputMediaTypeUnsupported  = errors.New("input media type unsupported")
    ErrInputModeUnsupported       = errors.New("input mode unsupported")
    ErrInputTooLarge              = errors.New("input too large")
    ErrInputDigestMismatch        = errors.New("input digest mismatch")
    ErrInputMaterializationFailed = errors.New("input materialization failed")
)

type InputError struct {
    Err       error
    Index     int
    Kind      input.Kind
    MediaType string
    Driver    string
    Limit     int64
}

InputError.Error() and wrapping must not contain source paths, URL credentials, file contents, opener errors containing secrets, or raw provider payloads. Detailed paths may be exposed only through an explicit host-owned diagnostic hook with its own redaction policy.

All validation/materialization/capability errors occur before provider launch and before prompt delivery. That makes them safe for the caller to correct and retry.

Thread, fingerprint, and retry semantics

• Inputs are turn content, like Prompt; individual input IDs, names, paths, sizes, digests, and bytes are not part of the Thread compatibility fingerprint.
• Driver configuration that changes how inputs are encoded may continue to participate in the existing Driver session/config fingerprint.
• On a resumed Thread, inputs are delivered exactly once for that invocation, not reattached to prior turns.
• All inputs must be materialized before the prompt may be delivered.
• If provider launch fails before prompt delivery, an existing safe fallback may reuse the materialized inputs.
• If prompt delivery is indeterminate or may have succeeded, core must not auto-replay the prompt or inputs. Existing no-replay-after-delivery rules remain authoritative.
• Prepare never resolves these inputs and does not create their directories.

Events, results, and redaction

• Input bytes and local paths never appear in public Events, Result, Raw result, Thread checkpoints, default logs, or error strings.
• A Driver may emit a redacted Notice containing input ID, kind, normalized media type, size and digest if the protocol needs auditable attachment acknowledgement.
• Result text remains the provider's final result; core does not synthesize a result from input metadata.
• Structured output continues to apply to provider output, not to input descriptors.

Built-in Driver requirements

Each built-in Driver should declare truthful support even if the first implementation is unsupported:

• Codex
• Claude
• Cursor
• CodeBuddy

For every declared kind/mode, adapter conformance must prove the actual provider request contains the corresponding native block or explicit workspace reference. A fixture that only verifies a prompt contains a path is not proof of native support.

Test requirements

Root API and materialization

• text-only existing calls remain source- and behavior-compatible;
• repeated WithInputs append in deterministic order and do not alias caller slices;
• path, fs.FS, and opener sources produce identical resolved metadata;
• actual size/digest are computed and expected mismatch fails before Driver invocation;
• max count, per-input bytes, total bytes, invalid kind/name/MIME, and required-mode mismatch are typed pre-launch failures;
• cancellation at open/copy/hash/Driver stages closes readers and removes run-owned files;
• success, Driver error, Stream close, Agent close, and panic-safe cleanup remove inputs only after event sources stop;
• parallel runs cannot observe or delete each other's inputs;
• symlink/junction/path replacement and mutation-during-copy tests use real filesystems on supported platforms;
• default logs and errors contain no input bytes or secret paths.

Thread and retry

• adding different inputs does not split an otherwise compatible Thread;
• one invocation delivers its inputs once on a resumed Thread;
• a safe pre-prompt launch retry reuses materialized inputs without reopening the source;
• an indeterminate post-prompt failure never replays the invocation;
• Prepare creates no input materialization.

Driver conformance

• every built-in Driver has explicit image/file/audio mode assertions;
• unsupported kind/MIME/mode fails before process launch;
• native fixtures inspect the real provider wire representation;
• workspace_reference fixtures prove the path is provider-visible and the capability is not reported as native;
• Driver does not retain a path after run cleanup;
• live tests remain opt-in and require existing authenticated local CLIs.

Acceptance criteria

• Existing text-only Run/Stream code compiles unchanged.
• One provider-neutral call can carry verified image, file, and audio refs.
• No new execution verb or root upload/HTTP client is introduced.
• WithInputs is call-only and individual inputs never affect Thread compatibility.
• Core materializes, verifies, limits, and cleans inputs under one lifecycle contract.
• Driver receives only immutable resolved local metadata through Request.Inputs.
• Static and live capabilities distinguish unsupported, native, and workspace-reference behavior truthfully.
• Unsupported/invalid/oversized/mismatched inputs fail with typed errors before provider launch and prompt delivery.
• Built-in Drivers pass adapter conformance for every advertised input mode.
• API reference, streaming, run policy, Thread/fingerprint, security, examples, changelog, and API golden are updated together.

Non-goals

• SDK-managed remote URL fetching.
• Persistent Agent-level default attachments.
• Storing input bytes in ThreadStore.
• Guaranteeing exactly-once external provider side effects.
• Claiming all providers/models support the same input kinds.
• Adding an upload service, daemon, or product-specific resource API to agent-adaptor.