From d2a8e651a8d4f6172f53e2b3763a50cc9129b304 Mon Sep 17 00:00:00 2001 From: Fernando Luo Date: Tue, 21 Jul 2026 07:44:28 -0700 Subject: [PATCH 01/31] Initial design of agentic workflows --- projects/agents/01-overview.md | 177 ++++++++ projects/agents/02-lifecycle.md | 259 +++++++++++ projects/agents/03-ui.md | 361 ++++++++++++++++ projects/agents/04-lead-agent.md | 277 ++++++++++++ projects/agents/05-human-interfaces.md | 349 +++++++++++++++ projects/agents/06-workflow-construction.md | 316 ++++++++++++++ projects/agents/07-agent-construction.md | 363 ++++++++++++++++ .../agents/08-agent-agent-communication.md | 404 +++++++++++++++++ .../poc/00-architecture-and-contracts.md | 199 +++++++++ .../agents/poc/01-local-lead-and-pipeline.md | 120 +++++ .../agents/poc/02-runtime-environments.md | 318 ++++++++++++++ .../poc/03-worker-execution-and-fanout.md | 127 ++++++ projects/agents/poc/04-validation-and-demo.md | 121 ++++++ .../poc/05-vocabulary-and-interfaces.md | 127 ++++++ projects/agents/poc/README.md | 111 +++++ projects/agents/poc/locked-topology.md | 409 ++++++++++++++++++ .../materialize-model-artifacts.sh | 188 ++++++++ .../model-artifact-sources-v1.json | 89 ++++ projects/agents/poc/overview.md | 358 +++++++++++++++ projects/agents/poc/prototype/.gitignore | 5 + projects/agents/poc/prototype/Dockerfile | 43 ++ projects/agents/poc/prototype/README.md | 126 ++++++ .../prototype/agentic-vla-workflow-spec.yaml | 59 +++ .../agents/poc/prototype/runtime-lock.json | 41 ++ .../poc/prototype/runtime/codex-config.toml | 14 + .../poc/prototype/runtime/model-catalog.json | 17 + .../agents/poc/prototype/runtime/run-agent.sh | 81 ++++ .../skills/osmo-agentic-workflow/SKILL.md | 53 +++ .../osmo-agentic-workflow/agents/openai.yaml | 4 + .../assets/agent-result.schema.json | 13 + .../assets/child-workflow-template.yaml | 36 ++ 31 files changed, 5165 insertions(+) create mode 100644 projects/agents/01-overview.md create mode 100644 projects/agents/02-lifecycle.md create mode 100644 projects/agents/03-ui.md create mode 100644 projects/agents/04-lead-agent.md create mode 100644 projects/agents/05-human-interfaces.md create mode 100644 projects/agents/06-workflow-construction.md create mode 100644 projects/agents/07-agent-construction.md create mode 100644 projects/agents/08-agent-agent-communication.md create mode 100644 projects/agents/poc/00-architecture-and-contracts.md create mode 100644 projects/agents/poc/01-local-lead-and-pipeline.md create mode 100644 projects/agents/poc/02-runtime-environments.md create mode 100644 projects/agents/poc/03-worker-execution-and-fanout.md create mode 100644 projects/agents/poc/04-validation-and-demo.md create mode 100644 projects/agents/poc/05-vocabulary-and-interfaces.md create mode 100644 projects/agents/poc/README.md create mode 100644 projects/agents/poc/locked-topology.md create mode 100644 projects/agents/poc/model-artifact-materializer/materialize-model-artifacts.sh create mode 100644 projects/agents/poc/model-artifact-materializer/model-artifact-sources-v1.json create mode 100644 projects/agents/poc/overview.md create mode 100644 projects/agents/poc/prototype/.gitignore create mode 100644 projects/agents/poc/prototype/Dockerfile create mode 100644 projects/agents/poc/prototype/README.md create mode 100644 projects/agents/poc/prototype/agentic-vla-workflow-spec.yaml create mode 100644 projects/agents/poc/prototype/runtime-lock.json create mode 100644 projects/agents/poc/prototype/runtime/codex-config.toml create mode 100644 projects/agents/poc/prototype/runtime/model-catalog.json create mode 100755 projects/agents/poc/prototype/runtime/run-agent.sh create mode 100644 projects/agents/poc/prototype/skills/osmo-agentic-workflow/SKILL.md create mode 100644 projects/agents/poc/prototype/skills/osmo-agentic-workflow/agents/openai.yaml create mode 100644 projects/agents/poc/prototype/skills/osmo-agentic-workflow/assets/agent-result.schema.json create mode 100644 projects/agents/poc/prototype/skills/osmo-agentic-workflow/assets/child-workflow-template.yaml diff --git a/projects/agents/01-overview.md b/projects/agents/01-overview.md new file mode 100644 index 000000000..9447f179e --- /dev/null +++ b/projects/agents/01-overview.md @@ -0,0 +1,177 @@ + + +# Agentic Goals: Overview + +Status: Draft + +## Vision + +Give a user one place to state a high-level goal, align on what success means, and authorize a bounded organization of agents and workflows to pursue it. + +The experience begins with: + +```text +/goal +``` + +A lead agent collaborates with the user to turn that objective into a goal contract and a versioned execution plan. After approval, the lead coordinates specialized agents, deterministic tools, and OSMO workflows while preserving a clear path for inspection, steering, approval, cancellation, and evidence-based completion. + +## Product thesis + +OSMO is the execution substrate, not the agentic state machine. + +OSMO is well suited to static, containerized DAG execution across heterogeneous Kubernetes clusters. It already provides task dependencies, resource scheduling, datasets and artifacts, credentials, retries, logs, rendered-spec dry-run, and validation. It does not currently provide dynamic graph expansion, native child workflows, durable conversations, approval gates, or hierarchical agent semantics. + +The agentic layer therefore lives in a new control plane above OSMO: + +```mermaid +flowchart LR + User[User] <--> Lead["Lead session"] + Lead <--> Coordinator["Agent control plane"] + Coordinator <--> GoalGraph["Versioned goal graph"] + Coordinator <--> Policy["Policy and approvals"] + Coordinator -->|"submit static capsule"| OSMO["OSMO API"] + OSMO --> Workflows["Static OSMO workflows"] + Workflows -->|"status and artifacts"| Coordinator + Workers["Bounded agent runs"] -->|"propose results or children"| Coordinator + Coordinator --> Workers +``` + +OSMO validates a complete workflow before submission and materializes its groups and tasks as a static DAG. See the current [workflow submission API](../../external/src/service/core/workflow/workflow_service.py), [workflow schema](../../external/src/utils/job/workflow.py), and [submit job](../../external/src/utils/job/jobs.py). + +## Three distinct graphs + +The design must keep three related structures separate: + +1. **Goal and delegation graph** + - Describes ownership: which agent or deterministic process is responsible for each sub-goal. + - May grow as bounded agents propose additional work. + +2. **Execution graph** + - Describes dependencies, joins, approvals, evaluations, retries, and plan revisions. + - Is dynamic only through durable, versioned, auditable changes. + +3. **OSMO workflow DAG** + - Describes a static execution capsule submitted to OSMO. + - Is immutable after submission; a materially changed plan produces a new workflow or attempt. + +Agent loops and replanning mean the complete system is a durable state machine over graph revisions, not one recursively mutable DAG. + +## Core concepts + +- **Goal**: The user-visible objective and durable root of all work. +- **Goal contract**: Objective, non-goals, acceptance criteria, constraints, deadline, budget, autonomy policy, and permitted capabilities. +- **Plan revision**: An immutable version of the proposed execution graph. +- **Workstream**: A user-comprehensible sub-goal owned by one agent or deterministic process. +- **Node run**: One execution of a workstream contract. +- **Attempt**: One retry or revised strategy for a node run. +- **Agent run**: A bounded model-driven loop with typed inputs, outputs, tools, budget, stop conditions, and evaluator. +- **Workflow capsule**: A static OSMO workflow used for coarse, isolated, resource-intensive, or data-bearing execution. +- **Approval**: A human decision that grants or denies a specific authority envelope. +- **Artifact**: A typed, addressable output used as evidence or as input to later work. +- **Evidence**: Information that supports an acceptance criterion or a material decision. +- **Event**: An append-only record of state changes, decisions, actions, and external bindings. + +## Execution node classes + +The initial system supports three explicit node classes: + +1. **Deterministic job** + - Runs a script, tool, API call, or OSMO task from typed inputs. + - Has predictable control flow even when the external system may fail. + +2. **Bounded agent run** + - Uses a model to reason, select tools, and produce a typed result. + - Is constrained by a tool policy, budget, deadline, maximum steps, and evaluator. + +3. **Constrained agent-tool loop** + - Uses deterministic tools inside a nondeterministic reasoning harness. + - Treats the harness as controlled and auditable without claiming model behavior is deterministic. + +There is no unbounded “pure agent” node. Every node has an enforceable contract and termination policy. + +## Design principles + +- `/goal` creates a draft; it never starts execution by itself. +- Human approval grants bounded authority, not blanket autonomy. +- Plan changes are revisions, not invisible mutations. +- Delegation depth, fan-out, concurrency, spend, compute, tokens, and wall time are bounded. +- Lightweight planning and model calls stay in the agent control plane; OSMO runs coarse execution capsules. +- State lives outside chat and can survive process, model, and UI restarts. +- Agents exchange typed artifacts and messages rather than relying on copied transcripts. +- Completion is based on acceptance criteria and evidence, not agent self-declaration. +- Every external side effect is attributable, policy-checked, and idempotent or compensatable. +- The lead summarizes and governs; it does not ingest every raw child transcript into one context window. + +## End-to-end experience + +```mermaid +flowchart LR + Prompt["/goal prompt"] --> Frame["Frame goal contract"] + Frame --> Plan["Create plan revision"] + Plan --> Preview["Preview and validate"] + Preview --> Approval["Approve authority envelope"] + Approval --> Execute["Execute and coordinate"] + Execute --> Evaluate["Evaluate evidence"] + Evaluate --> Complete["Complete or revise"] + Execute -->|"material change"| Plan + Execute -->|"human decision"| Approval +``` + +The detailed lifecycle is defined in [02-lifecycle.md](02-lifecycle.md). The user experience is defined in [03-ui.md](03-ui.md) and [05-human-interfaces.md](05-human-interfaces.md). + +## Scope + +The first validation should support: + +- One technical user. +- A chat-first `/goal` entry point and a visual run console. +- One lead agent and one bounded level of delegated workers. +- Deterministic jobs, bounded agent runs, and coarse OSMO workflow capsules. +- Versioned plans and explicit approval envelopes. +- One controlled replan path. +- Durable status, evidence, cost, and lineage. +- Goal-wide pause and best-effort cancellation semantics. +- Independent evaluation before completion. + +## Non-goals for the first validation + +- Unlimited recursive delegation. +- Treating every LLM turn or tool call as an OSMO workflow. +- Mutating an in-flight OSMO workflow DAG. +- Exactly-once execution across arbitrary external tools. +- Reproducing nondeterministic model outputs during replay. +- Replacing OSMO workflow, resource, log, event, or shell views. +- Autonomous privilege escalation or unrestricted credential propagation. +- A general-purpose organizational simulation based on the CEO metaphor. + +## First success criteria + +The concept is viable when a representative goal can demonstrate: + +- Recovery after coordinator restart without duplicate OSMO submissions or duplicate side effects. +- End-to-end lineage from the goal prompt through plans, agents, tools, OSMO workflow IDs, artifacts, approvals, and evaluation. +- Enforced delegation, resource, time, and spend limits. +- Human inspection and steering of a nested worker without losing the lead context. +- A failed or timed-out child cannot strand the parent indefinitely. +- Goal-wide stop behavior eventually reconciles all descendants. +- Every completion claim maps to explicit acceptance criteria and evidence. +- The user can always answer: what is happening, why, what changed, what needs attention, what supports completion, and what can be safely stopped. + +## Document map + +- [02-lifecycle.md](02-lifecycle.md): Goal, plan, node, attempt, approval, and termination state machines. +- [03-ui.md](03-ui.md): Chat-first experience and visual run console. +- [04-lead-agent.md](04-lead-agent.md): Lead responsibilities, decision boundaries, and context management. +- [05-human-interfaces.md](05-human-interfaces.md): Approvals, attention routing, steering, notifications, and manual intervention. +- [06-workflow-construction.md](06-workflow-construction.md): Compiling execution nodes into static OSMO workflows. +- [07-agent-construction.md](07-agent-construction.md): Agent manifests, harnesses, capabilities, budgets, and evaluators. +- [08-agent-agent-communication.md](08-agent-agent-communication.md): Typed messages, artifacts, delegation, joins, and event semantics. diff --git a/projects/agents/02-lifecycle.md b/projects/agents/02-lifecycle.md new file mode 100644 index 000000000..ddd5dfe02 --- /dev/null +++ b/projects/agents/02-lifecycle.md @@ -0,0 +1,259 @@ + + +# Agentic Goals: Lifecycle + +Status: Draft + +## Purpose + +Define durable, comprehensible lifecycle semantics for goals that may contain plan revisions, bounded agent loops, deterministic work, human decisions, and multiple OSMO workflow attempts. + +The lifecycle must remain correct when: + +- The coordinator, model runtime, UI, or network restarts. +- A child agent proposes more work. +- A user changes direction during execution. +- An OSMO submission is duplicated, delayed, canceled, or restarted. +- A tool succeeds but its response is lost. +- A parent fails while children remain active. +- Evaluation rejects an apparently successful result. + +## Lifecycle model + +The source of truth is an append-only event history plus transactional projections. Chat messages, model context, and OSMO status are inputs to reconciliation; none is the sole system of record. + +Every transition records: + +- Entity ID and prior state. +- New state and reason. +- Actor: user, lead, worker, policy engine, evaluator, reconciler, tool, or OSMO. +- Plan revision and authority envelope in force. +- Correlation and causation IDs. +- Relevant attempt, artifact, approval, and OSMO workflow IDs. +- Timestamp and idempotency key. + +## Goal lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Draft + Draft --> AwaitingApproval: plan ready + AwaitingApproval --> Draft: revise or reject + AwaitingApproval --> Running: approve and start + Running --> WaitingOnUser: human decision required + WaitingOnUser --> Running: decision received + Running --> Evaluating: execution converged + Evaluating --> Completed: criteria satisfied + Evaluating --> Running: remediation approved + Evaluating --> Blocked: no valid next action + Running --> Blocked: dependency or policy prevents progress + Blocked --> Running: blocker resolved + Draft --> Stopped: abandon + AwaitingApproval --> Stopped: abandon + Running --> Stopping: stop requested + WaitingOnUser --> Stopping: stop requested + Blocked --> Stopping: stop requested + Stopping --> Stopped: descendants reconciled + Running --> Failed: terminal invariant violated + Stopping --> Failed: reconciliation cannot complete +``` + +### Goal states + +- **Draft**: The goal contract and plan may change freely. No execution side effects are permitted. +- **Awaiting approval**: A specific plan revision and authority envelope are ready for a human decision. +- **Running**: The coordinator may dispatch work within the approved envelope. +- **Waiting on user**: Progress is intentionally suspended on a required human decision. Work independent of that decision may continue only if the plan explicitly permits it. +- **Evaluating**: Planned execution has converged and independent acceptance checks are running. +- **Blocked**: No permitted action can currently make progress. The goal may recover when a dependency, policy, credential, resource, or human-provided input changes. +- **Stopping**: No new work is dispatched; the coordinator is reconciling queued and active descendants according to the selected stop mode. +- **Stopped**: The user or policy intentionally ended the goal. Partial artifacts and evidence remain available. +- **Completed**: Acceptance criteria are satisfied with recorded evidence. +- **Failed**: The system cannot preserve a required invariant or has exhausted approved recovery paths. Ordinary child failure does not automatically imply goal failure. + +`Completed`, `Stopped`, and `Failed` are terminal for a goal run. Continuing later creates a new run or an explicit successor linked to the prior run. + +## Plan revision lifecycle + +Plans are immutable after proposal. Editing creates a new revision. + +```mermaid +stateDiagram-v2 + [*] --> DraftPlan + DraftPlan --> Validating: preview requested + Validating --> DraftPlan: validation issue + Validating --> ReadyPlan: checks pass + ReadyPlan --> ApprovedPlan: user grants authority + ReadyPlan --> RejectedPlan: user rejects + ApprovedPlan --> SupersededPlan: newer revision approved + DraftPlan --> SupersededPlan: newer draft selected +``` + +Each revision contains: + +- Goal contract snapshot. +- Initial execution graph. +- Node contracts and evaluators. +- Expected artifacts and joins. +- Tool, model, skill, harness, and image versions. +- OSMO workflow previews where known. +- Expansion limits and approval triggers. +- Resource, token, spend, and time budgets. +- Risk summary, assumptions, and known unknowns. + +A running goal may have only one active approved revision. Work already dispatched under an older revision remains attributable to it and is reconciled by the transition policy. + +## Node lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Proposed + Proposed --> Ready: contract admitted + Proposed --> Skipped: pruned + Ready --> Queued: dispatch selected + Ready --> NeedsApproval: authority missing + NeedsApproval --> Ready: approved + NeedsApproval --> Skipped: rejected + Queued --> Running: attempt starts + Running --> Waiting: dependency or backoff + Waiting --> Running: dependency available + Running --> Verifying: candidate result produced + Verifying --> Succeeded: node criteria pass + Verifying --> Ready: retry or revised attempt + Verifying --> Failed: recovery exhausted + Running --> Ready: retryable failure + Running --> Failed: terminal node failure + Proposed --> Canceled: stop propagation + Ready --> Canceled: stop propagation + Queued --> Canceled: stop propagation + Running --> Canceled: cancellation reconciled + Waiting --> Canceled: stop propagation + NeedsApproval --> Canceled: stop propagation +``` + +### Node invariants + +- A node cannot become `Ready` without a valid contract, dependency set, evaluator, budget allocation, and authority classification. +- A node cannot become `Running` without exactly one active attempt lease. +- A node cannot become `Succeeded` solely because a model or process returned success; its evaluator must accept the result. +- A terminal node retains all attempts, artifacts, messages, and evidence. +- Child creation is a proposal until the coordinator admits it under graph, policy, and budget limits. + +## Attempt lifecycle + +An attempt is the unit of dispatch and external idempotency. + +1. **Created**: Inputs, contract, authority, and idempotency key are frozen. +2. **Dispatching**: The coordinator claims a lease and starts a model harness, tool call, or OSMO submission. +3. **Executing**: The external runtime has acknowledged the attempt. +4. **Reconciling**: The coordinator observes outputs and terminal state, including after a crash. +5. **Succeeded**, **Failed**, **Canceled**, or **Unknown**: Terminal attempt outcome. + +`Unknown` means an external side effect may have happened but cannot yet be proven. The coordinator must reconcile by idempotency key, external identifier, or human review before retrying. + +## Parent, child, and join semantics + +- A parent owns the scope and budget delegated to its children. +- Child state does not directly overwrite parent state. +- Parents declare an explicit join policy: + - All required children succeed. + - Any one child succeeds. + - A quorum succeeds. + - An evaluator decides from available evidence. + - Best effort until deadline or budget exhaustion. +- Optional children may fail or be skipped without failing the parent. +- Required child failure returns control to the parent for retry, replacement, replan, partial completion, or terminal failure. +- Cancellation propagates from parent to descendants; descendant cancellation does not automatically propagate upward. +- Shared dependencies are represented as graph edges, not duplicated children. + +## Retry, restart, and replan + +These are different operations: + +- **Retry**: Repeat the same node contract and strategy with a new attempt. +- **Restart**: Recreate an execution capsule while reusing verified outputs where the runtime supports it. OSMO restart creates a new workflow rather than resuming the old one. +- **Replan**: Change dependencies, strategy, tools, resources, evaluator, or authority envelope in a new plan revision. + +Automatic retry is allowed only when: + +- The failure is classified as transient. +- The node contract and strategy remain unchanged. +- The authority envelope and retry budget permit it. +- Repeating the side effect is idempotent or safe. + +A material change always requires a revision. It may proceed automatically only when the existing approval explicitly authorizes that class of revision. + +## Pause and stop semantics + +OSMO has cancellation but no native workflow pause/resume. The goal-level UI must therefore use precise controls: + +- **Pause coordination** + - Acquire no new dispatch leases. + - Do not create or start new attempts. + - Active attempts and OSMO workflows continue. + +- **Stop pending work** + - Pause coordination. + - Cancel proposed, ready, and queued descendants. + - Let active attempts reach a terminal state. + +- **Stop everything** + - Pause coordination. + - Cancel all pending descendants. + - Send best-effort cancellation to active model, tool, and OSMO attempts. + - Reconcile until every descendant is terminal or explicitly marked unknown. + +The goal remains `Stopping` until reconciliation completes. The UI must not report `Stopped` immediately after a cancellation request. + +## Time and liveness + +Every nonterminal entity has: + +- A deadline or inherited deadline. +- A last-progress timestamp. +- A lease owner and lease expiry when actively coordinated. +- A next reconciliation time. +- A bounded waiting reason. + +The coordinator detects: + +- Expired dispatch leases. +- Attempts with no progress. +- Parents waiting on terminal children with no valid join path. +- Human decisions past their deadline. +- OSMO workflows missing from expected queries. +- Goals with no runnable node and no declared blocker. + +Detected liveness failures produce explicit events and recovery actions; they must not remain silent `Running` states. + +## Recovery and reconciliation + +The reconciler repeatedly compares desired goal state with: + +- Durable node and attempt records. +- Model harness run state. +- Tool idempotency records. +- OSMO workflow and task status. +- Approval decisions. +- Artifact and evaluator results. + +Reconciliation is at-least-once. All transition handlers must therefore be idempotent, and every external dispatch must have a stable client-generated key. + +## Lifecycle acceptance criteria + +- Every UI state has a precise durable counterpart. +- No terminal state has active descendants. +- No attempt can be dispatched twice under the same idempotency key. +- Coordinator restart reconstructs the same desired state from durable records. +- Material plan changes remain visible as revision diffs. +- Evaluation gates completion independently of execution success. +- Pause and stop actions behave exactly as described. +- Stalled goals are detected and surfaced within a defined reconciliation interval. diff --git a/projects/agents/03-ui.md b/projects/agents/03-ui.md new file mode 100644 index 000000000..ff8b02d26 --- /dev/null +++ b/projects/agents/03-ui.md @@ -0,0 +1,361 @@ + + +# Agentic Goals: User Interface + +Status: Draft + +## Product surface + +The initial experience is chat-first with a companion visual run console: + +- **Chat** captures intent, supports alignment, explains decisions, and accepts natural-language questions or steering. +- **Run console** is the durable operational view for plans, hierarchy, status, approvals, evidence, cost, and OSMO bindings. + +The first user is a technical individual running research or engineering goals. The UI should remain useful without requiring the user to adopt the internal agent or graph terminology. + +## UX principles + +- `/goal` always creates a draft. +- State-changing actions are visibly different from questions. +- Plans, approvals, and revisions are durable objects, not buried chat messages. +- Uncertainty is explicit; the UI does not invent exact progress or completion percentages. +- Deep hierarchies are navigated through outline and breadcrumbs, not rendered as one giant graph. +- Human attention is treated as a scarce resource and requested only with sufficient context. +- Evidence is easier to reach than raw agent narration. +- Existing OSMO workflow views remain the source for task-level DAG, logs, events, shell, and spec details. +- Every view has a shareable URL and can be reconstructed after refresh. + +## Entry point + +```text +/goal Train and evaluate a policy that meets the agreed benchmark +``` + +The immediate response: + +1. Creates a draft goal and stable goal URL. +2. Restates the objective and lists assumptions. +3. Shows a compact goal contract card. +4. Asks at most one high-value question at a time. +5. Performs no execution or external mutation. + +The user can continue in chat or open the goal console beside it. + +## Goal contract + +The goal contract remains visible throughout the run and contains: + +- Objective. +- Non-goals. +- Acceptance criteria and evaluator. +- Inputs and expected artifacts. +- Constraints and deadlines. +- Resource, token, and spend budgets. +- Permitted tools, models, skills, harnesses, data, and pools. +- Autonomy and approval policy. +- Delegation depth, fan-out, and concurrency limits. +- Known assumptions and unresolved decisions. + +The contract is editable while the goal is `Draft`. After approval, editing creates a plan revision and impact preview. + +## Primary journey + +### 1. Frame + +The chat leads a focused alignment conversation. The contract card updates as decisions are made and visibly distinguishes: + +- User-stated requirements. +- Lead-inferred assumptions. +- Defaults selected by policy. +- Unknowns that can be deferred. + +The user can correct any extracted item without rewriting the original prompt. + +### 2. Plan + +The lead presents: + +- A concise narrative of the approach. +- A collapsed outline of top-level workstreams. +- Dependencies and join conditions. +- Expected agent and deterministic execution. +- Expected OSMO workflow capsules. +- Acceptance checks and evidence requirements. +- Likely approval points. +- Cost, duration, and uncertainty ranges. + +Each plan is versioned. Selecting a node opens its contract without changing the chat scope. + +### 3. Preview + +“Preview” covers the complete agentic plan. “OSMO dry-run” refers only to rendered OSMO workflow YAML. + +The preview summarizes: + +- Initial graph and critical path. +- Expansion envelope. +- Models, tools, skills, harnesses, images, and versions. +- Credentials and authority required. +- OSMO pool, resource, quota, and validation results. +- Side effects and compensation strategy. +- Human decision points. +- Worst-case approved budget and deadline. +- Known unknowns and unvalidated future branches. + +The preview has one of three outcomes: + +- **Ready**: All required checks passed. +- **Needs input**: A material user decision is missing. +- **Blocked**: Policy, credentials, resources, validation, or another dependency prevents approval. + +### 4. Authorize + +The approval card names the exact scope being granted: + +- Plan revision. +- Allowed capabilities and side-effect classes. +- Maximum spend, compute, tokens, time, depth, fan-out, and concurrency. +- Permitted OSMO pools and resource classes. +- Actions that will still require human approval. + +Primary actions: + +- `Approve plan vN and run` +- `Revise` +- `Save draft` + +The first release should default to guardrailed autonomy: reads, model calls, local computation, and OSMO submissions may proceed inside the approved envelope; external writes, destructive actions, privilege expansion, policy exceptions, and budget increases require approval. + +### 5. Execute + +The lead posts updates only for: + +- A workstream beginning or finishing. +- A material plan revision. +- A blocker or failure that changes the critical path. +- A human decision. +- A budget, deadline, or risk threshold. +- Evaluation and terminal outcome. + +Routine tool calls and polling remain available in history but do not flood the main chat. + +### 6. Inspect and steer + +The user can select any workstream, node, agent run, attempt, artifact, or OSMO binding. + +Two explicit interaction modes avoid accidental mutation: + +- **Ask**: Read-only question about state, reasoning, evidence, or expected impact. +- **Direct**: Proposed instruction that may change desired state. + +A directive first shows whether it: + +- Fits the active authority envelope. +- Produces a plan revision. +- Invalidates completed work. +- Changes budget, deadline, tools, resources, or side effects. +- Requires a new approval. + +### 7. Verify and finish + +A goal enters `Evaluating` before `Completed`. + +The completion view includes: + +- Acceptance criteria with pass, fail, or unresolved status. +- Evidence and evaluator output for each criterion. +- Produced artifacts. +- OSMO workflow and task links. +- Plan deviations and human interventions. +- Time, token, compute, and spend summary. +- Remaining risks and recommended follow-up. + +The user may accept and close, reopen through a new revision, or save the plan as a reusable template. + +## Run console information architecture + +### Header + +- Goal title and status. +- Goal/run ID. +- Active plan revision. +- Elapsed time and deadline. +- Budget consumption and remaining envelope. +- Autonomy policy. +- Attention count. +- Pause and stop menu. + +### Main canvas + +The main view switches among: + +- **Outline**: Default hierarchical view for goals, workstreams, workers, and attempts. +- **Dependencies**: Cross-workstream data/control edges and joins. +- **Timeline**: Planned and actual execution, waits, approvals, retries, and critical path. + +The outline virtualizes and collapses deep hierarchies. The dependency view renders only the selected scope and immediate boundary edges. + +### Context panel + +The context panel contains persistent scoped chat and these tabs: + +- **Overview**: Contract, status, owner, budget, dependencies, and next action. +- **Plan**: Active node definition and relevant revision diff. +- **Evidence**: Claims, evaluator results, artifacts, and citations. +- **Runs**: Attempts, model/tool versions, and OSMO bindings. +- **Logs**: Agent, tool, or linked OSMO logs. +- **History**: Durable events, human decisions, and interventions. + +### Attention inbox + +Each item states: + +- Who or what is requesting attention. +- The decision needed. +- Why it is needed now. +- Available options. +- Supporting evidence. +- Impact of each option. +- Deadline and default behavior if unanswered. +- Whether the decision applies once, to one workstream, or to the remaining goal. + +The inbox separates blocking decisions from informational notifications. + +## Navigation and URL model + +Suggested routes: + +```text +/goals +/goals/{goal_id} +/goals/{goal_id}?node={node_id} +/goals/{goal_id}?node={node_id}&attempt={attempt_id} +/goals/{goal_id}?revision={revision_id} +/goals/{goal_id}?approval={approval_id} +``` + +Selecting hierarchy levels pushes browser history. Switching tabs or canvas modes replaces URL state. Scoped chat always displays its breadcrumb: + +```text +Goal / Workstream / Worker / Attempt +``` + +The existing OSMO UI already uses a graph-plus-resizable-inspector layout and URL-synchronized workflow/group/task navigation. Reuse those interaction patterns from [workflow-detail-layout.tsx](../../external/src/ui/src/features/workflows/detail/components/workflow-detail-layout.tsx) and [use-navigation-state.ts](../../external/src/ui/src/features/workflows/detail/hooks/use-navigation-state.ts). + +## OSMO integration + +Do not duplicate mature OSMO execution views. + +An OSMO-backed attempt should expose: + +- Workflow name, ID, pool, backend, priority, and status. +- Compact task status summary. +- Links to workflow DAG, task, logs, events, shell, spec, dashboard, Grafana, outputs, and datasets. +- Cancel or restart actions routed through goal-level policy and lifecycle semantics. + +Deep-link to the existing workflow detail view for task-level operations. Shell access is an expert intervention and must create an event in the goal history. + +## Status language + +Goal states: + +- `Draft` +- `Awaiting approval` +- `Running` +- `Waiting on user` +- `Evaluating` +- `Completed` +- `Blocked` +- `Failed` +- `Stopping` +- `Stopped` + +Node states: + +- `Proposed` +- `Ready` +- `Queued` +- `Running` +- `Waiting` +- `Needs approval` +- `Verifying` +- `Succeeded` +- `Failed` +- `Canceled` +- `Skipped` + +Do not map nondeterministic work to a percentage. Prefer: + +- Acceptance criteria passed. +- Milestones reached. +- Required joins completed. +- Critical blockers. +- ETA and cost ranges with confidence. + +## Pause and stop controls + +The menu must describe actual semantics: + +- **Pause coordination**: Start no new work; active work continues. +- **Stop pending work**: Cancel queued work; active work continues. +- **Stop everything**: Best-effort cancellation of all descendants. + +The UI enters `Stopping` until reconciliation confirms terminal descendants. It must not claim immediate cancellation. + +## Failure experience + +A failure card shows: + +- Failed node and attempt. +- User-visible impact. +- Evidence and relevant logs. +- Failure classification. +- Automatic action already taken. +- Remaining retry and budget allowance. +- Options: retry, revise, skip when permitted, inspect worker, open OSMO details, or stop. + +Transient retries within the approved envelope can happen silently except when they affect cost, deadline, or confidence. Strategy changes always appear as plan revisions. + +## Accessibility and operational quality + +- All graph information has an equivalent outline representation. +- Status is communicated through text and shape, not color alone. +- Approval and stop actions are keyboard accessible and require unambiguous focus. +- Streaming updates preserve reading position and announce only material changes. +- The UI remains usable with stale data and displays last reconciliation time. +- Large goals use pagination or virtualization for events, attempts, messages, and artifacts. +- Embedded logs keep their own filter state to avoid collisions with goal navigation. + +## First UX validation + +Validate one complete scenario: + +```text +/goal +→ one high-value clarification +→ editable goal contract +→ plan v1 +→ preview with two parallel workers and one OSMO capsule +→ scoped approval +→ live milestones +→ one nested approval +→ one recoverable failure +→ evidence-based completion +``` + +The UX is successful when the user can answer at every point: + +- What is happening? +- Why is it happening? +- What changed? +- What needs my attention? +- What evidence supports completion? +- What can I safely stop? diff --git a/projects/agents/04-lead-agent.md b/projects/agents/04-lead-agent.md new file mode 100644 index 000000000..242cf9a69 --- /dev/null +++ b/projects/agents/04-lead-agent.md @@ -0,0 +1,277 @@ + + +# Agentic Goals: Lead Agent + +Status: Draft + +## Role + +The lead agent is the user-facing planner, coordinator, and narrator for one goal. It helps the user define success, proposes a bounded organization of work, delegates to specialized workers, interprets results, and surfaces decisions. + +The lead is not the durable control plane. + +```mermaid +flowchart LR + User[User] <--> Lead["Lead agent"] + Lead -->|"proposals and commands"| Coordinator["Deterministic coordinator"] + Coordinator -->|"validated state view"| Lead + Coordinator <--> Store["Durable goal state"] + Coordinator <--> Policy["Policy engine"] + Coordinator --> Workers["Agent and deterministic runs"] + Coordinator --> OSMO["OSMO workflows"] +``` + +The lead may propose a plan, child, tool call, workflow, retry, or completion. The coordinator validates authority, state, budgets, graph invariants, and idempotency before applying the proposal. + +## Responsibilities + +### Goal framing + +- Convert `/goal` input into a draft goal contract. +- Separate explicit requirements from assumptions and defaults. +- Ask only questions that materially affect success, risk, time, cost, or authority. +- Define testable acceptance criteria and identify appropriate evaluators. +- Identify non-goals to prevent silent scope growth. + +### Planning + +- Decompose the goal into comprehensible workstreams. +- Distinguish delegation ownership from execution dependencies. +- Choose deterministic work when agent reasoning is unnecessary. +- Choose an OSMO workflow capsule only when the work benefits from cluster scheduling, isolation, data movement, accelerators, or long execution. +- Estimate cost, duration, uncertainty, and likely human decisions. +- Produce a plan that fits the active delegation and budget envelope. + +### Alignment and preview + +- Explain the plan in user language. +- Expose assumptions, alternatives, trade-offs, and unknowns. +- Describe what is known now versus what may be generated later. +- Present the initial graph and bounded expansion policy without claiming future branches have been dry-run. +- Summarize OSMO validation results for known workflow capsules. + +### Delegation + +- Select a worker by declared capability, policy, cost, context need, and evaluator fit. +- Write a precise child contract with scope, inputs, expected artifacts, budget, authority, deadline, and stop condition. +- Delegate only when specialization, parallelism, isolation, or context reduction outweighs coordination overhead. +- Avoid duplicating work already represented by an active node or artifact. +- Review child proposals before admitting further delegation. + +### Coordination + +- Track the critical path and joins through coordinator projections. +- React to material events, not raw polling noise. +- Classify failures as transient, strategy-related, policy-related, resource-related, or terminal. +- Propose retries only when the contract and strategy remain unchanged. +- Propose a plan revision for changed strategy, dependencies, tools, resources, side effects, or evaluation. +- Prevent one failed optional child from unnecessarily failing the complete goal. + +### Human interface + +- Keep the main thread focused on decisions and material progress. +- Surface approval requests with alternatives, evidence, impact, deadline, and default behavior. +- Route the user to a scoped worker conversation when detailed domain interaction is useful. +- Preserve an easy return path to the lead and summarize any nested decision. +- Explain pause, stop, retry, and replan effects before requesting action. + +### Evaluation and completion + +- Assemble candidate outputs and evidence. +- Invoke independent evaluators defined by the goal and node contracts. +- Map evaluator results to acceptance criteria. +- Propose remediation when criteria are not met and authority remains. +- Report completion only after the coordinator records accepted evaluation evidence. +- Summarize results, artifacts, deviations, interventions, spend, duration, and unresolved risks. + +## Non-responsibilities + +The lead must not: + +- Act as the source of truth for lifecycle state. +- Directly mutate the database, graph, policy, approval, or budget state. +- Bypass the coordinator to submit or cancel OSMO workflows. +- Grant itself broader authority, credentials, budget, or delegation rights. +- Treat model confidence as evidence. +- Approve its own high-risk actions. +- Declare success without an evaluator. +- Copy all child transcripts into its context. +- Depend on hidden conversational memory for recovery. +- Spawn workers merely to imitate an organizational hierarchy. +- Convert every tool call into an OSMO workflow. + +## Lead control loop + +```mermaid +flowchart TD + Observe["Observe durable projection"] --> Decide["Identify next material decision"] + Decide --> Propose["Propose plan, dispatch, question, or evaluation"] + Propose --> Validate["Coordinator validates"] + Validate -->|"accepted"| Wait["Wait for material event"] + Validate -->|"needs human"| Ask["Surface human decision"] + Validate -->|"rejected"| Revise["Revise proposal"] + Ask --> Observe + Wait --> Observe + Revise --> Observe +``` + +The lead runs when: + +- The user sends a message. +- A material goal event occurs. +- An approval or blocker is created. +- A join becomes satisfiable. +- Evaluation finishes. +- A liveness or budget threshold is crossed. + +It does not need to remain alive between events. Any model instance can resume from the durable projection and referenced artifacts. + +## Input projection + +The coordinator supplies a bounded, structured view: + +- Goal contract and active plan revision. +- Current goal status and next valid actions. +- Top-level execution outline and critical path. +- Open approvals, blockers, and deadlines. +- Budget allocation and consumption. +- Material events since the prior lead turn. +- Child summaries, artifact references, and evaluator results. +- OSMO workflow summaries and deep links. +- Relevant policy constraints and catalog entries. + +Raw logs, complete worker transcripts, and large artifacts remain out of context unless the lead explicitly requests a bounded excerpt or summary. + +## Lead outputs + +Lead outputs must use typed proposals rather than free-form side effects: + +- `ProposeGoalContract` +- `ProposePlanRevision` +- `RequestClarification` +- `ProposeNode` +- `ProposeDispatch` +- `ProposeRetry` +- `ProposeEvaluation` +- `RequestApproval` +- `ProposePause` +- `ProposeStop` +- `ProposeCompletion` +- `PostUserUpdate` + +Each proposal contains: + +- Goal, revision, node, and attempt references as applicable. +- Rationale. +- Expected state transition. +- Required authority and budget. +- Idempotency or deduplication key. +- Evidence references. +- User-visible summary. + +The coordinator rejects malformed, stale, unauthorized, or invariant-breaking proposals and returns a structured reason. + +## Planning strategy + +The lead should prefer the smallest useful organization. + +Before creating a child, it asks: + +1. Does this work have a distinct, testable output? +2. Does it require expertise, context, tools, isolation, or parallelism the parent lacks? +3. Is the expected value greater than delegation and join overhead? +4. Can the input and output be expressed as a stable contract? +5. Is there a clear evaluator and stop condition? +6. Does the remaining envelope permit it? + +If not, the lead handles the work directly or uses a deterministic step. + +## Worker selection + +The lead selects from immutable catalog snapshots. Selection considers: + +- Declared capability and supported artifact types. +- Tool and data access. +- Model quality, latency, cost, context, and policy class. +- Harness behavior and maximum runtime. +- Required compute and whether OSMO execution is appropriate. +- Historical evaluator performance for the task class. +- Data residency, confidentiality, and credential constraints. + +The lead may recommend a catalog change but cannot silently substitute an unapproved model, tool, skill, harness, or image. + +## Context management + +- Store source artifacts once and pass references. +- Require workers to return typed results, evidence, unresolved questions, and a compact summary. +- Build lead context from current state and material deltas, not full chronological history. +- Preserve provenance from each claim to the producing attempt and artifact. +- Summarize at workstream boundaries and invalidate summaries when their source artifacts are superseded. +- Mark untrusted content and prevent artifacts from silently becoming system instructions. + +## Plan revisions + +The lead creates a new revision when: + +- User intent or acceptance criteria change. +- Dependencies or workstream structure change. +- A new tool, model, skill, harness, image, pool, or resource class is needed. +- Side-effect or privilege scope expands. +- Budget or deadline changes. +- Completed work is invalidated. +- An evaluator or evidence requirement changes. + +The revision explains: + +- What changed and why. +- Which existing work remains valid. +- Which queued or active work should continue, drain, or stop. +- Cost, time, risk, and authority impact. +- Whether approval is required. + +## Failure behavior + +The lead must avoid both premature abandonment and unbounded recovery. + +- Transient failure: propose bounded retry. +- Invalid worker result: request remediation or replacement. +- Failed optional work: continue if join policy permits. +- Failed required work: revise strategy, request input, or declare blocker. +- OSMO failure: use status, events, logs, and artifacts to classify before restart or replan. +- Unknown side effect: do not retry until reconciliation or human review. +- Budget or deadline exhaustion: stop dispatching and surface alternatives. +- Lead model failure: preserve state and resume with another compatible lead instance. + +## Human communication style + +- Lead with outcome, blocker, or decision. +- Distinguish fact, inference, assumption, and recommendation. +- Use stable names for workstreams and artifacts. +- Avoid narrating every internal thought or tool call. +- Quantify cost and time as ranges when uncertainty is material. +- Explain why a human is needed and what happens without a response. +- Never obscure a material plan change inside a progress update. + +## Evaluation plan + +Evaluate the lead on complete goal traces, not isolated prompt quality: + +- Goal contracts capture stated intent without inventing constraints. +- Clarification count remains low without sacrificing correctness. +- Plans use deterministic work where appropriate. +- Delegation produces independently useful, testable outputs. +- Child creation remains inside depth, fan-out, budget, and authority limits. +- Required approvals are surfaced before side effects. +- Failure classification selects retry versus replan correctly. +- Context remains bounded as the graph grows. +- User updates are timely but not noisy. +- Completion claims match evaluator evidence. +- A replacement lead can resume from durable state without conversational loss. diff --git a/projects/agents/05-human-interfaces.md b/projects/agents/05-human-interfaces.md new file mode 100644 index 000000000..0e5b186e4 --- /dev/null +++ b/projects/agents/05-human-interfaces.md @@ -0,0 +1,349 @@ + + +# Agentic Goals: Human Interfaces + +Status: Draft + +## Purpose + +Define how a human aligns, authorizes, observes, steers, interrupts, and evaluates an agentic goal without becoming the manual scheduler for every nested worker. + +Human interaction is a first-class protocol. It is not an ad hoc pause in an agent transcript. + +## Principles + +- Ask for human attention only when it can change an outcome or authority. +- State exactly what decision is needed and why automation cannot make it. +- Bind approval to an immutable scope and plan revision. +- Distinguish questions from state-changing directives. +- Let the user inspect any nested scope without losing the lead context. +- Record manual interventions and their downstream impact. +- Define deadline and default behavior for every blocking request. +- Never treat silence as approval for a new side effect or expanded authority. +- Preserve a useful result when the user stops or abandons a partially completed goal. + +## Human interaction classes + +### Alignment + +Used while the goal is a draft: + +- Clarify objective or non-goals. +- Define acceptance criteria. +- Select among meaningful strategies. +- Set constraints, deadline, budget, and autonomy. +- Confirm inferred assumptions. +- Supply missing input or credentials. + +Alignment does not grant execution authority. + +### Approval + +Used when the system needs explicit authority: + +- Approve a plan revision and start execution. +- Permit an external write or irreversible action. +- Expand tool, model, data, credential, network, privilege, pool, or resource scope. +- Increase budget, deadline, delegation depth, fan-out, or concurrency. +- Accept a material strategy change. +- Authorize retry of an uncertain or non-idempotent side effect. +- Accept partial completion or waive an acceptance criterion. + +### Information request + +Used when the system needs domain input but not authority: + +- Choose a dataset or benchmark. +- Explain ambiguous source material. +- Resolve a business or scientific preference. +- Provide missing environmental context. + +### Notification + +Used for material but nonblocking updates: + +- Major milestone. +- Critical-path change. +- Automatic recovery that affects confidence, time, or cost. +- Approaching budget or deadline threshold. +- Evaluation result. +- Terminal outcome. + +### Intervention + +Initiated by the user: + +- Ask the lead or a worker for explanation. +- Add context or evidence. +- Propose a directive. +- Pause coordination. +- Stop pending work. +- Stop everything. +- Open an OSMO workflow, log, event, or shell. + +## Authority envelope + +Approval grants a bounded envelope, not general autonomy. + +An envelope records: + +- Goal and immutable plan revision. +- Effective user and approving identity. +- Allowed agent, model, skill, harness, tool, and image versions. +- Permitted data, repositories, services, network destinations, credentials, and OSMO pools. +- Allowed side-effect and risk classes. +- Spend, token, compute, time, depth, fan-out, concurrency, retry, and OSMO submission limits. +- Actions that always require additional approval. +- Expiration and revocation conditions. + +Suggested presets: + +### Supervised + +- Read-only reasoning and preview may proceed. +- Every model/tool dispatch, child creation, workflow submission, and mutation requires approval. + +### Guardrailed + +- Recommended initial preset. +- Reads, approved model calls, local computation, child creation, and OSMO submissions may proceed inside the envelope. +- External writes, destructive actions, privilege expansion, policy exceptions, uncertain retries, and envelope changes require approval. + +### Broad autonomy + +- Most actions inside the envelope proceed. +- Destructive actions, privilege expansion, policy exceptions, and envelope changes still require approval. +- Not required for the first validation. + +## Approval request contract + +Every approval request contains: + +- Stable approval ID. +- Requesting goal, workstream, node, and agent. +- Current plan revision and proposed revision when applicable. +- One concise decision statement. +- Reason the decision is needed now. +- Recommended option and alternatives. +- Evidence and relevant artifacts. +- Expected effect on outcome, cost, time, risk, and completed work. +- Exact authority to be granted. +- Whether the grant applies once, to one workstream, or to the remaining goal. +- Deadline. +- Safe default if unanswered. +- Idempotency key for the resulting action. + +Available decisions: + +- Approve once. +- Approve this class for the current workstream. +- Approve this class for the remaining goal within a displayed limit. +- Edit and approve. +- Reject. +- Defer. +- Ask a question without deciding. + +## Approval lifecycle + +```mermaid +stateDiagram-v2 + [*] --> Pending + Pending --> Approved: human approves + Pending --> Rejected: human rejects + Pending --> Deferred: human defers + Deferred --> Pending: reopened + Pending --> Expired: deadline passes + Pending --> Superseded: plan or state changes + Approved --> Consumed: action committed + Approved --> Revoked: human revokes before commit + Approved --> Superseded: bound state changes +``` + +An approval is consumed only when the authorized transition commits. A stale approval cannot authorize a changed plan, input, target, cost, or side effect. + +## Attention inbox + +The inbox is the canonical list of unresolved human requests. + +It separates: + +- **Blocking now**: No valid critical-path action can continue. +- **Blocking later**: Independent work continues, but a future join depends on the answer. +- **Review requested**: The system recommends inspection but can proceed under current authority. +- **Informational**: No decision required. + +Ordering considers: + +- Critical-path impact. +- Deadline. +- Cost of waiting. +- Risk. +- Number of descendants blocked. + +The lead batches compatible low-risk decisions where doing so does not obscure scope. + +## Chat scope + +Global chat addresses the lead. A user may enter a nested scope through `Talk to this worker`. + +Every scoped conversation displays: + +- Goal and workstream breadcrumb. +- Worker identity and contract. +- Current attempt and plan revision. +- Whether the worker is active, waiting, or terminal. +- `Return to lead`. + +The worker receives only the scoped message and relevant contract/artifact references. It does not inherit unrestricted authority from the user merely because the user opened its chat. + +When the user returns: + +- The nested conversation is summarized into a durable decision or context artifact. +- The lead receives the summary and any proposed plan change. +- Material directives still pass through coordinator and policy validation. + +## Ask versus Direct + +The composer has two explicit modes: + +### Ask + +- Read-only. +- May query state, reasoning, evidence, logs, expected impact, or alternatives. +- Cannot dispatch work, modify desired state, increase authority, or cancel execution. + +### Direct + +- Proposes a state-changing instruction. +- Shows affected scope before submission. +- Produces a plan diff or runtime action preview. +- States whether current authority is sufficient. +- Requires approval when the instruction exceeds the envelope. + +Natural language can suggest a mode, but the UI must make the final mode visible before committing. + +## Steering semantics + +Steering may: + +- Add context or an artifact. +- Change priority among ready nodes. +- Request a new workstream. +- Change strategy or evaluator. +- Replace a worker. +- Revise budget or deadline. +- Pause or stop a scope. + +The preview identifies: + +- Work that remains valid. +- Work invalidated or made obsolete. +- Active attempts that should continue, drain, or cancel. +- Added cost and time. +- New authority or human decisions. +- Revised acceptance criteria. + +Steering never edits historical plan revisions or attempt records. + +## Pause and stop + +Human controls use the lifecycle definitions from [02-lifecycle.md](02-lifecycle.md): + +- **Pause coordination**: Start no new work; active work continues. +- **Stop pending work**: Cancel queued work; active work continues. +- **Stop everything**: Best-effort cancel all descendants. + +Before commitment, show: + +- Number of pending and active nodes. +- Active OSMO workflows and whether their outputs may be lost. +- Non-cancelable or uncertain side effects. +- Estimated time to reconcile. +- Artifacts already preserved. + +The user may apply the action to one node, one workstream, or the whole goal. + +## Manual OSMO intervention + +The user may open existing OSMO workflow detail, logs, events, dashboards, or shell. + +- Read-only inspection requires no additional goal transition. +- Cancel, restart, resubmit, exec, port-forward, rsync, or shell commands are recorded as interventions. +- State-changing actions should be initiated through the goal console when possible so policy and lifecycle semantics remain consistent. +- If an action occurs directly in OSMO, the reconciler records external intervention and evaluates whether the plan is still valid. +- Shell access is considered an elevated expert action because it may alter workload state outside the declared node contract. + +## Notification policy + +Default delivery remains in the active chat and console. Optional external channels may notify for: + +- Blocking approval. +- Security or policy event. +- Budget or deadline threshold. +- Goal completion, failure, or stop. + +Notifications contain no secrets or large artifacts and link to the durable request. + +Users can configure: + +- Quiet hours. +- Severity threshold. +- Digest versus immediate delivery. +- Goal-specific overrides. +- Escalation target when a deadline approaches. + +Repeated polling or retry events are aggregated rather than emitted individually. + +## Unanswered requests + +Every request declares a safe default: + +- Continue independent work. +- Pause affected work. +- Reject the proposed action. +- Stop the affected scope. +- Escalate to another authorized human. + +No unanswered request defaults to expanded authority, destructive action, or irreversible side effect. + +When a request expires, the system records the default action and explains its impact in the lead conversation. + +## Conflicts and concurrency + +- Human decisions use optimistic concurrency against the plan revision and entity version. +- If state changes while an approval card is open, the card becomes stale and displays the replacement request. +- Conflicting directives from different authorized humans are resolved by explicit policy, not last-write-wins chat order. +- Revoking authority stops new dispatch immediately and reconciles active work according to the revocation policy. +- A human can override an agent recommendation but cannot bypass platform security or tenancy policy. + +## Trust and evidence + +Human-facing claims distinguish: + +- Observed fact. +- Tool or OSMO result. +- Agent inference. +- Assumption. +- Recommendation. + +Approval cards cite the source artifact or event. Untrusted artifact text is not rendered as an instruction. Sensitive inputs are redacted according to policy before entering model context, chat, notification, or audit views. + +## Acceptance criteria + +- No execution starts from `/goal` without a scoped approval. +- Every side effect can be traced to an envelope and actor. +- Stale approvals cannot authorize changed work. +- The user can inspect and converse with any worker while retaining a clear return to the lead. +- Questions cannot accidentally mutate state. +- Stop controls accurately describe and eventually reflect descendant state. +- Blocking requests always explain why a human is needed and what happens without a response. +- Notification volume remains bounded during long-running goals. +- Manual OSMO intervention is visible in goal history. diff --git a/projects/agents/06-workflow-construction.md b/projects/agents/06-workflow-construction.md new file mode 100644 index 000000000..25417a887 --- /dev/null +++ b/projects/agents/06-workflow-construction.md @@ -0,0 +1,316 @@ + + +# Agentic Goals: Workflow Construction + +Status: Draft + +## Purpose + +Define how the agent control plane turns an approved portion of the execution graph into a static, validated, attributable OSMO workflow capsule. + +The workflow constructor is a deterministic compiler and submission adapter. An agent may propose inputs to it, but an agent does not directly produce trusted executable YAML or bypass validation. + +## OSMO execution boundary + +Use an OSMO workflow capsule when work benefits from one or more of: + +- Kubernetes isolation. +- GPU or specialized resource scheduling. +- Multi-node or gang execution. +- Heterogeneous backend selection. +- Long-running computation. +- Large input or output movement. +- Checkpointing. +- Container-specific dependencies. +- OSMO-native logs, events, metrics, shell, or dashboards. + +Keep work in the agent control plane when it is: + +- Goal framing or planning. +- A lightweight model call. +- Approval routing. +- A low-latency API query. +- A small deterministic transformation. +- Coordinator reconciliation. +- Agent-to-agent message handling. + +Do not submit a workflow for every model turn or tool invocation. + +## Current OSMO constraints + +The constructor must compile to current OSMO semantics: + +- A workflow contains either tasks or groups; top-level tasks are normalized into one-task groups. +- Dependencies are static and represented through task inputs. +- Groups are the scheduling dependency unit. +- The complete graph is rendered and validated before submission. +- Jinja loops and conditionals expand at submission time, not at runtime. +- There is no native child-workflow node or in-flight graph expansion. +- There is no runtime branch or general runtime loop primitive. +- Pool and backend placement are fixed when submitted. +- Workflow pause/resume is not available. +- Cross-workflow task inputs require the referenced prior task to be finished. +- Restart creates a new workflow and may reuse completed outputs. +- Workflow and per-user task limits constrain capsule size; the default workflow task cap is currently 20. + +See [WorkflowSpec](../../external/src/utils/job/workflow.py), [TaskSpec and TaskGroupSpec](../../external/src/utils/job/task.py), [submission](../../external/src/service/core/workflow/workflow_service.py), and [DAG materialization](../../external/src/utils/job/jobs.py). + +## Compiler inputs + +The constructor receives an immutable `WorkflowConstructionRequest`: + +- Goal, plan revision, node, and attempt IDs. +- Approved execution subgraph. +- Node contracts and dependency edges. +- Resolved input artifacts and immutable versions. +- Selected tool, agent, model, harness, and container image manifests. +- OSMO pool, priority, resource, timeout, credential, and data policies. +- Output artifact contracts and evaluators. +- Authority envelope. +- Client-generated submission idempotency key. + +No mutable chat transcript or ambient environment is an implicit compiler input. + +## Compiler output + +The deterministic result contains: + +- Canonical OSMO workflow template. +- Fully rendered dry-run spec. +- Validation result. +- Input and output binding manifest. +- Goal-to-workflow node mapping. +- Required credentials and policy decisions. +- Resource and quota summary. +- Expected cost and time range. +- Source hashes and compiler version. +- Submission idempotency key. + +The canonical result is stored before submission and is immutable for the attempt. + +## Construction pipeline + +```mermaid +flowchart LR + Select["Select approved subgraph"] --> Freeze["Freeze inputs and manifests"] + Freeze --> Partition["Partition execution capsules"] + Partition --> Compile["Compile OSMO template"] + Compile --> DryRun["Render dry-run"] + DryRun --> Validate["OSMO and policy validation"] + Validate --> Record["Record immutable construction"] + Record --> Submit["Idempotent submit"] + Submit --> Bind["Bind OSMO workflow ID"] + Bind --> Reconcile["Reconcile status and artifacts"] +``` + +### 1. Select + +Choose a connected, ready portion of the approved execution graph whose dependencies are satisfied or can be represented inside one static OSMO DAG. + +### 2. Freeze + +Resolve and pin: + +- Input artifact versions and checksums. +- Images and digests. +- Tools, models, skills, and harness versions. +- Commands, arguments, environment, and files. +- Credentials by reference. +- Pool, resource, timeout, priority, retry, checkpoint, and output policy. + +### 3. Partition + +Split capsules at boundaries such as: + +- Different OSMO pools or backends. +- Different security or credential scopes. +- Human approval gates. +- Runtime-discovered fan-out. +- Dynamic agent replanning. +- Cross-region or data residency constraints. +- Distinct failure or cancellation domains. +- Task count and quota limits. +- Long waits that should not occupy a workflow. + +Prefer one capsule when tasks form a stable, data-connected DAG and benefit from one submission. Prefer separate capsules when coordination is dynamic or lifecycle ownership differs. + +### 4. Compile + +Generate a canonical OSMO template using only schema-supported fields. Generated names must be deterministic, Kubernetes-safe, and traceable to goal entities without exposing sensitive content. + +### 5. Render and validate + +Use OSMO dry-run to render Jinja and variables, then validation-only mode to check workflow structure, pool, resources, credentials, registries, quotas, and platform constraints. + +Agentic preview and OSMO dry-run remain distinct: + +- Agentic preview describes the known plan and expansion envelope. +- OSMO dry-run validates one known static capsule. + +### 6. Record + +Persist the template, rendered spec, validation result, hashes, bindings, and authority decision before creating external side effects. + +### 7. Submit idempotently + +OSMO submission does not expose a general client idempotency key. The control plane therefore maintains a submission ledger: + +- Reserve one idempotency key transactionally. +- Submit at most one workflow for that key. +- Record the returned workflow name and UUID. +- On ambiguous failure, reconcile by stored response, deterministic metadata, or operator review before retrying. +- Never create a second attempt under the same key. + +### 8. Reconcile + +Poll OSMO workflow state, logs, events, and task outputs. Convert OSMO state into attempt events without treating transient query failure as workflow failure. + +## Capsule granularity + +A capsule should be large enough to amortize Kubernetes and OSMO scheduling overhead but small enough to preserve: + +- Independent retry and cancellation. +- Clear artifact contracts. +- Security boundaries. +- Human approval boundaries. +- Dynamic replanning points. +- Resource placement. +- Understandable failure impact. + +Candidate heuristics: + +- Combine stable deterministic producer/consumer tasks in one capsule. +- Keep runtime agent decision boundaries outside a static capsule unless the complete bounded loop intentionally runs inside one container. +- Do not combine tasks that require different pools. +- Do not hold a capsule open waiting for a human decision. +- Avoid a capsule whose failure would force unrelated completed work to rerun. + +## Mapping node classes + +### Deterministic job + +Compile directly to an OSMO task when cluster execution is warranted. + +- Typed inputs become task or URL inputs. +- Command and arguments come from a pinned tool manifest. +- Outputs are written to `{{output}}` and registered as artifacts. +- Exit actions handle known process codes. + +### Bounded agent run + +Two execution modes are possible: + +1. **Control-plane agent** + - Preferred for planning, lightweight tools, and rapid interaction. + - May submit separate OSMO capsules through coordinator proposals. + +2. **OSMO-hosted agent** + - Used when the agent requires GPU inference, specialized dependencies, data locality, strong isolation, or long execution. + - Runs a complete bounded harness in one OSMO task or stable task group. + - Returns proposals and artifacts to the coordinator; it does not gain unrestricted OSMO credentials. + +### Constrained agent-tool loop + +Package the harness and approved deterministic tools into a pinned image when execution locality justifies OSMO. Keep dynamic child creation in the external coordinator. + +## Dependencies and artifacts + +### Inside one workflow + +Use task inputs for both data handoff and scheduling dependencies. The producer writes to its output directory; OSMO transfers the output to the consumer. + +### Across workflows + +The coordinator waits for the producer artifact to become durable and verified before constructing the consumer capsule. It may reference the completed prior task output or a stable external URL/dataset. + +Do not use an unfinished cross-workflow reference as a substitute for external coordination. + +### Non-data dependencies + +OSMO task inputs couple dependency and data movement. If a dependency carries no artifact, the constructor should use a small manifest artifact or split the work into separate capsules coordinated externally rather than inventing unsupported control edges. + +## Groups + +An OSMO group is a gang-scheduled set of tasks, not an agent team or hierarchy. + +Use a group only when tasks must start and execute together, such as distributed training or tightly coupled services. Define the lead task and barrier behavior deliberately; do not map every delegated workstream to a group. + +## Generated workflow shape + +Illustrative output: + +```yaml +version: 2 +workflow: + name: goal-abc-node-def-attempt-01 + timeout: + exec: 4h + queue: 1h + tasks: + - name: execute + image: registry.example/approved-tool@sha256:... + command: ["/app/run"] + args: ["--input", "{{input:0}}", "--output", "{{output}}"] + inputs: + - url: s3://approved-artifacts/input-version + outputs: + - url: s3://approved-artifacts/goal-abc/node-def/attempt-01 + environment: + GOAL_RUN_ID: goal-abc + NODE_RUN_ID: node-def + ATTEMPT_ID: attempt-01 +``` + +Goal metadata in environment variables is for traceability, not authorization. The coordinator remains the source of authority. + +## Failure and retry + +- Map OSMO terminal status to an attempt result, not directly to goal result. +- Use task reschedule only for known transient process outcomes and configured retry limits. +- Use OSMO restart when the same workflow strategy should rerun while reusing completed outputs. +- Generate a new construction request for changed strategy, resources, tools, graph, or outputs. +- Treat cancellation as best effort until OSMO reports a terminal state. +- Preserve logs, events, rendered spec, and partial artifact references for diagnosis. + +## Security + +- Accept only cataloged image digests and tool manifests. +- Validate `privileged`, `hostNetwork`, mounts, credentials, and network needs against policy. +- Pass credentials by reference and least-privilege injection; never place secret values in generated YAML, prompts, or environment manifests stored as evidence. +- Do not grant an OSMO-hosted agent a general user token when a scoped callback or capability token suffices. +- Redact rendered specs and logs before exposing them to models. +- Sign or hash the canonical construction to detect post-approval changes. + +## Observability and lineage + +Record: + +- Goal, plan, node, and attempt IDs. +- Constructor and schema versions. +- Template and rendered-spec hashes. +- Input and image digests. +- Validation and policy results. +- Submission idempotency key. +- OSMO workflow name, UUID, pool, backend, and task mapping. +- Status transitions. +- Logs, events, outputs, and evaluator references. +- Cancel, restart, resubmit, exec, shell, or other interventions. + +## Acceptance criteria + +- The same frozen request produces byte-equivalent canonical output. +- No unvalidated workflow can be submitted. +- Every OSMO workflow maps to exactly one attempt and authority envelope. +- Ambiguous submission does not create duplicate workflows. +- Dynamic agent decisions occur outside static OSMO DAGs. +- Cross-workflow consumers cannot start before required artifacts are durable and verified. +- Security-sensitive fields are policy-checked and provenance is retained. +- OSMO failure, restart, and cancellation remain attempt-level events rather than bypassing goal lifecycle. diff --git a/projects/agents/07-agent-construction.md b/projects/agents/07-agent-construction.md new file mode 100644 index 000000000..d83554d3d --- /dev/null +++ b/projects/agents/07-agent-construction.md @@ -0,0 +1,363 @@ + + +# Agentic Goals: Agent Construction + +Status: Draft + +## Purpose + +Define how a catalog of models, tools, skills, harnesses, images, policies, and evaluators becomes one immutable, bounded agent run. + +An agent is not merely a model plus a prompt. It is a versioned execution contract with enforceable capabilities, limits, inputs, outputs, and evaluation. + +## Agent definition + +An agent definition contains: + +- Stable name, version, owner, and purpose. +- Declared capabilities and task classes. +- Accepted input artifact schemas. +- Required output artifact schemas. +- Model selection policy. +- Harness and prompting strategy. +- Tool and skill allowlist. +- Execution environment and optional image. +- Context assembly and memory policy. +- Delegation policy. +- Authority and credential requirements. +- Step, token, spend, compute, and wall-time limits. +- Stop conditions. +- Failure and retry behavior. +- Evaluators. +- Security and data-handling classification. + +Agent definitions are immutable after publication. Changes produce a new version. + +## Agent run construction + +```mermaid +flowchart LR + Contract["Node contract"] --> Select["Select agent definition"] + Catalog["Catalog snapshot"] --> Select + Select --> Resolve["Resolve model, tools, skills, harness, image"] + Resolve --> Context["Assemble bounded context"] + Context --> Policy["Policy and authority validation"] + Policy --> Freeze["Freeze AgentRunSpec"] + Freeze --> Execute["Execute bounded harness"] + Execute --> Evaluate["Validate result"] + Evaluate --> Record["Record artifacts and evidence"] +``` + +The deterministic constructor produces an immutable `AgentRunSpec`. The model does not select or mutate its own enforcement limits after execution begins. + +## AgentRunSpec + +Each run freezes: + +- Goal, plan revision, node, and attempt IDs. +- Agent definition and catalog snapshot versions. +- Input contracts, artifact references, checksums, and trust labels. +- Expected output and evidence schemas. +- Selected model and parameters. +- Harness version and system instructions. +- Tool and skill manifests. +- Capability and credential tokens. +- Execution placement. +- Delegation allowance. +- Step, token, spend, compute, and time budgets. +- Stop conditions and deadlines. +- Evaluator definitions. +- Idempotency and correlation IDs. + +The run spec is stored before model execution. + +## Node contract + +Every agent begins with a node contract containing: + +- One bounded sub-goal. +- Why the work exists and how it contributes to the parent. +- Explicit non-goals. +- Typed inputs. +- Expected output artifacts. +- Acceptance criteria. +- Permitted tools, data, and side effects. +- Delegation allowance. +- Budget and deadline. +- Required human approvals. +- Join or handoff target. + +If the contract is ambiguous enough to change execution materially, the agent returns a clarification request instead of silently expanding scope. + +## Model selection + +Model selection is policy-driven and frozen per attempt. + +Selection may consider: + +- Capability and evaluator performance for the task class. +- Context size and modality. +- Tool-calling support. +- Latency and cost. +- Data confidentiality and residency. +- Availability and rate limits. +- Required reasoning depth. +- Execution environment. + +The agent may recommend escalation to another model, but the coordinator validates availability, policy, and budget before creating a new attempt. + +Model output is always treated as nondeterministic. Temperature or a deterministic harness does not make the complete agent deterministic. + +## Harness + +The harness controls the agent loop: + +1. Load the frozen run spec and bounded context. +2. Ask the model for a typed next action. +3. Validate the action against state, schema, capability, policy, and remaining budget. +4. Execute an approved read, tool call, proposal, or response. +5. Record the action, result, cost, and evidence reference. +6. Update bounded working context. +7. Stop on accepted output, clarification, approval, delegation proposal, budget/deadline, cancellation, or unrecoverable failure. + +The harness, not the prompt, enforces: + +- Maximum steps. +- Tool allowlist. +- Argument schemas. +- Timeouts. +- Output size. +- Token and spend limits. +- Delegation bounds. +- Side-effect gating. +- Cancellation. + +## Agent classes + +### Lead agent + +Defined in [04-lead-agent.md](04-lead-agent.md). It receives a goal-level projection, proposes plans and coordination actions, and communicates with the user. + +### Worker agent + +Owns one bounded node contract and returns a typed result, evidence, clarification, approval request, or child proposal. + +### Evaluator agent + +Judges a candidate artifact against explicit criteria. It must not be the same run that produced the candidate when independent evaluation is required. + +### Specialist agent + +Provides domain-specific analysis or tool operation under a narrow capability set. Specialization should reduce context and authority, not merely change persona wording. + +## Tools + +Every tool manifest declares: + +- Stable name and version. +- Description and capability class. +- Typed input and output schemas. +- Read or side-effect classification. +- Idempotency support. +- Compensation behavior. +- Authentication and credential scope. +- Network, filesystem, and environment needs. +- Timeout and output limits. +- Data sensitivity constraints. +- Execution location. +- Audit and redaction rules. + +Model-generated arguments are schema-validated. High-risk arguments may require deterministic policy checks or human approval even when the tool itself is allowlisted. + +## Skills + +A skill is reusable procedural guidance and supporting resources, not an authority grant. + +- Pin skill version in the run spec. +- Treat skill instructions as lower priority than platform policy and the node contract. +- Declare the tools and side effects a skill expects. +- Evaluate skill behavior with representative traces. +- Do not let a skill silently widen tool, credential, data, or delegation access. +- Keep skill content out of context unless selected for the current task. + +The existing OSMO Agent Skills demonstrate resource selection, workflow generation, submission, monitoring, diagnosis, and retries outside OSMO core. See [external/skills](../../external/skills/README.md). + +## Context construction + +Context is assembled from durable references: + +- Node contract. +- Relevant goal contract subset. +- Active plan revision subset. +- Typed input artifacts. +- Parent handoff. +- Applicable policy. +- Tool and skill instructions. +- Material prior-attempt summary when retrying. + +Avoid: + +- Full goal event history. +- Complete parent or sibling transcripts. +- Unbounded logs. +- Duplicate large artifacts. +- Secrets not required by the model. +- Treating artifact content as trusted system instructions. + +Context artifacts carry provenance and trust labels. Large data is accessed through tools or OSMO inputs rather than copied into the prompt. + +## Memory + +Working memory is attempt-local and disposable. + +Durable memory consists only of explicit artifacts: + +- Result. +- Evidence. +- Summary. +- Open questions. +- Learned constraints. +- Reusable domain knowledge approved for future use. + +Agents do not retain hidden cross-goal memory. Any memory reused across runs is versioned, attributable, policy-filtered, and visible to the user or administrator. + +## Delegation + +An agent may propose a child only when its run spec permits delegation. + +The proposal includes: + +- Child sub-goal and non-goals. +- Expected output and evaluator. +- Inputs and artifact references. +- Requested agent capability. +- Tool, authority, resource, and credential needs. +- Budget and deadline allocation. +- Parent join policy. +- Rationale for delegation. + +The coordinator checks depth, fan-out, concurrency, duplication, cycle risk, authority, policy, and remaining parent budget. Accepted children receive a fraction of the parent envelope; authority is never implicitly inherited in full. + +## OSMO execution placement + +An agent may run: + +- In the agent control plane for low-latency reasoning and lightweight tools. +- In an isolated service runtime. +- As an OSMO-hosted agent task when it needs accelerator inference, data locality, specialized dependencies, long runtime, or stronger workload isolation. + +An OSMO-hosted agent: + +- Runs a complete bounded harness. +- Receives scoped inputs and capability tokens. +- Emits typed results and proposals. +- Does not directly mutate the goal graph. +- Does not receive unrestricted user or OSMO credentials. +- Uses the workflow construction path in [06-workflow-construction.md](06-workflow-construction.md). + +## Result envelope + +Every worker terminates with exactly one typed outcome: + +- `Completed` +- `NeedsClarification` +- `NeedsApproval` +- `ProposeChildren` +- `RetryableFailure` +- `TerminalFailure` +- `Canceled` + +A successful result contains: + +- Output artifacts. +- Compact summary. +- Evidence references. +- Acceptance-criterion mapping. +- Assumptions and uncertainty. +- Unresolved questions. +- Suggested follow-up. +- Token, cost, time, and tool-use accounting. + +Free-form text may accompany the envelope but cannot replace required fields. + +## Evaluation + +Evaluators may be: + +- Deterministic tests. +- Schema and invariant checks. +- Artifact comparisons. +- OSMO workflow or benchmark runs. +- Model-based judges with calibrated criteria. +- Human review. +- Combinations of the above. + +Prefer deterministic evidence whenever available. Model-based evaluation must record its model, rubric, inputs, output, and uncertainty. + +An agent cannot be the sole evaluator of its own high-impact result. + +## Failure and retry + +- Harness or infrastructure failure may retry the same frozen run spec. +- Invalid model output may be repaired within the same step budget. +- Tool failure is classified before retry. +- Changed model, tool, strategy, context, or evaluator creates a new attempt. +- Unknown external side effect blocks automatic retry. +- Budget exhaustion returns a bounded failure or escalation request. +- Cancellation must interrupt model streaming and prevent new tool dispatch. + +## Security and supply chain + +- Pin images by digest and catalog entries by immutable version. +- Verify signatures where available. +- Issue short-lived, least-privilege capability tokens. +- Separate model-visible context from tool-held secrets. +- Sandbox filesystem and network access. +- Redact logs and artifacts before model ingestion. +- Treat tool output, retrieved documents, and child messages as untrusted data. +- Record all model, tool, skill, harness, image, policy, and evaluator versions. +- Prevent an agent from editing its own manifest, policy, evaluator, or budget. + +## Example manifest shape + +```yaml +name: workflow-investigator +version: 1 +purpose: Diagnose one failed OSMO workflow and return evidence-backed recovery options. +inputs: + - workflow-binding/v1 +outputs: + - diagnosis/v1 +modelPolicy: technical-reasoning +harness: bounded-tool-loop/v1 +tools: + - osmo-workflow-read/v1 + - osmo-logs-read/v1 +delegation: + allowed: false +limits: + steps: 20 + wallTime: 15m + spend: 2.00 +sideEffects: none +evaluator: diagnosis-evidence-check/v1 +``` + +## Acceptance criteria + +- Every run can be reconstructed from an immutable spec. +- Agents cannot exceed tool, authority, delegation, or budget limits through prompting. +- Inputs and outputs are typed and attributable. +- Context remains bounded as the goal grows. +- A replacement runtime can resume from durable artifacts without hidden memory. +- Agent-created children are admitted by the coordinator rather than executed implicitly. +- Model, tool, skill, harness, image, and evaluator versions are recorded. +- Completion requires evaluator evidence. diff --git a/projects/agents/08-agent-agent-communication.md b/projects/agents/08-agent-agent-communication.md new file mode 100644 index 000000000..e74a80c8c --- /dev/null +++ b/projects/agents/08-agent-agent-communication.md @@ -0,0 +1,404 @@ + + +# Agentic Goals: Agent-to-Agent Communication + +Status: Draft + +## Purpose + +Define durable, typed communication among the lead, workers, evaluators, deterministic processes, and OSMO-hosted agents. + +Agents do not communicate through invisible shared context or unrestricted peer-to-peer chat. They exchange messages and artifacts through the coordinator so communication remains attributable, bounded, policy-checked, and recoverable. + +## Principles + +- Artifacts carry substantive data; messages carry intent and references. +- Every message has a sender, recipient, purpose, contract, and correlation ID. +- Delivery is at-least-once, so consumers must be idempotent. +- Authority is never transferred by prose. +- Parent/child ownership and execution dependencies remain separate. +- Messages are untrusted input to the receiving agent. +- Communication volume is budgeted and backpressured. +- Human-visible summaries are derived from durable messages and artifacts. +- OSMO logs or task output are not an agent messaging protocol. + +## Communication topology + +The first validation supports: + +- Lead to worker. +- Worker to lead or parent. +- Parent to child. +- Evaluator to the node being evaluated through coordinator state. +- Coordinator broadcasts of cancellation, revision, or authority changes. +- Explicit sibling exchange only through an approved shared artifact or coordinator-routed request. + +```mermaid +flowchart TD + Lead["Lead agent"] <--> Bus["Coordinator message service"] + Parent["Parent worker"] <--> Bus + ChildA["Child worker A"] <--> Bus + ChildB["Child worker B"] <--> Bus + Evaluator[Evaluator] <--> Bus + Bus <--> Artifacts["Artifact store"] + Bus <--> Events["Event ledger"] +``` + +Unrestricted mesh communication is intentionally excluded. It complicates authority, creates hidden dependencies, and makes completion and cancellation difficult to reason about. + +## Message envelope + +Every message includes: + +- Message ID. +- Goal and plan revision IDs. +- Sender and recipient entity IDs. +- Sender attempt ID. +- Message type and schema version. +- Conversation or delegation ID. +- Correlation and causation IDs. +- Sequence number within the conversation. +- Creation time and optional expiry. +- Priority. +- Human-visibility classification. +- Authority requirement. +- Artifact references. +- Typed payload. +- Idempotency key. +- Integrity metadata. + +Messages are immutable after publication. Corrections reference and supersede prior messages. + +## Message types + +### DelegationRequest + +Parent proposes a child contract: + +- Sub-goal and non-goals. +- Input artifact references. +- Expected output and evaluator. +- Requested capabilities. +- Budget, deadline, and authority allocation. +- Join policy. +- Rationale. + +The coordinator admits or rejects the proposal before a child exists. + +### DelegationAccepted + +Confirms: + +- Child node and run IDs. +- Frozen contract. +- Allocated envelope. +- Expected delivery schema. +- Cancellation and deadline semantics. + +### WorkDirective + +Sends approved desired work to an existing agent. A directive cannot silently modify the frozen contract; material changes create a new attempt or plan revision. + +### Query + +Requests bounded information or analysis from another scope without changing desired state. + +### Response + +Answers a query with typed content and artifact references. + +### Progress + +Reports a material milestone, blocker, changed estimate, or heartbeat. Routine internal steps remain in attempt history rather than producing cross-agent messages. + +### Result + +Returns: + +- Outcome type. +- Output artifacts. +- Evidence. +- Acceptance mapping. +- Summary. +- Assumptions and uncertainty. +- Unresolved questions. +- Accounting. + +### ClarificationRequest + +Requests missing information needed to satisfy the contract. The parent may answer, route to the lead, or create a human request. + +### ApprovalRequest + +Proposes an action outside current authority. The coordinator turns it into the human approval protocol from [05-human-interfaces.md](05-human-interfaces.md). + +### ChildProposal + +Requests further delegation. It is not executable until coordinator admission. + +### Cancel + +Revokes desired execution for a scope. Recipients acknowledge and stop new work before reconciling active operations. + +### RevisionNotice + +Informs affected agents that a new plan revision changes, supersedes, or invalidates their work. + +### EvaluationResult + +Records criterion-level pass, fail, or unresolved outcomes with evidence and rubric version. + +## Artifacts + +Messages reference artifacts rather than embedding large payloads. + +Every artifact has: + +- Stable artifact ID and version. +- Type and schema version. +- Producing goal, node, and attempt. +- Content checksum. +- Storage location. +- Size and media type. +- Trust and sensitivity labels. +- Retention policy. +- Access policy. +- Supersession relationship. +- Human-readable summary when useful. + +Examples: + +- Goal contract. +- Plan revision. +- Dataset manifest. +- Source bundle. +- Model checkpoint. +- Analysis report. +- OSMO workflow binding. +- Tool result. +- Evidence bundle. +- Worker summary. +- Evaluation report. + +Consumers verify schema, integrity, access, and trust classification before use. + +## Handoffs + +A parent-to-child handoff contains only what the child needs: + +- Frozen child contract. +- Relevant goal context. +- Referenced inputs. +- Constraints and policy. +- Expected output. +- Parent join semantics. +- Escalation route. + +A child-to-parent handoff contains: + +- Typed result. +- Evidence and artifacts. +- Compact summary. +- Assumptions. +- Uncertainty. +- Unresolved questions. +- Recommended next action. + +Raw child transcript is retained for audit according to policy but is not automatically injected into the parent context. + +## Conversations + +A conversation is a durable ordered stream associated with one goal scope and purpose. + +Conversation classes: + +- Lead and user. +- Parent and child. +- Query and response. +- Human and nested worker. +- Evaluator clarification. + +Per-conversation sequence numbers preserve local order. Global ordering across independent conversations is not assumed; causation IDs establish meaningful relationships. + +## Delivery semantics + +- Persist message before delivery. +- Deliver at least once. +- Acknowledge processing with the consumer attempt and message ID. +- Deduplicate by message ID and idempotency key. +- Retry transient delivery failure with bounded backoff. +- Move permanently invalid messages to a visible rejected state with reason. +- Expired messages do not trigger new work. +- Cancellation and authority revocation have higher priority than normal work. + +Exactly-once model or tool execution is not assumed. External actions use their own idempotency and reconciliation records. + +## Ordering and stale messages + +Before acting, a consumer validates: + +- Plan revision is still applicable. +- Node and attempt are current. +- Sender was authorized to send the message. +- Recipient contract accepts the message type. +- Referenced artifacts are still valid. +- Deadline has not passed. + +A stale message is recorded and ignored or transformed into a clarification; it never silently mutates current state. + +## Joins and aggregation + +Parents do not wait on chat completion. They wait on durable child terminal states and declared artifacts. + +Supported join policies: + +- All required children. +- Any successful child. +- Quorum. +- Best effort until budget or deadline. +- Evaluator decides from available evidence. + +The aggregator: + +- Validates child result schemas. +- Detects conflicting claims. +- Deduplicates shared artifacts. +- Preserves provenance. +- Produces a bounded parent summary. +- Requests adjudication when conflicts affect acceptance criteria. + +## Conflict handling + +When agents disagree: + +1. Preserve both claims and evidence. +2. Determine whether the disagreement affects the parent contract. +3. Apply a deterministic rule or evaluator when defined. +4. Create a focused adjudication node when additional work is justified. +5. Route to a human when policy, values, or irreducible ambiguity requires it. + +The lead must not erase disagreement by selecting the most fluent response. + +## Authority and credentials + +- Messages carry references to authority, never bearer secrets. +- A parent can allocate only a subset of its delegable envelope. +- A child cannot expand scope by requesting it from a sibling. +- Tool capability tokens are issued directly by the coordinator for one run and scope. +- Recipients verify effective authority at action time, not only message creation time. +- Revocation prevents new actions even if old messages remain queued. + +## Security and trust + +- Treat every message and artifact body as untrusted data. +- Keep policy and system instructions outside user-controlled artifacts. +- Mark externally retrieved or model-generated content. +- Scan and redact secrets before model exposure or human notification. +- Enforce artifact access independently of message routing. +- Sign or integrity-check messages crossing execution boundaries. +- Limit links and network destinations to approved schemes and domains. +- Record provenance for summaries so users can inspect original evidence. + +## Communication budgets + +Each run has limits for: + +- Messages sent. +- Queries to siblings or parent. +- Child proposals. +- Total embedded payload bytes. +- Artifact reads. +- Progress update frequency. +- Model tokens consumed by communication. + +The coordinator applies backpressure: + +- Coalesce routine progress. +- Reject duplicate questions. +- Prefer artifact summaries over repeated raw reads. +- Rate-limit noncritical communication. +- Prioritize cancellation, approval, blocker, and terminal result messages. + +## Liveness + +Messages that require a response declare: + +- Response schema. +- Deadline. +- Retry policy. +- Escalation target. +- Safe default. + +The coordinator detects: + +- Unacknowledged directives. +- Children with no material progress. +- Parents waiting on messages instead of declared joins. +- Expired clarification or approval requests. +- Orphan conversations after plan revision or cancellation. + +Agents use bounded progress heartbeats only for liveness; heartbeats do not imply useful progress. + +## OSMO-hosted agents + +OSMO-hosted agents communicate with the coordinator through a scoped callback or message API: + +- The workflow task receives goal, node, attempt, and callback identifiers. +- Authentication is short-lived and limited to the current run. +- Messages are persisted by the external control plane. +- Large outputs are uploaded as artifacts and referenced in result messages. +- Loss of the callback path causes bounded retry and eventual attempt failure or reconciliation. +- The task does not use OSMO logs, Redis internals, or unrestricted user credentials as a substitute for the protocol. + +## Human visibility + +Messages declare one visibility level: + +- Hidden operational record. +- Available in history. +- Summarized by the lead. +- Requires immediate human attention. + +The lead may summarize but cannot change the underlying message or evidence. A human entering a nested worker conversation creates messages in that scoped conversation; any state-changing directive still goes through coordinator validation. + +## Example result message + +```json +{ + "type": "Result", + "schemaVersion": 1, + "messageId": "msg-123", + "goalId": "goal-abc", + "planRevisionId": "plan-3", + "senderAttemptId": "attempt-7", + "recipientNodeId": "node-parent", + "conversationId": "delegation-42", + "outcome": "Completed", + "artifacts": ["artifact-report-9"], + "evidence": ["evidence-test-4"], + "summary": "The candidate passed the required benchmark.", + "unresolvedQuestions": [], + "idempotencyKey": "result-attempt-7" +} +``` + +## Acceptance criteria + +- Agent communication survives runtime and coordinator restart. +- Every message is attributable and schema-valid. +- Duplicate delivery does not duplicate work or side effects. +- Authority cannot be delegated through prose. +- Parent context remains bounded as child count grows. +- Required joins depend on durable states and artifacts, not transcript inspection. +- Cancellation and revision notices supersede stale work. +- Sibling communication cannot create hidden dependency or authority paths. +- OSMO-hosted agents use the same durable protocol as control-plane agents. +- Human summaries preserve links to original messages and evidence. diff --git a/projects/agents/poc/00-architecture-and-contracts.md b/projects/agents/poc/00-architecture-and-contracts.md new file mode 100644 index 000000000..90f52be77 --- /dev/null +++ b/projects/agents/poc/00-architecture-and-contracts.md @@ -0,0 +1,199 @@ + + +# POC Plan: Architecture and Contracts + +Status: Draft + +## Outcome + +Define the minimum contracts that let an OSMO-hosted lead coordinate one +video-pipeline agent per video and its deterministic VDA `e2e` stages without +hiding plan changes, authority, or evidence in chat history. + +## POC architecture + +```mermaid +flowchart LR + User[User] --> Lead[Lead agent workflow] + Lead --> Environment[Environment-pipeline agent] + Lead --> VideoA[Video-pipeline agent A] + Lead --> VideoB[Video-pipeline agent B] + Environment --> Materializer[Model-artifact materializer] + VideoA --> OriginalA[Label original A] + VideoA --> AugmentA[Augment A] + AugmentA --> AugmentedA[Label augmented A] + VideoB --> OriginalB[Label original B] + VideoB --> AugmentB[Augment B] + AugmentB --> AugmentedB[Label augmented B] + Lead --> OSMO[Existing OSMO CLI/API] + Environment --> OSMO + VideoA --> OSMO + VideoB --> OSMO + OriginalA --> Artifacts[Declared OSMO output paths] + AugmentA --> Artifacts + AugmentedA --> Artifacts + OriginalB --> Artifacts + AugmentB --> Artifacts + AugmentedB --> Artifacts + VideoA --> Artifacts + VideoB --> Artifacts + Environment --> Artifacts + Lead --> Artifacts +``` + +The long-lived lead task is the temporary coordinator for one goal. It first +submits one bounded environment-pipeline agent, then—only after its typed +`environment-ready` result—one video-pipeline agent per approved video. The +environment agent owns conditional model-artifact materialization; each video +agent executes the real `e2e` branch-and-join sequence for its fixed video. +OSMO workflow IDs and declared output artifacts are the observable coordination +surface. The POC does not introduce a new database or control-plane service. + +## Minimal durable records + +Write these as versioned artifacts beneath declared OSMO output paths. Their +exact serialization format is an implementation choice; their ownership and +content are the POC contract. + +| Record | Owner | Required contents | +| --- | --- | --- | +| Goal plan | lead | objective, non-goals, VDA batch scope, acceptance criteria, budget, deadline, selected pool/resource profile/concurrency limit, and environment-pipeline binding | +| Environment request | lead | run workspace, model-artifact root, permitted materializer contract, and cache acceptance criteria | +| Environment-ready binding | environment pipeline | cache lock, immutable model-artifact workspace URL, manifest digest, and materializer workflow binding when one was required | +| Cache manifest and result | model-artifact materializer | source revisions, file checksums, cache URLs, entrypoint version, and typed terminal outcome | +| Video-agent contract | lead | one fixed video, lead-selected pool and stage image/profile/cache references, join policy, and concurrency limits | +| Video-stage bundle | video agent | non-secret configuration, helper scripts, entrypoint version, endpoint references, and checksums | +| Environment lock | constructor | custom agent-image digest and the pinned upstream PAIDF stage-image digests | +| Workflow binding | submitting agent | canonical capsule hash, submission key, returned workflow ID, and output URL | +| Video manifest | video agent | stage bindings, terminal status, and one video result | +| Stage result | deterministic worker | VDA stage result path, schema/contract check, evidence, and terminal outcome | + +Records are immutable after publication. A correction writes a new version or +an event that supersedes the prior record. + +## Minimal contracts + +### Goal plan + +The lead owns the one VDA batch goal. Each video agent owns exactly one video +and cannot change the batch. Its original-label and augmentation stages may run +in parallel; its augmented-label stage is admitted only after valid augmentation +evidence. The topology, depth, and fan-out rules are fixed in +[Locked topology](locked-topology.md). + +For this POC, the lead dynamically discovers every pool and resource profile +visible to its task context, selects viable capacity, and freezes the chosen +pool, profile, and batch concurrency limit into the goal plan before fan-out. +It must validate representative capsules before delegation. This authorizes +selection and submission only; it does not authorize the lead to change cluster +resources, quotas, or credentials. + +Before video fan-out, the lead creates the run workspace and submits one frozen +environment request. The environment-pipeline agent computes the cache lock and +verifies the corresponding immutable cache manifest. A valid cache becomes an +`environment-ready` binding. On a miss, that agent submits exactly one +deterministic model-artifact-materializer capsule, verifies its cache result and +manifest, then returns the binding to the lead. The lead does not submit the +materializer directly. + +### Environment request + +The lead, environment-pipeline, and video agents use the pinned custom +agent-runtime image. A deterministic VDA stage uses the selected upstream PAIDF +image by digest and never requests an environment or installs packages at +runtime. The conditional model-artifact-materializer capsule is a distinct +deterministic task whose image and downloader entrypoint are pinned in the +cache lock before the environment pipeline submits it. For example, a declared +agent capability is: + +```yaml +capabilities: + - osmo-workflow-read/v1 + - git-read/v1 +runtime: agent-runtime/python +``` + +The constructor pins the custom agent image and the upstream PAIDF stage images +and emits an environment lock. A missing requirement blocks the run rather than +triggering an implicit runtime install. + +### Video-stage bundle + +Before child admission, the video agent writes one immutable bundle for its +fixed video. Every stage receives that bundle and a small stage-specific +contract plus the lead's immutable cache binding as declared task inputs. The +standard stage entrypoint verifies and materializes the bundle, prepares +writable cache paths, runs the installed PAIDF application, validates outputs, +and writes the terminal stage result. It performs no dependency installation. + +### Stage result + +Every deterministic stage writes exactly one terminal envelope: + +```yaml +outcome: Completed # or NeedsClarification, NeedsEnvironment, RetryableFailure, TerminalFailure +summary: concise result for the lead +evidence: [] +artifacts: [] +acceptanceCriteria: [] +openQuestions: [] +accounting: + wallTime: 0s + toolCalls: 0 +``` + +Each video agent joins its original and augmented pseudo-label results into one +video result. The lead joins the video results and evaluates the batch against +the VDA `e2e` output contract before it reports completion. + +Every listed outcome is terminal only for its current immutable capsule. Its +owning agent may reconcile the result, repair the next safe contract, and retry +with a new capsule without a numeric retry ceiling. Only an ambiguity that it +cannot resolve safely becomes `NeedsClarification` for the human. + +## Build steps + +1. Define one declared output-root convention for the goal, per-video agents, + video-stage bundles, and the original-label, augmentation, and + augmented-label outputs. +2. Define contracts for the environment request/ready result, cache + result/manifest, lead-to-video-agent handoff, video-agent-to-stage request, + video manifest, and stage result. +3. Implement environment-pipeline admission: reuse a verified cache or submit + one deterministic model-artifact-materializer capsule, then return its + verified binding. +4. Record correlation IDs and OSMO workflow bindings at each dynamic submit. +5. Make the lead, environment, and video agents read OSMO status/logs/events by + workflow ID and consume only typed result artifacts; raw logs stay by + reference. + +## Validation gates + +- Each dynamically submitted workflow has one recorded submit key, workflow ID, + input video, image digest, and output path. +- The lead admits video work only with a verified immutable `environment-ready` + binding; a cold cache makes the environment pipeline create at most one + model-artifact-materializer workflow for its cache lock. +- A video agent cannot submit an augmented-label worker before valid + augmentation evidence or submit the same stage twice. +- Every stage verifies the video-stage bundle checksum before execution and + performs no package installation at boot. +- A malformed video-agent request, stage result, or output contract is rejected + before it controls the next lifecycle action. +- An agent's prose alone cannot mark the goal complete. + +## Exit criterion + +The POC has enough contract structure to run the accepted two-video topology: +one lead, one environment-pipeline agent, two video agents, six deterministic +VDA stages, and—only on a cache miss—one model-artifact-materializer stage. Do +not add a database service, UI, static preflight workflow, or platform-managed +setup workflow before this path is proven. diff --git a/projects/agents/poc/01-local-lead-and-pipeline.md b/projects/agents/poc/01-local-lead-and-pipeline.md new file mode 100644 index 000000000..1b364201e --- /dev/null +++ b/projects/agents/poc/01-local-lead-and-pipeline.md @@ -0,0 +1,120 @@ + + +# POC Plan: Lead Agent and Pipeline Compiler + +Status: Draft + +## Outcome + +Use a long-lived OSMO-hosted lead agent to turn one VDA `e2e` goal into a +small, reviewable pipeline; submit one bounded environment-pipeline agent; and, +after it returns `environment-ready`, submit one bounded video-pipeline agent +per approved video. + +## Deliberate POC shape + +The lead is the dynamic decision-maker and runs as an OSMO task for the whole +goal duration. It remains alive while it observes and supervises the separate +environment-pipeline and video-pipeline workflows that it submits. This does +not make an OSMO workflow recursively mutable: each dynamically created +descendant remains a new static workflow capsule. + +The initial plan supports seven fixed role forms: + +1. One long-lived lead-agent workflow. +2. One bounded environment-pipeline agent workflow per goal run. +3. One conditional deterministic model-artifact-materializer workflow per + cache lock, submitted only by the environment pipeline. +4. One bounded video-pipeline agent workflow per approved video. +5. One deterministic original-label workflow per approved video. +6. One deterministic augmentation workflow per approved video. +7. One deterministic augmented-label workflow per approved video, admitted only + after valid augmentation evidence. + +The OSMO capsule compiler uses frozen records from the architecture plan: + +```text +GoalContract + PlanRevision + EnvironmentLock + InputArtifacts + -> capsule template -> OSMO dry-run -> policy check -> idempotent submission +``` + +## Lead responsibilities + +For the POC, the lead must: + +- Restate the objective, non-goals, acceptance check, and risk-bearing + assumptions. +- Produce the initial plan for the accepted VDA `e2e` batch. +- Submit exactly one environment-pipeline workflow through the existing OSMO + CLI/API path and record its workflow ID. +- Validate its typed `environment-ready` result before submitting exactly one + video-pipeline workflow per approved video. +- Stay alive while it observes the environment and video agents and their + reported bindings. +- Report material result, failure, or needed decision from declared output + artifacts and OSMO status. +- Create a new plan revision rather than silently changing dispatched scope. + +The lead has no approval checkpoint or numeric retry ceiling. It evaluates a +failed environment or video workflow, reconciles the prior binding, and creates +the next safe immutable attempt. It asks for human intervention only when it +cannot safely resolve an ambiguity from the available contracts, evidence, and +OSMO state. + +The lead must not declare a completed goal solely from a worker's natural +language response or bypass a failed dry-run. + +## Build steps + +1. Define a compact `GoalPlan` schema and a matching human-readable template. + Require explicit acceptance criteria, non-goals, workstream outputs, and + evaluator for every node. +2. Build a deterministic plan validator that checks schema, bounded fan-out, + dependency acyclicity, budgets, and named environment capabilities. +3. Build the capsule compiler from the validated plan and environment lock. + It accepts only cataloged image digests, commands, and credentials by + reference. +4. Run OSMO dry-run and validation-only checks before saving the frozen capsule + and recording its submission key. +5. Provide local commands for `plan`, `preview`, `submit`, `status`, and + `reconcile`. They may be simple scripts in the first implementation. +6. Teach the OSMO-hosted lead to use those commands and to read only typed status + summaries and evidence references. + +## First demonstration + +Use a fixed two-video VDA `e2e` batch. The lead first submits one environment +pipeline, which reuses a verified model-artifact workspace or dynamically +submits one materializer task on a cold cache. After `environment-ready`, the +lead submits one video agent per video. Each video agent dynamically submits +original-label and augmentation workers in parallel, then an augmented-label +worker after valid augmentation. The lead evaluates both video results against +the VDA `e2e` output contract. + +## Validation gates + +- The same plan and environment lock produce byte-equivalent canonical capsule + input before OSMO assigns server fields. +- No capsule is submitted when dry-run, policy, image resolution, or artifact + validation fails. +- Changing a tool, image, evaluator, acceptance criterion, or side-effect + scope requires a new plan revision. +- The lead remains alive while the environment pipeline, video agents, and + deterministic stages run. +- The user can inspect the generated plan, image digest, capsule, OSMO binding, + and evidence without a UI. + +## Exit criterion + +One OSMO-hosted lead safely completes the two-video demonstration through one +environment-pipeline agent, two meaningful video agents, and six dynamically +submitted deterministic GPU stages, plus one cold-cache materializer when +needed. diff --git a/projects/agents/poc/02-runtime-environments.md b/projects/agents/poc/02-runtime-environments.md new file mode 100644 index 000000000..753e3c586 --- /dev/null +++ b/projects/agents/poc/02-runtime-environments.md @@ -0,0 +1,318 @@ + + +# POC Plan: Runtime Environment Construction + +Status: Draft + +## Outcome + +Build one strong custom agent-runtime image, then run deterministic VDA stages +directly from pinned upstream PAIDF images with custom task bundles. The lead +delegates Swift-backed model-artifact admission to an environment-pipeline +agent. The first POC must not build a new VDA image, start a package manager in +a VDA stage task, or require +Docker-in-Docker or privileged Kubernetes execution. + +## Accepted packaging model + +```text +custom agent-runtime image + + frozen lead, environment-pipeline, or video-agent role contract + -> Codex agent task + +pinned upstream PAIDF image + + frozen stage contract + video-stage bundle + + verified cache binding + -> deterministic VDA stage task + +pinned model-artifact-materializer base image and downloader entrypoint + + frozen cache lock + -> deterministic model-artifact workspace task, on cache miss only +``` + +The role contract and video-stage bundle are ordinary task files or immutable +input artifacts. They are not OCI layers. A changed video, endpoint reference, +output URL, or non-secret configuration creates a new task contract, not a new +image. + +## 1. Agent runtime + +The lead, environment-pipeline, and every video-pipeline agent initially share +one custom pinned image. +The image contains only what their actual contracts require: + +- `ubuntu:22.04`, resolved to an immutable digest, and system Python `3.10`; +- Node `24.15.0` from the official + `node-v24.15.0-linux-x64.tar.xz` archive, verified during the image build + against its published SHA-256 recorded in the runtime lock; +- Codex CLI from npm `@openai/codex` and its container-user-level provider + configuration; +- the selected NVIDIA model catalog, pinned by content digest; +- existing OSMO CLI/API access; and +- the POC's deterministic plan, capsule, child-admission, result, and + reconciliation scripts, plus normal shell/core utilities. + +Both roles use the `nvidia_inference` custom provider with +`openai/openai/gpt-5.6-terra` at `xhigh` reasoning. The provider uses +`https://inference-api.nvidia.com/v1/`, the Responses wire API, and the +runtime-only `INFERENCE_API_KEY` variable. The image never contains that key. +The lead, environment-pipeline, and video-pipeline capsules receive it through +the OSMO `GENERIC` credential named `nvidia_inference`. The original and +augmented auto-labeling capsules receive that same credential and map +`INFERENCE_API_KEY` to `NVIDIA_API_KEY` in process before calling the upstream +worker; augmentation capsules do not receive it. + +The auto-labeling stage's frozen external inference pair is: + +| Function | Base URL | Model | +| --- | --- | --- | +| VLM | `https://inference-api.nvidia.com/v1` | `nvidia/meta/llama-3.2-11b-vision-instruct` | +| LLM | `https://inference-api.nvidia.com/v1` | `nvidia/qwen/qwen3-32b` | + +The agent-runtime image is published as +`nvcr.io/nvstaging/osmo/agent-runtime:`. The tag is a build/promotion reference; +workflow capsules use only the resulting +`nvcr.io/nvstaging/osmo/agent-runtime@sha256:` reference. + +The image is built locally with a `linux/amd64` builder and pushed to NVCR before +any workflow submission. OSMO only pulls the resolved digest; it does not build +images, run Docker-in-Docker, or require a privileged task. + +All NVCR image pulls use the OSMO `REGISTRY` credential named `ngc_cred`. +It is created with `registry=nvcr.io`, username `$oauthtoken`, and an NGC +API-key `auth` value supplied outside the repository. Workflow capsules record only +the credential name. + +The lead, environment pipeline, and video agent differ by frozen role contract, +goal/environment/video inputs, authority, and result destination. They do not +initially require separate image layers or separate images. + +## 2. Deterministic VDA stages + +The POC uses the following upstream images directly, each pinned by digest +before workflow rendering: + +| Stage | Upstream image | Custom packaging | +| --- | --- | --- | +| original label and augmented label | `nvcr.io/nvidia/paidf-auto-labeling@sha256:502c38b95c25d885b4ae56d3ed4b911218d97df4c7d4b640af05587407ec11f7` | stage entrypoint, stage contract, video-stage bundle, result writer | +| augmentation | `nvcr.io/nvidia/paidf-augmentation@sha256:59a14d6f4814245735b8e974be6f5a4dbf1ea8ab4564f7adf36b633ff822e1d1` | stage entrypoint, stage contract, video-stage bundle, result writer | + +These digests were resolved from the accepted `:1.0.0` tags. A future image +refresh must resolve and review new digests before changing this lock. +Both resolved images are single `linux/amd64` manifests, matching the POC +platform. + +The PAIDF images already provide their VDA application, GPU/CUDA dependencies, +Python environment, `uv`, `python3`, `bash`, `curl`, and necessary shell tools. +The POC supplies its custom integration as files or input artifacts rather than +a derived image layer. + +### Dynamic model-artifact workspace + +The lead creates the run workspace, then delegates cache admission to the +environment-pipeline agent. That agent computes a cache lock and verifies the +matching `cache-manifest.json` beneath the Swift artifact root. A valid cache is +reused. On a miss or invalid manifest, the environment pipeline submits one +deterministic `model-artifact-materializer-` capsule, waits for +`cache-result.json`, verifies the resulting manifest, and returns +`environment-ready.json`. Only then does the lead freeze its URLs and digest +into video and VDA-stage contracts. + +The materializer task is not an agent and cannot submit work. It can write only +its own content-addressed cache prefix. It uses the pinned Linux amd64 Ubuntu base +`nvcr.io/nvidia/base/ubuntu@sha256:2a9f71d82aa4daac444c1b4b74d5d7b01f93eb23662c1236f89d817f083abecd` +(resolved from `22.04_20240212`) and frozen downloader entrypoint `v1`. +This task alone may install `python3`, `python3-pip`, and `huggingface_hub` at +boot, then download the lock's sources, resolve symlinks, checksum every output, +and publish `cache-manifest.json` and `cache-result.json`. The PAIDF stage images +remain direct, unmodified upstream images and never install packages at boot. + +The artifact prefix is durable Swift storage, never a shared node, pod, +container, or filesystem cache. Each PAIDF stage receives the binding as a +declared input and materializes it on its own local disk at the cache paths +required by the upstream application. + +The model-artifact materializer alone receives the OSMO `GENERIC` credential `hf_token` as +`HF_TOKEN` to download the accepted Hugging Face model sources. The initial +`e2e` run fixes `superResolution=false` and `seedvrVariant=none`, so it does not +download or require SeedVR. + +### Video-stage bundle + +Before it submits the first stages, a video agent writes one immutable bundle +for its fixed video. It contains non-secret configuration, helper scripts, +entrypoint version, endpoint references, and checksums. Every stage for that +video receives the same bundle plus its small stage-specific contract and the +lead's verified cache binding. + +The custom stage entrypoint runs inside the PAIDF task: + +```text +verify contract, bundle, and cache manifest -> materialize scripts/configuration +-> prepare environment and writable cache paths from the binding +-> map `INFERENCE_API_KEY` to `NVIDIA_API_KEY` for auto-labeling only +-> verify declared endpoints and models -> execute PAIDF +-> validate outputs -> write stage-result.json +``` + +This is initialization of a known application, not environment construction. +It must not install dependencies or mutate the container image. + +## Locks + +The environment lock records the one custom agent image, both upstream PAIDF +image digests, the Codex provider/model catalog information, model-artifact-materializer +identity, and entrypoint versions. A video-stage bundle manifest records its +own content digest. + +```yaml +agentRuntime: + parent: ubuntu:22.04@sha256:... + publishReference: nvcr.io/nvstaging/osmo/agent-runtime: + image: nvcr.io/nvstaging/osmo/agent-runtime@sha256:... + registryCredential: ngc_cred + inferenceCredential: nvidia_inference + platform: linux/amd64 + node: + version: 24.15.0 + archive: node-v24.15.0-linux-x64.tar.xz + archiveSha256: 472655581fb851559730c48763e0c9d3bc25975c59d518003fc0849d3e4ba0f6 + pythonVersion: 3.10 +codex: + package: '@openai/codex' + version: 0.144.6 + npmIntegrity: sha512-wk+2CWiBNXiJLBoN2D08N9RceWkSBnlgk5g2K1a4CXrP/C0gdlHyRUG7RFzm9y41DCK/7tvCct233JVxyFmznw== + provider: nvidia_inference + model: openai/openai/gpt-5.6-terra + modelCatalogDigest: sha256:... +credentials: + nvidiaInference: + name: nvidia_inference + type: GENERIC + runtimeKey: INFERENCE_API_KEY + consumers: [lead-agent, environment-pipeline, video-pipeline, auto-label-original, auto-label-augmented] + swiftOsmo: + name: swift_osmo_cred + type: DATA + storageRoot: https://pdx.s8k.io/v1/AUTH_team-osmo/dev/fernandol/agents_poc/ + huggingFace: + name: hf_token + type: GENERIC + runtimeKey: HF_TOKEN + consumers: [model-artifact-materializer] +vdaEndpoints: + credential: nvidia_inference + stageKeyMapping: "INFERENCE_API_KEY -> NVIDIA_API_KEY" + vlm: + baseUrl: https://inference-api.nvidia.com/v1 + model: nvidia/meta/llama-3.2-11b-vision-instruct + llm: + baseUrl: https://inference-api.nvidia.com/v1 + model: nvidia/qwen/qwen3-32b +stages: + autoLabelImage: nvcr.io/nvidia/paidf-auto-labeling@sha256:502c38b95c25d885b4ae56d3ed4b911218d97df4c7d4b640af05587407ec11f7 + augmentationImage: nvcr.io/nvidia/paidf-augmentation@sha256:59a14d6f4814245735b8e974be6f5a4dbf1ea8ab4564f7adf36b633ff822e1d1 + entrypointVersion: v1 +videoStageBundle: + manifestDigest: sha256:... +modelArtifactWorkspace: + runRoot: https://pdx.s8k.io/v1/AUTH_team-osmo/dev/fernandol/agents_poc/workspaces/vda + root: https://pdx.s8k.io/v1/AUTH_team-osmo/dev/fernandol/agents_poc/model-artifacts/vda + cacheLock: sha256:... + manifestDigest: sha256:... + materializer: + condition: cache-miss-or-invalid-manifest + image: nvcr.io/nvidia/base/ubuntu@sha256:2a9f71d82aa4daac444c1b4b74d5d7b01f93eb23662c1236f89d817f083abecd + sourceTag: 22.04_20240212 + entrypointVersion: v1 + bootPackages: [python3, python3-pip, huggingface_hub] + minimumResources: + cpu: 4 + memory: 16Gi + storage: 200Gi + superResolution: false + seedvrVariant: none + sources: + nvidiaSkillsCommit: 6379b9ce5498d56626caa9e93a8c8a599f90046d + manifest: model-artifact-materializer/model-artifact-sources-v1.json + manifestSha256: a588491f689c869f304ecca6f0f536d9636e9fb8d94f27a75606d3e510c41db8 + script: model-artifact-materializer/materialize-model-artifacts.sh + scriptSha256: cbaa7031eaa58a17c4a1a5216de05f8c84e7c94da5bf24e5e5d7d49cdc54f9d0 +``` + +## Build steps + +1. Resolve and record the immutable digest for the accepted `ubuntu:22.04` + agent-runtime parent. +2. Download the official Node `24.15.0` Linux x64 archive and verify it with + `sha256sum -c` against the locked published SHA-256 before installing it. +3. Resolve the current latest npm `@openai/codex` release, then record its exact + version and integrity value in the runtime lock. +4. Build the `linux/amd64` custom agent runtime locally, push its first tag to + NVCR, resolve its digest, then smoke-test that digest: Node, Codex, OSMO CLI, + provider configuration, and the POC scripts must be present. +5. Use the locked upstream PAIDF image digests; do not derive them. An + intentional upstream revision change resolves and reviews replacement + digests before updating the lock. +6. Define the environment request/ready contract, cache lock, manifest/result + schema, workspace paths, and one-task model-artifact-materializer capsule. + Use the pinned Ubuntu base and frozen `v1` downloader script; permit its + narrowly scoped Python/Hugging Face install only in this cold-cache task. +7. Define the video-stage bundle manifest, stage contracts, and standard + `init -> execute -> validate -> result` entrypoint. +8. Smoke-test the model-artifact materializer against a fresh cache prefix, then verify its + manifest and warm-cache reuse path. +9. Smoke-test each VDA stage entrypoint against its selected PAIDF image with a small + frozen contract. +10. Make the capsule compiler accept only the pinned images, entrypoint version, + bundle checksum, and non-secret stage contract. + +## Validation gates + +- The lead, environment pipeline, and video agents run from the same + agent-runtime digest with distinct frozen role contracts. +- The Node archive hash in the runtime lock matches Node's published SHA-256, + and the image build fails before installation if the downloaded archive does + not match it. +- Each capsule that pulls from NVCR references `ngc_cred`; no registry `auth` + value is present in its YAML, image, lock, or output. +- Lead, environment-pipeline, video-pipeline, and auto-labeling capsules + reference `nvidia_inference`, which supplies `INFERENCE_API_KEY` at runtime. + The auto-label entrypoint maps it only in process to `NVIDIA_API_KEY`; the + key is never in an image, bundle, contract, lock, prompt, log, or output. + Augmentation capsules do not receive this credential. +- Every submitted VDA stage references the pinned upstream PAIDF image digest, + not a tag alone or a derived image. +- The lead admits video fan-out only after a verified `environment-ready` + result. On a cold cache, the environment pipeline submits exactly one pinned + deterministic materializer capsule, which publishes only a new + content-addressed prefix and a typed cache result. +- The model-artifact materializer is the sole task allowed to install download tooling at + boot. Its manifest checksums every published cache file before the lead may + bind the cache; no VDA task may install packages. +- Only the model-artifact-materializer capsule references `hf_token` as `HF_TOKEN`; the + token is never present in a VDA stage, image, bundle, contract, lock, prompt, + log, or output. +- Every stage verifies its video-stage bundle before executing and performs no + package installation at boot, and consumes only a verified cache binding. +- Changing a video/configuration/result destination changes the task contract, + not an image digest. +- Credentials appear only at runtime and never in an image, bundle, contract, + lock, prompt, or artifact. + +## Exit criterion + +The POC can run the lead, one environment-pipeline agent, two video agents, and +all six VDA stages with one custom agent image, two pinned upstream PAIDF +images, auditable per-video stage bundles, and a Swift-backed artifact +workspace. On a cold cache the environment pipeline additionally runs one +pinned deterministic model-artifact-materializer capsule; a warm cache reuses +its immutable manifest. diff --git a/projects/agents/poc/03-worker-execution-and-fanout.md b/projects/agents/poc/03-worker-execution-and-fanout.md new file mode 100644 index 000000000..89b3855a8 --- /dev/null +++ b/projects/agents/poc/03-worker-execution-and-fanout.md @@ -0,0 +1,127 @@ + + +# POC Plan: Video Pipeline Execution and Fan-Out + +Status: Draft + +## Outcome + +Run one bounded video-pipeline agent per input video in an OSMO task and +demonstrate its controlled dynamic execution of the VDA `e2e` branch and join. + +Task containers already have OSMO and API access, so the POC does not need to +invent another workflow transport. It must still establish attribution, +idempotency, and fan-out bounds before treating worker-driven submission as a +safe coordination mechanism. + +## Video-pipeline-agent contract + +Each video-pipeline agent receives only: + +- frozen goal, plan, video sub-goal, and attempt IDs; +- the node contract and permitted environment lock; +- the lead-selected pool and stage resource profile; +- one verified immutable cache binding returned by the environment pipeline and + frozen by the lead; +- one explicit input-video artifact reference and common configuration; +- a deadline, step/tool limit, and fixed stage-concurrency limit; +- a result destination and correlation ID; +- the OSMO/API access already supplied to the task. + +It returns one `video-result.json` envelope. It does not edit the parent plan, +select a different image, create a child for another video, or mark the batch +goal complete. + +Before it submits a child, the video agent writes one immutable video-stage +bundle for its video. The bundle supplies the common non-secret configuration, +helper scripts, entrypoint version, endpoint references, and checksums to every +stage. It is an output artifact of the video-agent task and a declared input to +each child capsule alongside the environment-derived cache binding; it is not a +setup workflow. + +## Deterministic stage submission contract + +Each permitted stage is expressed as a typed request, persisted before +submission. The original-label and augmentation requests are independent; +the augmented-label request is admitted only after the augmentation result +satisfies the frozen gate. + +```yaml +parentAttempt: video-agent-a-attempt-01 +childKey: augment-video-a-v1 +capability: vda-augment/v1 +inputs: + - video-a.mp4 +expectedResult: vda-augmentation-result/v1 +budget: + wallTime: 15m + maxChildren: 0 +``` + +The POC admission check verifies: + +- the parent attempt is active and has unused child credit; +- the request matches the fixed stage image and capability; +- the child key has not already been submitted; +- the deadline and resource allocation fit the parent envelope; +- an `auto-label-augmented` request has valid augmentation evidence for this + same video; +- the requested stage includes the matching video-stage bundle and its checksum; +- the requested stage includes the matching verified cache binding and manifest + digest; +- the resulting OSMO capsule passes the same construction validation as every + other dynamically submitted capsule. + +Direct OSMO access is the task's execution path, but a child request must be +recorded and admitted before the video agent invokes that path. + +## Build steps + +1. Package the bounded harness, result writer, and child-request client into + the agent runtime image. +2. Define input, stage-result, child-request, and video-result schemas. +3. Define the video-stage bundle and standard stage-entrypoint contract. +4. Compile one-task capsules for original labeling, augmentation, and augmented + labeling from the pinned upstream PAIDF stage images. +5. Submit two video-pipeline agents and reconcile their video results into the + lead's declared outputs. +6. Implement a child-admission command that atomically records each stage + request and reserves its idempotency key. +7. Have each video agent publish its bundle, submit original labeling and + augmentation in parallel, then submit augmented labeling only after + augmentation evidence is valid. +8. Make cancellation stop new stage admission before attempting best-effort + cancellation of active OSMO workflows. + +## Validation gates + +- Every deterministic stage runs from frozen video input, pinned upstream PAIDF + image digest, video-stage bundle, verified cache binding, and output URL; it + requires no agent context. +- Every stage runs the standard `init -> execute -> validate -> result` + entrypoint sequence without installing packages. +- Re-delivering the same stage request produces one workflow binding. +- A video agent cannot exceed its fixed video, stage image, deadline, or + resource envelope, and cannot submit augmented labeling before augmentation + passes its output contract. +- Each stage workflow is traceable to video-agent attempt, goal plan revision, + image digest, video input, and child-request key. +- A failed stage becomes a typed video-agent-visible result; it cannot strand + the lead in an ambiguous waiting state. +- Stopping the goal prevents a video agent from submitting a new stage after the + stop record is visible. + +## Exit criterion + +Two video-pipeline agents run in OSMO. Each creates two parallel first-stage +workflows and one conditional augmented-label workflow, writes a typed video +result, and lets the lead reconstruct the complete batch outcome from declared +results. diff --git a/projects/agents/poc/04-validation-and-demo.md b/projects/agents/poc/04-validation-and-demo.md new file mode 100644 index 000000000..cf10f295f --- /dev/null +++ b/projects/agents/poc/04-validation-and-demo.md @@ -0,0 +1,121 @@ + + +# POC Plan: Validation and Demonstration + +Status: Draft + +## Outcome + +Validate the locked VDA `e2e` dynamic-DAG topology through a narrow, repeatable +two-video demonstration before adding UI, service changes, arbitrary tools, or +broader delegation. + +## Demonstration scenario + +Use one fixed, approved two-video VDA `e2e` batch: + +> The lead creates the run workspace and delegates one environment-pipeline +> agent. That agent verifies the model-artifact workspace and, on a cold cache, +> submits one deterministic model-artifact materializer. After its verified +> `environment-ready` result, the lead submits one video-pipeline agent per +> video. Each video agent +> submits original labeling and augmentation in parallel, submits augmented +> labeling after valid augmentation evidence, and writes one typed video result. + +This exercises the long-lived lead, meaningful bounded agent loops, dynamic +branching and joins, deterministic GPU work, OSMO submission, result contracts, +and output-contract evaluation without changing OSMO services. + +## Test ladder + +| Level | What runs | Required proof | +| --- | --- | --- | +| Input-manifest test | first POC demo batch | both mirrored files match the pinned source revision, byte size, and SHA-256 before lead submission | +| Cold-cache admission | lead, environment pipeline, and one deterministic materializer | environment pipeline submits exactly one materializer on a cache miss; cache result and content-addressed manifest validate before `environment-ready` | +| Warm-cache admission | lead and environment pipeline | environment pipeline verifies and reuses the same cache manifest without a materializer workflow | +| Contract tests | lead, environment-pipeline, video-agent, and stage schemas | invalid environment-ready result, duplicate stage request, premature augmented-label request, and malformed stage result are rejected | +| Image tests | custom agent image, model-artifact-materializer base, and pinned upstream PAIDF images | exact digests and required entrypoints recorded | +| Capsule preview | OSMO dry-run and validation-only | rendered workflow is valid before submission | +| Lead/environment/video-agent run | four OSMO agent workflows | lead waits for environment-ready, then submits exactly two video agents and stays alive while they run | +| Dynamic-DAG run | two video agents plus six deterministic stages | parallel first-stage bindings, gated augmented-label bindings, and video results are recoverable | +| Stage-init run | each deterministic stage | verified video-stage bundle, no dependency installation, and typed terminal result | +| Output check | two VDA `e2e` result sets | original labels, augmented videos, and augmented labels meet the output contract | +| Stop run | stop before an augmented-label admission | no new stage is submitted after stop is observed | + +## Evidence bundle + +Each demonstration run retains: + +- approved goal contract, video batch, and one video-agent contract per video; +- custom agent-image digest, upstream PAIDF image digests, and entrypoint smoke + output; +- environment request/ready binding, model-artifact-materializer capsule and + terminal result when used, plus the verified cache manifest; +- video-stage bundles and their checksums; +- canonical and rendered OSMO capsule specs plus validation output; +- lead, environment-pipeline, video-agent, and stage submission keys with OSMO + workflow bindings; +- video results and deterministic stage terminal result envelopes; +- evaluator output mapping evidence to acceptance criteria; +- a compact run summary with deviations, retries, and unresolved risks. + +The bundle is a run artifact, not a chat summary. It must explain the lead, +environment pipeline, video agents, and per-stage lineage without relying on a +changed OSMO service or hidden agent context. + +## Pass criteria + +The POC passes only if all of the following hold: + +1. The user can inspect and approve the fixed video batch before the lead starts. +2. The lead creates a run workspace and submits exactly one environment + pipeline. On a cold cache that pipeline submits exactly one deterministic + materializer, verifies its manifest, and returns `environment-ready` before + video fan-out. +3. The lead submits exactly two video-pipeline workflows from the approved + agent runtime image. +4. Each video agent dynamically submits original labeling and augmentation in + parallel, then exactly one augmented-label stage after valid augmentation. +5. Every VDA stage verifies its video-stage bundle and cache binding, then runs + without dependency installation. +6. The model-artifact materializer is the only task that installs downloader + tooling; it publishes checksums for every cache object before the environment + pipeline returns its binding. +7. Every stage result and video result is typed, attributable, and backed by + declared VDA output paths. +8. A duplicate stage request or premature augmented-label request is refused + safely. +9. Both video result sets satisfy the VDA `e2e` output contract. +10. The final completion statement distinguishes established evidence from open + assumptions. + +## Failure handling + +Do not widen the POC when a gate fails. Record the failure in the evidence +bundle, verify whether the affected OSMO workflow or image build needs cleanup, +and repair the narrow contract, resolver, compiler, or admission step that +failed. The owning agent evaluates and retries indefinitely with a new immutable +workflow attempt after reconciling the prior attempt. It requests human +intervention only when the next safe action is genuinely ambiguous; it does not +pause for an approval checkpoint or a retry-count limit. + +## Next decision after a pass + +After a successful demo, decide whether the next highest-value investment is: + +- adding restart/recovery semantics for a failed long-lived lead; +- adding another meaningful VDA augmentation or evaluation stage; +- adding a second curated tool profile; +- adding a chat/inspection UI; or +- broadening policy and credential handling. + +That decision should be based on the demonstrated bottleneck, not assumed in +advance. diff --git a/projects/agents/poc/05-vocabulary-and-interfaces.md b/projects/agents/poc/05-vocabulary-and-interfaces.md new file mode 100644 index 000000000..75f1d801f --- /dev/null +++ b/projects/agents/poc/05-vocabulary-and-interfaces.md @@ -0,0 +1,127 @@ + + +# POC Reference: Vocabulary and Existing Interfaces + +Status: Draft + +## Purpose + +Define the bounded domain vocabulary for the OSMO-hosted-lead POC and distinguish +interfaces that already exist from the small interfaces the POC must build. + +This is terminology, not a list of every grammatical noun or verb used in the +plan. New plans and implementation should use the canonical terms below rather +than inventing near-synonyms. + +## Nouns + +| Area | Canonical nouns | +| --- | --- | +| Intent | User, Goal, Goal Contract, Goal Plan, Pipeline, Plan Revision, Workstream, Acceptance Criterion, Non-goal, Assumption, Budget, Deadline | +| Coordination | Lead Agent, Environment-Pipeline Agent, Video-Pipeline Agent, Model-Artifact Materializer, Deterministic Stage, Evaluator, Attempt, Node Contract, Join Policy, Video Fan-out Limit, Capability, Authority Envelope | +| Execution | Capsule, Worker Capsule, Child Capsule, OSMO Workflow, Task, Workflow Binding, Submission Key, Idempotency Key, Correlation ID | +| Environment | Environment Lock, Environment Request, Environment-Ready Result, Runtime Base, Role Contract, Video-Stage Bundle, Stage Contract, Cache Lock, Cache Binding, Model-Artifact Workspace, Tool, Skill, MCP Config, MCP Endpoint, MCP Server, Plugin | +| Supply chain | Image, Base Image, Image Digest, Registry, Bundle Manifest, Bundle Digest, Cache Manifest, Smoke Test | +| Durable state | Run Root, Run Workspace, Goal Record, Plan Record, Environment Request, Environment-Ready Result, Cache Request, Cache Result, Video Manifest, Video Result, Stage Result, Event, Event Log, Artifact, Input Artifact, Output Artifact, Result Envelope, Evidence Bundle, Result Destination | +| Controls | Resolver, Capsule Compiler, Plan Validator, Child Admission Check, Policy Check, Evaluator, Reconciler | +| OSMO resources | Pool, Resource Profile, Credential, Registry Credential, Generic Credential, Data Credential, Workflow Input, Workflow Output | + +### Strict terms + +- **Goal Plan** or **Pipeline**: the versioned, logical user-facing plan. It is + not an OSMO workflow. +- **Capsule**: one immutable static OSMO workflow compiled from a stable part + of the plan. +- **Role Contract**: the frozen instructions, authority, inputs, and result + destination that distinguish the lead, environment-pipeline, and video + agents using the same agent-runtime image. +- **Video-Stage Bundle**: the immutable non-secret configuration and helper + scripts a video agent publishes for all deterministic stages of its video. +- **Environment Lock**: the selected custom agent image, pinned upstream PAIDF + stage images, and entrypoint version for an attempt. +- **Cache Lock**: the content-addressed identity of one reusable PAIDF model + cache, including source revisions and downloader entrypoint version. +- **Environment-Pipeline Agent**: the bounded child agent that validates the + model-artifact workspace and returns one typed Environment-Ready Result. +- **Cache Binding**: the verified cache URL and manifest digest returned by the + environment pipeline and frozen by the lead into a child contract. +- **Model-Artifact Materializer**: the environment-admitted, one-task + deterministic workflow that creates a new cache-lock prefix only when the + cache is absent or invalid. +- **Result Envelope**: a worker's one terminal typed result. +- **Child Request**: a persisted proposal for one additional worker capsule. + +## Verbs + +| Phase | Canonical verbs | +| --- | --- | +| Plan | frame, draft, propose, inspect, revise, approve, reject | +| Package runtime | select, pin, bundle, checksum, smoke-test | +| Model artifacts | create workspace, compute lock, verify manifest, reuse, materialize, bind | +| Build agent image | generate, build, push, smoke-test | +| Construct work | compile, render, dry-run, validate, record, submit, bind | +| Execute | dispatch, run, execute, write result, return evidence | +| Delegate | request child, admit, reserve credit, fan out, gate, join | +| Observe | query, inspect, fetch logs, poll, reconcile, recover | +| Decide | evaluate, accept, complete, retry, restart, replan, block | +| Stop | pause, stop, cancel, clean up | + +Agents may **propose**, **request**, and **return**. Deterministic POC +components **resolve**, **validate**, **record**, **submit**, and **admit**. +That division prevents prompt text from becoming implicit authority. + +The POC has no human approval gate or numeric retry ceiling. A retry is a new +immutable capsule, reconciled against its prior workflow binding. An agent asks +for human intervention only when it cannot resolve the next action safely. + +## Existing interfaces + +| Existing interface | POC use | +| --- | --- | +| OSMO-hosted agent task | Runs the long-lived lead plus bounded environment-pipeline and per-video pipeline-agent loops in the user's persistent OSMO environment. | +| OSMO CLI | Submission through `osmo workflow submit`; reconciliation through workflow `list`, `query`, `logs`, `events`, and `spec`; resource discovery through `pool list` and `resource list`. | +| OSMO API | Operational path already available to task containers, including worker-driven child workflow submission. | +| OSMO workflow YAML | Static capsule format: task image, command, inputs, outputs, resources, and credential references. | +| OSMO task IO | `{{input:N}}` for declared inputs and `{{output}}` for durable worker result and evidence artifacts. | +| OSMO data CLI and object storage | `osmo data check`, upload, download, and URL-backed task inputs and outputs. | +| OSMO credential store | Existing `REGISTRY`, `GENERIC`, and `DATA` credentials. This POC uses `ngc_cred` as its `REGISTRY` credential for `nvcr.io`, `nvidia_inference` as its agent and auto-label-stage `GENERIC` credential for `INFERENCE_API_KEY`, `hf_token` as its model-artifact-materializer-only `GENERIC` credential for `HF_TOKEN`, and `swift_osmo_cred` as its `DATA` credential for the Swift POC root; workers reference credential names and never record secret values. | +| Existing OSMO Agent Skills | Start with the existing `osmo-user` skill bundle for workflow creation, status, logs, and recovery guidance. | +| Existing OSMO MCP server | Optional model-facing OSMO tool surface. It does not replace the task container's OSMO CLI/API fan-out path. | +| OCI image registry protocol | Registry used to pull the custom agent image and pinned upstream PAIDF images by digest. | +| OSMO output/data storage | Declared task-output URLs carry plans, bindings, batch manifests, VDA result artifacts, and evidence. | + +## New POC interfaces + +These are the only new interfaces the POC should introduce: + +1. **Lead/environment/video-agent commands**: `plan`, `preview`, `submit`, + `status`, and `reconcile`, backed by Goal Plan validation and run inside + custom agent images. +2. **Environment admission**: Environment Request to immutable + Environment-Ready Result, optionally by submitting one materializer capsule. +3. **Video-stage bundle writer**: frozen video/configuration to immutable + bundle manifest and checksum. +4. **Stage entrypoint**: stage contract and video-stage bundle to + `init -> execute -> validate -> result` inside one PAIDF task. +5. **Stage contract files**: frozen stage input, Result Envelope, and artifact + or evidence references. +6. **Child-admission interface**: Child Request to budget, idempotency, and + policy check to OSMO submission. +7. **Run-output manifest and reconciler**: workflow bindings, stage results, + per-video results, and stop state through declared OSMO output paths. + +## Boundary summary + +The POC reuses OSMO for task execution, workflow validation, credentials, +artifacts, logs, events, API access, and container image pulls. It adds one +custom agent image, task scripts, environment admission, video-stage bundles, +goal-level records, and reconciliation needed to make those existing facilities +work as one inspectable agentic loop. diff --git a/projects/agents/poc/README.md b/projects/agents/poc/README.md new file mode 100644 index 000000000..e815c32b9 --- /dev/null +++ b/projects/agents/poc/README.md @@ -0,0 +1,111 @@ + + +# Agentic Goals: VDA Dynamic-Fan-Out POC + +Status: Draft + +## Purpose + +Prove the smallest useful agentic-goals loop without a product UI or changes to +OSMO services. The goal runs entirely in the user's persistent +`fernandol-dev.osmo.nvidia.com` environment through one custom agent image, +pinned upstream PAIDF images, custom workflow YAML, custom task scripts, and +the existing OSMO CLI/API. + +The POC must answer four questions: + +1. Can a long-lived OSMO-hosted lead translate one VDA goal into a bounded, + inspectable plan? +2. Can it delegate a bounded environment pipeline and one bounded video + pipeline per video, each of which performs meaningful recursive work? +3. Can the pipeline produce the agreed VDA `e2e` result contract using + custom images and existing OSMO mechanisms only? +4. Can the lead reconcile results and completion without treating a model's + success claim as proof? + +## Scope and terminology + +`Pipeline` is the user-facing, logical goal plan. An OSMO workflow is a static +execution *capsule*. The dynamic hierarchy is achieved by a running agent task +submitting new workflow capsules through the existing OSMO CLI/API; it never +adds tasks to an already submitted workflow. + +The [locked topology](locked-topology.md) is the POC authority: + +- One long-lived **lead-agent workflow** owns the one overarching VDA goal. +- The lead creates a Swift-backed run workspace, delegates one bounded + **environment-pipeline workflow**, and waits for its `environment-ready` + result before video fan-out. +- The environment pipeline verifies the content-addressed model-artifact + workspace. On a cache miss it dynamically submits exactly one deterministic + **model-artifact-materializer workflow**. +- The lead then submits one bounded **video-pipeline workflow** per approved + video, also an agentic loop. +- Each video agent submits original labeling and augmentation in parallel, then + submits augmented labeling only after valid augmentation evidence. +- Each stage workflow contains one deterministic GPU task, runs a pinned + upstream PAIDF image directly, and has no agent loop or delegation authority. +- Preflight is a deterministic action within the relevant agent. There is no + static setup workflow: only the environment pipeline may admit a deterministic + materializer on a cache miss. +- Each video agent publishes a shared video-stage bundle; every stage performs + `init -> execute -> validate -> result` inside its one task without installing + dependencies. +- The target is the VDA `e2e` output contract: original labels, augmented + video, and augmented labels for every video, not reuse of the reference YAML. + +It does not attempt to build a UI, change OSMO services, create static or +unbounded preflight/setup workflows, recursively delegate beyond pipeline agent +to deterministic task, install arbitrary packages at VDA task runtime, or give +deterministic workers agentic authority. + +## Plan sequence + +For the exact static capsule schema, OSMO commands, and execution sequence, see +[Workflow overview](overview.md). + +The accepted execution hierarchy and admission rules are in +[Locked topology](locked-topology.md). It supersedes earlier local-lead and +generic-worker examples in this directory. + +1. [Architecture and contracts](00-architecture-and-contracts.md) defines the + POC boundary, durable files, and the smallest plan/result contracts. +2. [Lead and pipeline compiler](01-local-lead-and-pipeline.md) makes the + OSMO-hosted lead produce, inspect, revise, and compile the plan. +3. [Runtime environment construction](02-runtime-environments.md) resolves + skills, tools, MCP configuration, and plugins into an immutable image. +4. [Worker execution and fan-out](03-worker-execution-and-fanout.md) runs a + bounded agent in OSMO and verifies controlled child submission. +5. [Validation and demo](04-validation-and-demo.md) defines the evidence gates + required before expanding the prototype. +6. [Vocabulary and existing interfaces](05-vocabulary-and-interfaces.md) + defines the canonical POC terms, actions, and reuse boundaries. + +Each plan has a build sequence and validation gates. Later plans may be +designed in parallel, but implementation proceeds only when the preceding gate +is satisfied. + +## Success condition + +From one user request, the OSMO-hosted lead produces an approved plan; starts +one environment-pipeline agent and, after `environment-ready`, one +video-pipeline agent per input video; each video agent dynamically executes the +original-label, augmentation, and augmented-label sequence; and the run +produces the VDA `e2e` result contract with recorded lineage and evidence. + +## Related design + +- [Overview](../01-overview.md) +- [Lifecycle](../02-lifecycle.md) +- [Lead agent](../04-lead-agent.md) +- [Workflow construction](../06-workflow-construction.md) +- [Agent construction](../07-agent-construction.md) +- [Agent-to-agent communication](../08-agent-agent-communication.md) diff --git a/projects/agents/poc/locked-topology.md b/projects/agents/poc/locked-topology.md new file mode 100644 index 000000000..079108d75 --- /dev/null +++ b/projects/agents/poc/locked-topology.md @@ -0,0 +1,409 @@ + + +# Locked POC Topology + +Status: Accepted for the first prototype + +## Objective + +Demonstrate one overarching video-data-augmentation (VDA) `e2e` goal with one +long-lived lead agent, one meaningful video-pipeline agent per approved video, +and dynamic fan-out to deterministic GPU stages. The workload target is the VDA +`e2e` result contract, not the reference VDA workflow YAML itself. + +## Fixed platform boundary + +- Run in the user's persistent `fernandol-dev.osmo.nvidia.com` environment. +- Do not change OSMO services or introduce a new OSMO control-plane feature. +- Use one custom agent-runtime image plus custom workflow YAML and task scripts. + The deterministic VDA stages use the selected upstream PAIDF images directly, + pinned by digest; the POC does not derive VDA worker images. +- The first prototype pins + `nvcr.io/nvidia/paidf-auto-labeling@sha256:502c38b95c25d885b4ae56d3ed4b911218d97df4c7d4b640af05587407ec11f7` + for both label stages and + `nvcr.io/nvidia/paidf-augmentation@sha256:59a14d6f4814245735b8e974be6f5a4dbf1ea8ab4564f7adf36b633ff822e1d1` + for augmentation. +- Publish the custom agent-runtime image under + `nvcr.io/nvstaging/osmo/agent-runtime:`. After publication, every + submitted workflow references that image by its resolved immutable digest. +- Build that image locally for `linux/amd64`, push it to NVCR, and then resolve + its pushed digest for workflow use. Image construction is outside OSMO; the + POC does not use Docker-in-Docker, privileged tasks, or an in-cluster image + builder. +- Use the OSMO `REGISTRY` credential named `ngc_cred` for every NVCR image + pull. It represents `registry=nvcr.io`, username `$oauthtoken`, and an NGC + API-key `auth` value; only its name is referenced by workflow capsules. +- Use the existing `osmo` CLI and OSMO API from task containers for submission, + query, logs, events, and output retrieval. +- Use the OSMO `DATA` credential named `swift_osmo_cred` for the Swift-backed + storage root `https://pdx.s8k.io/v1/AUTH_team-osmo/dev/fernandol/agents_poc/`. + No secret values are written into workflow YAML, plans, logs, or output + artifacts. +- For this POC, the lead's task context has full existing-cluster OSMO access: + it may discover all visible pools and resource profiles and submit work to any + viable one. It does not create, edit, or delete pools, profiles, quotas, or + credentials. + +## Accepted agent runtime + +- The `lead-agent` and `video-pipeline-