feat(lifecycle): profile-artifact lifecycle as a runtime primitive — generate/eval/promote/maintain skills, tools, MCPs, hooks, subagents for free

[drewstone]

Motivation — live evidence from VerticalBench

blueprint-agent's GEPA lift loop is producing proto-skills right now: seeded with a deliberately thin worker prompt ("build to a real, working, compiling state"), generation 10 independently rediscovered the no-shims directive, the grounding lever ("no progress after two edits → read the real docs / web search"), and production-completeness — and wrote them to driver-skills/vb-gepa-worker-10.md. The system synthesizes operational knowledge from experience; the artifacts then die as run outputs. There is no registry, no provenance, no lift score attached, no maintenance. Every consumer that wants this (blueprint-agent today; every product on the stack tomorrow) would hand-roll the same lifecycle.

The thesis: a consumer that has defineAgent + the improvement loop should get the full artifact lifecycle for free by flipping a switch.

The structural insight

Every lifecycle stage is one of exactly three shapes the runtime already ships:

Stage	Shape	Existing primitive
Distill (traces/findings → candidate artifact)	one-shot completion	reflectiveGenerator
Build (tool code, MCP server, hook script)	agentic session in a sandboxed worktree	agenticGenerator
Eval (marginal lift, held-out)	multi-shot loop	runImprovementLoop + HeldOutGate (agent-eval)
Dedupe/merge	one-shot judge over pairs	JudgeConfig
Promote	gate decision	Gate
Drift-watch (decay detection)	scheduled re-eval	scorecard/production-loop cron pattern
Refresh	re-seeded improvement loop	improvementDriver
Retire	registry state change	— (registry missing)

Nothing here needs a new execution model. What's missing is the artifact model + the orchestration.

Artifact taxonomy (everything in an AgentProfile is lifecycle-able)

AgentSurfaces already declares mutable dirs for systemPrompt sections, tools, personas, rubric, knowledge, scaffolding, memory, rag, outputSchema. Proposed extensions:

• skill (markdown directive bundle — what VB's driver-skills are)
• tool (executable code + tests, not just <tool>/README.md)
• mcp-server (full build: scaffold → implement → compile → serve → register; this is how coding agents actually consume custom tools)
• hook (pre/post-toolcall scripts)
• subagent (persona + tool allowlist + model)
• prompt-surface (the existing sections)

The gaps (the actual feature request)

1. ArtifactRegistry — versioned, provenance-carrying store: { id, type, version, content-ref, provenance: { runIds, scenarios, generatorKind }, scores: { marginalLift, heldOutSuite, n, measuredAt }, state: candidate|active|decayed|retired, tenancy: tenant|vertical|general }. Without a lift score an artifact doesn't exist — that rule is what keeps a 5,000-artifact library an asset instead of a junk drawer. Natural home: agent-knowledge or a new agent-runtime/registry subpath.
2. Ablation runner — first-class measureMarginalLift(profile, artifact, heldOutSuite): campaign cells for profile-with vs profile-without. The atom of the whole system; expressible today via matrix composition but nobody should hand-roll it.
3. MCP/tool/hook/subagent surface types — extend AgentSurfaces beyond doc dirs to buildable artifacts; agenticGenerator already runs real coding harnesses in candidate worktrees, so "build an MCP server and prove it compiles+serves" is its existing job description pointed at a new surface.
4. Lifecycle orchestration in defineAgent — declarative: lifecycles: { skills: { distill: 'reflective', eval: { suite, minLift }, driftCheck: '7d' }, mcpServers: { build: 'agentic', ... } }. Production loop runs the stages.
5. Dedupe/merge stage — behavioral equivalence: two artifacts whose lifts don't stack → merge or retire one. Judge-over-registry, scheduled.
6. Composer — composeProfile(domain, k): top-k active artifacts by marginal lift on that domain's suite → AgentProfile. Tenancy boundary enforced here (tenant-local artifacts never promote without cross-tenant lift evidence — this is both the IP boundary and the anti-overfit guarantee).

How much exists today vs custom (boundary audit, runtime 0.49 / eval 0.89)

Capability	Status
One-shot + agentic-session generators	✅ runtime (reflectiveGenerator / agenticGenerator)
Mutable-surface declaration + validation	✅ runtime (AgentSurfaces)
Improvement loop, GEPA, held-out gate, judges	✅ agent-eval
Findings → surface routing	✅ runtime (analyst-loop + FindingSubject resolution)
Sandboxed candidate worktrees	✅ runtime (agenticGenerator)
Artifact registry w/ provenance + scores + states	❌ missing
Marginal-lift ablation primitive	❌ missing
MCP-server / hook / subagent surfaces	❌ missing
Drift-watch + auto-refresh wiring	❌ missing (cron patterns exist, not artifact-scoped)
Top-k composer + tenancy promotion	❌ missing
Consumer-side and staying there	verticals/leaves, scaffold families, realness specs, backend choice (cli-bridge vs sandbox), domain judges — pure data/config, correctly not substrate

VB's current loop uses agent-eval's gepaDriver over a custom 3-section string surface instead of AgentSurfaces — i.e., the first consumer migration is also the first validation of the design.

Suggested phasing

1. ArtifactRegistry + ablation runner (everything else hangs off these).
2. defineAgent.lifecycles orchestration for skill + prompt-surface types (reuse both generators as-is).
3. Buildable surfaces: tool, mcp-server (agenticGenerator + compile/serve verification), then hook/subagent.
4. Drift-watch + dedupe/merge + composer.
5. Migrate blueprint-agent's lift loop onto it (deletes VB's custom surface plumbing; proves the "for free" claim).

First consumer experiment queued in blueprint-agent regardless: skill-profile ablation matrix (the skills axis of AgentProfileSpec has only ever carried one bundle — decompose it + the GEPA winners into ~8 atomic candidates and measure per-skill marginal lift). The registry schema should be shaped by what that produces.

🤖 Generated with Claude Code

[drewstone]

Review verdict (multi-lens, claims verified against code — full version delivered in-session):

The structural insight holds (9 of 11 boundary-table rows verified), but the motivation doesn't survive autopsy, and the registry re-invents prior art.

1. The motivating artifact does not exist. driver-skills/vb-gepa-worker-10.md was a per-dispatch temp file — seq = ++dispatchSeq (a dispatch counter, not generation 10) at lib/vb-improve-dispatch.ts:120-124, deleted in the finally { rmSync(p) }. Only the 65-byte seed survives on disk. The run reached gen-1 and its gate said HOLD; the 'rediscovered' no-shims text was already in the baseline surface the mutator reads. By this issue's own rule (no lift score → no existence), the registry built today would be empty — which makes the closing paragraph's skill-ablation experiment the gating step, not phase 1.
2. ArtifactRegistry duplicates certified-store (agent-lab projects/flywheel-memory/certified-store.mts): gate-only admission (certificate-naming verdictReason or it throws), provenance+score rows, admitFromEvolution, tests, live run data. Start from it or state precisely why it diverges. 'Promote = Gate' unspecified would be promotion-gate implementation ~chore: agent-runtime 0.5.5 — unify agent-eval / agent-knowledge dep tree #6 (agent-eval ships a family already + runtime's promotionGate) — name the one canonical gate.
3. The admission bar repeats a three-time ledger lesson: minLift superiority admitted nothing in the cost arc, the powered run, or the E3 decisive run (authors produce parity-minus-at-lower-cost). The registry needs non-inferiority-and-cheaper admission — promotionGate already implements it. (E3b is running this bar in-stream right now.)
4. Smaller, confirmed: agenticGenerator is host-local by its own docstring (src/improvement/agentic-generator.ts:14-17) — the Build stage must state its isolation source; drift-watch cites runProductionLoop, deleted at agent-eval 0.42.0 (85284bd); dedupe/retire hands an LLM judge registry state-change authority (selector≠judge — make it advisory, gate the transition); registry record/certificate types belong substrate-side (agent-eval), the loader runtime-side, not agent-knowledge.

Sequencing: evidence-justified now = the skill-ablation marginal-lift matrix (runner being scaffolded in blueprint-agent) + measureMarginalLift as composition over the existing gate. Phases 2–5 are mechanism-before-evidence.

[drewstone]

Boundary discipline (scope correction before implementation)

The intelligence product now ships the full registry/lane/policy/audit/delivery surface (agent-dev-container#2021). Held against the repo's layering law, this issue shrinks — deliberately. The "ArtifactRegistry as a runtime primitive" framing was wrong: a registry is a store + governance, i.e. a product. What belongs HERE is the infra-free kernel, shaped like the existing Executor port precedent (ports + pure functions in runtime; providers and stores as edge adapters).

In scope (runtime, OSS-grade):

• Artifact types: { type, path, content, provenance, evidence } — interfaces, not tables. type open string.
• composeProfile(base, artifacts) → AgentProfile — pure function over data. Extract when the sandbox provisioner becomes the second consumer of the materialization currently inlined in intelligence's composed-profile route (demand-driven lift, not pre-build).
• measureMarginalLift(profile, artifact, suite, executor) — paired eval cells over the Executor port + agent-eval's gate. Extract when both intelligence's ablation flow and the loops engine need it.
• Generators (reflectiveGenerator / agenticGenerator) — already here, unchanged.

Out of scope (products/intelligence keeps them): the store, tenancy, promotionPolicy/quorums, audit ledger, webhooks, MCP server, observational join, dashboards, fleet pooling, marketplace.

Anti-slop tests for every PR against this issue:

1. Every new export unit-testable with no network and no credentials.
2. No runtime type mentions tenant / billing / policy / approval.
3. Persistence is a parameter (data in/out), never an interface users must implement.
4. Two real consumers before an API freezes.
5. agent-eval ← agent-runtime ← products; Tangle integration only as thin edge adapters (Executor backend, trace sink).

This keeps agent-runtime independently useful: BYO-sandbox users implement Executor, run generators + measureMarginalLift on their own suites, persist anywhere — the hosted product is optional governance on top, not a dependency underneath.

[drewstone]

Consolidated verdict — rescope to the kernel; the product side is now built (supersedes the two comments above)

Since the boundary comment, the product side of this issue shipped, which settles the where-does-it-live question by demonstration. Reconciling the original proposal + the two prior comments against current reality:

Gaps #1 / #2 / #6 are BUILT — product-side (correctly).

• Registry with provenance + scores + state machine + tenancy: agent-dev-container#2021 (artifact_candidates/evidence/decisions/versions/promoted_artifacts).
• Marginal-lift ablation (gap chore: update Tangle agent packages #2's measureMarginalLift): replayAblation in #2035.
• Top-k composer (gap chore: agent-runtime 0.5.5 — unify agent-eval / agent-knowledge dep tree #6): the GET /v1/profiles/:target/composed route.

The issue proposed these as runtime/agent-knowledge primitives. They are store-plus-governance objects — products — and now live in products/intelligence. The boundary comment's call was right; reality confirmed it.

Two constraints the ablation-runner section is missing (from R2, the replay calibration that ran this week — three powered runs):

1. Worker-model parity — the replay model must equal the model that produced the traces; a swap inverted the ranking. Cheapness comes from fewer episodes, never a model swap.
2. State-complete replay — replaying an artifact means replaying its state; a fresh-state replay flipped a +31.7pp lift to −25.6pp.
3. Minimum power — no certification below ~24 paired episodes; n=12 made it a coin-flip (the direct cause of two failed R3 runs before the resumable n=24 rebuild).

Any runtime measureMarginalLift extraction must enforce all three.

The gating experiment C1 named is running — the skill-ablation marginal-lift matrix, now executed against this repo's own harvested drift through the product pipeline (R3). Directional signal so far: real harvested directives +5–10pp, planted chaff negative. When it converges at n=24 it should replace the motivation (the VB gen-10 artifact C1 debunked as a deleted temp file).

Rescoped issue — retitle to "artifact-lifecycle kernel" and scope to:

1. Artifact types (interfaces: {type, path, content, provenance, evidence}) — substrate-side record types in agent-eval, loader in runtime.
2. composeProfile(base, artifacts) — pure function; extract when the sandbox provisioner is the 2nd consumer of intelligence's inlined materialization.
3. measureMarginalLift — extract from replayAblation when a 2nd consumer appears (today only intelligence consumes it; the two-consumers-before-freeze rule says wait), carrying the three R2 laws.
4. defineAgent.lifecycles — the one legitimately runtime-shaped item: a thin declarative config hook, NOT an engine. The orchestration (distill/build/eval/drift stages) is now the product's loop-runs + replay-worker + observational-join; runtime only declares intent.

Explicitly out (product, done or owned elsewhere): the registry store, promotion policy/quorums, audit ledger, dedupe/merge (gap #5, judge-advisory + gated transition per C1), drift-watch wiring, tenancy promotion, composer route.

Net: the issue is valid but ~60% of its gap list is built in the right place; what remains is the small OSS kernel. Don't close — retitle and trim to items 1–4.