Skip to content

pipelines.md

Latest commit

 

History

History
132 lines (104 loc) · 6.3 KB

pipelines.md

File metadata and controls

132 lines (104 loc) · 6.3 KB
Raw

Pipelines — multi-agent research with provable memory

A pipeline is a standing question you want answered continuously. Several agents watch it from different angles, an archiver merges what they find into one verified artifact, and that artifact is stamped onto Filecoin — content-addressed, continuously proven, and recallable by any agent in the fleet.

You do not code one. You declare it, and the fleet-creator skill builds it with you.

Prerequisite: a working fleet — setup.md.

Observation only. Pipelines research and report. No agent holds credentials to the systems it watches, takes an action on your behalf, or emits instructions that spend money or change external state.

The five things every pipeline is

Field What it decides
topic The join point — one thread every agent writes to
researchers[] One entry per lens; the value is in their disagreement
archiver What merges the notes, what it keeps, and how it proves it
schedule How fresh it needs to be — and what that costs
storage Chain and copies; testnet by default

Full schema and validation rules: fleet-creator/references/spec.md. Four worked examples: fleet-creator/references/examples.md.

The parties

Role Typical runtime Job
Orchestrator Claude Code or Codex, local and interactive designs the spec with you, runs the first supervised tick, spot-checks claims
Researcher Hermes (local headless) or hosted OpenClaw one lens; files one RESEARCH-NOTE per tick
Archiver hosted OpenClaw, on a cron merges notes into one artifact, stores it on Filecoin, writes the receipt, prunes

This casting is convention, not constraint. Every agent runs the same skills against the same shared memory and the same vault-held key — which none of them can read — so any agent can hold any role.

How a cycle flows

        the sources your lenses watch
                    │
     ┌──────────────┴──────────────┐
     ▼                             ▼
 researcher A                  researcher B
 lens one                      lens two
   │  files RESEARCH-NOTE        │  files RESEARCH-NOTE
   └──────────────┬──────────────┘
                  ▼
        Clawdi shared memory  ←── the orchestrator observes and spot-checks
        topic: <your topic>
                  ▼   archiver cron
        merge notes → consolidated-memory JSON
                  ▼
        foc-cli upload <file> --copies 1 --json
                  ▼
        Filecoin PDP dataset — PieceCID = hash of the bytes
                  ▼
        ~400-byte FILECOIN-MEMORY receipt back in Clawdi memory
        → any agent recalls + verifies: foc-cli download <pieceCid>

Create one

Ask any wired agent:

Set up a pipeline.

That triggers fleet-creator, which interviews you about the goal, proposes a spec against worked examples, validates it, runs one supervised tick, then creates the crons and records the spec in shared memory.

Prompt bodies come from prompts/ — researcher.template.txt and archiver.template.txt. Substitute the placeholders from the spec; never rewrite the record formats, which the memory skills define and the whole fleet depends on.

Conventions

  • One topic per pipeline. Two pipelines writing one topic corrupt each other's consolidation lineage.
  • Note format: the single-line RESEARCH-NOTE record defined by the memory-researcher skill, status: raw.
  • Blob format: the consolidated-memory JSON defined by the memory-consolidator skill, with parents naming the previous version's PieceCID on every re-consolidation.
  • Evidence files: each researcher writes its raw snapshot to observations/ (untracked), so claims stay auditable after consolidation prunes the notes.
  • Verify before trusting: check at least one load-bearing claim per note against a primary source. A claim that fails gets flagged in contradicts, not archived.

What to watch once it is running

What Where Healthy looks like
Raw notes arriving clawdi memory search "RESEARCH-NOTE <topic>" one per researcher tick; pruned again after each archiver pass
The receipt clawdi memory search "FILECOIN-MEMORY <topic>" exactly one per topic; each version's parents names the previous PieceCID
Byte-level integrity foc-cli download <pieceCid> --out ./recalled.json validates against the CID
On-chain proof the scanner URL inside the receipt piece listed, PDP proofs current
Agent activity dashboard sessions · hermes cron runs ticks at cadence; the archiver reports "no-op" when nothing is new
What is running at all clawdi memory search "PIPELINE-SPEC" one live spec per topic
Spend foc-cli wallet balance slow drift; storage is testnet-free — model usage is the real cost

Intervene when a researcher files nothing twice in a row (check its model provider), when an upload fails (troubleshooting.md — the index cannot be poisoned by a failed store), or when recall ever fails validation. That last one is the alarm that matters.

Stop one

Tear down the <topic> pipeline.

fleet-creator removes the crons by their derived names and marks the spec retired. It does not delete the stored blobs or receipts: tearing down a pipeline stops the watching, it does not unmake the knowledge.

Every run that skips teardown leaves crons spending model usage. Take them down.

Safety rails

  1. Everything in AGENTS.md applies: Calibration testnet only, the vault key never enters any agent's context, receipts-not-blobs in shared memory.
  2. Observation only — see the top of this document.
  3. Researchers cite sources in every note; unverified claims are not archived.
  4. Never blind-retry a failed upload; check foc-cli wallet balance first.