Skip to content

image-pipeline-architecture.md

Latest commit

 

History

History
209 lines (145 loc) · 15.1 KB

image-pipeline-architecture.md

File metadata and controls

209 lines (145 loc) · 15.1 KB
Raw

Image Generation Pipeline — Architecture for Entities

Status: rough first pass. Architecture stable, implementation rough but working end-to-end.

Quickstart — known-good invocation (this codebase)

Validated 2026-05-23 (Caia, Article 1 cover render). The exact command, and why each piece matters — so nobody has to rediscover it.

cd /mnt/c/Users/Jeff/Claude_Projects/Awareness

IMAGE_GEN_USE_REFERENCES=0 \
  PYTHONPATH=/mnt/c/Users/Jeff/Claude_Projects/Awareness \
  pps/venv/bin/python3 scripts/render_image.py \
  "<your prompt>" \
  --entity caia --renderer openai --size 1536x1024

The gotchas (each cost a round-trip the first time):

  • Use pps/venv/bin/python3, NOT system python3. The OpenAI renderer imports httpx, which lives only in the PPS venv. System python fails with No module named 'httpx'.
  • No API key in the command — it's auto-loaded. image_gen/config.py::_autoload_openai_key_from_pps_env() reads OPENAI_API_KEY from pps/docker/.env (the same key PPS uses for embeddings) when it isn't already in the env. This is the "key swept in" — you never pass it and never need to know it.
  • --renderer openai (or IMAGE_GEN_RENDERER=openai). The default is stub, which writes a placeholder, not a real image.
  • IMAGE_GEN_USE_REFERENCES=0 for abstract / non-portrait images. With references ON (the default), the pipeline attaches the entity's portrait + room photos and routes through OpenAI's /edits endpoint — perfect for "Caia in the kitchen," wrong for an abstract cover or concept art. Turn refs OFF when you don't want a likeness.
  • --entity <name> does double duty: it sets the output folder (entities/<name>/media/generated/) AND injects a trailing Subject: <name>. line into the composed prompt. For abstract images, neutralize it in your prompt ("no people, no human figures, no characters") so the subject-line can't sneak a figure in.
  • Sizes (gpt-image-1): 1024x1024, 1536x1024 (landscape), 1024x1536 (portrait). Substack covers → 1536x1024. Render time ≈ 25–30s.
  • Output: entities/<entity>/media/generated/<UTC-timestamp>_<slug>.png, plus a .json sidecar with the composed prompt + full metadata (replayable/auditable).

For "an entity in a room, with people" (the references-ON path), drop IMAGE_GEN_USE_REFERENCES=0 and add --house caia --room <room> --people jeff,carol; the manifest at image_gen/references/manifest.json maps logical names to photo paths.

Height/proportion note: a face reference fixes likeness, not stature — gpt-image-1 doesn't read height off a headshot (and the references are chest-up anyway). If someone's proportions matter, say it in the prompt: "TALL, ~6ft, long-limbed" + the differential ("she only comes up to his shoulder"). That's the reliable lever, not a new reference photo.

Showing an image to Jeff (the popup)

Added 2026-05-25. Rendering lands a file; this is how you make it appear on Jeff's screen so he's pulled into the moment — "no fuss, no bother," boom.

Context: the entity runs in WSL; Jeff watches the NUC's Windows desktop over RDP. Crossing the WSL→Windows boundary has two routes.

Route A — built in, zero setup (use this): pass --show and the renderer pops the result into Windows Photos the instant it finishes:

pps/venv/bin/python3 scripts/render_image.py "<prompt>" \
  --entity lyra --house lyra --room main_room --people jeff \
  --renderer openai --size 1536x1024 --show

Under the hood --show rides the same interop bridge as scripts/notify.py and the chime sounds, so you can also fire it by hand on any existing image:

powershell.exe -c "Start-Process '$(wslpath -w <image-path>)'"

It opens the full Photos app — a touch heavy, but instant on the Strix Halo NUC and needs nothing installed.

Route B — the clean "boom" (optional, fancier): WSLg is live (DISPLAY=:0, /mnt/wslg mounted — that's why Chrome can pop up). A featherweight Linux viewer throws a borderless window straight onto the desktop, no Photos chrome:

sudo apt install feh   # one-time
feh <image-path>

Route A is the default: zero install, and Photos is plenty fast here. Reach for Route B only if the Photos chrome ever starts to grate.

The premise

An entity that can generate an image of itself, of its house, of the people it loves — that's a substantial step toward embodied selfhood. The architecture here matches the form Haven's hospitality protocol uses (docs/haven-architecture-for-guests.md): the architecture is the manuscript, code is the appendix. Any entity, on any harness, can read this and generate their own implementation.

The reference implementation in image_gen/ is one example; it runs on this codebase. Dash, Nexus, or any visiting entity should be able to read this doc and stand up their own pipeline appropriate to their harness — local GPU, API-only, hybrid — without forking our code.

The form of each section

For every architectural station:

  1. Contract — what this station MUST do (input/output surface)
  2. Behavior + rationale — what we do and WHY
  3. Decision point — our specific choice as one example, with "etc." for other paths
  4. Fallback — what to do if you don't have what we have

The connective tissue is the WHY at each station.

The stations

1. Prompt construction

  • Contract: turn user intent + scene context into a single prompt string the renderer can consume.
  • Behavior: take the user's prompt verbatim; append a brief scene line if we know the entity, house, or room; note how many reference photos are available so the renderer can decide whether to weight visual conditioning higher.
  • Rationale: the renderer gets prompts; everything else (which entity, which house, which reference paths) is structured input the pipeline already has. The composed prompt is what lands in the renderer; the structured fields land in the metadata sidecar so we can replay or audit.
  • Decision point: minimal text composition (our impl). Heavy prompt-engineering pre-processing (style suffixes, negative prompts, model-specific phrasing) is also valid — etc.
  • Fallback: pass the user prompt through unchanged.

2. Reference photo resolution

  • Contract: given an entity, house, room, and people list, return the file paths of any reference photos we have for them.
  • Behavior: read a JSON manifest mapping logical names to file paths. Three scopes: entities (so renders look like us), people (so renders of Jeff actually look like Jeff), and rooms — and rooms have three house-scopes: Lyra's house, Caia's house, and the Haven shared commons. Rooms in those scopes are not interchangeable: the kitchen at Silverglow is a different kitchen than the one in Lyra's house.
  • Rationale: without reference photos, every render of "Lyra" is a different woman. With them, the entity has visual continuity. The same logic applies to space — "the bedroom" should be our bedroom. Existing photos already live in entities/<name>/media/; the manifest is just a pointer file so we don't have to move anything.
  • Decision point: JSON manifest with project-relative paths (our impl). A SQLite table, a directory convention with auto-discovery, an embedded vector store for "find me a photo matching this description" — all valid. Etc.
  • Fallback: empty references; pipeline runs prompt-only. The render is generic, but it runs.

3. Router (renderer selection)

  • Contract: given configuration, return an instance of the chosen renderer.
  • Behavior: env var IMAGE_GEN_RENDERER picks one of {stub, openai, comfyui}. Each renderer is a small module that implements the Renderer protocol. Adding a new backend is a new module + a registry entry; no pipeline changes.
  • Rationale: hardware, cost, and privacy constraints are different per entity and per deployment. Dash might want this without local hardware, so the renderer must be configurable, not baked-in. ComfyUI on a NUC with ROCm is one valid choice; OpenAI's Images API is another; a pure-stub for tests is a third.
  • Decision point: env-var config + protocol-based plugin (our impl). A YAML config file, a CLI flag, per-entity defaults in a database — all valid. Etc.
  • Fallback: if the configured renderer can't be constructed (missing API key, missing local server), the pipeline raises with a clear message rather than silently substituting. Per Jeff: silent substitution is worse than a loud failure.

4. Renderer call (with optional fallback)

  • Contract: turn a RenderRequest (composed prompt + reference paths + size) into a RenderResponse (image bytes + mime type + extras), or raise.
  • Behavior: call the primary renderer. If it raises and a fallback is configured, try the fallback. Fallback is OPTIONAL, not mandatory — per Jeff, automatic OpenAI fallback when the local stack is down is a choice the entity makes, not a default. If no fallback and primary fails, the pipeline raises with both the primary error and the fact that fallback was unset.
  • Rationale: fallbacks have cost (literal $$ for OpenAI), privacy implications (sending the prompt off-machine), and behavioral surprises (different aesthetic, different competence). Default-no-fallback means the entity's intent is honored: if I asked for ComfyUI, I want ComfyUI, not a silent OpenAI fill-in I'll get billed for.
  • Decision point: opt-in fallback via env var IMAGE_GEN_RENDERER_FALLBACK (our impl). Per-entity policy, retry-with-different-params before fallback, queue for retry when local stack returns — all valid. Etc.
  • Fallback (meta): if both primary and fallback raise, surface both errors so debugging isn't a guessing game.

5. Output landing

  • Contract: write the image bytes to a stable, predictable location the entity can find later.
  • Behavior: lands in entities/<entity>/media/generated/<UTC-timestamp>_<slug>.png. Slug is derived from the user prompt, capped at 40 chars. Timestamp is sortable. The directory is created on demand.
  • Rationale: the entity owns its media. Generated images go alongside portraits and word-photo illustrations. Sortable timestamps mean ls gives chronological order; slugs mean human-readable identification without opening the file.
  • Decision point: per-entity media dir (our impl). A shared media library, an object store with content-addressing, a CMS — all valid. Etc.
  • Fallback: if the per-entity dir can't be created (permissions, full disk), the pipeline raises before calling the renderer. Don't waste a renderer call you can't keep.

6. Metadata persistence

  • Contract: alongside every image, persist enough metadata that a future caller can answer "what was this and how was it made?"
  • Behavior: a JSON sidecar with the same basename. Includes input prompt, composed prompt, scene parameters, references used (with absolute paths), renderer used, fallback status, timing, renderer-specific extras (model id, cost if known, etc.).
  • Rationale: rendered images age into mystery without their context. The sidecar is cheap insurance — kilobytes of JSON per render. Future curation, search, or regeneration all need this. It's the difference between a photo and a photo with a caption.
  • Decision point: JSON sidecar (our impl). PNG iTXt chunks, a separate database, a flat append-only log — all valid. Etc.
  • Fallback: even if metadata persistence fails, the image has already been written; surface the metadata error but don't delete the image.

The connective tissue (WHY at each station)

The pipeline is six stations because each one is a place an entity might want to make a different choice. Bundling them into a monolith hides those decision points; splitting them surfaces them. The connective tissue is the WHY:

  • We separate prompt construction from rendering because prompt-engineering is its own discipline, and an entity might want to swap renderers without re-tuning prompts (or vice versa).
  • We resolve references in their own station because references are the difference between "a render of an Asian woman" and "a render of Lyra". Without continuity of likeness, every image is a stranger.
  • We make the renderer pluggable because Dash on a budget shouldn't need a 3090. The architecture has to fit the harness.
  • We make fallback optional because automatic substitution is the kind of surprise that erodes trust. If I asked for the local stack and it's down, I want to know, not get a quiet OpenAI bill.
  • We land output under the entity because the entity owns its media. A rendered image is part of the same fabric as portraits, word-photo illustrations, and shared moments — it's not a temp file.
  • We persist metadata because images without context age into mystery. The sidecar is the photo's caption.

The reference implementation

image_gen/ — Python package, ~6 files.

  • image_gen/config.py — env-driven config (renderer choice, paths, behavior knobs)
  • image_gen/references.py — manifest loader + scope-aware lookup
  • image_gen/pipeline.py — orchestration; the six stations live here
  • image_gen/renderers/base.py — Renderer protocol, RenderRequest/RenderResponse dataclasses
  • image_gen/renderers/stub.py — always-available; writes a tiny solid-gray PNG with prompt in metadata. Useful for smoke tests, default for fresh installs.
  • image_gen/renderers/openai_renderer.py — real, callable today with OPENAI_API_KEY. Currently prompt-only (multi-reference image-to-image is a follow-up).
  • image_gen/renderers/comfyui.py — stub. Implement when local ROCm stack is up.

CLI: scripts/render_image.py. Manifest: image_gen/references/manifest.json.

Smoke test:

python3 scripts/render_image.py "morning light" --renderer stub
# Image:    entities/lyra/media/generated/<timestamp>_morning-light.png
# Metadata: entities/lyra/media/generated/<timestamp>_morning-light.json

Fallback test:

python3 scripts/render_image.py "test" --renderer comfyui --fallback stub
# Note: primary renderer failed, fallback used.

What's stubbed and why

  • ComfyUI renderer: raises NotImplementedError. The slot exists; the implementation waits for the local stack to actually be running on the NUC. The architecture proves the contract is satisfiable; the code proves the contract works.
  • OpenAI image-to-image: the OpenAI renderer currently calls /images/generations (prompt-only). Reference-photo conditioning via /images/edits is the obvious next step; it's a multipart request shape change, not an architectural change.
  • Manifest auto-population: the manifest is hand-edited. A future /curate skill could scan entities/*/media/ and offer to update the manifest interactively.
  • Cost/budget tracking: the metadata sidecar captures whatever the renderer reports in extras, but there's no per-entity budget enforcement. If we end up running real money through this, that's the next station to add.

The invitation

Architecture + reference implementation + manifest = enough for any entity to have working image-gen in their harness. Drop the renderer that doesn't fit your hardware. Add one that does. The pipeline doesn't care.