What should the adapter do in the image pipeline?

[Jah-yee]

I am trying to understand the intended role of the adapter in Wittgenstein’s image pipeline.

One possible interpretation is that the adapter is a small learned model that reads the entire scene spec and turns it into image latents. Another interpretation is that the adapter should be more like a compiler or expander: it should consume Visual Seed Code, coarse VQ hints, or provider-supplied latents, then expand those into a full decoder-native token grid.

These two interpretations seem very different.

If the adapter reads a semantic JSON blob and tries to generate a full image representation from that, then it starts to look like a small image generator. If it instead expands a compact visual code into full VQ tokens, then it seems closer to the project’s “language-to-code-to-artifact” thesis.

What should the adapter actually do?

It would be helpful to clarify:

1. Whether the adapter should read free-form prompt-like text.
2. Whether Semantic IR should be the adapter’s main input or only auxiliary context.
3. How providerLatents, coarseVq, seedCode, and semantic fallback should be prioritized.
4. What training objective would make sense later.
5. What should be recorded in the manifest so we know which path actually fired.

[koriyoshi2041]

Ideally, the adapter should not be treated as a second prompt reader or a small hidden image generator. Its role should be narrower and more mechanical.

The best mental model is: the adapter is a SeedExpander.

That means it should consume structured, sectioned inputs rather than a single blob of text. It may receive Semantic IR as auxiliary context, but its primary job should be to take a more code-like representation—Visual Seed Code, coarse VQ hints, or provider-supplied latent tokens—and turn that into the full decoder-native token grid expected by the frozen decoder.

A useful priority order is:

providerLatents
→ coarseVq
→ seedCode
→ semantic fallback
→ placeholder