Epic: AI actions — provider-agnostic capabilities on the first-class pipeline

[whaleen]

Epic: AI actions — provider-agnostic capabilities on the first-class action pipeline

Rewritten from the old "Flux image-actions UI scaffold" plan. That plan was 90% UI and assumed ModelsLab/Flux specifics. The real work is architectural: make AI editing a first-class, provider-agnostic part of the existing action system.

Vision

tiles already edits images and videos and composes them into tiles. The goal is to add AI-powered editing workflows as ordinary actions — provider-agnostically. ModelsLab/Flux is just the first implementation; the same actions must adapt to Replicate, Fal, etc.

Principles (decided)

1. Actions are generic capabilities (verbs); providers/models are configuration. image-edit, image-upscale, image-animate — never flux-*.
2. First-class lifecycle. AI runs as tiles-cli subcommands emitting TILES_PROGRESS → progress, batch, output-modes, Outputs page for free. (Flux's mistake: a Tauri reqwest command on a parallel path.)
3. One active provider at a time, chosen in Settings. Actions gate on media type AND active-provider capability support — reusing existing media-type gating.
4. Design for many, implement one. Build the capability/provider abstraction + per-provider credential map now; only declare/implement ModelsLab.
5. Descriptor is the single source of truth. Provider/capability/model descriptors live in Rust, exposed via a Tauri command (like list_actions). Settings, gating, forms, and payload preview all render from it.
6. Hybrid forms. Generic core hand-coded; provider-specific advanced + payload preview descriptor-driven.
7. Full media matrix eventually: image→image, image→video (on-brand), text→image, video→video, text→video — incrementally.

Sequencing — scaffold first, no API keys required

No accounts yet, so the only deferred piece is the live HTTP call. Everything else is built against a dry-run backend that validates inputs and emits the exact payload it would send. The app becomes fully clickable end-to-end before any key exists; the day a key arrives, only one inner function changes.

#5 (A) contract + dry-run backend   →  #6 (B) registry + gating + kill parallel path
        ↘ #8 (D) hybrid forms + payload preview (alongside B)
                                    →  #7 (C) matrix capabilities, incrementally
                                    →  [deferred] live API call once a key exists

Taxonomy

Capability (generic, user-facing)  →  Provider (active, in Settings)  →  Model
  image-edit / image-upscale            ModelsLab / Replicate / Fal       Flux / SDXL / …
  image-animate (image→video)           Replicate / Fal / …               Kling / SVD / …

Sub-issues

• AI actions A — the contract: provider/capability descriptors + credential map + dry-run backend #5 — A: provider/capability descriptors (Rust→Tauri command) + per-provider credential map + active-provider pref + dry-run CLI backend. Live call deferred.
• AI actions B — wire generic capabilities into the action registry + runner (two-factor gating) #6 — B: wire generic-capability actions into registry + runner, add output media type, two-factor gating, delete the bespoke Flux path. (works in dry-run)
• AI actions C — capability matrix (image→image, image→video, …) incrementally #7 — C: grow the capability matrix (image→image → image→video → video/text→*).
• AI actions D — capability forms: generic core + provider-specific advanced + payload preview #8 — D: hybrid forms — hand-coded core + descriptor-driven advanced + payload preview.

Status of existing code

The current flux-img2img (image_actions.rs + flux-img2img-form.tsx + single modelslab_api_key) is an unverified spike on the wrong path. Superseded: rewritten onto the CLI/dry-run path in A/B; single key → credential map. See #3 (closed) for history.

Out of scope

Frontend test coverage is tracked separately in #2 (Vitest), unaffected by this epic.

[whaleen]

Product-direction annotation: this epic aligns with the new agent-assisted workflow direction, but should remain framed as human-in-the-loop rather than autonomous video creation. AI/provider actions are one capability inside tiles' broader structured editing substrate. As this epic grows, prefer reviewable drafts, preview outputs, logs, and summaries; do not silently mutate active edits or source media. Agent workflows should be able to use the same first-class action pipeline, but the visible product promise is that the human stays in control of accepting or refining results.

[whaleen]

Triage update after the editor/agent-drivable project direction moved forward:\n\n- Scaffold phases A-D are complete in dry-run form (#5/#6/#7/#8). #5 is now closed as complete; live provider calls should become future provider-specific issues.\n- Keep this epic open as the broader AI actions umbrella, but treat it as lower priority than stabilizing the explicit timeline editor.\n- Any future AI/agent work should produce reviewable artifacts: output files, logs, payload previews, or draft compositions. It should not silently mutate active edits or source media.\n- Agent-assisted project workflows are broader than AI media generation: agents should eventually inspect structured project/composition data, propose draft timeline edits, run previews, and summarize changes. See docs/agent-workflows.md.\n- Recommended next AI-action step, when desired: pick one real provider/capability and create a narrow issue for live HTTP execution + verified response/error handling.