docs: add docs for image editing and image variations#1653
docs: add docs for image editing and image variations#1653Radheshg04 wants to merge 1 commit intomainfrom
Conversation
📝 WalkthroughSummary by CodeRabbit
WalkthroughDocumentation expanded to include image editing and variation capabilities across the provider matrix. Added new columns for Image Edit and Image Variation to the capability overview. Replicate provider documentation updated with comprehensive Image Edit section, replacing the Files API section and adjusting subsequent section numbering. Changes
Estimated code review effort🎯 2 (Simple) | ⏱️ ~10 minutes Poem
🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
🧪 Test Suite AvailableThis PR can be tested by a repository admin. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Fix all issues with AI agents
In `@docs/providers/supported-providers/overview.mdx`:
- Around line 18-38: The overview table is missing a row for Replicate; add a
new row for Replicate (`replicate/<model>`) to the providers matrix and populate
each column to match the capabilities documented in replicate.mdx (e.g., enable
✅ for Models, Text, Chat, Responses, Images, Image Edit/Image Variation,
Files/List Models where replicate.mdx shows support and use ❌ for features not
supported or streaming not present); ensure the row uses the same pipe-delimited
format as the other provider rows and mirrors the exact support state from
replicate.mdx for every column (Text (stream), Chat (stream), Responses
(stream), Images (stream), Image Edit (stream), Embeddings, TTS/STT, Batch,
Count tokens, etc.).
In `@docs/providers/supported-providers/replicate.mdx`:
- Around line 470-475: The table under the Image Generation section uses
abbreviated model names; update every model entry to the full Replicate
identifier format (owner/model) for consistency and clarity—e.g., replace
occurrences like `flux-1.1-pro-ultra`, `flux-kontext-pro`, `flux-kontext-max`,
`flux-dev`, `flux-fill-pro`, `flux-dev-lora`, `flux-krea-dev`, etc. with their
fully qualified forms (for example `black-forest-labs/flux-1.1-pro-ultra`,
`black-forest-labs/flux-kontext-pro`, `black-forest-labs/flux-kontext-max`,
`black-forest-labs/flux-dev`, etc.) so the table rows for `image_prompt`,
`input_image`, `image`, and `input_images` use owner/model identifiers
consistently.
- Around line 525-535: The documentation uses event type names
"image_edit.partial_image" and "image_edit.completed" but the code
(core/schemas/images.go) defines events as "n.partial_image" and "n.completed";
update the docs to use the exact event type strings "n.partial_image" for
partial chunks and "n.completed" for completion so the documentation matches the
implementation (ensure references in the Response/Streaming sections and any
examples are changed accordingly).
🧹 Nitpick comments (3)
docs/providers/supported-providers/overview.mdx (1)
18-18: Missing "Image Variation (stream)" column in the provider matrix.The table includes both non-streaming and streaming columns for Image Edit (
Image Edit/Image Edit (stream)) but only a non-streaming column for Image Variation. For consistency — and to future-proof the table if any provider adds streaming support for variations — consider adding anImage Variation (stream)column (all❌for now), or add a note explaining why the streaming column is intentionally omitted.docs/providers/supported-providers/replicate.mdx (2)
486-499: Curl example nestsimagesinsideinputbut placesoutput_formatandnum_inference_stepsat the top level.This is consistent with the Image Generation curl example (line 368), so the gateway API apparently accepts a flat structure for params. However, the parameter mapping table (lines 453-464) prefixes these with
params.*(e.g.,params.output_format), which could confuse readers about where these fields go in a raw HTTP request vs. the Go SDK.Consider adding a brief note (similar to what's done in the Chat Completions section) clarifying that for the gateway/curl interface, parameters are passed at the top level of the JSON body, while the
params.*notation in the mapping table reflects the Go SDK structure.
445-447: Anchor link#field-mapping-by-model-1in the introductory paragraph may be fragile.Line 447 links to
#field-mapping-by-model-1(the auto-generated suffix for the second "Field Mapping by Model" heading). If any heading is added or reordered above this section, the numeric suffix will change and break the link. Consider giving the Image Edit field mapping heading a unique name (e.g.,### Image Edit Field Mapping by Model) or using an explicit anchor/id to make the link resilient.
| | Provider | Models | Text | Text (stream) | Chat | Chat (stream) | Responses | Responses (stream) | Images | Images (stream) | Image Edit | Image Edit (stream) | Image Variation | Embeddings | TTS | TTS (stream) | STT | STT (stream) | Files | Batch | Count tokens | | ||
| |----------|--------|------|----------------|------|---------------|-----------|--------------------|--------|-----------------|------------|---------------------|-----------------|------------|-----|-------------|-----|--------------|-------|-------|--------------| | ||
| | Anthropic (`anthropic/<model>`) | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | | ||
| | Azure (`azure/<model>`) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | | ||
| | Bedrock (`bedrock/<model>`) | ✅ | ✅ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ❌ | | ||
| | Cerebras (`cerebras/<model>`) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Cohere (`cohere/<model>`) | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | | ||
| | Elevenlabs (`elevenlabs/<model>`) | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | | ||
| | Gemini (`gemini/<model>`) | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| | Groq (`groq/<model>`) | ✅ | 🟡 | 🟡 | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Hugging Face (`huggingface/<model>`) | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | | ||
| | Mistral (`mistral/<model>`) | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ✅ | ✅ | ❌ | ❌ | ❌ | | ||
| | Nebius (`nebius/<model>`) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Ollama (`ollama/<model>`) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | OpenAI (`openai/<model>`) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | | ||
| | OpenRouter (`openrouter/<model>`) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Parasail (`parasail/<model>`) | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Perplexity (`perplexity/<model>`) | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | SGL (`sgl/<model>`) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | | ||
| | Vertex AI (`vertex/<model>`) | ✅ | ❌ | ❌ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ | ❌ | ❌ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ✅ | | ||
| | xAI (`xai/<model>`) | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ | |
There was a problem hiding this comment.
Replicate is absent from the provider support matrix.
The replicate.mdx file documents multiple supported operations (Chat, Responses, Text, Image Generation, Image Edit, Files, List Models), yet Replicate does not appear as a row in the overview table. This may be a pre-existing gap, but since this PR adds Image Edit/Variation columns and Replicate is the first provider documented for Image Edit, it would be a good time to add a Replicate row.
#!/bin/bash
# Check if Replicate is mentioned anywhere in the overview file
rg -in 'replicate' docs/providers/supported-providers/overview.mdx🤖 Prompt for AI Agents
In `@docs/providers/supported-providers/overview.mdx` around lines 18 - 38, The
overview table is missing a row for Replicate; add a new row for Replicate
(`replicate/<model>`) to the providers matrix and populate each column to match
the capabilities documented in replicate.mdx (e.g., enable ✅ for Models, Text,
Chat, Responses, Images, Image Edit/Image Variation, Files/List Models where
replicate.mdx shows support and use ❌ for features not supported or streaming
not present); ensure the row uses the same pipe-delimited format as the other
provider rows and mirrors the exact support state from replicate.mdx for every
column (Text (stream), Chat (stream), Responses (stream), Images (stream), Image
Edit (stream), Embeddings, TTS/STT, Batch, Count tokens, etc.).
| | Field | Models | | ||
| |-------|--------| | ||
| | `image_prompt` | `black-forest-labs/flux-1.1-pro`, `flux-1.1-pro-ultra`, `flux-pro`, `flux-1.1-pro-ultra-finetuned` | | ||
| | `input_image` | `black-forest-labs/flux-kontext-pro`, `flux-kontext-max`, `flux-kontext-dev` | | ||
| | `image` | `black-forest-labs/flux-dev`, `flux-fill-pro`, `flux-dev-lora`, `flux-krea-dev` | | ||
| | `input_images` | All other models (default) | |
There was a problem hiding this comment.
Inconsistent model name formatting in the field mapping table.
The Image Generation section (lines 351-355) uses fully qualified model names (e.g., black-forest-labs/flux-1.1-pro-ultra, black-forest-labs/flux-kontext-pro), but this table abbreviates them (e.g., flux-1.1-pro-ultra, flux-kontext-max). Since these are Replicate model identifiers, use the full owner/model format for consistency and to avoid ambiguity.
Proposed fix
| Field | Models |
|-------|--------|
-| `image_prompt` | `black-forest-labs/flux-1.1-pro`, `flux-1.1-pro-ultra`, `flux-pro`, `flux-1.1-pro-ultra-finetuned` |
-| `input_image` | `black-forest-labs/flux-kontext-pro`, `flux-kontext-max`, `flux-kontext-dev` |
-| `image` | `black-forest-labs/flux-dev`, `flux-fill-pro`, `flux-dev-lora`, `flux-krea-dev` |
+| `image_prompt` | `black-forest-labs/flux-1.1-pro`<br/>`black-forest-labs/flux-1.1-pro-ultra`<br/>`black-forest-labs/flux-pro`<br/>`black-forest-labs/flux-1.1-pro-ultra-finetuned` |
+| `input_image` | `black-forest-labs/flux-kontext-pro`<br/>`black-forest-labs/flux-kontext-max`<br/>`black-forest-labs/flux-kontext-dev` |
+| `image` | `black-forest-labs/flux-dev`<br/>`black-forest-labs/flux-fill-pro`<br/>`black-forest-labs/flux-dev-lora`<br/>`black-forest-labs/flux-krea-dev` |
| `input_images` | All other models (default) |📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| | Field | Models | | |
| |-------|--------| | |
| | `image_prompt` | `black-forest-labs/flux-1.1-pro`, `flux-1.1-pro-ultra`, `flux-pro`, `flux-1.1-pro-ultra-finetuned` | | |
| | `input_image` | `black-forest-labs/flux-kontext-pro`, `flux-kontext-max`, `flux-kontext-dev` | | |
| | `image` | `black-forest-labs/flux-dev`, `flux-fill-pro`, `flux-dev-lora`, `flux-krea-dev` | | |
| | `input_images` | All other models (default) | | |
| | Field | Models | | |
| |-------|--------| | |
| | `image_prompt` | `black-forest-labs/flux-1.1-pro`<br/>`black-forest-labs/flux-1.1-pro-ultra`<br/>`black-forest-labs/flux-pro`<br/>`black-forest-labs/flux-1.1-pro-ultra-finetuned` | | |
| | `input_image` | `black-forest-labs/flux-kontext-pro`<br/>`black-forest-labs/flux-kontext-max`<br/>`black-forest-labs/flux-kontext-dev` | | |
| | `image` | `black-forest-labs/flux-dev`<br/>`black-forest-labs/flux-fill-pro`<br/>`black-forest-labs/flux-dev-lora`<br/>`black-forest-labs/flux-krea-dev` | | |
| | `input_images` | All other models (default) | |
🤖 Prompt for AI Agents
In `@docs/providers/supported-providers/replicate.mdx` around lines 470 - 475, The
table under the Image Generation section uses abbreviated model names; update
every model entry to the full Replicate identifier format (owner/model) for
consistency and clarity—e.g., replace occurrences like `flux-1.1-pro-ultra`,
`flux-kontext-pro`, `flux-kontext-max`, `flux-dev`, `flux-fill-pro`,
`flux-dev-lora`, `flux-krea-dev`, etc. with their fully qualified forms (for
example `black-forest-labs/flux-1.1-pro-ultra`,
`black-forest-labs/flux-kontext-pro`, `black-forest-labs/flux-kontext-max`,
`black-forest-labs/flux-dev`, etc.) so the table rows for `image_prompt`,
`input_image`, `image`, and `input_images` use owner/model identifiers
consistently.
| ### Response | ||
|
|
||
| Same as Image Generation: single URL → `data[0].url`, array of URLs → `data[i].url`, or data URIs. Response shape is `BifrostImageGenerationResponse` with `data[].url` or `data[].b64_json`. | ||
|
|
||
| ### Streaming | ||
|
|
||
| Image edit streaming is supported. Events use the same prediction log stream as image generation: | ||
|
|
||
| - **Partial chunks**: `type: "image_edit.partial_image"` with `b64_json` (or data URI) until completion. | ||
| - **Completed**: `type: "image_edit.completed"` with final image and usage. | ||
|
|
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# Search for image_edit event type strings in the codebase
rg -rn 'image_edit\.partial_image|image_edit\.completed' --type goRepository: maximhq/bifrost
Length of output: 216
🏁 Script executed:
# Search for image_edit and prediction log references
rg -rn 'image_edit|prediction.log' --type go | head -50Repository: maximhq/bifrost
Length of output: 526
🏁 Script executed:
# Search for event type definitions and streaming related code
rg -rn 'output.*done.*error|event.*type' --type go | head -50Repository: maximhq/bifrost
Length of output: 3813
Documentation event types image_edit.partial_image and image_edit.completed do not match the codebase definitions.
The actual event types defined in core/schemas/images.go are n.partial_image and n.completed, not image_edit.partial_image and image_edit.completed. Update the documentation to use the correct event type names to match the implementation.
🤖 Prompt for AI Agents
In `@docs/providers/supported-providers/replicate.mdx` around lines 525 - 535, The
documentation uses event type names "image_edit.partial_image" and
"image_edit.completed" but the code (core/schemas/images.go) defines events as
"n.partial_image" and "n.completed"; update the docs to use the exact event type
strings "n.partial_image" for partial chunks and "n.completed" for completion so
the documentation matches the implementation (ensure references in the
Response/Streaming sections and any examples are changed accordingly).
1 participant
Summary
Added support for Image Edit and Image Variation operations in the provider compatibility documentation, with detailed implementation for Replicate's Image Edit functionality.
Changes
Type of change
Affected areas
How to test
Test the Image Edit functionality with Replicate:
Breaking changes
Related issues
N/A
Security considerations
No new security implications.
Checklist
docs/contributing/README.mdand followed the guidelines