Problem
Axe currently only supports plain text content in messages to LLMs. The Message.Content field is a string, so there's no way to send images for vision/multimodal tasks (e.g., asking an LLM to describe an image or extract data from a screenshot).
Scope
- Extend
Message.Content from a plain string to a type that can hold both text blocks and image blocks (matching Anthropic content[] / OpenAI content[] formats).
- Update provider implementations (Anthropic, OpenAI, Ollama, Bedrock, OpenCode) to serialize image content blocks in their respective API formats.
- Add an input mechanism — e.g., a
--image flag or supporting image file paths in agent config, with binary detection and base64 encoding in the resolve layer.
Notes
- Not all models support vision — graceful error handling needed when a model doesn't support image input.
- Consider supported formats (PNG, JPEG, GIF, WebP) and size limits per provider.
- Sub-agent delegation should be able to pass image context if needed.
Problem
Axe currently only supports plain text content in messages to LLMs. The
Message.Contentfield is astring, so there's no way to send images for vision/multimodal tasks (e.g., asking an LLM to describe an image or extract data from a screenshot).Scope
Message.Contentfrom a plainstringto a type that can hold both text blocks and image blocks (matching Anthropiccontent[]/ OpenAIcontent[]formats).--imageflag or supporting image file paths in agent config, with binary detection and base64 encoding in the resolve layer.Notes