Skip to content

Support sending images to LLMs (vision/multimodal input) #71

Description

@jrswab

Problem

Axe currently only supports plain text content in messages to LLMs. The Message.Content field is a string, so there's no way to send images for vision/multimodal tasks (e.g., asking an LLM to describe an image or extract data from a screenshot).

Scope

  1. Extend Message.Content from a plain string to a type that can hold both text blocks and image blocks (matching Anthropic content[] / OpenAI content[] formats).
  2. Update provider implementations (Anthropic, OpenAI, Ollama, Bedrock, OpenCode) to serialize image content blocks in their respective API formats.
  3. Add an input mechanism — e.g., a --image flag or supporting image file paths in agent config, with binary detection and base64 encoding in the resolve layer.

Notes

  • Not all models support vision — graceful error handling needed when a model doesn't support image input.
  • Consider supported formats (PNG, JPEG, GIF, WebP) and size limits per provider.
  • Sub-agent delegation should be able to pass image context if needed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions