Skip to content

Shimmy Vision API

Jump to bottom
Michael A. Kuykendall edited this page Dec 16, 2025 · 1 revision

Shimmy Vision API Reference

Complete API documentation for Shimmy Vision endpoints.

Authentication

All Vision API requests require a valid license key in the request body:

{
  "license": "YOUR-LICENSE-KEY",
  ...
}

HTTP Status Codes

Code Meaning
200 Success
400 Bad request (missing url/image_base64)
402 Payment required (no license provided)
403 Forbidden (invalid/expired/suspended license)
422 Unprocessable entity (validation error)
502 Bad gateway (model error)
504 Gateway timeout (processing timeout)

Single Vision Request

Endpoint: POST /api/vision

Content-Type: application/json

Request Body

{
  "license": "YOUR-LICENSE-KEY",
  "url": "https://example.com",
  "image_base64": null,
  "mode": "brief",
  "screenshot": true,
  "viewport_width": 1920,
  "viewport_height": 1080,
  "model": "minicpm-v"
}

Parameters

Field Type Required Default Description
license string Yes - Your Shimmy Vision license key
url string One of url/image - URL to analyze (with optional screenshot)
image_base64 string One of url/image - Base64-encoded image data
mode string No brief Analysis mode: ocr, layout, brief, web, full
screenshot boolean No false Capture screenshot of URL before analysis
viewport_width integer No 1920 Browser viewport width for screenshots
viewport_height integer No 1080 Browser viewport height for screenshots
model string No auto Specific model to use

Response

{
  "image_path": null,
  "url": "https://example.com",
  "mode": "brief",
  "text_blocks": [
    {
      "text": "Example Domain",
      "confidence": 1.0
    }
  ],
  "layout": {
    "theme": "default",
    "regions": [],
    "key_ui_elements": []
  },
  "visual": {
    "background": "#fff",
    "accent_colors": ["#007bff"],
    "contrast": {
      "ratio": null,
      "compliant": null,
      "issues": []
    },
    "description": "A simple layout with white background"
  },
  "interaction": {
    "description": "Click elements for navigation"
  },
  "dom_map": [
    {
      "tag": "h1",
      "id": null,
      "class": null,
      "text": "Example Domain",
      "position": {
        "x": 0.2,
        "y": 0.15,
        "width": 0.6,
        "height": 0.05
      },
      "attributes": {},
      "colors": {}
    }
  ],
  "meta": {
    "model": "minicpm-v",
    "backend": "llama.cpp",
    "duration_ms": 5432,
    "parse_warnings": null
  },
  "raw_model_output": null
}

Batch Vision Request

Endpoint: POST /api/vision/batch

Content-Type: application/json

Response Type: text/event-stream (Server-Sent Events)

Request Body

{
  "license": "YOUR-LICENSE-KEY",
  "items": [
    {
      "id": "unique-item-id-1",
      "url": "https://example.com",
      "mode": "brief",
      "screenshot": true
    },
    {
      "id": "unique-item-id-2",
      "image_base64": "iVBORw0KGgoAAAANS...",
      "mode": "ocr"
    }
  ],
  "concurrency": 2,
  "model": "minicpm-v"
}

Parameters

Field Type Required Default Description
license string Yes - Your Shimmy Vision license key
items array Yes - Array of vision request items
items[].id string Yes - Unique identifier for this item
items[].url string One of - URL to analyze
items[].image_base64 string One of - Base64-encoded image
items[].mode string No brief Analysis mode
items[].screenshot boolean No false Capture screenshot
items[].viewport_width integer No 1920 Viewport width
items[].viewport_height integer No 1080 Viewport height
concurrency integer No 1 Number of parallel workers
model string No auto Model override for all items

SSE Events

batch_start

{"event": "batch_start", "data": {"total": 5, "concurrency": 2}}

item_start

{"event": "item_start", "data": {"id": "item-1", "index": 0}}

item_complete

{"event": "item_complete", "data": {"id": "item-1", "index": 0, "response": {...}}}

item_error

{"event": "item_error", "data": {"id": "item-1", "index": 0, "error": "Processing failed"}}

batch_complete

{"event": "batch_complete", "data": {"total": 5, "succeeded": 4, "failed": 1, "duration_ms": 25000}}

Example: Consuming SSE in JavaScript

const response = await fetch('http://localhost:11435/api/vision/batch', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    license: 'YOUR-LICENSE-KEY',
    items: [
      { id: '1', url: 'https://example.com', mode: 'brief', screenshot: true },
      { id: '2', url: 'https://google.com', mode: 'brief', screenshot: true }
    ]
  })
});

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  
  const text = decoder.decode(value);
  const lines = text.split('\n');
  
  for (const line of lines) {
    if (line.startsWith('data: ')) {
      const event = JSON.parse(line.slice(6));
      console.log(event.event, event.data);
    }
  }
}

Analysis Modes

ocr

Pure text extraction. Returns only text_blocks with extracted text.

Best for: Documents, receipts, text-heavy images

layout

Text extraction with structural information.

Best for: Forms, tables, structured documents

brief

Quick summary with key elements identified.

Best for: Web pages, general screenshots

web

Web-optimized analysis with DOM mapping.

Best for: Web scraping, automation testing

full

Complete analysis with all available information.

Best for: Detailed inspection, accessibility audits

Environment Variables

Variable Default Description
SHIMMY_VISION_DEV_MODE 0 Skip license validation (development only)
SHIMMY_VISION_MAX_LONG_EDGE 2560 Maximum image dimension
SHIMMY_VISION_MAX_PIXELS 5000000 Maximum total pixels
SHIMMY_VISION_TRACE 0 Enable detailed tracing

Rate Limits

Rate limits are enforced per license tier:

Tier Requests/Month Concurrent
Developer 2,500 2
Professional 10,000 5
Startup 50,000 10
Enterprise Unlimited Unlimited
Lifetime Unlimited Unlimited

Error Responses

All errors follow this format:

{
  "error": {
    "code": "ERROR_CODE",
    "message": "Human-readable description"
  }
}

Error Codes

Code HTTP Description
MISSING_LICENSE 402 No license key provided
INVALID_LICENSE 403 License invalid, expired, or suspended
MISSING_INPUT 400 Neither url nor image_base64 provided
INVALID_MODE 400 Unknown analysis mode
PROCESSING_ERROR 502 Model processing failed
TIMEOUT 504 Processing exceeded time limit

Clone this wiki locally