[Feat][Router] Add Multimodal Intent Recognition Support to Semantic Router Categories

[NPranitha]

FIX #153

This PR extends the vLLM Semantic Router with CLIP embedding-based multimodal classification and image generation intent detection.

Key Features

Multimodal Classification (Image → Text Routing):

• Native CLIP vision transformer integration for image embedding extraction
• Embedding fusion pipeline combining text (BERT/Qwen3) and image (CLIP) embeddings via weighted combination
• Zero-shot classification using cosine similarity matching against category descriptions
• Support for OpenAI-compatible API formats with Base64-encoded images
• Automatic fallback to llava:7b when vision transformer is unavailable

Image Generation Routing (Text → Image):

• Fine-tuned BERT binary classifier for image generation intent detection
• Automatic routing to Stable Diffusion inference service
• OpenAI-compatible response format with Base64-encoded image outputs

Technical Implementation

Rust/Candle Layer:

• CLIP vision transformer (ViT-B/32) integration for native image embeddings
• FFI bindings for Go interop with auto-initialization support
• Image preprocessing pipeline (decode, resize to 224x224, CLIP normalization)

Go Service Layer:

• Vision Embedding-based multimodal classification pipeline
• Weighted fusion with L2 normalization
• Cosine similarity-based category matching
• Fine-tuned BERT classifier integration for image generation detection

Configuration:

• New multimodal_enabled flag for categories
• Category descriptions for embedding-based classification
• Model capability specifications (capabilities: ["text", "image"])

Testing

• Comprehensive unit tests for embedding fusion and classification logic
• Backward compatibility validation with existing text-only workflows

Documentation

• Updated API documentation for multimodal endpoints
• Configuration examples for multimodal categories
• Implementation guide for vision transformer integration

Breaking Changes

None. This PR maintains full backward compatibility. Multimodal routing is opt-in via configuration.

---

• [x ] Make sure the code changes pass the pre-commit checks.
• Sign-off your commit by using -s when doing git commit
• Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

Detailed Checklist (Click to Expand)

Thank you for your contribution to semantic-router! Before submitting the pull request, please ensure the PR meets the following criteria. This helps us maintain the code quality and improve the efficiency of the review process.

PR Title and Classification

Please try to classify PRs for easy understanding of the type of changes. The PR title is prefixed appropriately to indicate the type of change. Please use one of the following:

• [Bugfix] for bug fixes.
• [CI/Build] for build or continuous integration improvements.
• [Doc] for documentation fixes and improvements.
• [Feat] for new features in the cluster (e.g., autoscaling, disaggregated prefill, etc.).
• [Router] for changes to the vllm_router (e.g., routing algorithm, router observability, etc.).
• [Misc] for PRs that do not fit the above categories. Please use this sparingly.

Note: If the PR spans more than one category, please include all relevant prefixes.

Code Quality

The PR need to meet the following code quality standards:

• Pass all linter checks. Please use pre-commit to format your code. See README.md for installation.
• The code need to be well-documented to ensure future contributors can easily understand the code.
• Please include sufficient tests to ensure the change is stay correct and robust. This includes both unit tests and integration tests.

DCO and Signed-off-by

When contributing changes to this project, you must agree to the DCO. Commits must include a Signed-off-by: header which certifies agreement with the terms of the DCO.

Using -s with git commit will automatically add this header.

What to Expect for the Reviews

[netlify]

❌ Deploy Preview for vllm-semantic-router failed.

Name	Link
🔨 Latest commit	e6f2f52
🔍 Latest deploy log	https://app.netlify.com/projects/vllm-semantic-router/deploys/693f8c58b8c13d00081b2c13

[rootfs]

@NPranitha thank you for the contribution! Looking forward to this feature!