[GAME]: Mixture-of-Models Recipes - the Unified MoM Model

[Xunzhuo]

Description

Create a comprehensive Mixture-of-Models (MoM) Recipes system that provides pre-configured, tested, and production-ready multi-model routing strategies for different optimization goals (accuracy, latency, cost).

A MoM Recipe is a complete configuration package that combines:

• Multiple OSS/Commercial models working together
• Signal-based decision logic and multi-model execution algorithms
• Plugin configurations (caching, safety, optimization)
• Verified evaluation results on standard benchmarks (MMLU-Pro, ARC Challenge, etc.)
• Complete documentation with use cases, trade-offs, and deployment guides

Context

Problem

• High barrier to entry: New users struggle to configure optimal multi-model routing
• No best practices: Lack of reference configurations for common scenarios
• Repeated work: Every user reinvents the wheel for accuracy/latency/cost optimization
• Unvalidated configs: Users don't know if their configuration performs well

Solution

Create a curated library of MoM Recipes - pre-configured, benchmarked, and documented routing strategies that users can:

1. Use directly via vllm-sr serve --config config/mom-recipes/accuracy-optimized/config.yaml
2. Customize as starting points for their specific needs
3. Learn from as reference implementations
4. Contribute back to the community

Reference

Recipe Structure

Each recipe should be a complete package under config/mom-recipes/{recipe-name}/:

config/mom-recipes/
├── README.md                          # Overview of all recipes
├── TEMPLATE.md                        # Template for creating new recipes
├── accuracy-optimized/
│   ├── README.md                      # Recipe documentation
│   ├── config.yaml                    # Complete router configuration
│   ├── evaluation/                    # Benchmark results
│   │   ├── mmlu-pro.json             # MMLU-Pro evaluation results
│   │   ├── arc-challenge.json        # ARC Challenge results
│   │   ├── gpqa.json                 # GPQA results (optional)
│   │   └── summary.md                # Performance summary & analysis
│   └── examples/
│       └── curl-examples.sh          # Usage examples
├── latency-optimized/
│   └── [same structure]
└── cost-optimized/
    └── [same structure]

Recipe Metadata (in README.md)

Each recipe must document:

0. Scenario & Optimization Goal

• Target use case (research, production, real-time, etc.)
• Primary optimization goal (accuracy, latency, cost, balanced)
• Workload characteristics (QPS, query complexity, latency tolerance)
• When to use this recipe vs alternatives

1. Models Used

• List of OSS and/or Commercial models
• Model sizes and providers
• Role of each model in the mixture (fast initial, high-accuracy fallback, etc.)
• Model selection algorithm (confidence, ratings, size-aware)

2. Evaluation Results

• Benchmark scores on standard leaderboards:
  • MMLU-Pro (accuracy %)
  • ARC Challenge (accuracy %)
  • GPQA (optional)
  • TruthfulQA (optional)
• Performance metrics:
  • Latency (P50, P95, P99)
  • Cost per 1M tokens
  • Token efficiency vs baseline
• Comparison to baseline (single-model) performance

3. Signal Design

• Keyword signals: patterns and use cases
• Embedding signals: semantic similarity thresholds
• Domain signals: category classification and model scores
• Fact check signals: hallucination detection configuration
• User feedback signals: if applicable
• Preference signals: if applicable
• Language signals: if applicable

4. Decision Logic & Plugins

• Decision rules: AND/OR combinations and priorities
• Algorithm configuration:
  • Confidence-based: threshold, confidence_method, on_error
  • Ratings: max_concurrent, on_error
  • Size-aware: param_size ordering
• Enabled plugins:
  • Semantic cache: backend, threshold, TTL
  • Hallucination detection: thresholds, actions
  • PII detection: policies
  • System prompts: per-decision customization

5. Runtime Configuration

• Complete config.yaml reference
• Required environment variables
• vLLM endpoint requirements
• Resource requirements (CPU, memory, GPU)

Usage Pattern

Users should be able to use recipes via vllm-sr CLI:

# Use a recipe directly
vllm-sr serve --config config/mom-recipes/accuracy-optimized/config.yaml

# Or copy and customize
cp config/mom-recipes/accuracy-optimized/config.yaml my-config.yaml
# Edit my-config.yaml...
vllm-sr serve --config my-config.yaml

Initial Recipe Set

Priority 1: Core Recipes

1. Accuracy-Optimized: Confidence-based escalation (small → large models), optimized for MMLU-Pro/ARC
2. Latency-Optimized: Fast model selection with domain routing, sub-200ms P95 latency
3. Cost-Optimized: Size-aware routing with aggressive semantic caching, 60%+ token reduction

Priority 2: Specialized Recipes
4. Balanced: Trade-off between accuracy, latency, and cost
5. Research-Focused: Maximum reasoning capability with extended thinking
6. Production-Scale: High-throughput with load balancing and failover

Evaluation Integration

Recipes must be validated using existing evaluation framework or even public model arena, to evaluate if it is a good MoM model.

Documentation Updates

Update website documentation:

• Add "MoM Recipes" section to sidebar (website/sidebars.ts)
• Create overview page: website/docs/recipes/overview.md
• Create individual recipe pages: website/docs/recipes/{recipe-name}.md
• Update main intro to mention recipes as quick-start option

Acceptance Criteria

• Directory structure created under config/mom-recipes/
• README.md with overview of all recipes
• TEMPLATE.md for creating new recipes
• At least 3 complete recipes (accuracy, latency, cost optimized)
• Each recipe includes:
  • Complete config.yaml
  • Comprehensive README.md with all required sections
  • Evaluation results from MMLU-Pro and ARC Challenge
  • Usage examples
• Website documentation updated with recipes section
• Recipes validated with vllm-sr serve --config command
• CI/CD integration for recipe validation (optional, future work)

Future Enhancements

• Recipe versioning and changelog
• Community recipe contributions
• Automated recipe generation from evaluation results
• Recipe recommendation system based on user requirements
• Recipe performance comparison dashboard

[Xunzhuo]

This issue is important and can be assigned to multiple forks concurrently, if anyone is interested, please comment /assign to assign your self. Looking forward to the magic in the community.

[szedan-rh]

/assign

[samzong]

/assign I can handle part of this.

[madhav-murali]

/assign

[haowu1234]

/assign

[R3hankhan123]

/assign

[yike21]

/assign

[rootfs]

Let's break it down into sub tasks.

@samzong @yike21 @Xunzhuo @R3hankhan123 @madhav-murali @haowu1234 @szedan-rh Please make a one line description of what you can help and which item you can do in a timely fashion.

[Xunzhuo]

@rootfs maybe we can turn this into a discussion from an issue? My initial idea is that every one can share how they build the mixture of models and see if there are any magics😉

[rootfs]

that's a good idea! Let's use this discussion to brainstorm sub tasks and ownership

@samzong @yike21 @Xunzhuo @R3hankhan123 @madhav-murali @haowu1234 @szedan-rh

[rootfs]

Also want to relate the recipes for verification process through public beta. Would be great recipe proposals can be encapsulated in a the operational process that #1030 can provision for verification.

[Xunzhuo]

great!

[srampal]

/assign