Skip to content
This repository was archived by the owner on Jun 5, 2025. It is now read-only.
This repository was archived by the owner on Jun 5, 2025. It is now read-only.

Muxing Support for Model Routing #566

Closed
15 of 15 issues completed
Closed
Story 🗺️
15 of 15 issues completed
@lukehinds

Description

@lukehinds

Introduce support for "Muxing" in CodeGate, enabling users to route specific types of logic to different large language models (LLMs) based on file type, individual files and workspace. This would provide cost savings , a common complaint from users is the expense of tokens and how to optimise those costs along with a way of isolation use to specific models (based on privacy, security , protection of IP)

It would also enable context switching (context window refers to the amount of text data a language model can consider at one time when generating responses) , whereby we could switch models during a long protracted session of prompt / response.

For example:

  • Per Workspace: A user can set a LLM choice for the
  • Per File Type: Documentation files (e.g., .md, .rst) could be processed by a free, local model, while source code files (e.g., .py, .js) are handled by an advanced model like Claude Sonnet 3.5.
  • Per File: Users could specify models for particular files within a project based on complexity or other criteria.

Why is this feature important?

  1. Granular Control: Developers can tune model usage based on file type or project needs.
  2. Resource Efficiency: Allocate lightweight or local models to simpler tasks, while reserving advanced models for more complex challenges, providing more cost efficiency.
  3. Workspace Integration: Aligns with the workflow of managing multiple workspaces

NOTE: As always, start small, simple and validate, the following acts as a guideline of where this could lead.

Possible Solution

Expand CodeGate’s functionality to include:

  1. Repository-Level Configuration:

    • Enable users to set model preferences for an entire workspace.
    • Example: Repository A -> local_model, Repository B -> advanced_model.
  2. File-Type-Based Routing:

    • Route tasks based on file extensions or MIME types.
    • Example: .md -> local_model, .py -> claude_sonnet_3.5.
  3. Per-File Routing:

    • Allow specific files in a workspace to be linked to a designated model.
    • Example: config.py -> local_model_v2.

Challenges and Considerations

  • Extensible: It's likely a lot of features will sit on top of muxing, and some synergies will be mapped to pipelines.
  • Latency and Performance: Optimize routing logic to prevent delays in large or multi-repository setups.
  • User Experience: Provide clear documentation and possibly a UI component for managing these configurations.

Sub-issues

Metadata

Metadata

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions