Muxing Support for Model Routing

[lukehinds]

Introduce support for "Muxing" in CodeGate, enabling users to route specific types of logic to different large language models (LLMs) based on file type, individual files, or the context of a Git codebase. This would provide cost savings , a common complaint from users is the expense of tokens and how to optimise those costs.

For example:

• Per File Type: Documentation files (e.g., .md, .rst) could be processed by a free, local model, while source code files (e.g., .py, .js) are handled by an advanced model like Claude Sonnet 3.5.
• Per File: Users could specify models for particular files within a project based on complexity or other criteria.
• Per Repository: For one Git codebase, users might choose a local model for privacy reasons, while another repository could leverage a different model for advanced coding tasks.

Why is this feature important?

1. Granular Control: Developers can tune model usage based on file type or project needs.
2. Resource Efficiency: Allocate lightweight or local models to simpler tasks, while reserving advanced models for more complex challenges, providing more cost efficiency.
3. Git Integration: Aligns with the workflow of managing multiple repositories, providing seamless configuration across projects. See

NOTE: As always, start small, simple and validate, the following acts as a guideline of where this could lead.

Possible Solution

Expand CodeGate’s functionality to include:

1. File-Type-Based Routing:

   • Route tasks based on file extensions or MIME types.
   • Example: .md -> local_model, .py -> claude_sonnet_3.5.

2. Per-File Routing:

   • Allow specific files in a project to be linked to a designated model.
   • Example: config.py -> local_model_v2.

3. Repository-Level Configuration:

   • Enable users to set model preferences for an entire repository.
   • Example: Repository A -> local_model, Repository B -> advanced_model.

Challenges and Considerations

• Git Context Awareness: Ensure the muxing system can recognize the active repository and apply the appropriate routing.
• Conflict Resolution: Define how conflicts between repository-level and file-level rules are handled.
• Latency and Performance: Optimize routing logic to prevent delays in large or multi-repository setups.
• User Experience: Provide clear documentation and possibly a UI component for managing these configurations.

[lukehinds]

Moving this to ready, as I think we can at least start prototyping and thinking about this, albeit we should land #454 first to really make it useful.

[jtenniswood]

Filter by file: can do support wildcards (eg doc*.txt), as well as specific file names?
Would we also want to know the directory structure too?

[jtenniswood]

Would we want to specify a version of a model, or something broader (eg claude_sonnet vs claude_sonnet_3.5).
What would happen if 3.5 wasn't available?

[jtenniswood]

Would users select their repo, or just type it in (eg can we see a list or is it just manual?)

[jtenniswood]

Would this be on a global level or per workspace?

[jhrozek]

Filter by file: can do support wildcards (eg doc*.txt), as well as specific file names?

I think it would be nice, I can envision being OK to send tests/database.sql but not prod/database.sql (or anything under prod)

Would we also want to know the directory structure too?

We would want to but I don't think it was easy. Tagging @aponcedeleonch and @JAORMX who were looking into this. I suspect it's not fully possible until codegate is hooked into the IDE, but I don't know 100%

[jhrozek]

(Similar to the other reply, this is my 2 öre, not a canonical reply)

Would we want to specify a version of a model, or something broader (eg claude_sonnet vs claude_sonnet_3.5).

This is a question about model aliases right_

What would happen if 3.5 wasn't available?

I think we could start with just being specific - in other words, just require the full model name to by typed. Many providers have aliases anyway, e.g. anthropic supports claude-3-5-sonnet-latest. OpenAI has the same. The "major versions" are usually different enough I would let the user type the version.

But I can also see that codegate having some logic to pick the best model might be good, e.g. "for simple requests, use anthropic haiku, for more complex changes use anthropic sonnet, for very complex tasks use anthropic opus".

[JAORMX]

We would want to but I don't think it was easy. Tagging @aponcedeleonch and @JAORMX who were looking into this. I suspect it's not fully possible until codegate is hooked into the IDE, but I don't know 100%

That's right, codegate can't know unless it's actually hooked into the IDE, which it isn't