Add a way to disallow specific LLM models

[buger]

Right now you can only whitelist models, but it would be handy to blacklist instead. For example some expensive model.
I wonder if already possible using current patterns. Or maybe we can reuse current patterns, like if it starts with ! or similar

[probelabs]

Thanks for the suggestion!

Hello @buger, thank you for opening this issue and suggesting an enhancement for managing LLM models.

I understand you're proposing a way to blacklist or disallow specific models, as opposed to the current system which only supports whitelisting. This is a great idea for providing administrators with more granular control, especially for managing costs by blocking access to more expensive models.

Current Implementation

Based on my analysis of the codebase, you are correct that we currently only support whitelisting. The relevant logic is located in the proxy service:

• File: proxy/model_validator.go
• Function: IsModelAllowed(modelName string) bool

This function checks if a given model name matches any of the regex patterns defined in the AllowedModels list for an LLM configuration. If the list is empty, all models are permitted.

// proxy/model_validator.go
func (mv *ModelValidator) IsModelAllowed(modelName string) bool {
	if len(mv.allowedModels) == 0 {
		return true // If no models specified, allow all
	}

	for _, pattern := range mv.allowedModels {
		matched, err := regexp.MatchString(pattern, modelName)
		if err == nil && matched {
			return true
		}
	}
	return false
}

Proposed Solution

Your suggestion to reuse the existing AllowedModels field and introduce a prefix like ! for blacklisting is an excellent and intuitive approach. We could adapt the IsModelAllowed function to handle this logic.

The implementation could work as follows:

1. Partition the AllowedModels list into a whitelist and a blacklist based on the ! prefix.
2. First, check if the requested model matches any pattern in the blacklist. If it does, deny the request immediately.
3. If it doesn't match the blacklist, then:
   • If the whitelist is empty, allow the request.
   • If the whitelist is not empty, the model must match at least one pattern to be allowed.

This would provide the flexibility you're looking for. It seems like a reasonably straightforward enhancement that would be contained primarily within the model_validator.go file.

Thanks again for this valuable feedback!

---

Powered by Visor from Probelabs

💡 TIP: You can chat with Visor using /visor ask <your question>