-
Notifications
You must be signed in to change notification settings - Fork 338
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FEAT: Add Unify Integration for Multi-Provider LLM Support #353
Comments
There are several providers for "all-in-one" platforms, e.g., Poe is another one. It's not obvious to me if there are pros/cons to any of them. Any thoughts? Are you proposing this as something you want to contribute or as a feature request that you would like to see (but you don't want to contribute yourself)? |
Hello @romanlutz , About Poe, I guess Unify is set apart because it provides all LLM providers with one single API key, also adding features like optimisation, benchmarking , routing etc. |
Got it! Thanks for confirming. Need to check with our team and will get back to you on Tuesday (Monday is a holiday in the US). |
Alright @romanlutz . |
@KatoStevenMubiru we discussed it and suggest you go ahead with a pull request. You can use any other target as a starting point. Since we don't have Unify accounts we can't add notebooks at this time (as we would be unable to run them). The PR should have at least basic unit tests. Is it fair to assume that you'll be willing to maintain the target and investigate issues if any should arise? |
This is great @romanlutz . Yes we are willing to maintain the target and investigate incase of any potential issues. Best regards👏🏾 |
Is your feature request related to a problem? Please describe.
PyRIT currently lacks built-in support for easily using and comparing multiple LLM providers. This makes it challenging for users to leverage and evaluate different models efficiently within the PyRIT framework.
Describe the solution you'd like
Integrate Unify's functionality into PyRIT to enable:
Universal API access to multiple LLM providers through a single interface
Built-in benchmarking and evaluation tools for comparing LLM performance across providers
Intelligent routing capabilities to select optimal models based on specific tasks, quality, cost, and speed
The integration should align with PyRIT's existing architecture and follow best practices for extensibility.
Describe alternatives you've considered, if relevant
Implementing provider-specific integrations individually within PyRIT
Using a different multi-provider LLM library instead of Unify
Creating a custom solution for multi-provider support within PyRIT
Additional context:
Unify documentation: [link to Unify docs] Key features to consider implementing:
Single interface for accessing multiple LLM providers
Custom testing and evaluation framework for benchmarking
Routing system based on quality, cost, and speed metrics
Integration with PyRIT's existing testing and evaluation capabilities
This integration would significantly enhance PyRIT's capabilities for AI red teaming across multiple LLM providers and models.
The text was updated successfully, but these errors were encountered: