Skip to content

Conversation

martin-gaievski
Copy link
Member

@martin-gaievski martin-gaievski commented Jul 4, 2025

Added capabilities for setting up model that is natively compatible with OpenAI API and hosted locally. That applies to:

  • github CI, added new action that sets up model and executes integration tests that retrieves judgment ratings from LLM
  • local development and testing. model is started in local server, and ml-commons model connector can be setup to use it as remote model.

Part of this change are slight improvements in LLM judgment logic, I have added multiple formats of possible response to support basic models.

Resolves issue

#181

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@martin-gaievski martin-gaievski force-pushed the poc/use_hosted_llm_for_tests branch 4 times, most recently from e1436a4 to 8ca7521 Compare July 8, 2025 00:09
Signed-off-by: Martin Gaievski <[email protected]>
@martin-gaievski martin-gaievski force-pushed the poc/use_hosted_llm_for_tests branch 2 times, most recently from 64df79e to 55c9375 Compare July 8, 2025 05:29
@martin-gaievski martin-gaievski force-pushed the poc/use_hosted_llm_for_tests branch from 55c9375 to 07d5aa3 Compare July 8, 2025 05:30
Signed-off-by: Martin Gaievski <[email protected]>
@martin-gaievski martin-gaievski force-pushed the poc/use_hosted_llm_for_tests branch from ac5ff99 to b2e7dc0 Compare July 8, 2025 05:43
Signed-off-by: Martin Gaievski <[email protected]>
@martin-gaievski martin-gaievski force-pushed the poc/use_hosted_llm_for_tests branch 3 times, most recently from fd8bc44 to 71f2fbe Compare July 9, 2025 00:50
@martin-gaievski martin-gaievski force-pushed the poc/use_hosted_llm_for_tests branch from 71f2fbe to cfde6b3 Compare July 9, 2025 01:04
Signed-off-by: Martin Gaievski <[email protected]>
@martin-gaievski martin-gaievski force-pushed the poc/use_hosted_llm_for_tests branch 2 times, most recently from 510e09f to 57ab4f6 Compare July 9, 2025 19:25
@martin-gaievski martin-gaievski force-pushed the poc/use_hosted_llm_for_tests branch 2 times, most recently from 5c37f37 to 33cd659 Compare July 9, 2025 22:56
@fen-qin
Copy link
Collaborator

fen-qin commented Jul 9, 2025

Really cool to see Ollama can be used for CI tests. Glad to know Ollama has compatibility with OpenAI Chat - OpenAI compatibility and can bridge with OpenAI schema. Great Idea, Martin

@martin-gaievski martin-gaievski force-pushed the poc/use_hosted_llm_for_tests branch from 33cd659 to dc277e1 Compare July 10, 2025 00:17
@martin-gaievski
Copy link
Member Author

Really cool to see Ollama can be used for CI tests. Glad to know Ollama has compatibility with OpenAI Chat - OpenAI compatibility and can bridge with OpenAI schema. Great Idea, Martin

thanks for feedback and for pointers Fen. I pushed commit that implemented direct connection between ml-commons and ollama for CI and local testing.
I especially like the local testing feature, so we don't need to rely on behemoth models hosted remotely and can use fully local setup for simple tasks

@martin-gaievski martin-gaievski changed the title Testing CI improvements Add local Ollama setup and testing scripts for native OpenAI API compatibility Jul 11, 2025
@martin-gaievski martin-gaievski force-pushed the poc/use_hosted_llm_for_tests branch 2 times, most recently from 4dbbf96 to cffb40e Compare July 12, 2025 00:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants