Tests overhaul#61
Merged
sheltoncyril merged 7 commits intotrustyai-explainability:mainfrom Mar 12, 2026
Merged
Conversation
This comment was marked as off-topic.
This comment was marked as off-topic.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This commit introduces a comprehensive end-to-end testing environment for the llama-stack-provider-ragas distribution on OpenShift. It includes a new Containerfile for building the test image, deployment and teardown scripts, configuration manifests, and a test suite using pytest. The setup ensures that the necessary resources are created and validated, facilitating robust testing of the provider's functionality.
This commit introduces several improvements to the testing framework for the llama-stack-provider-ragas. Key changes include: - Updated the pre-commit configuration to run unit tests for "unit or lls_integration". - Added shared evaluation helper classes (`SmokeTester` and `EvalTester`) to `base_eval_tests.py` for better test organization and reusability. - Enhanced inline evaluation tests to utilize the new helper classes, improving clarity and maintainability. - Removed outdated Kubeflow integration tests and streamlined the test structure for better focus on current testing strategies. These changes aim to improve the robustness and clarity of the testing process, ensuring better coverage and easier maintenance.
dmaniloff
force-pushed
the
tests_overhaul
branch
from
91e5a2a to
3bf8ef1
Compare
These files were superseded by tests/cluster-deployment/ in PR trustyai-explainability#59. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Drop the `embedding/` prefix so test defaults match the configmap value `nomic-ai/nomic-embed-text-v1.5`. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a model configuration section to TESTING.md and inline comments in each test module explaining why the defaults differ across suites. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
dmaniloff
requested review from
RobGeada,
m-misiura,
ruivieira,
saichandrapandraju and
sheltoncyril
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improvements to the testing framework for the llama-stack-provider-ragas. Key changes include:
SmokeTesterandEvalTester) tobase_eval_tests.pyfor better test organization and reusability.tests/TESTING.md