feat: added tests for dir fairness metrics#390
feat: added tests for dir fairness metrics#390adolfo-ab merged 3 commits intoopendatahub-io:mainfrom
Conversation
…fairness.py modified: tests/model_explainability/trustyai_service/trustyai_service_utils.py
📝 WalkthroughSummary by CodeRabbit
WalkthroughThe fairness test suite was refactored to support parameterized testing over multiple fairness metrics, specifically SPD and DIR. A new metric identifier "DIR" was added to the fairness metrics class. Test methods were updated to use parameterization, enabling iteration over both metrics in various test scenarios. Changes
Suggested labels
✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
|
The following are automatically added/executed:
Available user actions:
Supported labels{'/verified', '/wip', '/hold', '/lgtm', '/cherry-pick', '/build-push-pr-image'} |
|
/verified |
for more information, see https://pre-commit.ci
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (9)
tests/model_explainability/trustyai_service/fairness/test_fairness.py (9)
66-74: Update class documentation to reflect multiple metrics.The docstring still specifically mentions "(spd)" but the class now tests both SPD and DIR metrics due to parameterization.
class TestFairnessMetricsWithPVCStorage: """ - Verifies all the basic operations with a fairness metric (spd) available in TrustyAI, using PVC storage. + Verifies all the basic operations with fairness metrics (SPD, DIR) available in TrustyAI, using PVC storage. 1. Send data to the model and verify that TrustyAI registers the observations. 2. Apply name mappings - 3. Send metric request (spd) and verify the response. + 3. Send metric request and verify the response. 4. Send metric scheduling request and verify the response. 5. Send metric deletion request and verify that the scheduled metric has been deleted. """
107-117: Fix formatting issues in parameterized test method.Static analysis detected formatting issues with the method signature.
@pytest.mark.parametrize("metric_name", FAIRNESS_METRICS) def test_fairness_metric_with_pvc_storage( - self, admin_client, current_client_token, trustyai_service_with_pvc_storage, onnx_loan_model, metric_name + self, + admin_client, + current_client_token, + trustyai_service_with_pvc_storage, + onnx_loan_model, + metric_name, ):
119-129: Apply consistent formatting to parameterized test method.For consistency with the previous suggestion, format the method signature properly.
@pytest.mark.parametrize("metric_name", FAIRNESS_METRICS) def test_fairness_metric_schedule_with_pvc_storage( - self, admin_client, current_client_token, trustyai_service_with_pvc_storage, onnx_loan_model, metric_name + self, + admin_client, + current_client_token, + trustyai_service_with_pvc_storage, + onnx_loan_model, + metric_name, ):
131-140: Apply consistent formatting to parameterized test method.For consistency with the previous suggestions, format the method signature properly.
@pytest.mark.parametrize("metric_name", FAIRNESS_METRICS) def test_fairness_metric_prometheus( - self, admin_client, model_namespace, trustyai_service_with_pvc_storage, onnx_loan_model, prometheus, metric_name + self, + admin_client, + model_namespace, + trustyai_service_with_pvc_storage, + onnx_loan_model, + prometheus, + metric_name, ):
142-151: Apply consistent formatting to parameterized test method.For consistency with the previous suggestions, format the method signature properly.
@pytest.mark.parametrize("metric_name", FAIRNESS_METRICS) def test_fairness_metric_delete_with_pvc_storage( - self, admin_client, current_client_token, trustyai_service_with_pvc_storage, onnx_loan_model, metric_name + self, + admin_client, + current_client_token, + trustyai_service_with_pvc_storage, + onnx_loan_model, + metric_name, ):
166-175: Update class documentation to reflect multiple metrics.Similar to the PVC storage class, this docstring should be updated to reflect that multiple metrics are now tested.
class TestFairnessMetricsWithDBStorage: """ - Verifies all the basic operations with a fairness metric (spd) available in TrustyAI, using MariaDB storage. + Verifies all the basic operations with fairness metrics (SPD, DIR) available in TrustyAI, using MariaDB storage. 1. Send data to the model and verify that TrustyAI registers the observations. 2. Apply name mappings - 3. Send metric request (spd) and verify the response. + 3. Send metric request and verify the response. 4. Send metric scheduling request and verify the response. 5. Send metric deletion request and verify that the scheduled metric has been deleted. """
208-218: Apply consistent formatting to parameterized test method.For consistency with the PVC storage class methods, format the method signature properly.
@pytest.mark.parametrize("metric_name", FAIRNESS_METRICS) def test_fairness_metric_with_db_storage( - self, admin_client, current_client_token, trustyai_service_with_db_storage, onnx_loan_model, metric_name + self, + admin_client, + current_client_token, + trustyai_service_with_db_storage, + onnx_loan_model, + metric_name, ):
220-230: Apply consistent formatting to parameterized test method.For consistency, format the method signature properly.
@pytest.mark.parametrize("metric_name", FAIRNESS_METRICS) def test_fairness_metric_schedule_with_db_storage( - self, admin_client, current_client_token, trustyai_service_with_db_storage, onnx_loan_model, metric_name + self, + admin_client, + current_client_token, + trustyai_service_with_db_storage, + onnx_loan_model, + metric_name, ):
232-241: Apply consistent formatting to parameterized test method.For consistency, format the method signature properly.
@pytest.mark.parametrize("metric_name", FAIRNESS_METRICS) def test_fairness_metric_delete_with_db_storage( - self, admin_client, current_client_token, trustyai_service_with_db_storage, onnx_loan_model, metric_name + self, + admin_client, + current_client_token, + trustyai_service_with_db_storage, + onnx_loan_model, + metric_name, ):
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (2)
tests/model_explainability/trustyai_service/fairness/test_fairness.py(3 hunks)tests/model_explainability/trustyai_service/trustyai_service_utils.py(1 hunks)
🧰 Additional context used
🧠 Learnings (2)
📓 Common learnings
Learnt from: dbasunag
PR: opendatahub-io/opendatahub-tests#338
File: tests/model_registry/rbac/test_mr_rbac.py:24-53
Timestamp: 2025-06-06T12:22:57.057Z
Learning: In the opendatahub-tests repository, prefer keeping test parameterization configurations inline rather than extracting them to separate variables/constants, as it makes triaging easier by avoiding the need to jump between different parts of the file to understand the test setup.
tests/model_explainability/trustyai_service/fairness/test_fairness.py (2)
Learnt from: dbasunag
PR: opendatahub-io/opendatahub-tests#338
File: tests/model_registry/rbac/test_mr_rbac.py:24-53
Timestamp: 2025-06-06T12:22:57.057Z
Learning: In the opendatahub-tests repository, prefer keeping test parameterization configurations inline rather than extracting them to separate variables/constants, as it makes triaging easier by avoiding the need to jump between different parts of the file to understand the test setup.
Learnt from: adolfo-ab
PR: opendatahub-io/opendatahub-tests#334
File: tests/model_explainability/trustyai_service/test_trustyai_service.py:52-65
Timestamp: 2025-06-05T10:05:17.642Z
Learning: For TrustyAI image validation tests: operator image tests require admin_client, related_images_refs, and trustyai_operator_configmap fixtures, while service image tests would require different fixtures like trustyai_service_with_pvc_storage, model_namespace, and current_client_token.
🪛 Flake8 (7.2.0)
tests/model_explainability/trustyai_service/fairness/test_fairness.py
[error] 112-112: continuation line with same indent as next logical line
(E125)
[error] 112-112: whitespace before ')'
(E202)
🪛 Pylint (3.3.7)
tests/model_explainability/trustyai_service/fairness/test_fairness.py
[refactor] 111-111: Too many arguments (6/5)
(R0913)
[refactor] 111-111: Too many positional arguments (6/5)
(R0917)
[refactor] 122-122: Too many arguments (6/5)
(R0913)
[refactor] 122-122: Too many positional arguments (6/5)
(R0917)
[refactor] 134-134: Too many arguments (7/5)
(R0913)
[refactor] 134-134: Too many positional arguments (7/5)
(R0917)
[refactor] 217-217: Too many arguments (6/5)
(R0913)
[refactor] 217-217: Too many positional arguments (6/5)
(R0917)
[refactor] 229-229: Too many arguments (6/5)
(R0913)
[refactor] 229-229: Too many positional arguments (6/5)
(R0917)
🔇 Additional comments (2)
tests/model_explainability/trustyai_service/trustyai_service_utils.py (1)
35-35: LGTM! Clean addition of the DIR fairness metric.The new DIR metric constant follows the same pattern as the existing SPD metric and integrates well with the test parameterization.
tests/model_explainability/trustyai_service/fairness/test_fairness.py (1)
37-37: Good parameterization approach for testing multiple fairness metrics.The constant definition allows both SPD and DIR metrics to be tested consistently across all test methods. This aligns well with the PR objective of adding DIR fairness metric tests.
|
Status of building tag latest: success. |
6 participants
This PR Parametrized the existing test cases for DIR Fairness metrics
Description
How Has This Been Tested?
Merge criteria: