Skip to content

test(maas): add subscription enforcement tests for TinyLlama#1163

Merged
sheltoncyril merged 5 commits intoopendatahub-io:mainfrom
SB159:feature/maas_subscription_test2
Mar 3, 2026
Merged

test(maas): add subscription enforcement tests for TinyLlama#1163
sheltoncyril merged 5 commits intoopendatahub-io:mainfrom
SB159:feature/maas_subscription_test2

Conversation

@SB159
Copy link
Copy Markdown
Contributor

@SB159 SB159 commented Mar 3, 2026

Pull Request

Summary

This PR adds TinyLlama premium subscription tests:

  • test_subscribed_user_gets_200
  • test_invalid_subscription_header_gets_429
  • test_explicit_subscription_header_works

Related Issues

  • Fixes:
  • JIRA:

How it has been tested

  • Locally
  • Jenkins

Summary by CodeRabbit

  • Tests
    • Added subscription enforcement tests for premium models covering authenticated access, explicit subscription headers, and invalid subscription scenarios.
  • Refactor
    • Updated subscription fixtures to use grouped owners and explicit token rate limits (free: 100/min, premium: 1000/min).
  • Chores
    • Tests now wait for subscriptions to reach Ready status before proceeding.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 3, 2026

The following are automatically added/executed:

  • PR size label.
  • Run pre-commit
  • Run tox
  • Add PR author as the PR assignee
  • Build image based on the PR

Available user actions:

  • To mark a PR as WIP, add /wip in a comment. To remove it from the PR comment /wip cancel to the PR.
  • To block merging of a PR, add /hold in a comment. To un-block merging of PR comment /hold cancel.
  • To mark a PR as approved, add /lgtm in a comment. To remove, add /lgtm cancel.
    lgtm label removed on each new commit push.
  • To mark PR as verified comment /verified to the PR, to un-verify comment /verified cancel to the PR.
    verified label removed on each new commit push.
  • To Cherry-pick a merged PR /cherry-pick <target_branch_name> to the PR. If <target_branch_name> is valid,
    and the current PR is merged, a cherry-picked PR would be created and linked to the current PR.
  • To build and push image to quay, add /build-push-pr-image in a comment. This would create an image with tag
    pr-<pr_number> to quay repository. This image tag, however would be deleted on PR merge or close action.
Supported labels

{'/cherry-pick', '/wip', '/build-push-pr-image', '/hold', '/lgtm', '/verified'}

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Mar 3, 2026

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to data retention organization setting

📥 Commits

Reviewing files that changed from the base of the PR and between 0334ae5 and e8237de.

📒 Files selected for processing (1)
  • tests/model_serving/model_server/maas_billing/maas_subscription/test_maas_sub_enforcement.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • tests/model_serving/model_server/maas_billing/maas_subscription/test_maas_sub_enforcement.py

📝 Walkthrough

Walkthrough

Updated MaaS subscription fixtures to change owner to a groups list, replace tokensPerMinute with tokenRateLimits, and add readiness polling; added a new test module containing three subscription-enforcement tests and two constants.

Changes

Cohort / File(s) Summary
Fixture updates
tests/model_serving/model_server/maas_billing/maas_subscription/conftest.py
Modified maas_subscription_tinyllama_free and maas_subscription_tinyllama_premium: replaced owner single-Group object with {"groups":[{"name": ...}]}, replaced tokensPerMinute with tokenRateLimits list ([{"limit":..., "window":"1m"}]), and added wait_for_condition(..., timeout=300) after create/yield.
Subscription enforcement tests
tests/model_serving/model_server/maas_billing/maas_subscription/test_maas_sub_enforcement.py
Added new test module defining MAAS_SUBSCRIPTION_HEADER and INVALID_SUBSCRIPTION and TestSubscriptionEnforcementTinyLlama with three tests: subscribed user gets 200, explicit subscription header works, and invalid subscription header yields 429/403. Uses fixtures and a 60s HTTP timeout.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 40.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (2 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: adding subscription enforcement tests for TinyLlama in the MaaS billing module.
Description check ✅ Passed The description includes a summary of the test cases added and testing methods, but omits required JIRA ticket information and additional requirements checklist.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/model_serving/model_server/maas_billing/maas_subscription/conftest.py`:
- Around line 176-196: The fixture maas_auth_policy_no_subscription is defined
but not used; either remove this unused fixture or wire it into the appropriate
test(s) by adding maas_auth_policy_no_subscription as a parameter to the test
function(s) that should validate authenticated-but-no-subscription behavior
(e.g., tests that reference MaaSModel maas_model_tinyllama_premium or rely on
MaaSAuthPolicy); if keeping it, ensure the test imports/accepts the fixture name
so the with-block in maas_auth_policy_no_subscription (which creates a
MaaSAuthPolicy resource and waits for Ready) actually executes, otherwise delete
the maas_auth_policy_no_subscription fixture to clean up dead code.

In
`@tests/model_serving/model_server/maas_billing/maas_subscription/test_maas_sub_enforcement.py`:
- Around line 98-127: The test test_auth_pass_no_subscription_gets_429 never
updates last_resp inside the TimeoutSampler loop so the pytest.fail message is
unhelpful; inside the loop where resp is received from TimeoutSampler (the
iteration calling request_session_http.post), assign last_resp = resp on each
iteration (e.g., immediately after receiving resp) so the final failure message
can show the most recent response status and body; adjust references in the loop
surrounding TimeoutSampler, resp, and last_resp accordingly.
- Around line 98-127: Rename or clarify the test name and fix the unassigned
response variable: change the test function
test_auth_pass_no_subscription_gets_429 to reflect that it is validating
behavior when the x-maas-subscription header is omitted (e.g.,
test_auth_pass_missing_subscription_header_gets_429) or, if you intended to test
a truly unsubscribed actor, switch the parametrized ocp_token_for_actor to an
actor type/group that has no subscription; additionally, inside the
TimeoutSampler loop in test_auth_pass_no_subscription_gets_429 (or the renamed
function) assign last_resp = resp on each iteration so the final failure message
reports the real last response status/text.

ℹ️ Review info

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Cache: Disabled due to data retention organization setting

Knowledge base: Disabled due to data retention organization setting

📥 Commits

Reviewing files that changed from the base of the PR and between 0b1ffc2 and 112932b.

📒 Files selected for processing (2)
  • tests/model_serving/model_server/maas_billing/maas_subscription/conftest.py
  • tests/model_serving/model_server/maas_billing/maas_subscription/test_maas_sub_enforcement.py

Comment thread tests/model_serving/model_server/maas_billing/maas_subscription/conftest.py Outdated
threcc
threcc previously approved these changes Mar 3, 2026
brettmthompson
brettmthompson previously approved these changes Mar 3, 2026
Copy link
Copy Markdown
Contributor

@brettmthompson brettmthompson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@SB159 SB159 dismissed stale reviews from brettmthompson and threcc via e8237de March 3, 2026 20:09
Copy link
Copy Markdown
Contributor

@brettmthompson brettmthompson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Copy link
Copy Markdown
Contributor

@sheltoncyril sheltoncyril left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@sheltoncyril sheltoncyril enabled auto-merge (squash) March 3, 2026 20:43
@sheltoncyril sheltoncyril merged commit 394a748 into opendatahub-io:main Mar 3, 2026
8 checks passed
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 3, 2026

Status of building tag latest: success.
Status of pushing tag latest to image registry: success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants