Skip to content

test: New tests to validate postgres password autogeneration#1200

Merged
dbasunag merged 3 commits intoopendatahub-io:mainfrom
dbasunag:db_pass
Mar 11, 2026
Merged

test: New tests to validate postgres password autogeneration#1200
dbasunag merged 3 commits intoopendatahub-io:mainfrom
dbasunag:db_pass

Conversation

@dbasunag
Copy link
Copy Markdown
Collaborator

@dbasunag dbasunag commented Mar 10, 2026

Pull Request

Summary

Related Issues

  • Fixes:
  • JIRA:

How it has been tested

  • Locally
  • Jenkins

Additional Requirements

  • If this PR introduces a new test image, did you create a PR to mirror it in disconnected environment?
  • If this PR introduces new marker(s)/adds a new component, was relevant ticket created to update relevant Jenkins job?

Summary by CodeRabbit

  • Tests

    • Added a suite verifying model catalog DB secret lifecycle, secret recreation behavior, and pod readiness after secret changes.
    • Added fixtures to capture secret values, simulate secret recreation with polling, and enforce test ordering.
    • Added smoke tests to assert secret presence, password rotation, and pod readiness after recreation.
  • Test Helpers

    • Added utilities to decode and log secret data and a package init marker for the test folder.

@github-actions
Copy link
Copy Markdown

The following are automatically added/executed:

  • PR size label.
  • Run pre-commit
  • Run tox
  • Add PR author as the PR assignee
  • Build image based on the PR

Available user actions:

  • To mark a PR as WIP, add /wip in a comment. To remove it from the PR comment /wip cancel to the PR.
  • To block merging of a PR, add /hold in a comment. To un-block merging of PR comment /hold cancel.
  • To mark a PR as approved, add /lgtm in a comment. To remove, add /lgtm cancel.
    lgtm label removed on each new commit push.
  • To mark PR as verified comment /verified to the PR, to un-verify comment /verified cancel to the PR.
    verified label removed on each new commit push.
  • To Cherry-pick a merged PR /cherry-pick <target_branch_name> to the PR. If <target_branch_name> is valid,
    and the current PR is merged, a cherry-picked PR would be created and linked to the current PR.
  • To build and push image to quay, add /build-push-pr-image in a comment. This would create an image with tag
    pr-<pr_number> to quay repository. This image tag, however would be deleted on PR merge or close action.
Supported labels

{'/lgtm', '/verified', '/cherry-pick', '/hold', '/build-push-pr-image', '/wip'}

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai bot commented Mar 10, 2026

📝 Walkthrough

Walkthrough

Adds tests and fixtures that fetch, decode, delete and poll for recreation of the model-catalog-postgres Secret, plus a utility to extract base64-encoded Secret values and tests that verify secret rotation and pod readiness. Decoding and logging of secret keys/values may expose sensitive data (CWE-200).

Changes

Cohort / File(s) Summary
Module marker
tests/model_registry/model_catalog/db_check/__init__.py
Adds package initializer for database check tests.
Secret fixtures & lifecycle
tests/model_registry/model_catalog/db_check/conftest.py
New pytest fixtures to fetch the model-catalog-postgres Secret, capture decoded values, delete the Secret and poll (up to 120s, 10s interval) for operator-driven recreation, returning extracted values.
Integration tests
tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py
Three tests: assert Secret presence, verify recreated Secret preserves db name/user but rotates password, and confirm model-catalog pods become ready after Postgres pod deletion; uses pytest dependency markers and DynamicClient for pod ops.
Test utilities
tests/model_registry/model_catalog/db_check/utils.py
Adds extract_secret_values to base64-decode Secret data with fallback on decode errors and a module LOGGER; logs captured keys.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 1 | ❌ 1

❌ Failed checks (1 inconclusive)

Check name Status Explanation Resolution
Description check ❓ Inconclusive The PR description follows the template structure but the Summary section lacks concrete details about changes, and no Related Issues or JIRA tickets are provided. Fill in the Summary section with specific details about what tests were added and why they are needed.
✅ Passed checks (1 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately describes the main change: adding tests for PostgreSQL password autogeneration.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 5

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/model_registry/model_catalog/db_check/conftest.py`:
- Around line 14-20: The fixture model_catalog_postgres_secret currently uses
py_config["model_registry_namespace"] which bypasses dynamic namespace
resolution; change the fixture to accept the existing pytest fixture
model_registry_namespace (add it as a parameter to
model_catalog_postgres_secret) and use that value for the Secret namespace
instead of py_config["model_registry_namespace"]; apply the same change to the
other fixtures in this module that reference py_config for the model registry
namespace so they all use the model_registry_namespace fixture (keep the Secret
creation and ensure_exists behavior unchanged).
- Around line 38-56: The loop currently treats any Secret with exists==True as
the recreated secret but delete() is asynchronous and the first sampled Secret
may be the old instance; capture the original secret identity from
model_catalog_postgres_secret (e.g., metadata.uid or metadata.resource_version)
before calling delete(), then in the TimeoutSampler loop for Secret (the Secret
class used with admin_client and resource_name) wait until secret.exists is True
AND the secret's metadata.uid (or resource_version) differs from the original
value to ensure you observed a newly-created Secret instance; update the
condition that sets recreated_secret to check both existence and a changed
uid/resource_version.

In `@tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py`:
- Around line 57-60: The test currently calls the namespace-wide
wait_for_pods_running which can be tripped by unrelated pod restarts; replace
that call with the targeted helper
wait_for_model_catalog_pod_ready_after_deletion and import it from
tests.model_registry.model_catalog.db_check.utils, keeping the existing
assertion wait_for_model_catalog_pod_created(client=admin_client,
model_registry_namespace=model_registry_namespace) and then invoking
wait_for_model_catalog_pod_ready_after_deletion(...) with the same admin_client
and model_registry_namespace (and any required pod identifier params) so the
check is scoped to the model-catalog pod under test rather than the whole
namespace.
- Line 46: The call to get_postgres_pod_in_namespace in the test is using the
default namespace; update the invocation to pass the test's
model_registry_namespace variable so it targets the correct namespace (e.g.,
replace get_postgres_pod_in_namespace(admin_client=admin_client) with
get_postgres_pod_in_namespace(admin_client=admin_client,
namespace=model_registry_namespace)); keep the admin_client argument and ensure
any downstream references use the returned postgres_pod as before.

In `@tests/model_registry/model_catalog/db_check/utils.py`:
- Around line 59-68: The helper currently only checks pods[0] (catalog_pod) for
readiness which is order-dependent; update the logic to iterate over the pods
list and consider the helper successful if any pod meets the readiness criteria.
Replace the single-use of catalog_pod with a loop over pods, and for each pod
check pod.status == Pod.Status.RUNNING and pod.instance.status.conditions, then
inspect condition.type == "Ready" and condition.status == "True" and return when
found; keep LOGGER.info(f"Model catalog pod {pod.name} is ready") for the pod
that passes.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 99f90098-5f4e-41eb-9b18-7c05ab52e60a

📥 Commits

Reviewing files that changed from the base of the PR and between 68d9aef and 2e6a723.

📒 Files selected for processing (4)
  • tests/model_registry/model_catalog/db_check/__init__.py
  • tests/model_registry/model_catalog/db_check/conftest.py
  • tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py
  • tests/model_registry/model_catalog/db_check/utils.py

Comment thread tests/model_registry/model_catalog/db_check/conftest.py Outdated
Comment thread tests/model_registry/model_catalog/db_check/conftest.py
Comment thread tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py Outdated
Comment thread tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py Outdated
Comment thread tests/model_registry/model_catalog/db_check/utils.py Outdated
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (2)
tests/model_registry/model_catalog/db_check/conftest.py (2)

31-58: ⚠️ Potential issue | 🟠 Major

Wait for a new Secret instance, not just exists == True.

delete() is asynchronous. The first sampled object can still be the original Secret, so this loop can return stale data before recreation happens.

Suggested fix
 def recreated_model_catalog_postgres_secret(
-    admin_client: DynamicClient, model_catalog_postgres_secret: Secret
+    admin_client: DynamicClient,
+    model_catalog_postgres_secret: Secret,
+    model_registry_namespace: str,
 ) -> dict[str, str]:
     """Delete model-catalog-postgres secret and wait for it to be recreated"""
-    model_registry_namespace = py_config["model_registry_namespace"]
     resource_name = "model-catalog-postgres"
 
     LOGGER.info(f"Deleting secret {resource_name} in namespace {model_registry_namespace}")
+    original_uid = model_catalog_postgres_secret.instance.metadata.uid
     model_catalog_postgres_secret.delete()
@@
     ):
-        if secret.exists:
-            LOGGER.info(f"Secret {resource_name} has been recreated")
-            recreated_secret = secret
-            break
+        if not secret.exists:
+            continue
+
+        if secret.instance.metadata.uid == original_uid:
+            LOGGER.info(
+                f"Secret {resource_name} still has the original UID; waiting for recreation"
+            )
+            continue
+
+        LOGGER.info(f"Secret {resource_name} has been recreated")
+        recreated_secret = secret
+        break
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/model_registry/model_catalog/db_check/conftest.py` around lines 31 -
58, The current recreated_model_catalog_postgres_secret function may return the
original secret because delete() is async; change the wait logic to detect a new
Secret instance by comparing a stable identifier from the original secret (e.g.,
model_catalog_postgres_secret.metadata.uid or .resource_version) against the
sampled Secret returned by TimeoutSampler (the sampled Secret must have a
different uid/resource_version before accepting it), and only call
extract_secret_values when you confirm the sampled secret is a new instance (not
the pre-deletion object).

14-20: ⚠️ Potential issue | 🟠 Major

Use the live model_registry_namespace fixture here.

This fixture hard-codes the namespace from py_config, so it can read or delete the wrong Secret when the registry namespace is overridden by cluster state.

Suggested fix
 `@pytest.fixture`(scope="class")
-def model_catalog_postgres_secret(admin_client: DynamicClient) -> Secret:
+def model_catalog_postgres_secret(
+    admin_client: DynamicClient, model_registry_namespace: str
+) -> Secret:
     """Get the model-catalog-postgres secret from model registry namespace"""
     return Secret(
         client=admin_client,
         name="model-catalog-postgres",
-        namespace=py_config["model_registry_namespace"],
+        namespace=model_registry_namespace,
         ensure_exists=True,
     )
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/model_registry/model_catalog/db_check/conftest.py` around lines 14 -
20, The model_catalog_postgres_secret fixture currently hard-codes the namespace
from py_config; change it to accept and use the live model_registry_namespace
fixture instead so tests use the actual cluster namespace. Update the
model_catalog_postgres_secret signature to accept model_registry_namespace and
replace namespace=py_config["model_registry_namespace"] with
namespace=model_registry_namespace (retain ensure_exists=True and the Secret
construction). Ensure any callers of model_catalog_postgres_secret in tests are
compatible with the fixture change.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@tests/model_registry/model_catalog/db_check/conftest.py`:
- Around line 31-58: The current recreated_model_catalog_postgres_secret
function may return the original secret because delete() is async; change the
wait logic to detect a new Secret instance by comparing a stable identifier from
the original secret (e.g., model_catalog_postgres_secret.metadata.uid or
.resource_version) against the sampled Secret returned by TimeoutSampler (the
sampled Secret must have a different uid/resource_version before accepting it),
and only call extract_secret_values when you confirm the sampled secret is a new
instance (not the pre-deletion object).
- Around line 14-20: The model_catalog_postgres_secret fixture currently
hard-codes the namespace from py_config; change it to accept and use the live
model_registry_namespace fixture instead so tests use the actual cluster
namespace. Update the model_catalog_postgres_secret signature to accept
model_registry_namespace and replace
namespace=py_config["model_registry_namespace"] with
namespace=model_registry_namespace (retain ensure_exists=True and the Secret
construction). Ensure any callers of model_catalog_postgres_secret in tests are
compatible with the fixture change.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 883df9fb-256d-449e-8f54-ac0f62e6c514

📥 Commits

Reviewing files that changed from the base of the PR and between 2e6a723 and d3c84b6.

📒 Files selected for processing (4)
  • tests/model_registry/model_catalog/db_check/__init__.py
  • tests/model_registry/model_catalog/db_check/conftest.py
  • tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py
  • tests/model_registry/model_catalog/db_check/utils.py
🚧 Files skipped from review as they are similar to previous changes (3)
  • tests/model_registry/model_catalog/db_check/utils.py
  • tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py
  • tests/model_registry/model_catalog/db_check/init.py

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@tests/model_registry/model_catalog/db_check/utils.py`:
- Line 29: The current LOGGER.info call logs secret_values.keys() which exposes
sensitive secret key names; change the logging to avoid listing keys by either
logging only the count (e.g., len(secret_values)) or moving the message to DEBUG
level and omit the key names; update the reference in this file to replace the
LOGGER.info(f"Captured secret with keys: {list(secret_values.keys())}") usage
with a safer message that references LOGGER and secret_values (e.g., log the
number of secrets or a debug-only message) so no secret key names are emitted.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: 9d8f9997-3f81-4256-bd6e-49f204aa88ef

📥 Commits

Reviewing files that changed from the base of the PR and between d3c84b6 and e2707b1.

📒 Files selected for processing (4)
  • tests/model_registry/model_catalog/db_check/__init__.py
  • tests/model_registry/model_catalog/db_check/conftest.py
  • tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py
  • tests/model_registry/model_catalog/db_check/utils.py
🚧 Files skipped from review as they are similar to previous changes (3)
  • tests/model_registry/model_catalog/db_check/init.py
  • tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py
  • tests/model_registry/model_catalog/db_check/conftest.py

Comment thread tests/model_registry/model_catalog/db_check/utils.py Outdated
Comment thread tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py Outdated
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

♻️ Duplicate comments (2)
tests/model_registry/model_catalog/db_check/conftest.py (2)

31-35: ⚠️ Potential issue | 🟠 Major

Use the live namespace fixture here as well.

Line 35 still bypasses model_registry_namespace, so this fixture can delete or poll the wrong Secret when the registry namespace is overridden by cluster state.

Suggested fix
 `@pytest.fixture`(scope="class")
 def recreated_model_catalog_postgres_secret(
-    admin_client: DynamicClient, model_catalog_postgres_secret: Secret
+    admin_client: DynamicClient,
+    model_catalog_postgres_secret: Secret,
+    model_registry_namespace: str,
 ) -> dict[str, str]:
     """Delete model-catalog-postgres secret and wait for it to be recreated"""
-    model_registry_namespace = py_config["model_registry_namespace"]
     resource_name = "model-catalog-postgres"
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/model_registry/model_catalog/db_check/conftest.py` around lines 31 -
35, The fixture recreated_model_catalog_postgres_secret currently reads
model_registry_namespace from py_config (bypassing the live namespace); update
the fixture signature to accept the model_registry_namespace fixture (e.g., add
model_registry_namespace: str parameter) and replace the local
py_config["model_registry_namespace"] usage with that parameter so the secret
deletion and polling use the actual live namespace when the registry namespace
is overridden.

38-56: ⚠️ Potential issue | 🟠 Major

Wait for a new Secret instance, not just exists == True.

Line 39 deletes asynchronously. Lines 53-56 can still capture the pre-delete Secret, so the rotation check becomes nondeterministic and may read stale credentials.

Suggested fix
     LOGGER.info(f"Deleting secret {resource_name} in namespace {model_registry_namespace}")
+    original_uid = model_catalog_postgres_secret.instance.metadata.uid
     model_catalog_postgres_secret.delete()
@@
     ):
-        if secret.exists:
-            LOGGER.info(f"Secret {resource_name} has been recreated")
-            recreated_secret = secret
-            break
+        if not secret.exists:
+            continue
+
+        if secret.instance.metadata.uid == original_uid:
+            LOGGER.info(f"Secret {resource_name} still has the original UID; waiting for recreation")
+            continue
+
+        LOGGER.info(f"Secret {resource_name} has been recreated")
+        recreated_secret = secret
+        break
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@tests/model_registry/model_catalog/db_check/conftest.py` around lines 38 -
56, Capture the original Secret identity before deleting (e.g., read
model_catalog_postgres_secret or Secret(...).metadata.uid/resource_version) and
after deletion, change the TimeoutSampler loop (which currently checks only
secret.exists) to wait for a Secret instance whose uid/resource_version differs
from the original; specifically, in the loop that iterates
Secret(client=admin_client, name=resource_name,
namespace=model_registry_namespace) replace the exists-only check with a check
that secret.exists is True AND secret.metadata.uid (or
metadata.resource_version) != original_uid so you only accept a recreated
Secret, not the pre-delete instance.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Duplicate comments:
In `@tests/model_registry/model_catalog/db_check/conftest.py`:
- Around line 31-35: The fixture recreated_model_catalog_postgres_secret
currently reads model_registry_namespace from py_config (bypassing the live
namespace); update the fixture signature to accept the model_registry_namespace
fixture (e.g., add model_registry_namespace: str parameter) and replace the
local py_config["model_registry_namespace"] usage with that parameter so the
secret deletion and polling use the actual live namespace when the registry
namespace is overridden.
- Around line 38-56: Capture the original Secret identity before deleting (e.g.,
read model_catalog_postgres_secret or Secret(...).metadata.uid/resource_version)
and after deletion, change the TimeoutSampler loop (which currently checks only
secret.exists) to wait for a Secret instance whose uid/resource_version differs
from the original; specifically, in the loop that iterates
Secret(client=admin_client, name=resource_name,
namespace=model_registry_namespace) replace the exists-only check with a check
that secret.exists is True AND secret.metadata.uid (or
metadata.resource_version) != original_uid so you only accept a recreated
Secret, not the pre-delete instance.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited), Organization UI (inherited)

Review profile: CHILL

Plan: Pro

Run ID: a20dcf9a-3ba4-42a8-8297-7046f2bbd88d

📥 Commits

Reviewing files that changed from the base of the PR and between e2707b1 and 280798e.

📒 Files selected for processing (3)
  • tests/model_registry/model_catalog/db_check/conftest.py
  • tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py
  • tests/model_registry/model_catalog/db_check/utils.py
🚧 Files skipped from review as they are similar to previous changes (2)
  • tests/model_registry/model_catalog/db_check/utils.py
  • tests/model_registry/model_catalog/db_check/test_model_catalog_secrets.py

Copy link
Copy Markdown
Contributor

@fege fege left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

Copy link
Copy Markdown
Contributor

@kpunwatk kpunwatk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@dbasunag dbasunag merged commit d90df16 into opendatahub-io:main Mar 11, 2026
13 checks passed
@dbasunag dbasunag deleted the db_pass branch March 11, 2026 16:27
@github-actions
Copy link
Copy Markdown

Status of building tag latest: success.
Status of pushing tag latest to image registry: success.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants