Make PMM-T1087's postgres-db drop deterministic by travagliad · Pull Request #1293 · percona/pmm-qa

travagliad · 2026-09-02T15:46:53Z

Failures fixed (investigator)

source: Percona-Lab/pmm-submodules PR #4543 — run 33638736972, check E2E / Instances UI tests / e2e tests: @fb-instances
tests:
- codeceptjs-e2e/tests/verifyRemoteInstances_test.js:316 / @fb-instances — PMM-T1087 Verify adding PostgreSQL remote instance without postgres database

What failed

The only red job of 30 (Launchable gate: 1 actionable, 0 quarantined; 31 passed, 1 failed, 15 skipped), and it failed both attempts of Feature(...).retry(1):

expected element [role="alert"], [role="status"] to include
  "Connection check failed: pq: database "postgres" does not exist"

      + expected - actual
      -Service “arbustum_service” added to your inventory
       Your PostgreSQL service instance is now ready to be monitored.

The test points a remote PostgreSQL service at postgresnodb with database postgres
and expects PMM's pre-add connection check to reject it. Instead the service was added.

Root cause — the QA setup, not the product

postgresnodb had a postgres database in that run. From the run's own
pmm-managed.log:

CheckConnectionRequest: DSN: postgres://...@postgresnodb:5432/postgres?connect_timeout=2&sslmode=disable
CheckConnection response: .
ServiceInfo response: version:"15.19 - Percona Distribution"
  database_list:"postgres" database_list:"not_default_db"

The connection check succeeded, pg_database listed postgres, and the service's
postgres_exporter log has no connection errors. PMM did the right thing.

That database is supposed to be removed during setup by the one-shot
postgresremovedefaultdb container:

command: >
  psql -Utest -hpostgresnodb -dnot_default_db -c 'DROP DATABASE postgres WITH (FORCE);'

One attempt, no verification — and docker compose up -d never looks at its exit
code, so when that psql fails the run continues with the wrong precondition and the
test fails ~12 minutes later with a message that reads like a product regression.

Reproduced

On a throwaway Linode VM running the FB server image perconalab/pmm-server-fb:PR-4543-018d0e8,
AddService against the same postgresnodb container, same payload the UI sends:

`postgres` database	Result
absent (the state T1087 expects)	`HTTP 400` — `Connection check failed: pq: database "postgres" does not exist (3D000).`
present (the state the FB run had)	`200` — service created, exactly the toast the test saw

So the outcome is entirely determined by whether the setup dropped that database;
the product behaves correctly in both cases.

The drop itself succeeded 6/6 on an idle box, so the original psql failure was
transient and is not captured in the CI logs. What is reproducible is the failure
mode — the old one-shot dies on the first hiccup and says nothing:

psql: error: could not translate host name "postgresnodb" to address: Name or service not known

Fix

drop_default_db.sh (run by the one-shot) retries the drop until it is confirmed
gone via pg_database, so a transient resolve/connect error self-heals. Verified
on the VM: with postgresnodb down when the one-shot starts, it recovers on attempt 6
and exits 0 — the old command would have exited non-zero at attempt 1 and left the
database in place.
verify_no_default_db.sh, called from db_setup.sh, asserts the precondition on the
host (and surfaces the one-shot's exit code and logs). Verified both ways: exit 0 in
the good state, exit 1 with postgresnodb still has a postgres database … after
re-creating it.
Both sides are time-bounded (PGCONNECT_TIMEOUT, timeout around docker wait and
the query), so an unreachable postgresnodb fails within a known budget instead of
hanging the setup step — see the review thread.

The assertion in the test is untouched — the postgres database really must be absent
for T1087 to mean anything, so the setup is what had to become reliable.

CI on this branch

In E2E tests Matrix run 33650775022 on cdc7d9d, the
job FB E2E tests / Instances UI tests / e2e tests: @fb-instances completed
successfully (16:27:25 UTC) — 35 tests found, 35 passed, 0 failed at
LAUNCHABLE_CONFIDENCE: 100%, so the whole spec including T1087 ran, with
Setup PMM Server (which now runs the assertion) green. That is the end-to-end
confirmation the suite itself can give. The run as a whole reads cancelled
because the follow-up commit c6c8ab7 superseded it while three unrelated jobs
were still going; the Instances job had already finished green by then.

c6c8ab7 (timeouts, inlined literals, explicit exit propagation) is re-running as
33655528833.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PCsB6xKBJ8gxnjJUKmRVjV

The `postgres` database on postgresnodb was dropped by a fire-and-forget one-shot container: no retry, no verification, and `docker compose up -d` never sees its exit code. When the drop does not take effect, PMM correctly accepts the remote service and PMM-T1087 fails ~12 minutes later with a message that reads like a product regression. Retry the drop until it is confirmed, and assert the state from db_setup.sh so a failure fails the setup step instead. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCsB6xKBJ8gxnjJUKmRVjV Signed-off-by: travagliad <davi.travaglia@percona.com>

coderabbitai · 2026-09-02T15:51:26Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Team

Run ID: 1e561847-beef-4276-a4a5-9a0b0ea23d45

📥 Commits

Reviewing files that changed from the base of the PR and between 30eaf90 and cdc7d9d.

📒 Files selected for processing (5)

codeceptjs-e2e/docker-compose-nomad.yml
codeceptjs-e2e/docker-compose.yml
codeceptjs-e2e/testdata/db_setup.sh
codeceptjs-e2e/testdata/docker-db-setup-scripts/drop_default_db.sh
codeceptjs-e2e/testdata/docker-db-setup-scripts/verify_no_default_db.sh

🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

percona/pmm-qa (manual)
percona/pmm (manual)

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

The PostgreSQL default database removal logic now runs from a mounted drop_default_db.sh script in both Compose configurations. The script retries removal and verifies the database state. db_setup.sh invokes verify_no_default_db.sh, which waits for the removal container, reports failures, and checks pg_database to confirm that the postgres database no longer exists.

Sequence Diagram(s)

sequenceDiagram
  participant db_setup.sh
  participant verify_no_default_db.sh
  participant postgresremovedefaultdb
  participant drop_default_db.sh
  participant postgresnodb
  db_setup.sh->>verify_no_default_db.sh: start verification
  verify_no_default_db.sh->>postgresremovedefaultdb: wait for completion
  postgresremovedefaultdb->>drop_default_db.sh: execute mounted script
  drop_default_db.sh->>postgresnodb: force-drop postgres database
  verify_no_default_db.sh->>postgresnodb: query pg_database
  postgresnodb-->>verify_no_default_db.sh: return database state

Merge Risk: ⚪ Minimal · up to cdc7d

The change makes the test database setup deterministic and fails early when the required precondition is not met; no actionable merge-blocking risk remains after normal checks and review.

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely summarizes the main change: making the PMM-T1087 PostgreSQL database drop deterministic.
Description check	✅ Passed	The description directly explains the setup failure, root cause, retry and verification changes, and validation results for PMM-T1087.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Review follow-up: inline the literals the compose file already declares, set PGCONNECT_TIMEOUT and bound `docker wait` so a black-holed postgresnodb fails within a known budget instead of hanging the setup step, propagate the assertion's exit status explicitly from db_setup.sh, and trim the comments to the invariant. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PCsB6xKBJ8gxnjJUKmRVjV Signed-off-by: travagliad <davi.travaglia@percona.com>

travagliad · 2026-09-02T17:19:51Z

MongoDB SSL tests / e2e tests: @ssl-mongo was red on c6c8ab7 (run 33655528833). It is not this PR's failure, and there is nothing here to port or patch.

It failed in Run Setup for E2E Tests — pmm-framework --database ssl_psmdb, in qa-integration/, which this PR does not touch — while building the build_member image:

[MIRROR] percona-backup-mongodb-2.15.0-1.el9.x86_64.rpm: Interrupted by header callback:
  Inconsistent server data, reported file Content-Length: 42193810,
  repository metadata states file length: ...
[FAILED] percona-backup-mongodb-2.15.0-1.el9.x86_64.rpm: No more mirrors to try
Error: Error downloading packages

That is a publish/mirror-sync race in Percona's own pbm-testing repo, not a test problem. percona-backup-mongodb-2.15.0-1.el9.x86_64.rpm was republished at 16:32:16 UTC, about six minutes before the job's dnf transaction (metadata check 16:38:31), so the repodata a mirror served still described the previous file length and every mirror attempt aborted. The repo has since settled — the served file and the current repodata both state 42193810 bytes with file time 16:32:16 — so I have re-run the failed job once.

Everything relevant to this PR is green on the same commit: the 29 other jobs passed, Setup PMM Server (which runs the new assertion via db_setup.sh) succeeded in the failing job too, and FB E2E tests / Instances UI tests / e2e tests: @fb-instances — the suite containing PMM-T1087 — passed again with 35 tests found, 35 passed, 0 failed.

Generated by Claude Code

travagliad requested review from peterSirotnak and yurkovychv as code owners September 2, 2026 15:46

travagliad mentioned this pull request Sep 2, 2026

PMM-15031: Email logo Percona-Lab/pmm-submodules#4543

Open

2 tasks