Skip to content

test: add Inpatient morbidity/ICD-10 tracker import perf scenario - #24814

Open
jason-p-pickering wants to merge 4 commits into
masterfrom
test/tracker-inpatient-icd10-perf
Open

test: add Inpatient morbidity/ICD-10 tracker import perf scenario#24814
jason-p-pickering wants to merge 4 commits into
masterfrom
test/tracker-inpatient-icd10-perf

Conversation

@jason-p-pickering

@jason-p-pickering jason-p-pickering commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What this adds

A new, self-contained SingleEventTest.java performance test for single-event-program tracker imports -- event programs without tracked entities, imported one event per HTTP request from many independent concurrent user sessions. It currently exercises one Sierra Leone demo DB program: Inpatient morbidity and mortality (eBAyeGv0exc), which uses a ~14,000-option ICD-10 diagnosis option set (eUZ79clX7y1, data element K6uUAvq500H). This is meant to stress-test typical single-event import against a large option set specifically -- companion perf coverage for #24815 (master port of #24813), which fixes an N+1 in how tracker preheat resolves option codes.

This is deliberately a new class rather than a fourth program inside TrackerTest: sharing TrackerTest's importEntitiesPerRequest/importUsers/importRequestsPerUser config broke in three independent ways once single-event-per-request behavior was forced through it (batch size floors to 0 and crashes the whole Gatling simulation; shared concurrency knobs contaminate MNCH/Child/ANC's own results; and the shared admin login exceeds max.sessions.per_user under enough concurrent users). TrackerTest.java itself is untouched by this PR -- it stays an exact match of master.

Unlike MNCH/Child/ANC (S3-fetched Synthea data, for per-patient realism), the Inpatient event payload is a single template embedded directly in SingleEventTest.java. The goal here is consistent pressure on the ICD-10 option set through the import path, not data variety across the ~14k codes, so no Synthea generation or S3 fetch is needed.

Threshold dependency on #24815 -- please read before merging

SINGLE_EVENT_IMPORT_P95 (SMOKE: 250ms, LOAD: 1500ms) is calibrated from real performance-tests-compare.yml runs using #24815's candidate (fixed) performance, not current master's. On unfixed master this scenario measures ~1,204ms p95 (SMOKE) and ~32,183ms p95 with a persistent trickle of hard 60s timeouts (LOAD) -- both well past these thresholds.

This branch is not stacked on #24815 (it's a plain new Gatling test file with no code dependency on the fix), so the two merge independently. That's fine as long as merge order is respected: #24815 should land before this, or at minimum before SingleEventTest is added to performance-tests-scheduled.yml's nightly matrix (it isn't in that matrix yet, so there's no live CI risk today -- this note is so the risk doesn't get rediscovered the hard way once someone wires it in).

@jason-p-pickering jason-p-pickering added the run-perf-tests Enables performance tests label Aug 8, 2026
Single-event programs (e.g. Inpatient morbidity and mortality, exercising
the ~14,000-option ICD-10 diagnosis option set) are imported one event per
request from many independent user sessions in production -- a
fundamentally different traffic shape than TrackerTest's tracked-entity
programs (MNCH, Child), which batch a patient's enrollments and events
together per request via a shared importEntitiesPerRequest/importUsers
configuration.

Forcing single-event behavior out of TrackerTest's shared knobs either
crashes the whole simulation (a tracked-entity program's line-batching
floors to zero at batch size 1, and Gatling's feeder rejects that) or
contaminates the other programs' own concurrency (importUsers is shared
across all of them). SingleEventTest.java is a fully self-contained
simulation instead, decoupled from TrackerTest entirely -- TrackerTest.java
itself is unchanged by this PR.

Verified end-to-end against a real DHIS2 2.41 instance: login-once-then
-reuse-session works (no Basic Auth on measured requests), each request's
`stats.created` is asserted to confirm a real event is persisted (not just
HTTP 200), and a direct database check confirmed the event table's row
count increases by exactly the configured request count per run.

SINGLE_EVENT_IMPORT_P95 is a provisional placeholder pending a real
performance-tests-compare.yml baseline-vs-candidate run against this
scenario specifically.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jason-p-pickering
jason-p-pickering force-pushed the test/tracker-inpatient-icd10-perf branch from 0967d74 to 4c4aef7 Compare August 9, 2026 09:00
jason-p-pickering and others added 2 commits August 9, 2026 11:12
Default (10) is too low once a scenario's concurrent-user count exceeds it:
Gatling scenarios share one admin/replica account across many virtual
users, and Spring Security's concurrent-session control evicts older
sessions past the cap, redirecting (302) any request still holding an
evicted session instead of serving it -- observed as "status.find.is(200),
but actually found 302" errors and near-50% success rates on TrackerTest's
import scenarios once importUsers was pushed to 20 across four concurrent
scenarios (80 total sessions under one account). SingleEventTest's own
LOAD profile (users=20) hits the same wall on its own, single-scenario.

Test-only override via the live-mounted docker/dhis.conf; production
deployments keep the real default.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
SMOKE (run 31307558297, 1 user): baseline p95 1,204ms, candidate p95 105ms.
LOAD (run 31305382629, 20 concurrent users): baseline p95 32,183ms with a
persistent trickle of hard 60s timeout failures (0.3% of requests never
completed), candidate p95 666ms with 100% success and zero timeouts --
a ~48x improvement, and the fix eliminates the timeouts entirely, not
just the latency.

Both runs are unconfounded: this simulation shares no configuration with
TrackerTest's tracked-entity programs, so these numbers measure exactly
what they claim to.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@jason-p-pickering

Copy link
Copy Markdown
Contributor Author

Export threshold failures during investigation — stale Postgres stats after bulk import, not related to the import-path fix

While the Inpatient/ICD-10 scenario was still living inside TrackerTest.java (before it moved to
the dedicated SingleEventTest.java this PR now adds), running performance-tests-compare.yml
against it to calibrate a threshold hit the same assertion failure on every run:

Get Inpatient events / Search by date range: 95th percentile of response time is less than or equal to X : false

This failed identically on unpatched master (baseline) and on #24815's candidate, in both SMOKE and LOAD profiles — so it was never a regression from the import-path fix:

Run Threshold Baseline actual Candidate actual
SMOKE (run 31295093484) 41ms 84ms 82ms
LOAD, 500 events/request (run 31296609465) 509ms 3445ms 3413ms

Likely mechanism: TrackerTest's import and export scenarios run back-to-back inside one continuous Gatling simulation (per the class javadoc: "imports first..., then exports"). run-simulation.sh's vacuum analyze only runs once, before the whole measured run starts — there's no ANALYZE between the import phase and the export phase within a run. Inpatient was a brand-new program with zero pre-existing rows in the demo DB image, unlike MNCH/Child/ANC which have real historical data already baked in and stats already collected on it. So by the time "Get Inpatient events / Search by date range" fired, every matching row had been inserted moments earlier by the same test, and Postgres's planner still had whatever stats existed when the DB image was baked (i.e., none for this program) — a cardinality misestimate there can flip a join/plan choice rather than just add linear overhead, which matches the cliff-shaped numbers above better than gradual degradation would.

Current status: this PR moves the Inpatient scenario to its own SingleEventTest.java, which is import-only (no export scenario at all) — so this specific manifestation no longer exists in the codebase, and it's not blocking anything here. Flagging anyway because the underlying gap is general to TrackerTest.java, not specific to Inpatient: any newly-introduced program that gets bulk-imported and then queried in the same run's export phase would hit the same stale-stats cliff. ANC's own export scenario ("Get ANC events") is currently insulated only because ANC already has real historical data baked into the demo DB image, not because the gap is fixed — a future program added the same way Inpatient was could reintroduce this exact failure mode. Options if it comes up again: an explicit ANALYZE between the import and export phases for newly-introduced programs, seeding some pre-existing rows in the demo DB image ahead of the test, or just loosening/removing export-side thresholds for brand-new programs until addressed.

@sonarqubecloud

Copy link
Copy Markdown

@codecov

codecov Bot commented Aug 11, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.22%. Comparing base (e15b4d9) to head (0e7a415).
⚠️ Report is 10 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff              @@
##             master   #24814      +/-   ##
============================================
+ Coverage     58.05%   64.22%   +6.16%     
+ Complexity     1414      661     -753     
============================================
  Files          3725     3725              
  Lines        144739   144761      +22     
  Branches      16875    16878       +3     
============================================
+ Hits          84033    92975    +8942     
+ Misses        53550    44454    -9096     
- Partials       7156     7332     +176     
Flag Coverage Δ
integration 50.13% <ø> (+10.35%) ⬆️
unit 35.94% <ø> (+0.16%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 684 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed1f585...0e7a415. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-perf-tests Enables performance tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant