feat(server): reconstruct the narrative by robertmitchellv · Pull Request #1369 · CDCgov/dibbs-ecr-refiner

robertmitchellv · 2026-06-13T00:15:52Z

🔀 PULL REQUEST

💡 Summary

Note

narrative.py was pulled out of the section module in the ecr/ service and is not its own module next to both section/ and specification/ (some of this is just a clean move out of the original narrative.py

The package is organized around a clean three-layer DRY seam, plus shared writers:

elements.py — low-level primitives. Namespace-qualified element builders (_make_element/_sub_element), <text> placement into the correct CDA R2 xs:sequence slot, comment scrubbing. Everything emitted must carry the urn:hl7-org:v3 namespace or it silently fails NarrativeBlock.xsd.
footnote.py — the per-section provenance footnote (configured‑vs‑actual, tied to the augmentation run timestamp via xs:ID).
writers.py — the three <text> swap operations, all routed through one _place_section_text helper: removal notice, reconstruction swap, generic‑path restoration, plus the nullFlavor="NI" minimal‑section stub.
reconstruction.py — the new work, structured as:
1. Layer 1 (shared primitives):
  - render_typed_value — one place that branches over the closed CDA data‑type set (CD/PQ/ST/IVL_TS/PIVL_TS), handling both xsi:type‑tagged and monomorphic values.
  - extract_fields — reads a flat FieldSpec list off one anchor.
  - build_table — emits an XSD‑valid <text><table> with a block‑level "machine‑derived, not clinician‑attested" marker.
2. Layer 2 (data): per‑statement FieldSpec maps (PANEL_FIELDS, SPECIMEN_FIELDS, RESULT_FIELDS) — label, relative xpath, kind.
3. Layer 3 (per‑section joins): reconstruct_results — the only section implemented so far. The row is each result observation, joined up to its organizer (panel) and sideways to a sibling procedure (specimen).
4. Dispatch: a flat LOINC → reconstructor dict (SECTION_RECONSTRUCTORS, currently just 30954-2 Results). Adding a section is "one field map + one function + one dict entry."

The narrative action ("retain"/"remove"/"reconstruct", the DbNarrativeAction type) is configured per‑section and threaded straight through process_section → the matching engines. In entry_matching.py it's applied after prune+enrich:

On "reconstruct" it calls the pure reconstruct_narrative(section).
Falls back to a removal notice when there's no registered reconstructor or nothing survived.

The engine reports a narrative_disposition (retained/removed/reconstructed) that refine._interpret_run_result maps to the REFINED_NARRATIVE_RECONSTRUCTED outcome.

New unit suite test_service_narrative_reconstruction.py
New integration scenario covid_results_reconstruction (full eICR/RR fixtures + trace + test_narrative_reconstruction.py), wired into conftest.py and the scenario REPORT.md/build_report.py.
Snapshots regenerated across existing scenarios (footnote metadata changed).

Known Open Items (per your memory notes)

reconstruction.py flags the TODO for problems/immunizations/medications reconstructors.
There are deferred decisions on the unguarded retain+reconstruct combo and entry↔narrative linkage.

🔗 Related Issue

Fixes #1159

[USER STORY] Reconstruct Results section narrative #1159

✅ Acceptance Criteria

🧪 How to test

Integration tests

From the project root make sure you're in the refiner/ directory
Activate a virtual environment using your tool of choice
Make sure that both requirements.txt and dev-requirements.txt are installed
Run the scenario integration tests with pytest -vv tests/integration/scenarios/
Additionally you can check refined output in scenarios/snapshots/all_sections_covid_influenza/covid_results_reconstruction/ to see the raw output of the results being reconstructed

ℹ️ Additional Information

…tests pass

…e treatment

…for narrative reconstruciton

…t was also added to conftest

github-actions · 2026-06-13T00:18:29Z

🔒 Security Scan Results

⚠️ Found 15 vulnerabilities

Severity	Total
🟠 High	12
🟡 Medium	3

📦 refiner-app

✅ No vulnerabilities found

📦 refiner-lambda

Severity	Count
🟠 High	4

📦 refiner-ops

Severity	Count
🟠 High	8
🟡 Medium	3

View detailed results: Security tab
Last updated: 2026-06-22 20:52:57 UTC

rogeruiz

lgtm 🌈

Left a few comments on this. Great job on this @robertmitchellv!

rogeruiz · 2026-06-22T19:22:19Z

+          <text>
+            <!-- Narrative reconstructed by the eCR Refiner from surviving clinical entries: machine-derived, not clinician-attested. -->
+            <table border="1">
+              <thead>
+                <tr>
+                  <th>Panel</th>
+                  <th>Specimen</th>
+                  <th>Test</th>
+                  <th>Result</th>
+                  <th>Interpretation</th>
+                  <th>Date</th>
+                </tr>
+              </thead>
+              <tbody>
+                <tr>
+                  <td>SARS-CoV+SARS-CoV-2 (COVID-19) Ag [Presence] in Specimen</td>
+                  <td/>
+                  <td>SARS-CoV+SARS-CoV-2 (COVID-19) Ag [Presence] in Specimen</td>
+                  <td>Detected (qualifier value) (260373001)</td>
+                  <td>Abnormal (A)</td>
+                  <td>20201107</td>
+                </tr>
+              </tbody>
+            </table>
+            <footnote ID="ecr-refiner-30954-2-20260101000000">
+              <paragraph>
+                <content styleCode="Bold">eCR Refiner — Jurisdiction Configuration</content>
+              </paragraph>
+              <table border="1">
+                <thead>
+                  <tr>
+                    <th>Section (LOINC)</th>
+                    <th>Section Name</th>
+                    <th>Included</th>
+                    <th>Action</th>
+                    <th>Retain Narrative</th>
+                    <th>Config Version</th>
+                    <th>Source</th>
+                    <th>Outcome</th>
+                  </tr>
+                </thead>
+                <tbody>
+                  <tr>
+                    <td>30954-2</td>
+                    <td>Results Section</td>
+                    <td>Yes</td>
+                    <td>refine</td>
+                    <td>No</td>
+                    <td>v9</td>
+                    <td>Configured by jurisdiction</td>
+                    <td>Refined; narrative reconstructed</td>
+                  </tr>
+                </tbody>
+              </table>
+            </footnote>
+          </text>


rogeruiz · 2026-06-22T19:33:41Z

+    if el is None:
+        return ""
+
+    xsi_type = el.get(f"{{{_XSI}}}type")
+
+    # coded value (CD)--declared via xsi:type, or monomorphic (a coded
+    # element such as interpretationCode that carries @code with no xsi:type)
+    if xsi_type == "CD" or (xsi_type is None and el.get("code")):
+        disp, code = el.get("displayName"), el.get("code")
+        if disp and code:
+            return f"{disp} ({code})"
+        return disp or code or ""
+
+    # physical quantity (PQ)--declared via xsi:type, or monomorphic
+    # (doseQuantity and friends are PQ by the model)
+    if xsi_type == "PQ" or (xsi_type is None and el.get("unit") is not None):
+        val, unit = el.get("value"), el.get("unit")
+        return f"{val} {unit}".strip() if val else ""


This looks to be what I was expecting. The fallbacks are good and the cleverness around PQ is pretty neat.

rogeruiz · 2026-06-22T19:37:27Z

+# TODO:
+# next we will need to add problems, immunizations, and medications
+SECTION_RECONSTRUCTORS: dict[str, SectionReconstructor] = {
+    "30954-2": reconstruct_results,  # Results
+}


Thanks for this. I'll be migrating this over to refiner/app/services/ecr/policy.py so we're only defining these in one spot.

oh right i forgot i already added this!

robertmitchellv added 6 commits June 11, 2026 16:57

create narrative module and separate out seams; fix imports so local …

af60301

…tests pass

merge in main and fix a couple of conflicts

481364e

adding results reconstruction and connecting to new enum for narrativ…

8e0bb20

…e treatment

regenerate snapshots; footnote metadata changed + new scenario added …

d25604b

…for narrative reconstruciton

update the report builder, docs, and the report with new scenario tha…

0af47a9

…t was also added to conftest

add updated tests

2564f2a

robertmitchellv and others added 3 commits June 12, 2026 17:18

ruff format fix

40a2ab2

Merge branch 'main' into robert/1159-reconstruct-results

fb1ebc3

Merge branch 'main' into robert/1159-reconstruct-results

f5dd54c

robertmitchellv marked this pull request as ready for review June 17, 2026 19:35

robertmitchellv requested review from fzhao99, jakewheeler, lina-roth and rogeruiz as code owners June 17, 2026 19:35

rogeruiz self-assigned this Jun 18, 2026

rogeruiz approved these changes Jun 22, 2026

View reviewed changes

robertmitchellv added 2 commits June 22, 2026 13:25

Merge branch 'main' into robert/1159-reconstruct-results

716f499

Merge branch 'main' into robert/1159-reconstruct-results

b3ea54c

robertmitchellv added this pull request to the merge queue Jun 22, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 22, 2026

robertmitchellv added this pull request to the merge queue Jun 22, 2026

Merged via the queue into main with commit 5f133e4 Jun 22, 2026
21 checks passed

robertmitchellv deleted the robert/1159-reconstruct-results branch June 22, 2026 22:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(server): reconstruct the narrative#1369

feat(server): reconstruct the narrative#1369
robertmitchellv merged 11 commits into
mainfrom
robert/1159-reconstruct-results

robertmitchellv commented Jun 13, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jun 13, 2026 •

edited

Loading

Uh oh!

rogeruiz left a comment

Uh oh!

rogeruiz Jun 22, 2026

Uh oh!

rogeruiz Jun 22, 2026

Uh oh!

rogeruiz Jun 22, 2026

Uh oh!

robertmitchellv Jun 22, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

robertmitchellv commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔀 PULL REQUEST

💡 Summary

Known Open Items (per your memory notes)

🔗 Related Issue

✅ Acceptance Criteria

🧪 How to test

ℹ️ Additional Information

Uh oh!

github-actions Bot commented Jun 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔒 Security Scan Results

⚠️ Found 15 vulnerabilities

📦 refiner-app

📦 refiner-lambda

📦 refiner-ops

Uh oh!

rogeruiz left a comment

Choose a reason for hiding this comment

Uh oh!

rogeruiz Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

rogeruiz Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

rogeruiz Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

robertmitchellv Jun 22, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

robertmitchellv commented Jun 13, 2026 •

edited

Loading

github-actions Bot commented Jun 13, 2026 •

edited

Loading