BFD-4193: Restructure bfd-model-idr, set up formatting/linting, and add CI

.github/workflows/ci-pipeline-synthetic.yml

@@ -0,0 +1,35 @@
+name: CI - Synthetic Pipeline
+
+on:
+  pull_request:
+    paths:
+      - "apps/bfd-model-idr/*"
+      - "apps/bfd-pipeline/bfd-pipeline-idr/*"
+      - ".github/workflows/ci-pipeline-synthetic.yml"
+  merge_group:
+
+jobs:
+  run-synthetic:
+    runs-on: ubuntu-24.04
+    steps:
+    - name: Checkout code
+      id: code-checkout
+      uses: actions/checkout@v4
+    - name: Install uv
+      uses: astral-sh/setup-uv@v5
+      with:
+          version: "0.8.10"
+    - name: Install node
+      uses: actions/setup-node@v4
+      with:
+        node-version: '22'
+    - name: Install dependencies
+      run: |
+        npm install -g fsh-sushi
+        cd ${{ github.workspace }}/apps/bfd-model-idr
+        sushi build ./sushi
+        uv run patient_generator.py
+        uv run claims_generator.py --benes ./out/SYNTHETIC_BENE_HSTRY.csv
+        cd ../bfd-pipeline/bfd-pipeline-idr
+        ./run-db.sh
+        uv run pipeline.py local

apps/.gitignore

@@ -9,11 +9,5 @@
 **/nb-configuration.xml
 **/nbactions.xml
 
-#ignore artifacts from FHIR Mapping Language
-bfd-model/bfd-model-idr/StructureMaps/*
-bfd-model/bfd-model-idr/outputs/*
-bfd-model/bfd-model-idr/sushi/fsh-generated/*
-bfd-model/bfd-model-idr/validator_cli.jar
-
 # Ignore Python bytecode files
 *.pyc

apps/bfd-model-idr/.gitignore

@@ -0,0 +1,6 @@
+#ignore artifacts from FHIR Mapping Language
+StructureMaps/*
+out/*
+!out/.gitkeep
+sushi/fsh-generated/*
+validator_cli.jar

apps/bfd-model/bfd-model-idr/README.md → apps/bfd-model-idr/README.md

@@ -1,15 +1,16 @@
-### Welcome to the README!
-
-This file will be extended, but for this step, we're consolidating multiple steps into a single script. 
+# IDR Model
 
 Downloading the FHIR validator is necessary to run the following scripts, along with installing sushi
 
 To download the FHIR Validator:
 https://github.com/hapifhir/org.hl7.fhir.core/releases/latest/download/validator_cli.jar
-  - place the jar file in the `beneficiary-fhir-data/apps/bfd-model/bfd-model-idr` directory
-  - ensure the name file name is `validator_cli.jar`
 
-#### Install sushi + fhirpath.js + yaml
+```sh
+curl -L https://github.com/hapifhir/org.hl7.fhir.core/releases/latest/download/validator_cli.jar > validator_cli.jar
+```
+
+## Install sushi + fhirpath.js
+
 ```sh
 # Check if npm is installed
 npm --version
@@ -19,10 +20,10 @@ brew install npm
 ```
 
 ```sh
-npm install -g fsh-sushi yaml fhirpath
+npm install -g fsh-sushi fhirpath
 ```
 
-#### Install packages  (via uv)
+## Install packages  (via uv)
 ```sh
 # Check if uv is installed
 uv --version
@@ -36,18 +37,24 @@ curl -LsSf https://astral.sh/uv/install.sh | sh
 uv sync
 ```
 
-### Create FHIR files with synthetic data
+## Create FHIR files with synthetic data
+
+To easily compile all resources:
 
-EOB Institutional Inpatient:
 ```sh
-uv run compile_resources.py \
-    -m maps/EOB-Base.map \
-    -i sample-data/EOB-Base-Sample.json \
-    -o outputs/ExplanationOfBenefit.json \
-    -r https://bfd.cms.gov/MappingLanguage/Maps/ExplanationOfBenefit-Base \
-    --test
+./compile-all-resources.sh
 ```
-Patient:
+
+To compile a specific resource:
+
+Pass along map with -m
+pass along the sample file with -i
+pass along the output file with -o
+pass along the resource url with -r
+pass along --test to run conformance tests
+
+Example (Patient):
+
 ```sh
 uv run compile_resources.py \
     -m maps/patient.map \
@@ -56,36 +63,24 @@ uv run compile_resources.py \
     -r https://bfd.cms.gov/MappingLanguage/Maps/Patient \
     --test
 ```
-Coverage (part A/B):
-```sh
-uv run compile_resources.py \
-    -m maps/Coverage-Base.map \
-    -i sample-data/Coverage-FFS-Sample.json \
-    -o outputs/Coverage-FFS.json \
-    -r https://bfd.cms.gov/MappingLanguage/Maps/Coverage-Base \
-    --test
-```
 
-Coverage (part C/D):
-```sh
-uv run compile_resources.py \
-    -m maps/Coverage-Base.map \
-    -i sample-data/Coverage-PartC-Sample.json \
-    -o outputs/Coverage.json \
-    -r https://bfd.cms.gov/MappingLanguage/Maps/Coverage-Base \
-    --test
-```
+### Synthetic data generation
 
+Quick start:
 
-Pass along map with -m
-pass along the sample file with -i
-pass along the output file with -o
-pass along the resource url with -r
-pass along --test to run conformance tests
+```sh
+uv run patient_generator.py
+
+uv run claims_generator.py \
+    --sushi \
+    --benes out/SYNTHETIC_BENE_HSTRY.csv
+```
 
+#### Patient Data
 
 To generate synthetic patient data, the patient_generator.py script is used.
 To utilize it:
+
 ```sh
 uv run patient_generator.py
 ```
@@ -109,6 +104,8 @@ SYNTHETIC_BENE_MAPD_ENRLMT.csv
 
 The patient generator creates synthetic beneficiary data with realistic but SYNTHETIC MBIs, coverage information, and historical records. It can generate multiple MBI versions per beneficiary and handles beneficiary cross-references with kill credit switches.
 
+#### Claims data
+
 To generate synthetic claims data, the claims_generator.py script is used. 
 To utilize it:
 ```sh
@@ -134,10 +131,13 @@ SYNTHETIC_CLM_ANSI_SGNTR.csv
 
 These files represent the schema of the tables the information is sourced from, although for tables other than CLM_DT_SGNTR, the CLM_UNIQ_ID is propagated instead of the 5 part unique key from the IDR.
 
-Data Dictionary Notes:
+### Data Dictionary
+
 Generally, the data dictionary will source definitions from the IDR's table definitions. There are instances where this may not be the definition we wish to publish. To overwrite the definition from the IDR, or populate a definition not available from the IDR, populate the "definition" key for the relevant concept in the relevant StructureDefinition. 
 
 Sometimes a field may be condensed at the IDR level, and fanned into multiple discrete components at the BFD / FHIR layer. An example is BENE_MDCR_STUS_CD. This code can indicate several interesting characteristics, such as ESRD status and disability status. A field, nameOverride, is available to directly populate names in the BFD DD for these fields that do not surface through a StructureDefinition. 
 To generate the data dictionary:
-python gen_dd.py
 
+```sh
+uv run gen_dd.py
+```

apps/bfd-model/bfd-model-idr/StructureDefinitions/Source/LineItemComponent.json → apps/bfd-model-idr/StructureDefinitions/Source/LineItemComponent.json

@@ -674,18 +674,6 @@
                     }
                 ]
             },
-            {
-                "id": "LineItemComponent.CLM_LINE_FROM_DT",
-                "path": "LineItemComponent.CLM_LINE_FROM_DT",
-                "label": "First Expense Date",
-                "min": 0,
-                "max": "1",
-                "type": [
-                    {
-                        "code": "string"
-                    }
-                ]
-            },
             {
                 "id": "LineItemComponent.CLM_LINE_INGRDNT_CST_AMT",
                 "path": "LineItemComponent.CLM_LINE_INGRDNT_CST_AMT",