Skip to content

README.md

Latest commit

 

History

History
338 lines (241 loc) · 14 KB

README.md

File metadata and controls

338 lines (241 loc) · 14 KB
Raw

IDR Model

synthetic-data

The synthetic-data directory contains the synthetic data loaded into each of our environment's databases.

When adding new fields, take care to pass every CSV into the patient_generator.py (see below) or else the resulting data may be invalid.

Generating data

Downloading the FHIR validator is necessary to run the following scripts, along with installing sushi

To download the FHIR Validator: https://github.com/hapifhir/org.hl7.fhir.core/releases/download/6.7.10/validator_cli.jar

curl -L https://github.com/hapifhir/org.hl7.fhir.core/releases/download/6.7.10/validator_cli.jar > validator_cli.jar

Install sushi + fhirpath.js

# Check if npm is installed
npm --version

# If not then install
brew install npm
npm install -g fsh-sushi fhirpath

Install packages (via uv)

# Check if uv is installed
uv --version

# If not then install
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install dependencies
uv sync

Compile FSH Resources

To compile the .fsh files from this folder

cd sushi && sushi build && cd ..

This will generate the StructureDefinition and CodeSystem resources necessary for synthetic data generation. Running compile_resources.py is not necessary to generate synthetic data.

Get Matchbox up and running

To reduce dependencies on tx.fhir.org as well as improve the speed of validation, we use matchbox to run a local FHIR server. Read more about matchbox at https://ahdis.github.io/matchbox/

Note: Matchbox uses a good bit of memory. Allocating at least 8GB of RAM is recommended. When new version dependencies are added, the heap allocated required should be re-evaluated. The compose setup is optimized to GC aggressively to reduce the impact.

To start matchbox, run

docker compose up -d

Note, it takes several minutes and requires a good bit of RAM. Once the uploads from matchbox-setup are completed, matchbox should be ready to use. Additionally, matchbox has a health check available at:

curl -X GET "http://localhost:8080/matchboxv3/actuator/health"

Referencing New/Updated Dependencies for Matchbox

As new versions of IGs are released, they may have multiple nested dependencies. This takes up a significant amount of memory if loaded directly into Matchbox. To eliminate heap errors while still being able to accurately validate profiles and terminology, we download the FHIR Packages locally, untar them, and upload relevant resources directly to Matchbox. The list of resources + packages are in matchbox_profiles.txt. To add a new IG reference, follow the syntax in that file. Packages are only uploaded using docker compose up (by calling setup_matchbox.py), so restart the composition if adding more dependencies.

Generating Sample JSON from Synthetic CSVs

To generate or update sample data files from the generated synthetic CSVs (located in out/), you can use the sample generation scripts. This only exists for prior auth data for now - it will be expanded to other use cases gradually.

Prior Authorization Sample Generator

To generate a prior authorization JSON sample from the synthetic CSVs based on tracking number.

python generate_prior_auth_sample.py --utn=<utn-here>

This will search for the specified UTN in out/SYNTHETIC_PRAUC.csv, collect the segments, and get it into the format that we map using FML.

Create FHIR files with synthetic data

Requires Matchbox to be active.

To easily compile all resources:

./compile-all-resources.sh

To compile a specific resource:

Pass along map with -m pass along the sample file with -i pass along the output file with -o pass along the resource url with -r pass along --test to run conformance tests pass along --skip-structure-map-generation to skip generating the structure map. Only use this in the context of sequential transformations that re-use a structure map. pass along --profileType or -p to specify the profile type (Basis, Regular, or CMS) to filter for. Defaults to CMS.

Example (Patient):

uv run compile_resources.py \
    -m maps/patient.map \
    -i sample-data/Beneficiary-Sample.json \
    -o outputs/Patient.json \
    -r https://bfd.cms.gov/MappingLanguage/Maps/Patient \
    --test

Patient Data - patient_generator.py

patient_generator.py usage
usage: patient_generator.py [-h] [--patients PATIENTS] [--claims]
                            [--exclude-empty | --no-exclude-empty]
                            [--force-ztm-static-rows | --no-force-ztm-static-rows]
                            [files ...]

Generate synthetic patient data

positional arguments:
  files                 CSVs that will be regenerated/updated with new
                        columns. Updates are idempotent, meaning that passing
                        in an existing table/CSV without any new columns being
                        added to the synthetic data generation will result in
                        a byte-identical output file. Take care to avoid
                        providing a partial set of tables with foreign key
                        constraints (e.g. BENE_SK) without providing the root
                        table as this could result in broken output data

options:
  -h, --help            show this help message and exit
  --patients PATIENTS   Number of patients to generate. Ignored if
                        SYNTHETIC_BENE_HSTRY is provided via 'files'
  --claims              Automatically generate claims after patient generation
                        using the generated SYNTHETIC_BENE_HSTRY.csv file
  --exclude-empty, --no-exclude-empty
                        Treat empty column values as non-existant and allow
                        the generator to generate new values
  --force-ztm-static-rows, --no-force-ztm-static-rows
                        Allow "zero-to-many" rows (e.g. BENE_ENTLMT, c/d data,
                        etc.) for a patient loaded from a file to be
                        generated. This will introduce new rows for patients
                        that previously had none. Useful if not all tables for
                        a patient have been generated yet.
Generating patient data

To generate synthetic patient data, the patient_generator.py script is used. To utilize it to generate an entirely new set of data from nothing:

uv run patient_generator.py

Or, to load the v3 synthetic data (to add new fields):

uv run patient_generator.py synthetic-data/*.csv

NOTE: the bene_id column in SYNTHETIC_BENE_HSTRY.csv is to reference the bene_id field used in V1/V2. It's not used for sample data generation here.

The files output will be in the out folder:

  • SYNTHETIC_BENE_HSTRY.csv
  • SYNTHETIC_BENE_MBI_ID.csv
  • SYNTHETIC_BENE_MDCR_ENTLMT_RSN.csv
  • SYNTHETIC_BENE_MDCR_ENTLMT.csv
  • SYNTHETIC_BENE_MDCR_STUS.csv
  • SYNTHETIC_BENE_TP.csv
  • SYNTHETIC_BENE_XREF.csv
  • SYNTHETIC_BENE_CMBND_DUAL_MDCR.csv
  • SYNTHETIC_BENE_LIS.csv
  • SYNTHETIC_BENE_MAPD_ENRLMT_RX.csv
  • SYNTHETIC_BENE_MAPD_ENRLMT.csv

The patient generator creates synthetic beneficiary data with realistic but synthetic MBIs, coverage information, and historical records. It can generate multiple MBI versions per beneficiary and handles beneficiary cross-references with kill credit switches.

Claims data - claims_generator.py

Important

Synthetic claims data is much larger in size relative to patient data, and so it is not stored in the repository under ./synthetic-data. If you are looking to regnerate this data, please reach out in #bfd so that the existing dataset can be provided to you.

claims_generator.py usage

Usage: claims_generator.py [OPTIONS] [PATHS]...

  Generate synthetic claims data. Provided file PATHS will be updated with new
  fields.

Options:
  --sushi / --no-sushi            Generate new StructureDefinitions. Use when
                                  testing locally if new .fsh files have been
                                  added.
  --min-claims INTEGER            Minimum number of claims to generate per
                                  person
  --max-claims INTEGER            Maximum number of claims to generate per
                                  person
  --force-pac-claims / --no-force-pac-claims
                                  Generate _new_ partially-adjudicated claims
                                  when existing pac claims tables exist in the
                                  synthetic data provided
  --help                          Show this message and exit.

Generating claims data

Warning

Either SYNTHETIC_CLM.csv or SYNTHETIC_BENE_HSTRY.csv must be provided as claims data generation requires an existing BENE_SK or CLM to generate/regenerate data. It is recommended that SYNTHETIC_CNTRCT_PBP_NUM.csv also be provided so new claims use the same generated contracts. If not provided, newly generated claims will instead use randomly selected contract nums and pbp nums. This may result in claims referencing contracts that do not exist in the generated contract tables.

To generate synthetic claims data, the claims_generator.py script is used.

The synthetic claims data generated will be written to the ./out folder in the form of CSVs, one per-table:

  • SYNTHETIC_CLM.csv
  • SYNTHETIC_CLM_RLT_COND_SGNTR_MBR.csv
    • This file contains an extra column, CLM_UNIQ_ID, that is purely metadata used by the synthetic claims generator and is not consumed by the IDR Pipeline
  • SYNTHETIC_CLM_LINE.csv
  • SYNTHETIC_CLM_LINE_RX.csv
  • SYNTHETIC_CLM_VAL.csv
  • SYNTHETIC_CLM_DT_SGNTR.csv
  • SYNTHETIC_CLM_PROD.csv
  • SYNTHETIC_CLM_INSTNL.csv
  • SYNTHETIC_CLM_LINE_INSTNL.csv
  • SYNTHETIC_CLM_DCMTN.csv
  • SYNTHETIC_CLM_FISS.csv
  • SYNTHETIC_CLM_PRFNL.csv
  • SYNTHETIC_CLM_LINE_PRFNL.csv
  • SYNTHETIC_CLM_ANSI_SGNTR.csv
  • SYNTHETIC_PRVDR_HSTRY.csv
  • SYNTHETIC_CNTRCT_PBP_NUM.csv
  • SYNTHETIC_CNTRCT_PBP_CNTCT.csv

These files represent the schema of the tables the information is sourced from, although for tables other than CLM_DT_SGNTR, the CLM_UNIQ_ID is propagated instead of the 5 part unique key from the IDR.

Using SYNTHETIC_BENE_HSTRY.csv and SYNTHETIC_CNTRCT_PBP_NUM.csv

The below will generate entirely new claims for the given BENE_SKs in the provided file:

uv run claims_generator.py \
    --sushi \
    out/SYNTHETIC_BENE_HSTRY.csv out/SYNTHETIC_CNTRCT_PBP_NUM.csv
Regenerating existing claims data

The below will re-generate existing claims data (assume <PATH_TO_CLAIMS_DATA> is a local directory containing synthetic claims data):

uv run claims_generator.py \
    --sushi \
    ./synthetic-data <PATH_TO_CLAIMS_DATA>

If any claims-related tables have had columns added to their respective generation functions, those new columns will be populated with values without impacting existing values in other columns.

Caution

If an existing column value must be updated, that column value MUST BE DELETED from the respective table CSV first so that the values can be regenerated.

--sushi

--sushi is not strictly needed, if you have a local copy of the compiled shorthand files, but recommended to reduce drift. To specify a list of benes, pass in a .csv file containing a column named BENE_SK.

Testing Mapping Changes

Verifying FML Map Changes

To test updates to your FML map files, run compile_resources.py to generate a resource in the out/ directory and verify the output:

uv run compile_resources.py \
    -m maps/ExplanationOfBenefit-Base.map \
    -i sample-data/EOB-Carrier-MCS-Sample.json \
    -o out/ExplanationOfBenefit-MCS.json \
    -r https://bfd.cms.gov/MappingLanguage/Maps/ExplanationOfBenefit-Base \
    --test

Updating Structure Definitions

FHIR Mapping Language is used to go from source to target. The source data structure must be defined, and we define them using FHIR Shorthand (https://hl7.org/fhir/uv/shorthand/N2/). When a new source field is added, add it to the relevant .fsh file (or create a new one) in the sushi/input folder. StructureDefinitions for input data structures should follow the naming convention of StructureDefinition-.fsh

Resource Augmentation

Due to FML limitations, some complex mappings that require more than simple lookups are handled via augment_sample_resources.py. For example, CLM_AUDT_TRL_STUS_CD on an EOB is derived by combining the status code, location code (CLM_AUDT_TRL_LCTN_CD), and source (META_SRC_SK). The augmentation script resolves these fields into the claim status code. To test augmentation logic independently:

python augment_sample_resources.py {your_sample_file}.json [Basis|Regular|CMS]

Note

The uv run compile_resources.py command mentioned above also executes this script. You can inspect the output at out/temporary-sample.json and the compiled resource.

Data Dictionary

Generally, the data dictionary will source definitions from the IDR's table definitions. There are instances where this may not be the definition we wish to publish. To overwrite the definition from the IDR, or populate a definition not available from the IDR, populate the "definition" key for the relevant concept in the relevant StructureDefinition.

Sometimes a field may be condensed at the IDR level, and fanned into multiple discrete components at the BFD / FHIR layer. An example is BENE_MDCR_STUS_CD. This code can indicate several interesting characteristics, such as ESRD status and disability status. A field, nameOverride, is available to directly populate names in the BFD DD for these fields that do not surface through a StructureDefinition. To generate the data dictionary:

./compile-all-resources.sh
uv run gen_dd.py

If the gen_dd.py script produces warnings about missing tables or columns, run the following query to retrieve the latest updates for the affected table from IDR. Run:

DESCRIBE VIEW CMS_VDM_VIEW_MDCR_PRD.{TABLE_NAME}

Export the results as a CSV named {TABLE_NAME}.csv and save it under ReferenceTables.