feat(clinical_report): add FDA NME compilation as a provider

[ireneisdoomed]

Summary

This PR adds a connector to FDA NME compilation:

This dataset provides a listing of all New Molecular Entities (NMEs) approved from 1985 – 2025 and regulated by the Center for Drug Evaluation and Research (CDER). The listed NMEs include both small molecule drugs approved under a New Drug Application (NDA) and new biological products approved under a Biologics License Application (BLA). The product information, indications, and regulatory characteristics described in the compilation reflect the state of each application at the time of original marketing approval.

Notes

The most important thing to know here is that this is:

• A good source for keeping track of new drugs approved in the US
• A bad source for keeping track of approvals for existing drugs

In the input data there are interesting fields I am not capturing, most importantly if it is an orphan drug or if it went through accelerated approval. Should I include this?

The intention of the PR is to add the connector. To include that as part of a release process, changes in pipeline are necessary.

Data metrics

Despite this fact, I thought it would be interesting to know if it was worth integrating, since the data comes in an Excel format very similar to the EMA Human Drugs data. It would be easy to ingest and it is updated every day of the week.

Here's a summary of the results:

FDA NME metric	Result
Approval reports	1,292
Unique application IDs	1,292
Reports with a mapped drug	1,284 (99.4%)
Reports with a mapped disease	1,184 (91.6%)
Reports with both mapped	1,176 (91.0%)
Reports with neither mapped	0
Unique mapped drug IDs	1,314
Unique mapped disease IDs	620
Approval years populated	1,292 (100%)
Year range	1985–2025

Approval coverage

The main sources to compare this data with are ChEMBL's FDA and Dailymed. We have a low coverage of approved drugs where provider = ChEMBL + source = FDA (541). However, provider = ChEMBL + source = Dailymed (1,482) is not bad in terms of drug coverage (unclear how up-to-date we are in terms of drug/indication pairs).

Top 2 diagrams is a comparison against ChEMBL data / bottom represents other approvals

Even though the overlap is big, I think this dataset is interesting because:

1. It is an up-to-date source of approved medicines in the US that is easy to maintain
2. Approval date is timestamped - something we are complety missing for ChEMBL references
3. Drug/disease mapping coverage is really good - ensuring that we are processing almost the entirety of this resource

Approval timestamp coverage

• ChEMBL FDA/DailyMed: none of the 878 overlapping drugs or 383 overlapping drug–disease pairs has a year or start date. FDA NME therefore supplies the timestamp for all of them.
• All other approval sources: 726 of 1,207 overlapping drugs are timestamped elsewhere (60.1%); 481 only receive a timestamp from FDA NME.
• At drug–disease level: 607 of 995 overlapping pairs are timestamped elsewhere (61.0%); 388 only receive a timestamp from FDA NME.

Example clinical reports

1. elamipretide and Barth syndrome - covered by trials but approval was missing

[{'id': 'nda215244',
  'clinicalStage': 'APPROVAL',
  'origin': 'REGULATORY_AGENCY',
  'type': 'INDICATION',
  'source': 'FDA NME Compilation',
  'provider': 'FDA',
  'drugs': [{'drugFromSource': 'elamipretide', 'drugId': 'CHEMBL3833370'}],
  'trialDescription': None,
  'trialOverallStatus': None,
  'year': 2025,
  'trialPrimaryPurpose': None,
  'trialLiterature': None,
  'trialSponsor': None,
  'trialPhase': None,
  'countries': None,
  'trialWhyStopped': None,
  'phaseFromSource': 'NDA',
  'trialNumberOfArms': None,
  'diseases': [{'diseaseFromSource': 'barth syndrome',
    'diseaseId': 'MONDO_0010543'}],
  'url': 'https://www.accessdata.fda.gov/scripts/cder/daf/index.cfm?event=overview.process&ApplNo=215244',
  'sideEffects': None,
  'trialStudyType': None,
  'trialStartDate': None,
  'trialDetailedDescription': None,
  'trialOfficialTitle': None}]

2. sofpironium and primary axillary hyperhidrosis - new drug with an unknown clinical status; unfortunately disease is not mapped

[{'id': 'nda217347',
  'clinicalStage': 'APPROVAL',
  'origin': 'REGULATORY_AGENCY',
  'type': 'INDICATION',
  'source': 'FDA NME Compilation',
  'provider': 'FDA',
  'drugs': [{'drugFromSource': 'sofpironium', 'drugId': 'CHEMBL3707224'}],
  'trialDescription': None,
  'trialOverallStatus': None,
  'year': 2024,
  'trialPrimaryPurpose': None,
  'trialLiterature': None,
  'trialSponsor': None,
  'trialPhase': None,
  'countries': None,
  'trialWhyStopped': None,
  'phaseFromSource': 'NDA',
  'trialNumberOfArms': None,
  'diseases': [{'diseaseFromSource': 'primary axillary hyperhidrosis',
    'diseaseId': None}],
  'url': 'https://www.accessdata.fda.gov/scripts/cder/daf/index.cfm?event=overview.process&ApplNo=217347',
  'sideEffects': None,
  'trialStudyType': None,
  'trialStartDate': None,
  'trialDetailedDescription': None,
  'trialOfficialTitle': None}]

3. gemifloxacin mesylate and bronchitis - latest drug indication for a drug we didn't have any clinical precedence for

[{'id': 'nda21158',
  'clinicalStage': 'APPROVAL',
  'origin': 'REGULATORY_AGENCY',
  'type': 'INDICATION',
  'source': 'FDA NME Compilation',
  'provider': 'FDA',
  'drugs': [{'drugFromSource': 'gemifloxacin mesylate',
    'drugId': 'CHEMBL1200621'}],
  'trialDescription': None,
  'trialOverallStatus': None,
  'year': 2003,
  'trialPrimaryPurpose': None,
  'trialLiterature': None,
  'trialSponsor': None,
  'trialPhase': None,
  'countries': None,
  'trialWhyStopped': None,
  'phaseFromSource': 'NDA',
  'trialNumberOfArms': None,
  'diseases': [{'diseaseFromSource': 'community - acquired pneumonia',
    'diseaseId': None},
   {'diseaseFromSource': 'bronchitis', 'diseaseId': 'MONDO_0003781'}],
  'url': 'https://www.accessdata.fda.gov/scripts/cder/daf/index.cfm?event=overview.process&ApplNo=21158',
  'sideEffects': None,
  'trialStudyType': None,
  'trialStartDate': None,
  'trialDetailedDescription': None,
  'trialOfficialTitle': None}]

Caveats

The format of this dataset is very easy to ingest, very similar to EMA: except for how the indication is reported. indicationText contains a brief sentence with the main description of the drug intended use coming from the drug label. This means that this field is free text, therefore cannot be mapped directly.

How I partially addressed this is by doing NER on this free text field. This worked relatively well: 91% of FDA references have a disease mapped. However, because the NER identifies conditions but lacks context, sometimes we extract the background condition as being associated with the drug.

That is what is happening here for Verteporfin/choroidal neovascularisation. That is a symptom of the particular cohort of patients with macular degeneration, not the primary indication. I've reviewed the examples and don't see anything blatantly wrong, so I believe the gain is positive. To address this, we need a model for therapeutic relationships. I think that developing this is possible and within our scope.

[{'id': 'nda21119',
  'clinicalStage': 'APPROVAL',
  'origin': 'REGULATORY_AGENCY',
  'type': 'INDICATION',
  'source': 'FDA NME Compilation',
  'provider': 'FDA',
  'drugs': [{'drugFromSource': 'verteporfin', 'drugId': 'CHEMBL5314346'}],
  'trialDescription': None,
  'indicationText': 'treatment of age-related macular degeneration in patients with predominantly classic subfoveal choroidal neovascularization',
  'trialOverallStatus': None,
  'year': 2000,
  'trialPrimaryPurpose': None,
  'trialLiterature': None,
  'trialSponsor': None,
  'trialPhase': None,
  'countries': None,
  'trialWhyStopped': None,
  'phaseFromSource': 'NDA',
  'trialNumberOfArms': None,
  'diseases': [{'diseaseFromSource': 'choroidal neovascularization',
    'diseaseId': 'MONDO_0810000'},
   {'diseaseFromSource': 'macular degeneration',
    'diseaseId': 'MONDO_0003004'}],
  'url': 'https://www.accessdata.fda.gov/scripts/cder/daf/index.cfm?event=overview.process&ApplNo=21119',
  'sideEffects': None,
  'trialStudyType': None,
  'trialStartDate': None,
  'trialDetailedDescription': None,
  'trialOfficialTitle': None}]

[ireneisdoomed]

Here's the disease label distribution, showing that most cases involve <3 disease labels:

┌────────────┬─────────────────────┐
│ statistic  ┆ disease_label_count │
│ ---        ┆ ---                 │
│ str        ┆ f64                 │
╞════════════╪═════════════════════╡
│ count      ┆ 1292.0              │
│ null_count ┆ 0.0                 │
│ mean       ┆ 2.445046            │
│ std        ┆ 1.872608            │
│ min        ┆ 1.0                 │
│ 25%        ┆ 1.0                 │
│ 50%        ┆ 2.0                 │
│ 75%        ┆ 3.0                 │
│ max        ┆ 20.0                │
└────────────┴─────────────────────┘

Here's the most extreme case:

[{'id': 'nda20760',
  'clinicalStage': 'APPROVAL',
  'origin': 'REGULATORY_AGENCY',
  'type': 'INDICATION',
  'source': 'FDA NME Compilation',
  'provider': 'FDA',
  'drugs': [{'drugFromSource': 'alatrofloxacin mesylate',
    'drugId': 'CHEMBL1200498'}],
  'indicationText': 'indicated for the treatment of patients with mild to moderate infections caused by susceptible strains of designated microorganisms in the following conditions: nosocomial pneumonia, community acquired pneumonia, acute bacterial exacerbation of chronic bronchitis, acute sinusitis, complicated intra-abdominal infections including post-surgical infections, gynecologic and pelvic infections including endomyometritis, parametritis, septic abortion and post-partum infections, prophylaxis of infection associated with elective colorectal surgery, vaginal and abdominal hysterectomy, uncomplicated skin and skin structure infection including diabetic foot infections, uncomplicated urinary tract infections, chronic bacterial prostatitis, uncomplicate urethral gonorrhea in males and endocervical and rectal gonorrhea in females, pelvic inflammatory disease',
  'year': 1997,
  'phaseFromSource': 'NDA',
  'diseases': [{'diseaseFromSource': 'diabetic foot infections',
    'diseaseId': None},
   {'diseaseFromSource': 'gynecologic and pelvic infections',
    'diseaseId': None},
   {'diseaseFromSource': 'endomyometritis', 'diseaseId': 'MONDO_0042451'},
   {'diseaseFromSource': 'intra - abdominal infections', 'diseaseId': None},
   {'diseaseFromSource': 'pelvic inflammatory disease',
    'diseaseId': 'MONDO_0000922'},
   {'diseaseFromSource': 'bronchitis', 'diseaseId': 'MONDO_0003781'},
   {'diseaseFromSource': 'skin and skin structure', 'diseaseId': None},
   {'diseaseFromSource': 'bacterial prostatitis', 'diseaseId': None},
   {'diseaseFromSource': 'post - partum infections', 'diseaseId': None},
   {'diseaseFromSource': 'nosocomial pneumonia', 'diseaseId': None},
   {'diseaseFromSource': 'endocervical and rectal gonorrhea',
    'diseaseId': None},
   {'diseaseFromSource': 'infections', 'diseaseId': 'EFO_0000544'},
   {'diseaseFromSource': 'parametritis', 'diseaseId': 'MONDO_0006887'},
   {'diseaseFromSource': 'post - surgical infections', 'diseaseId': None},
   {'diseaseFromSource': 'community acquired pneumonia', 'diseaseId': None},
   {'diseaseFromSource': 'urinary tract infections',
    'diseaseId': 'MONDO_0100338'},
   {'diseaseFromSource': 'infection', 'diseaseId': 'EFO_0000544'},
   {'diseaseFromSource': 'sinusitis', 'diseaseId': 'MONDO_0005961'},
   {'diseaseFromSource': 'septic abortion', 'diseaseId': 'EFO_1001177'},
   {'diseaseFromSource': 'urethral gonorrhea', 'diseaseId': None}],
  'url': 'https://www.accessdata.fda.gov/scripts/cder/daf/index.cfm?event=overview.process&ApplNo=20760',

The extracted diseases are genuine indications, not comorbidities. The only wrong condition I see is skin and skin structure but this is NER's fault.

[d0choa]

Pulled the compilation down and ran your parser over it to check a few things. It holds up well — the ingredient splitting in particular is clearly written against this file: all 69 multi-ingredient rows are genuine combinations and come out right, including the awkward ones (imipenem, cilastatin, and relebactam, ombitasvir, paritaprevir, ritonavir; dasabuvir (co-packaged)). Nice.

One real gap, plus a few notes.

Secondary application numbers are dropped (provider/fda.py:67). id and url are built from Application Number(1) only, but in the current file 59 rows populate Application Number(2) and 6 populate (3). Those applications vanish silently. Worth either expanding to a row per application number, or capturing them so the link back to Drugs@FDA is complete.

The input should come through PIS. config.yaml:37 points at a year-stamped file under $HOME, so today this only runs on your machine. There's a stable URL, so it can be a plain copy step like copy aact dump:

- name: copy fda nme compilation
  source: https://www.fda.gov/media/177921/download?attachment
  destination: input/clinical_report/fda_nme.xlsx

That URL returns exactly the file your config names. It's served as .xlsx despite the page calling it CSV, so pl.read_excel is correct.

Please don't route it via gs://ot-team/irene/clinical_mining/... instead. That glob is the easy path, but personal buckets aren't collectively managed and become a single point of failure — PIS already carries a # TODO: move other clinical report files to team bucket about exactly this, and I'd rather not add a new source to it.

On "updated every day of the week" — the FDA page says updates "will be made annually after the new approvals are confirmed with the system of record". I think that's Drugs@FDA being conflated in. Doesn't change the value here (the timestamps are the point), but worth pinning a version rather than relying on freshness. FWIW the file is 1,387 rows today against the 1,292 in your description, so it has moved since your run.

On orphan drug / accelerated approval — yes, worth capturing, and the columns are already sitting in the sheet: Orphan Drug Designation is 461 Yes / 916 No, Accelerated Approval 152 Yes / 1045 No / 182 N/A. One catch: both carry values like "Yes (indication [B] only)" (10 and 8 rows respectively), so they're per-indication rather than per-drug — a plain boolean on the drug would be wrong for those. Probably its own PR given that.

On the background-condition NER — agree, merge as-is. The verteporfin case is a precision cost rather than a correctness bug, and I think you're right that the net gain is positive.

As we discussed, a good way to put a number on that: take the drugs that appear both here and in ChEMBL/DailyMed and use an LLM judge to check whether the conditions line up. The FDA indications come from NER over free text while DailyMed's are curated, so the overlap is a natural precision test for the extraction — and it would turn the background-condition question into a measurement rather than spot checks.