Babel is switching its Publication compendium to this tool's NDJSON export (NCATSTranslator/Babel branch incorporate-pubmed2db). One thing Babel's own XML parser produced that the export has no equivalent for is publication status: for every PMID it collected every PubStatus attribute found on any <PubMedPubDate> across the baseline and update files, and wrote them as {"id": "PMID:1234", "statuses": ["accepted", "pubmed", "retracted", ...]} one line per PMID. The code is at https://github.com/NCATSTranslator/Babel/blob/a3ae3e4d/src/createcompendia/publications.py#L188-L283 (the pubmed_status / pubdates_with_pubstatus lines), for reference; nothing downstream ever read the file, so Babel dropped it rather than keep it alive on the new path.
The goal it was reaching for is a per-version record with a status, so a consumer can ask whether the PMID it holds has been retracted, updated or superseded — e.g. PMID:1234.1 "Updated", PMID:1234.2 "Latest", PMID:4321 "Retracted". pubmed2db already has most of the raw material: article keeps every version keyed by (pmid, source_file) with file_order_key ordering, history holds the PubMedPubDate statuses, deleted_pmid records deletions, and publication_type would show "Retraction of Publication" / "Retracted Publication". What is missing is an export that surfaces it — either extra fields on the metadata record (e.g. statuses, version, is_latest, retracted) or a separate per-version export — and a decision on how a version is named.
Filing this so the design can happen here, where the data is; the Babel side (NCATSTranslator/Babel issue linked below) will be written against whatever this exports rather than reviving the XML-era code.
Babel is switching its Publication compendium to this tool's NDJSON export (NCATSTranslator/Babel branch
incorporate-pubmed2db). One thing Babel's own XML parser produced that the export has no equivalent for is publication status: for every PMID it collected everyPubStatusattribute found on any<PubMedPubDate>across the baseline and update files, and wrote them as{"id": "PMID:1234", "statuses": ["accepted", "pubmed", "retracted", ...]}one line per PMID. The code is at https://github.com/NCATSTranslator/Babel/blob/a3ae3e4d/src/createcompendia/publications.py#L188-L283 (thepubmed_status/pubdates_with_pubstatuslines), for reference; nothing downstream ever read the file, so Babel dropped it rather than keep it alive on the new path.The goal it was reaching for is a per-version record with a status, so a consumer can ask whether the PMID it holds has been retracted, updated or superseded — e.g.
PMID:1234.1"Updated",PMID:1234.2"Latest",PMID:4321"Retracted". pubmed2db already has most of the raw material:articlekeeps every version keyed by(pmid, source_file)withfile_order_keyordering,historyholds thePubMedPubDatestatuses,deleted_pmidrecords deletions, andpublication_typewould show "Retraction of Publication" / "Retracted Publication". What is missing is an export that surfaces it — either extra fields on the metadata record (e.g.statuses,version,is_latest,retracted) or a separate per-version export — and a decision on how a version is named.Filing this so the design can happen here, where the data is; the Babel side (NCATSTranslator/Babel issue linked below) will be written against whatever this exports rather than reviving the XML-era code.