feat: full HIPAA Safe Harbor coverage (all 18 identifiers), 80+ CUI Registry subcategories, regex bug fixes

Expand enhanced_pii.py to cover all 18 HIPAA Safe Harbor de-identification
identifiers per 45 CFR 164.514(b)(2), adding fax numbers, zip codes, vehicle
IDs, device IDs (UDI), web URLs, IP addresses, biometric identifiers,
full-face photo references, professional licenses, DEA numbers, ages over 89,
dates of death, and admission/discharge dates. Fix CAC number and EIN/TIN
regex patterns that failed to match natural-language variants.

Expand cui_detector.py from ~24 to 80+ CUI categories organized per the NARA
CUI Registry, covering critical infrastructure, defense, intelligence, law
enforcement, legal, privacy, financial, nuclear, patent, public health, safety,
statistical, technology, security, geospatial, and transportation groupings.
Add 10 new legacy marking patterns (CEII, UCNI, NNPI, FTI, SBIR, STTR,
COMSEC, SAFETY_ACT, CHEM/CFATS, DELIBERATIVE).

Fix sanitizer.py hidden XLSX sheet detection to handle both XML attribute
orderings. Fix export_control.py foreign person patterns to match plural forms.
Update audit_trail.py CEF version to 0.2.1. Add 37 new tests (213 total, all
passing). Update README and bump version to 0.2.1.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Author: marc-shade

README.md

@@ -6,12 +6,12 @@ Defense-grade document ingestion with CUI detection, ITAR/EAR export control scr
 
 | Framework | Standard | Coverage |
 |-----------|----------|----------|
-| CUI Program | 32 CFR Part 2002 | CUI marking detection, category validation, marking deficiency checks |
+| CUI Program | 32 CFR Part 2002 | CUI marking detection, 80+ category/subcategory validation, marking deficiency checks |
 | NIST 800-171 | SP 800-171 Rev 2 | 20+ security controls mapped (3.1.x, 3.3.x, 3.5.x, 3.8.x, 3.13.x) |
 | NIST 800-53 | SP 800-53 Rev 5 | AU, SI, AC, MP, SC control families |
 | ITAR | 22 CFR 120-130 | USML category detection (I-XXI), technical data screening |
 | EAR | 15 CFR 730-774 | ECCN pattern detection, CCL classification |
-| HIPAA | 45 CFR 160, 164 | PHI detection (MRN, health plan IDs, patient IDs, dates of service) |
+| HIPAA | 45 CFR 164.514(b)(2) | **All 18 Safe Harbor identifiers** -- full de-identification standard coverage |
 | Privacy Act | 5 USC 552a | PII detection with regulatory mapping |
 | FedRAMP | AU Family | Tamper-evident audit trail with SHA-256 hash chain, CEF export |
 | DFARS | 252.204-7012 | CUI protection requirements for defense contractors |
@@ -53,17 +53,45 @@ tests/
 
 ### CUI Detection (32 CFR Part 2002)
 - Detects CUI markings: `CUI`, `CUI//SP-xxx`, `CUI//REL TO`
-- Validates against CUI Registry categories (CTI, PRVCY, INTEL, EXPT, ITAR, etc.)
+- Validates against **80+ CUI Registry categories and subcategories** from the NARA CUI Registry, including:
+  - Critical Infrastructure: CTI, DCRIT, PCII, CEII, SSI
+  - Defense: ITAR, EXPT, SAMI, UCNI, NNPI, TFNI
+  - Intelligence: INTEL, FISA, HUMINT, SIGINT, GEOINT, OSINT, MASINT
+  - Law Enforcement: LES, LESI, GRAND_JURY, INFORMANT, WITNESS, SURVEIL
+  - Legal: LEGAL, ATTY_WORK, ATTY_CLIENT, DELIBERATIVE
+  - Privacy: PRVCY, PII, HIPAA, GENE, SORN, EDUCATIONAL, SUBSTANCE
+  - Financial: TAX, FTI, BANK_SECRECY, PROPIN, PROCUREMENT
+  - Nuclear: UCNI, NNPI, NNSA, NUCLEAR
+  - Science/Technology: SBIR, STTR, RESEARCH
+  - Security: OPSEC, COMSEC, PHYS, INFOSEC, VULN, PENTEST, INCIDENT
+- Detects 15+ legacy marking formats (FOUO, SBU, LES, SSI, PCII, CEII, UCNI, NNPI, FTI, COMSEC, etc.)
 - Detects classification banners: UNCLASSIFIED, CONFIDENTIAL, SECRET, TOP SECRET, TS//SCI
 - Detects dissemination controls: NOFORN, REL TO, ORCON, PROPIN, FISA, IMCON
 - Identifies marking deficiencies (missing banners, contradictory markings, legacy FOUO)
 - Generates handling recommendations per NIST 800-171
 - Risk scoring with NIST control mapping
 
 ### Enhanced PII/PHI Detection
-- **30+ detection categories** with confidence scoring (high/medium/low)
-- Standard PII: email, phone, SSN, credit card, DOB, driver's license, passport, address
-- HIPAA PHI: medical record numbers, health plan IDs, patient IDs, dates of service, diagnoses, prescriptions
+- **42+ detection categories** with confidence scoring (high/medium/low)
+- **Full HIPAA Safe Harbor coverage** -- all 18 identifier categories per 45 CFR 164.514(b)(2):
+  1. Names (via NER)
+  2. Geographic subdivisions (zip codes, addresses)
+  3. Dates (DOB, admission/discharge, death, ages >89)
+  4. Telephone numbers
+  5. Fax numbers
+  6. Email addresses
+  7. Social Security numbers
+  8. Medical record numbers
+  9. Health plan beneficiary numbers (including subscriber/group IDs)
+  10. Account numbers
+  11. Certificate/license numbers (driver's license, professional license, DEA)
+  12. Vehicle identifiers (VIN)
+  13. Device identifiers (UDI, serial numbers)
+  14. Web URLs
+  15. IP addresses
+  16. Biometric identifiers (fingerprint, voiceprint, retinal scan)
+  17. Full-face photographs (reference detection)
+  18. Other unique identifiers
 - Defense PII: DoD ID (EDIPI), CAC numbers, security clearance references, CAGE codes, DUNS numbers, SAM UEI
 - Financial PII: bank routing numbers, SWIFT/BIC codes, EIN/TIN, bank accounts, IBAN
 - Export control markers: ITAR markings, EAR markings, controlled technical data
@@ -356,6 +384,7 @@ pytest tests/ -v
 
 ## Version History
 
+- **0.2.1** -- Full HIPAA Safe Harbor coverage (all 18 identifiers), 80+ CUI Registry subcategories, regex bug fixes, 213 passing tests
 - **0.2.0** -- Defense compliance upgrade: CUI detection, enhanced PII/PHI, document sanitization, ITAR/EAR screening, FedRAMP audit trails
 - **0.1.34** -- Multi-format support, semantic compression, basic PII detection
 

docsingest/compliance/audit_trail.py

@@ -142,7 +142,7 @@ def to_cef(self) -> str:
         ext_str = ' '.join(extensions)
 
         return (
-            f"CEF:0|docsingest|ComplianceAudit|0.2.0|"
+            f"CEF:0|docsingest|ComplianceAudit|0.2.1|"
             f"{self.event_type}|{self.action}|{cef_severity}|{ext_str}"
         )
 

docsingest/compliance/cui_detector.py

@@ -21,32 +21,135 @@
 
 
 class CUICategory(Enum):
-    """CUI Registry categories per 32 CFR Part 2002."""
+    """CUI Registry categories per 32 CFR Part 2002 and NARA CUI Registry.
+
+    Organized by CUI Registry groupings from archives.gov/cui/registry/category-list.
+    Includes both CUI Basic and CUI Specified categories.
+    """
+    # --- Critical Infrastructure ---
     CTI = "Controlled Technical Information"
-    PRVCY = "Privacy"
-    INTEL = "Intelligence"
-    EXPT = "Export Controlled"
+    DCRIT = "Critical Infrastructure"
+    PCII = "Protected Critical Infrastructure Information"
+    CEII = "Critical Energy Infrastructure Information"
+    SSI = "Sensitive Security Information"
+
+    # --- Defense ---
     ITAR = "International Traffic in Arms Regulations"
+    EXPT = "Export Controlled"
+    SAMI = "Controlled Technical Information - Space"
+    NOFORN_DATA = "Not Releasable to Foreign Nationals Data"
+    UCNI = "Unclassified Controlled Nuclear Information"
+    NNPI = "Naval Nuclear Propulsion Information"
+    TFNI = "Transclassified Foreign Nuclear Information"
+    DoD_UCTI = "DoD Unclassified Controlled Technical Information"
+
+    # --- Export Control ---
+    EI = "Export Information"
+
+    # --- Financial ---
+    TAX = "Federal Taxpayer Information"
+    BUDGT = "Budget"
+    FTI = "Federal Tax Information"
+    BANK_SECRECY = "Bank Secrecy"
     PROPIN = "Proprietary Business Information"
+    PRIV_FIN = "Privileged Financial Information"
+    PROCUREMENT = "Source Selection Information"
+
+    # --- Immigration ---
+    IMMIG = "Immigration"
+    VISA = "Visa Information"
+
+    # --- Intelligence ---
+    INTEL = "Intelligence"
+    FISA = "Foreign Intelligence Surveillance Act"
+    HUMINT = "Human Intelligence"
+    SIGINT = "Signals Intelligence"
+    GEOINT = "Geospatial Intelligence"
+    OSINT = "Open Source Intelligence"
+    MASINT = "Measurement and Signature Intelligence"
+
+    # --- International Agreements ---
+    INTL_AGREE = "International Agreement Information"
+
+    # --- Law Enforcement ---
     LES = "Law Enforcement Sensitive"
+    LESI = "Law Enforcement Sensitive Investigation"
+    GRAND_JURY = "Grand Jury Information"
+    INFORMANT = "Confidential Informant Identity"
+    WITNESS = "Witness Protection Information"
+    SURVEIL = "Surveillance Information"
+    DEA_SENS = "DEA Sensitive Information"
+
+    # --- Legal ---
+    LEGAL = "Legal Privilege"
+    ATTY_WORK = "Attorney Work Product"
+    ATTY_CLIENT = "Attorney-Client Privilege"
+    DELIBERATIVE = "Deliberative Process"
+
+    # --- Natural & Cultural Resources ---
+    ARCH = "Archaeological Resource Information"
+    CULTURAL = "Cultural Resource Information"
+    SPECIES = "Endangered Species Information"
+
+    # --- Nuclear ---
+    OCA = "Original Classification Authority"
+    NNSA = "NNSA Information"
+    NUCLEAR = "Nuclear Security Information"
+
+    # --- Operations Security ---
+    OPSEC = "Operations Security"
+    COMSEC = "Communications Security"
+
+    # --- Patent ---
+    PATENT = "Patent Application Information"
+    INVENTION = "Invention Secrecy Act"
+
+    # --- Privacy ---
+    PRVCY = "Privacy"
+    PII = "Personally Identifiable Information"
+    HIPAA = "Health Insurance Portability and Accountability Act"
+    GENE = "Genetic Information"
+    SORN = "System of Records Notice Information"
+    EDUCATIONAL = "Student Educational Records (FERPA)"
+    SUBSTANCE = "Substance Abuse Treatment Records (42 CFR Part 2)"
+
+    # --- Provisional (Legacy) ---
     FOUO = "For Official Use Only"
     SBU = "Sensitive But Unclassified"
-    SSI = "Sensitive Security Information"
-    PCII = "Protected Critical Infrastructure Information"
+
+    # --- Public Health ---
     PHLTH = "Public Health"
-    TAX = "Federal Taxpayer Information"
-    LEGAL = "Legal Privilege"
-    OPSEC = "Operations Security"
+    SELECT_AGENT = "Select Agent and Toxin Information"
+    BSAT = "Biological Select Agents and Toxins"
+    PANDEMIC = "Pandemic Preparedness Information"
+
+    # --- Safety ---
+    SAFETY_ACT = "SAFETY Act Information"
+    CHEM = "Chemical Facility Anti-Terrorism Standards"
+
+    # --- Statistical ---
+    CENSUS = "Census"
+    CIPSEA = "CIPSEA Statistical Information"
+
+    # --- Technology & Science ---
+    SBIR = "Small Business Innovation Research"
+    STTR = "Small Business Technology Transfer"
+    RESEARCH = "Controlled Research Information"
+
+    # --- Physical & Information Security ---
     PHYS = "Physical Security"
     INFOSEC = "Information Systems Vulnerability Information"
-    BUDGT = "Budget"
-    CENSUS = "Census"
-    DCRIT = "Critical Infrastructure"
-    FISA = "Foreign Intelligence Surveillance Act"
-    GENE = "Genetic Information"
+    VULN = "Vulnerability Assessment Information"
+    PENTEST = "Penetration Testing Information"
+    INCIDENT = "Cybersecurity Incident Information"
+
+    # --- Geospatial ---
     GEO = "Geospatial"
-    PII = "Personally Identifiable Information"
-    SAMI = "Controlled Technical Information - Space"
+    GEO_PROD = "Geospatial Product Information"
+
+    # --- Transportation ---
+    SSTI = "Sensitive Surface Transportation Information"
+    RAIL = "Rail Security Information"
 
 
 class ClassificationLevel(Enum):
@@ -165,6 +268,16 @@ class CUIDetector:
         re.compile(r'\b(LES|LAW\s+ENFORCEMENT\s+SENSITIVE)\b', re.IGNORECASE): CUICategory.LES,
         re.compile(r'\b(SSI|SENSITIVE\s+SECURITY\s+INFORMATION)\b', re.IGNORECASE): CUICategory.SSI,
         re.compile(r'\b(PCII|PROTECTED\s+CRITICAL\s+INFRASTRUCTURE\s+INFORMATION)\b', re.IGNORECASE): CUICategory.PCII,
+        re.compile(r'\b(CEII|CRITICAL\s+ENERGY\s+INFRASTRUCTURE\s+INFORMATION)\b', re.IGNORECASE): CUICategory.CEII,
+        re.compile(r'\b(UCNI|UNCLASSIFIED\s+CONTROLLED\s+NUCLEAR\s+INFORMATION)\b', re.IGNORECASE): CUICategory.UCNI,
+        re.compile(r'\b(NNPI|NAVAL\s+NUCLEAR\s+PROPULSION\s+INFORMATION)\b', re.IGNORECASE): CUICategory.NNPI,
+        re.compile(r'\b(FTI|FEDERAL\s+TAX\s+INFORMATION)\b', re.IGNORECASE): CUICategory.FTI,
+        re.compile(r'\b(SBIR|SMALL\s+BUSINESS\s+INNOVATION\s+RESEARCH)\b', re.IGNORECASE): CUICategory.SBIR,
+        re.compile(r'\b(STTR|SMALL\s+BUSINESS\s+TECHNOLOGY\s+TRANSFER)\b', re.IGNORECASE): CUICategory.STTR,
+        re.compile(r'\b(COMSEC|COMMUNICATIONS\s+SECURITY)\b', re.IGNORECASE): CUICategory.COMSEC,
+        re.compile(r'\b(SAFETY\s+ACT\s+(?:PROTECTED|INFORMATION))\b', re.IGNORECASE): CUICategory.SAFETY_ACT,
+        re.compile(r'\b(CHEM[-\s]?SECURITY|CFATS|CHEMICAL\s+FACILITY\s+ANTI[-\s]?TERRORISM)\b', re.IGNORECASE): CUICategory.CHEM,
+        re.compile(r'\b(DELIBERATIVE\s+(?:PROCESS|PRIVILEGE))\b', re.IGNORECASE): CUICategory.DELIBERATIVE,
     }
 
     # Classification banner patterns