Add TES API v2.0.0 support with backward compatibility

[Copilot]

Overview

Implements support for TES API version 2.0.0 while maintaining full backward compatibility with v1.0.0. The new API version introduces significant structural changes that required updates to filtering logic, code extraction, and resource grouping.

Problem

The TES API v2.0.0 introduces breaking changes:

1. Filtering fails: v2.0.0 ValueSets have numeric IDs (e.g., "1995") instead of "rs-grouper-*" format
2. Multiple entries per condition: v2.0.0 returns 6 separate ValueSets per condition instead of 1
3. Mixed structure: Main groupers use ValueSet references while additional context groupers contain direct codes
4. New grouper types: Additional context groupers have different useContext codes

Solution

Key Changes

1. Version Parameter Support

   # Use v1.0.0 (default)
   python seed_terminology_db.py
   
   # Use v2.0.0
   python seed_terminology_db.py 2.0.0
   # or
   TES_API_VERSION=2.0.0 python seed_terminology_db.py

2. Smart Filtering Logic

   • v1.0.0: ID prefix matching (rs-grouper-*)
   • v2.0.0: useContext-based filtering (condition-grouper, additional-context-grouper)

3. Enhanced Code Extraction

   • Handles ValueSet references (main groupers) - logged but skipped for now
   • Processes direct concept codes (additional context groupers)
   • Combines codes from multiple ValueSets per condition

4. Multi-ValueSet Grouping

   • Groups related ValueSets by condition (typically 6 per condition in v2.0.0)
   • Merges codes from all ValueSets into single database entries
   • Preserves main grouper titles over additional context titles

Implementation Details

# New filtering logic
def is_relevant_grouper(self, resource):
    if self.api_version == "1.0.0":
        return resource.get("id", "").startswith("rs-grouper")
    else:
        # v2.0.0: Check useContext for grouper types
        use_context = resource.get("useContext", [])
        for context in use_context:
            if context.get("code", {}).get("code") == "task":
                codings = context.get("valueCodeableConcept", {}).get("coding", [])
                for coding in codings:
                    if coding.get("code") in ["condition-grouper", "additional-context-grouper"]:
                        return True
        return False

Database Impact

• No schema changes required - preserves existing database structure
• Multiple ValueSets merged into single grouper entries
• All terminology systems supported (LOINC, SNOMED, ICD-10, RxNorm)

Testing

• 12 existing tests continue to pass (100% backward compatibility)
• 6 new tests for v2.0.0 functionality
• 3 integration tests for full workflow validation
• Comprehensive edge case coverage including malformed data handling

Backward Compatibility

All existing functionality is preserved:

• v1.0.0 remains the default
• No breaking changes to existing APIs
• Original filtering logic intact for v1.0.0

Files Changed

• refiner/scripts/seed_terminology_db.py - Core implementation
• refiner/scripts/README.md - Documentation updates
• refiner/tests/test_seed_terminology_v2.py - New test suite

Migration Guide

No migration required. Users can:

1. Continue using v1.0.0 (default behavior unchanged)
2. Opt into v2.0.0 via command line argument or environment variable
3. Switch between versions as needed

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.