Fix conditional enum filtering and sync workflow

This commit fixes the conditional enum filtering system to work with Synapse's
limitations and consolidates the sync scripts.

## Problem
1. Conditional filtering used $defs/$refs which Synapse doesn't support
2. Two sync scripts (sync_model_systems.py and sync_model_systems_enhanced.py) were confusing
3. Weekly workflow didn't regenerate JSON schemas after syncing data
4. modules/Sample/generated/ folder had no documentation

## Solution

### 1. Replace sync_model_systems.py with enhanced version
- Merged sync_model_systems_enhanced.py functionality into sync_model_systems.py
- Added antibody and genetic reagent syncing to the enhanced script
- Deleted the "enhanced" version to avoid confusion
- Updated weekly workflow to use standard name

### 2. Fix add_conditional_enum_filtering.py to inline enums
- Changed from using $refs pointing to $defs
- Now directly inlines enum values in if/then conditionals
- Reads from modules/Sample/generated/*.yaml files
- Creates conditionals like:
  ```
  if: {modelSystemType: "cell line", modelSpecies: "Homo sapiens", ...}
  then: {modelSystemName: {items: {enum: ["90-8", "ST88-14", ...]}}}
  ```
- No $defs section in output (Synapse-compatible)

### 3. Update weekly-model-system-sync.yml workflow
- Added step to regenerate JSON schemas after syncing data
- Now runs add_conditional_enum_filtering.py + gen-json-schema-class.py
- Ensures schemas stay in sync with latest cell lines/models
- Updated PR description to mention schema regeneration

### 4. Document modules/Sample/generated/ folder
- Added README.md explaining purpose and build process
- Clarifies these are source files, not runtime files
- Documents the cascading filter approach for staying under 100-value limit

## Result
- ✅ Conditional filtering works without $defs (Synapse-compatible)
- ✅ Single sync script handles all resource types
- ✅ Weekly workflow keeps schemas synchronized
- ✅ Clear documentation for generated enum files

## Files Changed
- utils/sync_model_systems.py - Now the main sync script (was "enhanced")
- utils/sync_model_systems_enhanced.py - Deleted (merged into main)
- utils/add_conditional_enum_filtering.py - Inline enums instead of $defs
- .github/workflows/weekly-model-system-sync.yml - Add schema regeneration
- modules/Sample/generated/README.md - New documentation

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

Author: BelindaBGarana

.github/workflows/weekly-model-system-sync.yml

@@ -40,7 +40,7 @@ jobs:
 
       - name: Sync model system data with conditional filtering
         run: |
-          python utils/sync_model_systems_enhanced.py --synapse-id ${{ env.SYNAPSE_TOOLS_TABLE_ID }}
+          python utils/sync_model_systems.py --synapse-id ${{ env.SYNAPSE_TOOLS_TABLE_ID }}
       
       - name: Check for changes
         id: changes
@@ -61,10 +61,20 @@ jobs:
           git clone --depth 1 https://github.com/anngvu/retold.git
           pip install linkml==v1.8.1
           npm install -g json-dereference-cli
-          
+          pip install jsonref  # Required for schema generation
+
           # Rebuild the data model
           make NF.yaml
-          bb ./retold/retold as-jsonld --dir modules --out NF.jsonld 
+          bb ./retold/retold as-jsonld --dir modules --out NF.jsonld
+
+      - name: Regenerate JSON schemas
+        if: steps.changes.outputs.changes == 'true'
+        run: |
+          # Add conditional filtering (inlines enum values into schemas)
+          python utils/add_conditional_enum_filtering.py
+
+          # Regenerate all JSON schemas (this will inline everything properly)
+          python utils/gen-json-schema-class.py --skip-validation 
       
       - name: Create Pull Request
         if: steps.changes.outputs.changes == 'true'
@@ -89,15 +99,17 @@ jobs:
             - Updated genetic reagents in `modules/Experiment/GeneticReagent.yaml`
             - Added/updated `source` links to NF Tools Central detail pages
             - Rebuilt data model artifacts (`NF.jsonld`, `dist/NF.yaml`)
+            - Regenerated JSON schemas with updated enum values
 
             ### Files Modified:
-            - `modules/Sample/CellLineModel.yaml` (638 cell lines)
-            - `modules/Sample/AnimalModel.yaml` (123 animal models)
+            - `modules/Sample/CellLineModel.yaml` (cell lines)
+            - `modules/Sample/AnimalModel.yaml` (animal models)
             - `modules/Sample/generated/*.yaml` (29 filtered enum subsets)
             - `modules/Experiment/Antibody.yaml`
             - `modules/Experiment/GeneticReagent.yaml`
             - `NF.jsonld`
             - `dist/NF.yaml`
+            - `registered-json-schemas/*.json` (63 schemas)
 
             ### What This Includes:
             - ✅ Cell line and animal model names from NF Tools Central (syn51730943)

modules/Sample/generated/README.md

@@ -0,0 +1,63 @@
+# Filtered Enum Subsets for Conditional Filtering
+
+This directory contains auto-generated filtered enum subsets used for conditional enum filtering in JSON schemas.
+
+## Purpose
+
+Synapse has a **100-value limit** for enum fields. The full `CellLineModel` enum has 638 values, which exceeds this limit. To work around this, we use **conditional filtering** with cascading dropdowns:
+
+```
+modelSystemType → modelSpecies → cellLineCategory → cellLineGeneticDisorder → modelSystemName
+```
+
+Based on the user's selections in the filter fields, the JSON schema shows only the relevant subset of model systems (always <100 values).
+
+## Files
+
+Each file contains a filtered subset of cell lines or animal models:
+
+- `CellLineHomosapiensCancercelllineNeurofibromatosistype1Enum.yaml` - 54 human NF1 cancer cell lines
+- `CellLineHomosapiensInducedpluripotentstemcellNeurofibromatosistype1Enum.yaml` - 32 human NF1 iPSCs
+- etc. (29 files total)
+
+## Build Process
+
+These files are **source files** for the build process, not runtime files:
+
+1. **Weekly Sync** (`weekly-model-system-sync.yml`)
+   - Runs `utils/sync_model_systems.py`
+   - Queries syn51730943 (NF Tools Central) for latest cell lines/models
+   - Generates these filtered enum YAML files
+   - Updates `CellLineModel.yaml`, `AnimalModel.yaml`, etc.
+
+2. **Schema Generation** (manual or in CI)
+   - Runs `utils/add_conditional_enum_filtering.py`
+   - Reads these YAML files
+   - Creates if/then conditional rules in JSON schemas
+   - **Inlines enum values directly** (no $refs - Synapse doesn't support them)
+
+3. **Upload to Synapse**
+   - Only the final `registered-json-schemas/*.json` files are uploaded
+   - These contain fully inlined enum values from this directory
+
+## Data Source
+
+- Table: **syn51730943** (NF Tools Central)
+- Columns: `resourceName`, `species`, `cellLineCategory`, `cellLineGeneticDisorder`
+- Maintained by the NFTC team
+
+## Maintenance
+
+These files are auto-generated weekly. **Do not edit manually.**
+
+To regenerate:
+```bash
+python utils/sync_model_systems.py --synapse-id syn51730943
+python utils/add_conditional_enum_filtering.py
+python utils/gen-json-schema-class.py
+```
+
+## See Also
+
+- [Conditional Enum Filtering Plan](../../../docs/conditional-enum-filtering-plan.md)
+- Issue #797 (100-value enum limit)

utils/add_conditional_enum_filtering.py

@@ -221,12 +221,13 @@
 ]
 
 
-def create_conditional_rule(mapping: Dict[str, str]) -> Dict[str, Any]:
+def create_conditional_rule(mapping: Dict[str, str], enum_values: List[str]) -> Dict[str, Any]:
     """
     Create an if/then conditional rule for a filter combination.
 
     Args:
         mapping: Dict with filter field values and enum name
+        enum_values: List of enum values to inline directly (no $refs)
 
     Returns:
         Dict with if/then structure for allOf
@@ -252,11 +253,12 @@ def create_conditional_rule(mapping: Dict[str, str]) -> Dict[str, Any]:
         if_properties["cellLineGeneticDisorder"] = {"const": mapping["cellLineGeneticDisorder"]}
         required_fields.append("cellLineGeneticDisorder")
 
-    # Build the then condition - reference the filtered enum
+    # Build the then condition - inline enum values directly (Synapse doesn't support $refs/$defs)
     then_properties = {
         "modelSystemName": {
             "items": {
-                "$ref": f"#/$defs/{mapping['enum']}"
+                "enum": enum_values,
+                "type": "string"
             }
         }
     }
@@ -280,31 +282,18 @@ def add_conditionals_to_schema(schema: Dict[str, Any]) -> Dict[str, Any]:
         schema: The JSON schema dict
 
     Returns:
-        Modified schema with conditionals added
+        Modified schema with conditionals added (enum values inlined, no $defs)
     """
     # Check if this schema has modelSystemName field
     if "properties" not in schema or "modelSystemName" not in schema.get("properties", {}):
         return schema  # No modelSystemName, skip
 
     print(f"  Adding {len(CONDITIONAL_MAPPINGS)} conditional rules...")
 
-    # Create allOf rules
-    conditional_rules = [create_conditional_rule(mapping) for mapping in CONDITIONAL_MAPPINGS]
-
-    # Add to schema
-    if "allOf" in schema:
-        # Append to existing allOf
-        schema["allOf"].extend(conditional_rules)
-    else:
-        # Create new allOf
-        schema["allOf"] = conditional_rules
-
-    # Add the filtered enum definitions to $defs
-    if "$defs" not in schema:
-        schema["$defs"] = {}
-
-    # Load enum definitions from generated files
+    # Load enum values from generated files
     generated_dir = Path("modules/Sample/generated")
+    conditional_rules = []
+
     for mapping in CONDITIONAL_MAPPINGS:
         enum_name = mapping["enum"]
         enum_file = generated_dir / f"{enum_name}.yaml"
@@ -318,11 +307,23 @@ def add_conditionals_to_schema(schema: Dict[str, Any]) -> Dict[str, Any]:
             if "enums" in enum_data and enum_name in enum_data["enums"]:
                 enum_values = list(enum_data["enums"][enum_name]["permissible_values"].keys())
 
-                # Add to $defs
-                schema["$defs"][enum_name] = {
-                    "enum": enum_values,
-                    "type": "string"
-                }
+                # Create conditional rule with inlined enum values
+                rule = create_conditional_rule(mapping, enum_values)
+                conditional_rules.append(rule)
+            else:
+                print(f"    Warning: Could not find enum '{enum_name}' in {enum_file}")
+        else:
+            print(f"    Warning: Enum file not found: {enum_file}")
+
+    # Add to schema
+    if "allOf" in schema:
+        # Append to existing allOf
+        schema["allOf"].extend(conditional_rules)
+    else:
+        # Create new allOf
+        schema["allOf"] = conditional_rules
+
+    print(f"    ✓ Added {len(conditional_rules)} conditional rules with inlined enum values")
 
     return schema