Auto-update SILO database config from LAPIS API

[gordonkoehn]

Problem

The SILO database configuration (resources/silo/database_config.yaml) is currently a static file that must be manually updated when the upstream SILO database schema changes. This can lead to:

• Schema drift between sr2silo and the actual SILO database
• Test failures when the upstream config changes
• Manual maintenance burden

Proposed Solution

Automatically fetch and validate the database config from the LAPIS API endpoint:

https://lapis.wasap.genspectrum.org/swagger-ui/index.html#/info-controller/getDatabaseConfigAsJson

Options to Consider

Option A: CI-based auto-update

• Add a scheduled GitHub Action that fetches the config and creates a PR if it changed
• Keeps the config in version control for reproducibility

Option B: Runtime fetch with fallback

• Fetch config at runtime (with caching)
• Fall back to bundled config if API is unavailable

Option C: Test-time validation only

• Keep static config but add a CI test that compares it against the live API
• Fail CI if they drift apart

Tasks

• Decide on approach (A, B, or C)
• Implement the chosen solution
• Update test_database_config_validation.py to use/validate against live API
• Add appropriate error handling and caching
• Update documentation

Acceptance Criteria

• Schema drift between sr2silo and SILO database is detected automatically
• No manual intervention required to keep config in sync
• Tests remain reliable (not flaky due to network issues)

Related Files

• test_database_config_validation.py - Current validation tests
• resources/silo/database_config.yaml - Static config file