Notable infrastructure changes will be kept in this file.
- Moves the initContainers processes on the
pipeline-runnerthat sync variant reference data used in the pipeline (both hail tables and VEP tarballs) into cronjobs.
- The initContainers change resolves an bug reported here relating to the initial migrations
when creating a seqr installation for the first time. By offloading the initContainers, the
pipeline-runnerpod can immediately listen to requests from the seqr migrations.
- Moves variant-level reference data sources directly into ClickHouse Materialized Views. Note that this will temporarily increase the memory used by the ClickHouse server process by 50% while we work to remove those sources from the loading pipeline.
- Deprecates the ClickHouse loading service, moving loading directly into the loading pipeline as a Luigi task.
- Updates the
hail-search->ClickHousemigration script to load directly into ClickHouse after generating parquets.