Merge pull request #5537 from broadinstitute/dev

Dev

Author: hanars

.github/workflows/unit-tests.yml

@@ -140,5 +140,5 @@ jobs:
         working-directory: ./ui
       - run: npm run lint
         working-directory: ./ui
-      - run: npm test
+      - run: npm test -- --coverage
         working-directory: ./ui

CLAUDE.md

@@ -0,0 +1,45 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Overview
+
+seqr is a web-based rare disease genomics tool (Broad Institute): Django (Python) server + React (JS) client, backed by **postgres** (`seqrdb` + `reference_data_db`), **clickhouse** (variant search, read-only from the app), and optional **redis** cache. Dataset loading is done by an external pipeline-runner service (not in this repo). Elasticsearch and Hail-backend search are deprecated in favor of ClickHouse — don't build new features against ES.
+
+## Commands
+
+**Backend:** `pip install -r requirements-dev.txt -r requirements.txt`, `./manage.py migrate`, `./manage.py migrate --database=reference_data`, `./manage.py runserver`
+
+Run all backend tests (matches CI): `./manage.py test -p '*_tests.py' reference_data clickhouse_search seqr matchmaker panelapp`
+Single test: `./manage.py test seqr.views.apis.family_api_tests.FamilyAPITest.test_update_family`
+`clickhouse_search` tests need a running ClickHouse + `CLICKHOUSE_READER_USER`/`CLICKHOUSE_WRITER_USER`/`CLICKHOUSE_SERVICE_HOSTNAME`. CI enforces `--fail-under=99` coverage.
+
+**Frontend (`ui/`):** `npm install`, `npm run start` (dev server), `npm run build`, `npm test`, `npm run lint`
+Single test: `npx jest path/to/File.test.js`. Coverage threshold 40% lines/statements. Node 14 required.
+
+## Architecture
+
+- **`seqr/`** — core app (users, projects, families, samples, saved variants, permissions). `models.py` is the central data model. Function-based views (no DRF) in `views/apis/*.py`, one module per resource, each paired with `*_tests.py`, wired by name in `urls.py`. No service layer — logic lives in `views/apis`/`utils/`.
+- **`reference_data/`** — external gene/variant reference datasets (OMIM, gencode, HPO, PanelApp, etc.), lives in the separate `reference_data` Postgres DB via `ReferenceDataRouter`; refreshed by `update_all_reference_data`.
+- **`matchmaker/`** — GA4GH Matchmaker Exchange (MME) protocol (`views/external_api.py`) plus seqr's own MME UI endpoints (`views/matchmaker_api.py`).
+- **`panelapp/`** — Genomics England PanelApp gene panel integration.
+- **`clickhouse_search/`** — the variant search backend (sole supported implementation; ES only remains in old migrations). Custom Django DB backend (`backend/`), ClickHouse-side models (`models/`), query construction in `search.py`/`managers.py` called from `seqr`'s `variant_search_api.py`. Routed via `ClickHouseRouter`.
+- **`vlm/`** — separate standalone `aiohttp` service (own requirements, own Auth0 auth), not part of the Django URLconf, deployed independently.
+
+**DB routing:** four connections — `default` (seqrdb), `reference_data`, `clickhouse`/`clickhouse_write` — via `DATABASE_ROUTERS`. No cross-DB joins at the DB level.
+
+**Auth:** Google OAuth2 / Azure AD v2 via `social_django`, plus `django-guardian` object-level permissions.
+
+### Frontend (`ui/`)
+
+- `app.jsx` entry: Redux `Provider` + react-router-dom v5.
+- `pages/<Page>/` — one dir per route, each with `X.jsx`, `reducers.js`, `selectors.js`(+`.test.js`), `constants.js`, `fixtures.js`, `components/`.
+- `shared/` — cross-page components/utils. `redux/` — classic ducks pattern, `redux/utils/reducerFactories.js` for generic reducers.
+- Module aliases via `babel-plugin-module-resolver`: `shared -> ./shared/`, `pages -> ./pages/`.
+- Semantic UI React + `styled-components`. Codebase is mostly JS/JSX; **new files should be written in TypeScript**.
+- Jest + Enzyme, colocated `*.test.js`. Prefer deep-rendering connected components over shallow-rendering unconnected ones in tests.
+
+
+## Development Notes
+
+Never delete a file that is not in source control without confirming first

clickhouse_search/all_search_tests.py

@@ -367,17 +367,24 @@ def test_multi_project_search(self):
         )
 
     def test_all_project_search(self):
-        request_body = {'allGenomeProjectFamilies': '38'}
         self._assert_expected_search_error(
-            'No data available for genome version "GRCh38"', request_body=request_body, check_login=self.check_require_login,
+            'No data available for genome version "GRCh37"', request_body={'allGenomeProjectFamilies': '37'},
+            check_login=self.check_require_login,
+        )
+
+        request_body = {'allGenomeProjectFamilies': '38'}
+        additional_response = {'familiesByGuid': {'F000001_1': mock.ANY}}
+        self._assert_expected_search(
+            [FAMILY_3_VARIANT, FAMILY_1_VARIANT], project_families=[
+                {'projectGuid': 'R0001_1kg', 'familyGuids': ['F000001_1', 'F000003_3', 'F000005_5']},
+            ], request_body=request_body, additional_response=additional_response,
         )
 
         self.login_collaborator()
         project_families = [{'projectGuid': 'R0001_1kg', 'familyGuids': [
             'F000001_1', 'F000002_2', 'F000003_3', 'F000004_4', 'F000005_5', 'F000006_6', 'F000007_7', 'F000008_8',
             'F000009_9', 'F000010_10', 'F000013_13',
         ]}]
-        additional_response = {'familiesByGuid': {'F000001_1': mock.ANY}}
         self._assert_expected_search(
             [VARIANT1, VARIANT2, MULTI_FAMILY_VARIANT, VARIANT4, GCNV_VARIANT1, GCNV_VARIANT2,
              GCNV_VARIANT3, GCNV_VARIANT4, FAMILY_1_VARIANT, MITO_VARIANT1, MITO_VARIANT2, MITO_VARIANT3],
@@ -1613,7 +1620,7 @@ def _cached_lookup_variant(self, variant, expected_individuals=None):
     def test_get_single_variant(self):
         url_template = (reverse(query_single_variant_handler, args=['variant_id']) + '?familyGuid={}').replace('variant_id', '{}')
         url = url_template.format('21-3343353-GAGA-G', 'F000001_1')
-        self.check_collaborator_login(url)
+        self.check_require_login(url)
 
         response = self.client.get(url)
         self.assertEqual(response.status_code, 200)
@@ -1630,7 +1637,12 @@ def test_get_single_variant(self):
             ]},
         })
 
-        response = self.client.get(url_template.format(VARIANT_IDS[1], 'F000002_2'))
+        family_2_url = url_template.format(VARIANT_IDS[1], 'F000002_2')
+        response = self.client.get(family_2_url)
+        self.assertEqual(response.status_code, 403)
+
+        self.login_collaborator()
+        response = self.client.get(family_2_url)
         self.assertEqual(response.status_code, 400)
         self.assertDictEqual(response.json(), {'error': 'Variant 1-91511686-TCA-G not found'})
 
@@ -1652,8 +1664,8 @@ def test_get_single_variant(self):
         self.assertEqual(response.status_code, 200)
         self.assertDictEqual(response.json()['variantsById'], {'7-143270172-A-G': GRCH37_VARIANT})
 
-        self.assertTrue(all(call.args[0].startswith('projects__') for call in self.mock_redis.get.mock_calls))
-        self.assertTrue(all(call.args[0].startswith('projects__') for call in self.mock_redis.set.mock_calls))
+        self.assertTrue(all(call.args[0].startswith('project_analysis_groups__') for call in self.mock_redis.get.mock_calls))
+        self.assertTrue(all(call.args[0].startswith('project_analysis_groups__') for call in self.mock_redis.set.mock_calls))
 
     def test_frequency_filter(self):
         sv_callset_filter = {'sv_callset': {'af': 0.05}}
@@ -2389,6 +2401,12 @@ def test_gene_variant_lookup(self):
             check_login=self.check_require_login,
         )
 
+        locus = {'rawItems': 'ENSG00000097046'}
+        self._assert_expected_search(
+            [], request_body={'allGenomeProjectFamilies': '37', 'includeNoAccessProjects': True},
+            locus=locus, project_families=[], response_search={'no_access_project_genome_version': '37'},
+        )
+
         annotations = {
             'missense': ['missense_variant'],
             'other': ['non_coding_transcript_exon_variant'],
@@ -2398,30 +2416,30 @@ def test_gene_variant_lookup(self):
             'gnomad_genomes': {'af': 0.003},
             'gnomad_exomes': {'af': 0.003},
         }
-        locus = {'rawItems': 'ENSG00000097046'}
         response_search = {'no_access_project_genome_version': '38'}
         variant4 = {**VARIANT4, 'selectedMainTranscriptId': 'ENST00000350997', 'numFamilies': 3}
         del variant4['familyGuids']
         del variant4['genotypes']
+        project_families = [
+            {'projectGuid': 'R0001_1kg', 'familyGuids': ['F000001_1', 'F000003_3', 'F000005_5']},
+        ]
         self._assert_expected_search(
             [variant4], request_body=request_body, response_search=response_search,
             cached_variant_fields=[{'selectedTranscript': CACHED_CONSEQUENCES_BY_KEY[4][1]}],
-            annotations=annotations, freqs=freqs, locus=locus, project_families=[], export_data=[
+            annotations=annotations, freqs=freqs, locus=locus, project_families=project_families, export_data=[
                 EXPORT_DATA[0][:27], EXPORT_DATA[4][:24] + ['3 Families', '', ''],
             ], gene_counts={'ENSG00000097046': {'total': 1, 'families': {}}},
         )
 
         freqs = {'callset': freqs['callset']}
-        variant3 = {**VARIANT3, 'selectedMainTranscriptId': 'ENST00000497611', 'numFamilies': 4}
-        del variant3['familyGuids']
-        del variant3['genotypes']
+        variant3 = {**FAMILY_3_VARIANT, 'selectedMainTranscriptId': 'ENST00000497611'}
         self._assert_expected_search(
             [variant3, variant4], request_body=request_body, response_search=response_search,
             cached_variant_fields=[
                 {'selectedTranscript': CACHED_CONSEQUENCES_BY_KEY[3][3]},
                 {'selectedTranscript': CACHED_CONSEQUENCES_BY_KEY[4][1]},
             ],
-            annotations=annotations, freqs=freqs, locus=locus, project_families=[],
+            annotations=annotations, freqs=freqs, locus=locus, project_families=project_families,
         )
 
         self._assert_expected_search_error(
@@ -2430,22 +2448,22 @@ def test_gene_variant_lookup(self):
         )
 
         self._assert_expected_search(
-            [], request_body=request_body, response_search=response_search, project_families=[],
+            [], request_body=request_body, response_search=response_search, project_families=project_families,
             annotations=annotations, freqs=freqs, locus=locus, inheritance_mode='homozygous_recessive',
         )
 
-        variant3['numFamilies'] = 1
         variant4['numFamilies'] = 1
+        self.maxDiff = None
         self._assert_expected_search(
             [variant3, variant4], request_body=request_body, response_search=response_search,
             cached_variant_fields=[
                 {'selectedTranscript': CACHED_CONSEQUENCES_BY_KEY[3][3]},
                 {'selectedTranscript': CACHED_CONSEQUENCES_BY_KEY[4][1]},
             ],
-            annotations=annotations, freqs=freqs, locus=locus, inheritance_mode='de_novo', project_families=[],
+            annotations=annotations, freqs=freqs, locus=locus, inheritance_mode='de_novo', project_families=project_families,
             gene_counts={
-                'ENSG00000097046': {'total': 2, 'families': {}},
-                'ENSG00000177000': {'total': 1, 'families': {}},
+                'ENSG00000097046': {'total': 2, 'families': {'F000003_3': 1}},
+                'ENSG00000177000': {'total': 1, 'families': {'F000003_3': 1}},
             },
         )
 

clickhouse_search/managers.py

@@ -1067,12 +1067,10 @@ def _filter_project_families(self, entries, sample_data):
     def _sample_family_q(cls, sample_type, families):
         return Q(family_guid__in=families)
 
-    def _search_call_data(self, entries, sample_data, inheritance_mode=None, inheritance_filter=None, qualityFilter=None, pathogenicity=None, exclude_projects=None, annotate_carriers=False, annotate_hom_alts=False, **kwargs):
+    def _search_call_data(self, entries, sample_data, inheritance_mode=None, inheritance_filter=None, qualityFilter=None, pathogenicity=None, annotate_carriers=False, annotate_hom_alts=False, **kwargs):
        multi_sample_type_families = None
        if sample_data:
            entries, multi_sample_type_families = self._filter_project_families(entries, sample_data)
-       elif exclude_projects:
-           entries = entries.exclude(project_guid__in=exclude_projects)
 
        inheritance_q = None
        quality_q = None

clickhouse_search/search.py

@@ -64,32 +64,35 @@ def get_clickhouse_variants(families, user, genome_version=None, sort=None, samp
         except InvalidDatasetTypeException:
             continue
 
+        sample_data = sample_data_by_dataset_type[dataset_type]
+        exclude_dt_keys = (exclude_keys or {}).get(dataset_type)
+        dataset_results = []
+
+        if sample_data:
+            logger.info(f'Loading {dataset_type} data for {sample_data["num_families"]} families', user)
+
+            if inheritance_mode != COMPOUND_HET:
+                dataset_results += _get_search_results(
+                    entry_qs, variants_qs, sample_data, inheritance_mode=inheritance_mode, exclude_keys=exclude_dt_keys, **search, **parsed_filters,
+                )
+
+            run_x_linked_male_search = has_x_linked and not (inheritance_mode == X_LINKED_RECESSIVE and sample_data.get('samples'))
+            if run_x_linked_male_search:
+                dataset_results += _get_x_linked_male_search_results(
+                    entry_qs, variants_qs, dataset_type, user, sample_data, exclude_keys=exclude_dt_keys,
+                    **search, **parsed_filters,
+                )
+
         if dataset_type == Dataset.DATASET_TYPE_VARIANT_CALLS and no_access_project_genome_version:
             results += _get_no_access_search_results(
                 entry_qs, variants_qs, has_comp_het, user, **search, **parsed_filters,
-                exclude_projects=sample_data_by_dataset_type[dataset_type].get('project_guids'), inheritance_mode=inheritance_mode,
+                exclude_keys=[r['key'] for r in dataset_results] + (exclude_dt_keys or []), inheritance_mode=inheritance_mode,
             )
             searched_dataset_types.add(dataset_type)
 
-        sample_data = sample_data_by_dataset_type[dataset_type]
         if not sample_data:
             continue
 
-        logger.info(f'Loading {dataset_type} data for {sample_data["num_families"]} families', user)
-
-        dataset_results = []
-        if inheritance_mode != COMPOUND_HET:
-            dataset_results += _get_search_results(
-                entry_qs, variants_qs, sample_data, inheritance_mode=inheritance_mode, exclude_keys=(exclude_keys or {}).get(dataset_type), **search, **parsed_filters,
-            )
-
-        run_x_linked_male_search = has_x_linked and not (inheritance_mode == X_LINKED_RECESSIVE and sample_data.get('samples'))
-        if run_x_linked_male_search:
-            dataset_results += _get_x_linked_male_search_results(
-                entry_qs, variants_qs, dataset_type, user, sample_data, exclude_keys=(exclude_keys or {}).get(dataset_type),
-                **search, **parsed_filters,
-            )
-
         if has_comp_het:
             dataset_results += _get_data_type_comp_het_results_queryset(
                 entry_qs, variants_qs, sample_data, user, parsed_filters, **search,

matchmaker/views/external_api_tests.py

@@ -42,7 +42,7 @@ def test_mme_metrics_proxy(self):
         self.assertEqual(response.status_code, 200)
         self.assertDictEqual(response.json(), {
             'metrics': {
-                'numberOfCases': 4,
+                'numberOfCases': 5,
                 'numberOfSubmitters': 2,
                 'numberOfUniqueGenes': 3,
                 'numberOfUniqueFeatures': 4,

requirements-dev.txt

@@ -1,83 +1,81 @@
 #
-# This file is autogenerated by pip-compile with Python 3.9
+# This file is autogenerated by pip-compile with Python 3.11
 # by the following command:
 #
 #    pip-compile requirements-dev.in
 #
-asgiref==3.6.0
+asgiref==3.12.1
     # via
     #   -c requirements.txt
     #   django
-build==0.10.0
+build==1.5.0
     # via pip-tools
-certifi==2024.7.4
+certifi==2026.6.17
     # via
     #   -c requirements.txt
     #   requests
-charset-normalizer==3.0.1
+charset-normalizer==3.4.9
     # via
     #   -c requirements.txt
     #   requests
-click==8.1.3
+click==8.4.2
     # via pip-tools
 coverage==6.5.0
     # via -r requirements-dev.in
-django==4.2.27
+django==4.2.30
     # via
     #   -c requirements.txt
     #   django-appconf
+    #   django-compressor
     #   django-debug-toolbar
-django-appconf==1.0.5
+django-appconf==1.2.0
     # via django-compressor
-django-compressor==4.3.1
+django-compressor==4.6.0
     # via -r requirements-dev.in
-django-debug-toolbar==3.2.4
+django-debug-toolbar==6.3.0
     # via -r requirements-dev.in
-idna==3.7
+idna==3.18
     # via
     #   -c requirements.txt
     #   requests
-mock==5.0.1
+mock==5.2.0
     # via -r requirements-dev.in
-packaging==24.0
+packaging==26.2
     # via
     #   -c requirements.txt
     #   build
-pip-tools==6.12.2
+    #   wheel
+pip-tools==7.5.3
     # via -r requirements-dev.in
-pyproject-hooks==1.0.0
-    # via build
-rcssmin==1.1.1
+pyproject-hooks==1.2.0
+    # via
+    #   build
+    #   pip-tools
+pyyaml==6.0.3
+    # via responses
+rcssmin==1.2.2
     # via django-compressor
-requests==2.32.4
+requests==2.34.2
     # via
     #   -c requirements.txt
     #   responses
-responses==0.22.0
+responses==0.26.2
     # via -r requirements-dev.in
-rjsmin==1.2.1
+rjsmin==1.2.5
     # via django-compressor
-sqlparse==0.5.4
+sqlparse==0.5.5
     # via
     #   -c requirements.txt
     #   django
     #   django-debug-toolbar
-toml==0.10.2
-    # via responses
-tomli==2.0.1
-    # via
-    #   build
-    #   pyproject-hooks
-types-toml==0.10.8.5
-    # via responses
-urllib3==1.26.19
+urllib3==2.7.0
     # via
     #   -c requirements.txt
     #   requests
     #   responses
 urllib3-mock==0.3.3
     # via -r requirements-dev.in
-wheel==0.46.2
+wheel==0.47.0
     # via pip-tools
 
 # The following packages are considered to be unsafe in a requirements file:

requirements.in

@@ -1,11 +1,11 @@
 Django>=4.2,<4.3                  # core server-side framework
 django-anymail                    # for sending emails using cloud-based mail service providers
 django-clickhouse-backend
-django-csp                        # for setting CSP headers
+django-csp>=3.7,<4.0              # for setting CSP headers
 django-guardian                   # object-level permissions for database records. Behind a major version due to missing Python 2 support
 django-hijack                     # allows admins to login as other user
 django-notifications-hq           # notification app
-django-cors-headers               # allows CORS requests for client-side development
+django-cors-headers>=3.13,<4.0    # allows CORS requests for client-side development
 django-storages[google]           # alternative GCS storage backend for the django media_root
 social-auth-app-django>5.0.0      # the package for Django to authenticate users with social medieas
 social-auth-core                  # the Python social authentication package. Required by social-auth-app-django