Dataset model

[hanars]

Pull request overview

This PR replaces the legacy per-individual Sample model with a new Dataset model (supporting multiple active/inactive individuals per dataset) and updates the backend APIs, ClickHouse search integration, pipeline loading commands, and UI/Redux state to consume datasetsByGuid instead of samplesByGuid.

Changes:

• Introduce Dataset Django model + migration to populate it from existing Sample rows and remove Sample.
• Update API endpoints/utilities/selectors to return and consume dataset-centric JSON (datasetsByGuid, active/inactive individual lists, updated project overview counts).
• Refactor pipeline/management commands + ClickHouse search code/tests to use datasets/individual active dataset relationships.

Motivation

There are now over 400,000 Sample models in seqr, which is starting to stretch the limits of postgres performance and we have already had to create several aggregations to return to the frontend due to the http responses getting too large to be able to return quickly and efficiently process them in the client (i.e. the actual response streaming and encoding is a bottleneck to page rendering). Moreover, almost all the metadata stored on the sample is shared by the batch of samples loaded together (data source, date loaded, sample type, dataset type) so this data is heavily duplicated. After mapping all the Sample models to the new Dataset models, there are only about 2,100 datasets which is a more manageable order of magnitude

[Copilot]

Pull request overview

This PR replaces the legacy per-individual Sample model with a new Dataset model (supporting multiple active/inactive individuals per dataset) and updates the backend APIs, ClickHouse search integration, pipeline loading commands, and UI/Redux state to consume datasetsByGuid instead of samplesByGuid.

Changes:

• Introduce Dataset Django model + migration to populate it from existing Sample rows and remove Sample.
• Update API endpoints/utilities/selectors to return and consume dataset-centric JSON (datasetsByGuid, active/inactive individual lists, updated project overview counts).
• Refactor pipeline/management commands + ClickHouse search code/tests to use datasets/individual active dataset relationships.

Reviewed changes

Copilot reviewed 51 out of 51 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
ui/shared/components/panel/fixtures.js	Rename fixture state key from samplesByGuid to datasetsByGuid.
ui/redux/selectors.js	Add dataset selectors/grouping and adapt per-individual dataset derivations.
ui/redux/rootReducer.js	Swap reducer key samplesByGuid → datasetsByGuid.
ui/pages/Search/fixtures.js	Update Search page fixture data to new dataset shape.
ui/pages/Project/selectors.js	Add dataset-based counts by type and switch RNA counts to rnaSampleCounts.
ui/pages/Project/reducers.js	Remove project export loading of samples (now datasets).
ui/pages/Project/fixtures.js	Update Project page fixture state to datasetsByGuid.
ui/pages/Project/components/ProjectOverview.jsx	Render dataset sections from dataset-type + RNA counts; wire new selector.
seqr/views/utils/variant_utils.py	Replace Sample references with Dataset for dataset-type semantics and stats aggregation.
seqr/views/utils/test_utils.py	Replace SAMPLE_FIELDS with DATASET_FIELDS expectations for JSON.
seqr/views/utils/pedigree_info_utils.py	Switch sample validation queries to active dataset relationships on Individuals.
seqr/views/utils/orm_to_json_utils_tests.py	Remove Sample JSON test and update imports/field expectations.
seqr/views/utils/orm_to_json_utils.py	Add get_json_for_datasets and adjust aggregations for distinct results.
seqr/views/utils/individual_utils.py	Update individual deletion and “active search sample” validation to use datasets.
seqr/views/utils/anvil_metadata_utils.py	Refactor AnVIL export metadata generation to operate on dataset-annotated individuals.
seqr/views/apis/variant_search_api.py	Replace trio-family sample-count logic and family sample-type aggregation with dataset-aware equivalents.
seqr/views/apis/summary_data_api.py	Update clickhouse dataset type constant usage from Sample → Dataset.
seqr/views/apis/saved_variant_api.py	Use Dataset constants for default dataset_type when creating manual saved variants.
seqr/views/apis/report_api_tests.py	Update expected exported sample/discovery rows after dataset/individual-id changes.
seqr/views/apis/report_api.py	Update stats aggregation to count active dataset individuals; adjust export wiring to new AnVIL utils signature.
seqr/views/apis/project_api_tests.py	Update project overview response shape and remove project samples endpoint test.
seqr/views/apis/project_api.py	Replace samplesByGuid/sample counts with datasetsByGuid and rnaSampleCounts; remove project_samples endpoint.
seqr/views/apis/individual_api_tests.py	Update individual edit/delete tests to manipulate dataset active/inactive relationships.
seqr/views/apis/family_api_tests.py	Update family page expectations for datasets and remove individual sampleGuids field expectations.
seqr/views/apis/family_api.py	Return datasets for family page and swap Sample dataset-type constants to Dataset.
seqr/views/apis/data_manager_api_tests.py	Update delete-family assertions and fixture expectations for dataset world.
seqr/views/apis/data_manager_api.py	Refactor project loading eligibility and deactivate-search operations to operate on datasets.
seqr/views/apis/dashboard_api.py	Replace sample-type counts with dataset-annotated individual aggregation.
seqr/views/apis/anvil_workspace_api.py	Swap Sample constants to Dataset constants for workspace data loading.
seqr/utils/vcf_utils.py	Use Dataset dataset-type constants for file expectations.
seqr/utils/search/utils.py	Use Dataset constants when deriving dataset type for variants.
seqr/utils/search/add_data_utils.py	Refactor “disable search” operations to move individuals from dataset active → inactive relations.
seqr/urls.py	Remove project/.../get_samples route.
seqr/models.py	Replace Sample model with Dataset model and update related constants/usages.
seqr/migrations/0087_dataset.py	Create/populate Dataset model (including m2m relations) and delete Sample model.
seqr/management/tests/transfer_families_to_different_project_tests.py	Update expectations around dataset splitting and inactive dataset relations.
seqr/management/tests/reload_saved_variant_genotypes_tests.py	Update fixtures/tests to set dataset active individuals instead of sample FK fields.
seqr/management/tests/check_for_new_samples_from_pipeline_tests.py	Rewrite pipeline loading tests for dataset creation (new deterministic IDs, timestamps).
seqr/management/commands/update_individuals_sample_qc.py	Update CLI argument validation to Dataset constants.
seqr/management/commands/transfer_families_to_different_project.py	Split datasets that span moved + remaining families; update references.
seqr/management/commands/tag_seqr_prioritized_variants.py	Rework variant prioritization command inputs to use Individual active datasets rather than Sample rows.
seqr/management/commands/reload_saved_variant_genotypes.py	Use Dataset objects to determine dataset types present for reload.
seqr/management/commands/check_for_new_samples_from_pipeline.py	Replace per-sample activation with per-dataset creation + inactivation of previous datasets’ individuals.
seqr/fixtures/1kg_project.json	Update fixture data from seqr.sample rows to seqr.dataset rows (active/inactive individual lists).
seqr/admin.py	Register Dataset model instead of Sample in Django admin.
clickhouse_search/search_tests.py	Update test setup/helpers to clone datasets and adjust expectations for dataset-based behavior.
clickhouse_search/search.py	Replace Sample-based lookups/validation with Individual active dataset metadata and Dataset constants.
clickhouse_search/models/search_models.py	Swap constants in clickhouse model maps from Sample → Dataset.
clickhouse_search/models/gt_stats_models.py	Swap dataset-type mapping constants from Sample → Dataset.
clickhouse_search/management/tests/set_saved_variant_key_tests.py	Update SV key test fixture setup to activate datasets via m2m.
clickhouse_search/management/commands/set_saved_variant_key.py	Update dataset-type constants and “has search data” checks to use active datasets.

Comments suppressed due to low confidence (1)

ui/redux/selectors.js:86

• getProjectDatasetTypes treats activeIndividuals as a truthy/falsy flag, but it’s an array. An empty array is truthy, so dataset types can be added even when there are 0 active individuals. Consider checking Array.isArray(activeIndividuals) and activeIndividuals.length > 0 (or normalizing the API to always return arrays) before adding datasetType.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hanars]

this was used when downloading all samples from a project, no longer needed because the project overview now returns all datasets instead of a subset of the samples