Create versioned AnVIL export packages from Dashboard data

[HadleyKing]

Summary

Add a Dashboard-managed AnVIL export package workflow that creates an immutable, reproducible release artifact from schema-valid Dashboard data.

This feature must generate GREGoR-compatible TSV tables, a release manifest, and AnVIL validation configuration files without modifying source records, staging files, or calling AnVIL APIs.

Background

The historical transfer workflow performed source-data cleanup and table reconstruction before submission. Those responsibilities should now belong to the Dashboard’s normal GREGoR schema-driven table creation, import, and validation process.

The export package should therefore be a deterministic serialization of valid Dashboard records, not a second source of truth or a repair pipeline.

Goals

• Create a release candidate from a defined Dashboard submission scope.
• Freeze the included records and source schema version.
• Validate release-level table relationships and submission completeness.
• Generate GREGoR-compatible TSV files.
• Generate a manifest with row counts, hashes, source identifiers, and schema version.
• Generate the JSON configuration required by the existing AnVIL validation workflow.
• Provide a human-readable export preview and readiness report.
• Preserve complete provenance for every generated artifact.

Scope

Release model

Create models equivalent to:

AnvilSubmissionProfile
AnvilRelease
AnvilReleaseTable
AnvilReleaseArtifact
AnvilReleaseValidationRun

Suggested responsibilities:

AnvilSubmissionProfile
- GREGoR data-model version
- intended target workspace name
- required tables
- table inclusion rules
- release-specific validation rules
- validation workflow configuration template

AnvilRelease
- release identifier
- submission profile
- status
- created by
- created timestamp
- frozen source scope
- model version
- manifest location and hash

AnvilReleaseTable
- table name
- generated TSV location
- row count
- SHA-256 hash
- validation status

AnvilReleaseArtifact
- artifact type
- path
- hash
- size
- generation timestamp

Release workflow

Implement a release workflow with the following states:

draft
→ validating
→ validation_failed
→ ready_for_review
→ approved
→ exported

The initial implementation ends at exported.

Validation

Use the GREGoR Data Model schema as the basis for release validation.

Release validation must include:

• Required table presence according to the submission profile.
• Referential closure across included records.
• No unresolved participant, family, analyte, experiment, alignment, aligned-set, or callset references.
• Required participant/family/phenotype relationships for the selected scope.
• Required paired metadata relationships, such as alignment records and referenced experiments.
• No duplicate primary keys in exported tables.
• No invalid values that bypassed normal Dashboard validation.
• No file URI fields that are malformed or empty when required by the exported table.

Validation findings must identify the source record, table, field or relationship, severity, and suggested corrective action.

Export artifacts

Generate:

participant.tsv
family.tsv
phenotype.tsv
analyte.tsv
experiment_*.tsv
aligned_*.tsv
called_variants_*.tsv
genetic_findings.tsv
manifest.json
validation_report.json
AnVIL validation workflow configuration JSON

Only tables applicable to the selected submission profile and release scope should be emitted.

The manifest must include:

release ID
submission profile
GREGoR model version
Dashboard source snapshot identifier
creation time
creating user
included table names
row counts
artifact paths
SHA-256 hashes
validation result

Frontend

Add an AnVIL Releases page with:

• List of releases and statuses.
• Create Release action.
• Submission-profile selection.
• Release-scope selection.
• Validation findings grouped by severity.
• Export preview with table row counts.
• Download links for generated TSVs, manifest, and validation report.
• Approval action restricted to authorized Dashboard roles.

Acceptance Criteria

• An authorized user can create a draft AnVIL release from a selected Dashboard scope.
• The release stores an immutable snapshot of included source records or an equivalent immutable source representation.
• Release validation identifies broken references and prevents export when blocking errors exist.
• The generated TSV files are deterministic for identical source snapshots and schema versions.
• Each exported table has a recorded row count and SHA-256 hash.
• The generated manifest identifies the source snapshot, schema version, creating user, and all release artifacts.
• Export generation does not alter Dashboard participant, family, phenotype, analyte, experiment, alignment, or findings records.
• Validation findings link back to the source Dashboard records requiring correction.
• Automated tests cover valid export creation, broken-reference failures, missing required table failures, deterministic output, and manifest hashing.

Non-goals

• Do not upload files to an AnVIL workspace bucket.
• Do not invoke Google Cloud Storage APIs.
• Do not import Terra Data Tables.
• Do not submit or monitor AnVIL workflows.
• Do not silently normalize, repair, infer, or discard invalid source data during export.

Dependencies

• Existing GREGoR schema-driven validation and table-generation infrastructure.
• A defined initial AnVIL submission profile for the UCI GREGoR workflow.
• Agreement on release-scope semantics, such as incremental submission versus full replacement submission.