Add requirements for 10X multiome

[brianraymor]

Pending completion of Discovery. Draft available by October 12

[jahilton]

I have created a PR for the schema to facilitate comments/discussion & updates.

Proposal currently has fragment files as optional for a is_primary_data:True Dataset that is all 10x multiome. We could change this to required, especially if we successfully recover fragments for all existing cases.

One key decision is to accept unpaired scATAC data. This is based on many users finding them valuable, especially because 10x multiome data can be of poor quality.
unpaired scATAC-seq Datasets will be the gene activity matrix (not a peak matrix). Paired scATAC-seq (eg 10x multiome) Datasets will be the gene expression matrix (RNA data)
Matrix Layers table - Accessibility (e.g. ATAC-seq, mC-seq) can be specified to unpaired Accessibility (e.g. ATAC-seq, mC-seq)
Will need to communicate this distinction clearly to the user outside the schema

This does prevent submission of a gene activity matrix for paired assays. Accepting these would add complexity to "link" the two modalities & validate obs metadata across them - we have 2 Datasets impacted:

• https://cellxgene.cziscience.com/collections/ceb895f4-ff9f-403a-b7c3-187a9657ac2c
• https://cellxgene.cziscience.com/collections/5900dda8-2dc3-4770-b604-084eac1c2c82

Could possibly go in layers of the expression Dataset if a contributor wished, but in both cases, the ATAC has fewer genes and also has ~2k genes that are not in the expression Dataset

The proposal doesn't allow for any redundant submissions of unpaired data - no overlapping subsets within a Collection or reused data that appear in other Collections - no existing submissions. We could relax this if we see requests for this. Shouldn't be a lot of added burden on curators.

Given the definitions of paired vs unpaired assays in the proposal, we should review EFO to update to better ensure future terms will be linked correctly to "scRNA-seq" (Smart-like, 10x transcription profiling, etc. are not linked to any RNA/expression assay term)

Fragment File Submission

• Submission of a fragment file via the Discover API requires a collection_id and the valid dataset_id in the Collection that the fragment file is meant to attach to.
• Once the fragment file is uploaded, it will be queued for validation/processing once the associated Dataset reaches point X in its upload process.
• point X is some time after validation and the information necessary for fragment file validation has been gathered (assay, is_primary_data, feature_reference, obs index values)
• If the Dataset fails to reach point X (eg fails validation) then the fragment file remains queued.
• A queued fragment file could be deleted at any time
• If the fragment file fails validation, "Publish" will be blocked until either the fragment file is deleted or is replaced and successfully uploaded/validated/processed.
• If a Dataset with a fragment file asset is revised and no longer meets eligibility criteria for a fragment file asset, then the Dataset is "invalid". Attempt to revise the fragment file will fail. The fragment file can be deleted or the .h5ad can be revised. If the .h5ad was determined otherwise valid, then it will be valid upon deletion of the fragment file.
• If a Dataset with a fragment file asset is revised, once the re-uploaded .h5ad reaches point X in its upload process, the fragment file must be partially re-validated to confirm the obs index still matches. If that step fails then a curator can revise either the fragment file or the .h5ad

Fragment File Access

• Similar to the .h5ad & .rds assets, the fragment file & index are named by the dataset version id, with .tsv.gz & .tsv.gz.tbi extensions respectively.
• From Discover API, the Dataset schema includes the file name, file size, file type and presigned url for the fragment file & index - suggest adding to existing assets
• From Collection page, a user can easily see which Dataset(s) has a fragment file asset (without say clicking each "Download" button to see what options are there)
• From Collection page, a user can download the fragment file & index for a Dataset
• From /collections search page, a user can filter for Collections that have at least one Dataset with a fragment file asset
• From /datasets search page, a user can filter for Datasets with a fragment file asset
• From the Census API, a user can filter for observations that have associated fragments.
• From the Census API, a user can output a fragment file & index for their query, if at least one observation in their query has associated fragments.
• From the Census API, a user can filter for fragments based on chromsome, start coordinate, end coordinate.
• New Product- users can view 'universal peaks' that are a product of a CELLxGENE corpus-wide analysis of all fragments.

Genome Track Submission

• Submission & validation will be similar to fragments file assets.
• At initial submission, it would be OK to require all genome track assets are submitted together.
• Open question as to how a curator specifies which genome track file associates with which peak_grouping_column value
• During Revision, it would be preferred to allow for updating of just some genome track assets, rather than resubmit all.

Genome Track Access

• Similar to the .h5ad & .rds assets, the fragment file & index are named by the dataset version id. Name should also include the peak_grouping_column value that they correspond to.
• Discover API, Collection page, /collections search page, /datasets search page access mirrors what is proposed for fragments files
• No Census consumption of genome track assets is anticipated/recommended.
• From Explorer, alongside the current embeddings visualization, a user can view each of the submitted genome tracks in a genome browser.
  • The user can navigate to a region of the genome based on coordinates and gene symbol.
  • The user can change the range to zoom in/out.
  • The user can collapse any of the tracks to "hide" them.
  • The user can view the GENCODE track from the pinned annotation version in the schema
  • The user can view the most recent CELLxGENE 'universal peaks' track (proposed above)

[brianraymor]

@jahilton - During our earlier chat with @Bento007, we discussed the possibility of embedding the fragment file as a ndarray in uns. Was that not feasible? Linking files is extremely brittle and adds complexity to the product.

[jahilton]

@jahilton - During our earlier chat with @Bento007, we discussed the possibility of embedding the fragment file as a ndarray in uns. Was that not feasible? Linking files is extremely brittle and adds complexity to the product.

Discovery Table 3 shows the .h5ad size increasing from 0.7 G (no fragments in uns) to 5.4 G (31M fragments in uns) to 25.8 G (144M fragments in uns. Considering this one Dataset would require 824M fragments total, this is not a feasible option.
SnapATAC2 stores a CSR in obsm (see import_data) and that same 0.7 G .h5ad increased to 4.9 G with all 824M fragments. But this requires knowledge of the reference genome, a mechanism to extract back to TSV for user access, would require updating both fragments and non-fragment components of the .h5ad if either requires updating, and is more likely to lead to error without the ability to QA (eg obs are not ordered consistently)

[brianraymor]

@jahilton - reviewing your serving suggestions above, are there any updates that you're proposing to the Curation UX or is fragment submission absolutely dependent on the Discover API? (Beyond the UX blocking publication as needed.)

[brianraymor]

@jahilton - what are your thoughts about the revision process? For example, if a dataset with an attached fragment file is revised, does the process require:

1. That the new dataset continue to demonstrate that it meets the requirements for an attached fragment file.
2. That the fragment file remains valid.

[brianraymor]

@jahilton - is there a requirement to version fragment files similar to dataset files for reproducibility? Currently, datasets include an embedded citation to their dataset version. How would a consumer find the fragment file associated with that version?

[jahilton]

are there any updates that you're proposing to the Curation UX or is fragment submission absolutely dependent on the Discover API? (Beyond the UX blocking publication as needed.)

Fragment submission is dependent on Discover API. The only other UX update would be to communicate to the curator when a fragment file submission has failed, in a similar way to how it currently does for .h5ad upload.

---

Revision process - added some notes above.

---

is there a requirement to version fragment files similar to dataset files for reproducibility? Currently, datasets include an embedded citation to their dataset version.

No requirement to version the fragment file independently of the Dataset. It is considered a part of the Dataset. So if a Revision occurs, and the only change is replacement of the fragment file attached, then that's a new Dataset version.

---

How would a consumer find the fragment file associated with that version?

The API will return the URL for all assets, including the fragment file, when querying a version ID.
Strong suggestion to add md5sum to each asset (regardless of this proposal). That can help a consumer check if a fragment file has changed when they see a new version of the Dataset.

[BAevermann]

When it comes to the "paired" vs "unpaired" scATAC data, I am curious as to why we are retaining the gene activity matrix approach? It was my understanding that these are not reused and relatively uninformative. Will we not be attempting to collect the peak matrix? (the peak matrix being a better representation of the local experimental result, while the fragment file is for integration/reuse)

In addition, why rename scATAC and other "unpaired" assays when the "paired" versions always have more specific names, such as 10x Multiome and snATACseq-multiome? (note that HubMAP has updated this in their interface).

[jahilton]

@BAevermann the part of the proposal that is not presented yet is the genome track data product. That will be the primary product to explore the local experiment result as it is preferable to the peak matrix. So the gene activity matrix is included as it will provide some exploration value in absence of paired expression data and is a mechanism to host the obs metadata that pairs with the fragments - while also limiting the complexity of the schema (no new feature space in a Dataset)

I am not following the renaming question. The proposal speaks in general terms based on the EFO structure, and relies on EFO labels as with expression assays, etc.
(what is snATACseq-multiome?)

[BAevermann]

snATACseq-multiome is the term that HuBMAP is now using in its interface for its brand of multiome data ( it was previously snare-seq or share-seq I believe)

[BAevermann]

I was referring to this line:
"Matrix Layers table - Accessibility (e.g. ATAC-seq, mC-seq) can be specified to unpaired Accessibility (e.g. ATAC-seq, mC-seq)"

As the assay terms imply paired vs unpaired, do we need to hard code it into the layer name?