Best practice for using DRS and Data Connect together

[briandoconnor]

Background

Feature Branch: https://github.com/ga4gh/data-repository-service-schemas/tree/feature/issue-394-drs-plus-connect-docs-v1

I'm opening this issue based on followup to the April 20th, 2023 GA4GH Connect meeting "DRS and Data Connect" session. This session looked at exploring how standards from the Cloud and Discovery work streams can be used together to identify the two needs identified in the aims listed below:

• Address the need to obtain additional data about a DRS object
• Revisit how Data Connect handles the need for bundles

Some resources of interest:

• Original issue for extra metadata in DRS: adding/standardizing field providing reference to external metadata #336
• Pull request - intent is to close, still has useful for review of requirements: Feature/issue 343 metadata endpoint #390
• DRS and metadata exploration to revisit
  • https://github.com/ga4gh/fasp-scripts/blob/master/notebooks/drs/altMetadata.ipynb
  • https://github.com/ga4gh/fasp-scripts/tree/master/notebooks/drs

Key Takeaways from GA4GH Connect

Metadata + DRS

We agreed that best practices for working with metadata were important, and largely agreed on two guiding principles:

• 1. DRS doesn’t know about metadata, and shouldn’t. Instead, we should lean into the fact that systems that use DRS typically have some database-like component that does know about object metadata.
• 2. No new APIs (or API changes for DRS) are needed. Instead, we should add an appendix to the DRS spec documenting best practices for building systems that use DRS and care about metadata.

Compound Objects

We agreed with the way the DRS 1.3.0 develop branch frames the need for compound object support:

• Some content (e.g. DICOM images) is best represented as a compound object consisting of a structured collection of atomic DrsObjects.
• Each compound object should have a DRS ID, that clients can use to retrieve the object structure and its constituent atomic objects.
  We discussed two possible ways to represent and retrieve compound object contents, but didn’t have time to discuss their tradeoffs:
• 1. The approach documented in the develop branch (Best Practice: Manifests), where the compound object’s DRS ID provides access to a manifest file listing the object contents. Manifest format is datatype-specific and outside the scope of the DRS spec (but could for example be a JSON file).
• 2. An alternate approach where the compound object’s DRS ID provides access to a Data Connect table describing the object contents. Table format is datatype-specific and outside the scope of the DRS spec.

Goal for this Issue

This issue is to give us a place to discuss the use of Data Connect and DRS together (and link PRs to). The immediate goal of this Issue is to get a corresponding PR that addresses the best practice of using Data Connect together with DRS to provide 1) more metadata about DRS objects and 2) a scalable alternative to bundles. The intention is a documentation only change with a best practice appendix to the DRS spec.

[ianfore]

Three suggestions:

1. The use of Data Connect to obtain lists of DRS ids as an alternative to bundles relies is only feasible because of the ability to make DRS calls in bulk. Where a Data Connect query returns multiple DRS ids it is likely to be common practice to request all DRS ids for a specific DRS server in bulk.

2. It is a Data Connect rarher than a DRS matter, but a specific data type for DRS URIs in Data Connect schema is likely to be helpful in indicating that a given table column contains DRS URIs.

3. It is expected that Data Connect tables store DRS URIs as opposed to DRS ids.

[briandoconnor]

To respond to Ian's comment 1, we now have DRS bulk so hopefully item 1 is solved.

Agree with 2

Agree with DRS URIs

Do we have bi-directional links between a DRS object and a Data Connect query? Do we have info in the service-info about the Data Connect server linked to this DRS server?