externalInfo display

[will-moore]

Since we are starting to use externalInfo in IDR and possibly for other users - see https://github.com/ome/omero-cli-zarr/pull/167/files, it may be useful to display this in web. Glencoe already use this.

Updated description based on discussions below:

• Various UI options have been considered. E.g. show nothing unless you do $(".debug").show() in dev Console, or put it under the ../../ Fileset button, or even show a Zarr button.

• We aim to load externalInfo along with the original objects (instead of an extra lazy loading call)

• getObject() and getExternalInfo() load externalInfo omero-py#453 updates getObject() etc to include externalInfo

• We use that PR to display in the right panel (nothing is shown if externalInfo is not found)

• NB: if the omero-py PR is missing, then this will fail silently since obj.getExternalInfo will not be found. No need for this PR to depend on getObject() and getExternalInfo() load externalInfo omero-py#453

Zarr button:

(includes temp BioNGFF viewer link used for demo):

Adding it to Fileset Info:

cc @knabar @dominikl

[sbesson]

@will-moore thanks for starting this conversation. As noted above, Glencoe is using ExternalInfo quite extensively so we are interested in ways to expose this metadata via the OMERO.web API and/or UI.

Starting with with your last question, my immediate feeling is that it would be most beneficial to include this functionality in the OMERO.web JSON API. A few reasons for that:

• as explained in https://omero.readthedocs.io/en/stable/developers/Web.html, the /webclient component should be considered as internal only. If the intent is to expose this metadata as JSON for consumption (both internal and external), api/ is the location of choice
• omero-marshal already has support for ExternalInfo encoding/decoding so there is no complex cross-component work
• the usage of ExternalInfo is not limited to Image and can be applied to any object. Glencoe uses this on Mask objects to represent labels for instance. Having api/v0/<images,rois,...>/<id> loading and exposing the external info in a unified manner would be consistent.

On the query itself, my impression is that externalInfo should be treated the same way as other details. I don't expect the additional join to be expensive in terms of query time given these are relatively shallow objects with only a few attributes but that should certainly be reviewed as par of the functional testing.

The UI questions is probably the one where I am the less opinionated and I have a suspicion the answer might be "it depends". For Zarr data, we have used ExternalInfo as an mechanism to store the location of the data. The closest equivalent would be the legacy OMERO pyramids which are not exposed in the UI. The most relevant existing icon would be the ../../. in the right-hand panel which lists the Fileset entries.
However other ExternalInfo objects might be more naturally suited either as Image details or as additional attributes

[knabar]

Agree with @sbesson; I would probably not create a separate view for retrieving externalInfo, but get it with the other object details on an existing call. Any performance hit should be minimal compared to having a whole separate call.

[will-moore]

@sbesson This is now deployed on idr-testing. e.g. https://idr-testing.openmicroscopy.org/webclient/?show=image-15160031

[joshmoore]

As a quick sidenote before I disappear for a week, I'd be more than happy if we hide the acronym "LSID" from users.

[dominikl]

👍 Looks good to me. What does LSID actually stand for, and what is the ExternalInfo actually used for generally @joshmoore ?

[joshmoore]

LSID stands for "Life Science Identifier". https://www.lsid.info/ et al. https://zenodo.org/records/46804 is a pretty good read.

The ExternalInfo object wasn't used a lot previously but was intended to hold info of the objects that felt "too detailed" to expose in the main object. It also helps that if you have an opaque identifier that you don't have to check every table to find the object but you can look in the one external info table.

[will-moore]

https://en.wikipedia.org/wiki/LSID.
@joshmoore "Hide LSID from users" - what about CLI users? - in ome/omero-cli-zarr#167 we have --lsid and in general we try to make the UI reflect the data model.

Do we use another term or just e.g.

http://path/to/my/image.zarr
entityType: com.glencoesoftware.ngff.multiscales
entityId: 3

[joshmoore]

in general we try to make the UI reflect the data model.

"Attribues"? "Attachments"? 😄

what about CLI users?

I certainly worry about CLI users less.

But if we need to keep it, that's fine, just more than the other terms, "LSID" was a mistake.

[will-moore]

As discussed in web meeting, we probably don't want ExternalInfo to be so much "in your face" since it's not of interest to most users.
Could look at hiding it under ../../ or elsewhere.

[will-moore]

Deployed latest changes to idr-testing and updated screenshot above.

[will-moore]

To help testing, you can now set ExternalInfo via the CLI - see ome/omero-py#453

[will-moore]

In IDR (and regular OMERO if/when omero-zarr-pixel-buffer becomes adopted) we want it to be nice and easy for a user to get the OME-Zarr URL for an image they are looking at, as they can do for e.g. https://idr.github.io/ome-ngff-samples/.

We probably don't want to show "External Info" or com.glencoesoftware.ngff.multiscales but for Images where the extInfo EntityType is com.glencoesoftware.ngff.multiscales, we should show the OME-Zarr URL with a handy "copy to clipboard" button.
This could be in addition to the ExternalInfo display as above, which would support all Objects (not just images) and other EntityTypes.

[will-moore]

As discussed in web meeting today:

• Exposing externalInfo to users is not useful in IDR since this is a path/to/zarr rather than a public s3.
• It is only useful for debugging etc, so let's hide it with a .debug: {display: none} css then we can easily show all such UI elements with:

$(‘.debug’).show()

[will-moore]

As discussed with @jburel...

Since we have addressed s3 backend issues in omero-zarr-pixel-buffer, we can expect to use genuine public URLs for external-info, so this should be easier for the user to find.

This also gives us the chance to add "Open with" various zarr viewers.
However, the stored externalInfo lsid doesn't include endpoint info. E.g. s3://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr/?anonymous=true can't be used in web based viewers. Instead we need e.g. https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr. Can we assume that we just replace s3 with https to get this URL (and remove ?anonymous=true)?

[sbesson]

Note that everything that is discussed is conditional to the fact the externalInfo:
1- is of the correct entityType and entityId
2- is pointing at a remote resource rather than a local path

However, the stored externalInfo lsid doesn't include endpoint info. E.g. s3://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr/?anonymous=true can't be used in web based viewers

Not sure I understand. Unlike the s3://<bucket>/<prefix> URI used e.g. by tools like the AWS CLI, to be usable by omero-zarr-pixel-buffer, the externalinfo.lsid must include an endpoint after s3://

Can we assume that we just replace s3 with https to get this URL

At least, this should be correct for Amazon S3. The service supports to types of requests: Path-style requests and Virtual-hosted–style requests. So https://s3.<region-code>.amazonaws.com/<bucket-name>/<zarr_prefix> is a valid path-style request.
Other S3 compatible object stores might behave differently.

and remove ?anonymous=true

If the intent is to create an HTTP URL using by any a third-party application, then such transformation is probably only applicable to public data. No query parameter in a S3 lsid value indicates that credentials are expected to read the data.

[jburel]

@will-moore we cannot change the value to https or remove anonymous as @sbesson indicated.

We only discussed showing external info to help identifying the "registered" one not reformatting it. to use it in the web, the application will need to parse it

[will-moore]

I understand that omero-web will need to parse it - I'm just asking if it will always work to replace s3 with https?
That's what I've implemented above and it seems to work for various lsids.

[joshmoore]

I'm just asking if it will always work to replace s3 with https?

Definitely not in the general case.

[will-moore]

To be discussed at next web meeting...

[will-moore]

NB: with ome/omero-py#483 this should now be obj.getDetails().getExternalInfo()

[will-moore]

NB: with ome/omero-py#483 this should now be obj.getDetails().getExternalInfo()

[will-moore]

@dominikl How do you like the current Zarr button (as deployed on merge-ci)?

[dominikl]

Thanks Will! Looks good to me! I think this would great to have on IDR, makes it so much easier to quickly check where an image is actually coming from! 👍