Reviewing StreamResource and StreamDatum from the storage, access perspective

[danielballan]

During the discussions that developed StreamResource and StreamDatum documents, the "producer" side was being built, but much of the "consumer" side remained to be developed. I think we agreed to stay open to adjustments when the time came to build out the consumer side. I hope that window is still open.

Here's a StreamResource from DLS I22, courtesy of @DiamondJoseph:

  {
    "name": "stream_resource",
    "doc": {
      "uid": "158167d9-796b-4e12-afff-5b3dd903815c",
      "data_key": "waxs-sum",
      "spec": "AD_HDF5_SWMR_SLICE",
      "root": "/dls/i22/data/2023/cm33873-5",
      "resource_path": "i22-723667-waxs-hdf.h5",
      "resource_kwargs": {
        "path": "/entry/instrument/NDAttributes/StatsTotal",
        "multiplier": 1,
        "timestamps": "/entry/instrument/NDAttributes/NDArrayTimeStamp"
      },
      "path_semantics": "posix",
      "run_start": "617184e5-5e24-40b1-80f7-0056d51e44f8"
    }
  }

I can see nothing to change about these:

• uid, unique key for the document
• data_key, tells us which field (column) in the Event Stream this maps to
• run_start, tells us which BlueskyRun this belongs to

I might suggest re-evaluating that names of:

• spec, a key in a registry that tells us which reader ("handler") to use
• resource_kwargs, additional configuration for the reader

In Tiled, we use MIME types, including standard ones such as image/tiff and custom ones named like application/x-hdf5-smwr-slice, in the role of spec. Maybe it's worth considering whether we want to use mimtype here, as it is a recognized standard way to spell, "This is a format, which tells you which I/O code to read it."

Taking a broader-than Python look at the world, resource_kwargs (emphasis on "kwargs") is a bit of a Python-ism. These are JSON-serializable parameters that are needed by the code that reads the data. This code does not necessary have to be in Python. In Tiled, the analogous entity is simply called parameters.

The existing names aren't doing any harm, so we have to weigh the pain of change against the marginal benefit of maybe-better names. If we leave them as is, I think that would be fine. Just worth considering.

The way we spell the location of the data seems more problematic:

• root, an absolute filepath
• resource_path, a relative filepath (sometimes filename)
• path_semantics, a two-member enum that is win or posix

I think we should consider replacing all of this with a URI, like file://localhost/dls/i22/data/2023/cm33873-5/i22-723667-waxs-hdf.h5".

1. We foresee data being available in protocols other than file:, such as s3:.
2. The partitioning of the path between root and resource_path is a guess about what parts of the path will be stable over time and what parts may change as the data moves. Different people make different guesses, and at NSLS-II this has been a mess.
3. IIRC, path_semantics is completely vestigial, based on a misunderstanding that Windows required backslashes in paths. In fact, anything newer than Windows 95 is fine with forward slashes.

In Tiled, the same logical resource may be available in multiple places. This is expressed as a one-to-many relation between logical datasets and URIs (effectively). If the StreamResource just had a URI, this would be simpler and more future-proof.

Here is a StreamDatum:

  {
    "name": "stream_datum",
    "doc": {
      "stream_resource": "158167d9-796b-4e12-afff-5b3dd903815c",
      "uid": "158167d9-796b-4e12-afff-5b3dd903815c/2",
      "seq_nums": {
        "start": 6,
        "stop": 7
      },
      "indices": {
        "start": 5,
        "stop": 6
      },
      "descriptor": "8ff32942-9b21-4542-aee1-16c04291950d"
    }
  }

The descriptor is on StreamDatum instead of on StreamResource with data_key. I think it's worth revisiting whether we really really need that, because life would be simplify if the descriptor and data_key were together on the same document.

[tacaswell]

I'm agnostic to the key name changes.

Adding the file: vs s3: vs ... should be safe to do in a back-compatible way if we assume that anything with out a protocol is file:. I think this can be done independent of any other changes.

I'm hesitant to give up all information about which parts of the uri are mounting / hosting details and which are "real". Although this came from a time when we had the same filesystems mounted under very different paths on different machines with much higher regularity, having the ability to "make the path relative" (for lack of a better term) seems generically useful (maybe we should have gone with a single integer rather than two parts of the path). However, given the growing usage of containers and the trivial ability to be re-mount any level of path on the host to any level of path in the container it may make sense to lift all of this up to be a client configuration problem (which opens the door to much more complex re-writing rules).

"it has not been well used" is not a compelling argument to remove it (it is a compelling argument for better docs!), but "it does not actually solve the problem" is.

[callumforrester]

I'm also agnostic to the key name changes, maybe leaning slightly in the direction of pro, especially for mimetype and general reduction of the specialness of the documents.

I'll hold my hands up and say I hadn't noticed that stream_datum referenced descriptor. I agree that's not necessarily needed. If we move it to stream_resource do we need to keep the reference to run_start as well?

I agree with @tacaswell that the advent of containers is likely to make different path mounting more well-used. In fact I strongly encourage it because baked in assumptions of multiple machines using the same mount point has bitten me many times!

Also happy with the file:// and s3:// idea. If I were designing from scratch I might make it more structured

{
    "protocol": "file",
    "location": "/data/foo/bar"
}

But if :// is better for backwards compatibility then so be it :)

[danielballan]

Exactly. The existing names aren't bad, but if we can use widely-known names and standards, developers don't even have to notice them.

Let's talk about the location of descriptor on a future call. I think we should keep run_start either way, to effectively save on a JOIN (a literal JOIN in SQL, a metaphorical JOIN in MongoDB...).

I like structure things, but if we care about covering s3: and others, we'd need to cover the full URL spec, as in:

{'scheme': 'file',
 'netloc': 'localhost',
 'path': '/a/b/c',
 'params': '',
 'query': '',
 'fragment': ''}

and I think the string form is probably the easiest thing to pass around in documents. Not a strongly-held opinion, though.

[callumforrester]

I'm not too attached to structure either but I don't think the full URL spec is a terrible thing to pass around. I think it might be best to see if anyone does hold strong opinions either way.

[danielballan]

That is fair…and making me wonder whether Tiled should store the URI in a structured form at rest. That could enable indexing and more efficient filtering by scheme or netloc, which could come up…

[danielballan]

From discussion:

• Consider moving descriptor from stream_datum to stream_resource. This can be done by:
  • Updating schemas in event-model
  • Updating RE with a simple change: to tack the descriptor on to a partial stream_resource received from ophyd, and re-emit when it sees stream_datum pointing to a stream_resource in a stream that hasn't had one before.
  • No changes to ophyd at all
• [For @tacaswell: Let the record show that @danielballan did not suggest this renaming.] Should we rename stream_resource and stream_datum because "stream" is overloaded? No good suggestions were made for a better name.

[tacaswell]

If we move the descriptor to StreamResource that implies that we can have multiple StreamResource with overlapping keys for the same stream to support changing the configuration mid-stream. I suspect that is going to bring a bunch of complications down the line a well.

[danielballan]

Even so, think it is the lesser evil. We can discuss. There is a tension here between what is easy for the producer and what is simpler (and performant) for the consumer.

[callumforrester]

I must confess to getting a bit lost among text descriptions of relational hierarchies, so I drew it out as a sanity check, does this look right?

Top is current state, bottom is proposed change:

Also tagging @coretl

[coretl]

If we move the descriptor to StreamResource that implies that we can have multiple StreamResource with overlapping keys for the same stream to support changing the configuration mid-stream. I suspect that is going to bring a bunch of complications down the line a well.

@tacaswell we already reduced the scope of StreamResource so that it only points to a single data_key, so I don't think keys could overlap? We also already need multiple StreamResource documents per Descriptor because we emit a new one per HDF file, and pausing/resuming is planned to start writing into a new file.

I don't think I have many opinions either way here, both options are the same complexity to produce in ophyd-async, and not much different in the run bundler, so happy with either.

[coretl]

Also tagging @DiamondJoseph for comment

[danielballan]

Filling in the motivation for my terse notes above:

The Tiled adapter that loads Bluesky documents has to chase a lot of references, illustrated in Callum’s diagram, to answer the simple question, “What data do you have?” The overhead is especially unfortunate when it is asked to return pages of search results describing ~300 datasets at a time.

Placing descriptor and data_key in separate documents sets us up for more of this. It is maximally-normalized, but adds complexity and a JOIN. I don’t feel the trade-offs have been fully explored, but I think we should examine whether a little de-normalization (reissuing StreamResource for each Descriptor) could be worthwhile here.

[callumforrester]

Also also tagging @garryod

[DiamondJoseph]

Discussed with TC just before lunch, we have assumptions about Descriptors which motivate keeping Descriptor reference on StreamDatum.

• A new Descriptor for an existing stream can be emitted during a scan
• A Datum cares about which Descriptor it is relevant for, while a Resource only cares about which stream/data_key

If a new Descriptor being emitted also emitted a new Resource: requires that the Device know when the Descriptor changes (for the DatumFactory), or that the Datums are modified to point to the correct Resource (although I suppose currently they have their Descriptor injected?)

[danielballan]

keeping Descriptor reference on StreamResource

Do you mean keeping the Descriptor reference on the StreamDatum, or moving it to the StreamResource? I think you mean the former, just checking.