Representing Non-trivial Scan Shapes

[genematx]

A summary of what we have discussed with @hyperrealist.

Some plans collect data points in order that can not be easily reshaped from the acquired 1D sequence to a scientifically meaningful 2D (or higher-dimensional) representation, e.g. snake-scans, spiral scans, etc. It was decided to produce this data representation via custom endpoints, e.g. /custom/snake instead of /array/full; the associated processing code can be implemented separately and declared in the server config (see the /documents endpoint in databroker for a precedent).

A few points that still need to be clarified, which would have impact on TiledWriter.

1. Where to mark the stream (or a specific data key) as adhering to a certain scan spec. "Start" document has been proposed as the obvious solution, but I would like to explore other available options. "Descriptors" maybe?

2. How to encode the information about the scan spec in an efficient way, so we don't bloat the bluesky docs and metadata with unnecessary information. Saving the explicit order of point indices seems redundant. Perhaps, the scan spec can be parameterized, e.g. in the simplest case, by (1) enum type of scan ("snake", "spiral", etc) and (2) the number of points in a single row. This information, {"scan_spec": {"kind": "snake", "nrows": 42}}, along with the total number of acquired points, which is known from the data shape, is enough to fully reconstruct the trajectory.

3. Where to store the "scan_spec" information in Tiled. Apart from keeping it in the Start document, it seems to be useful to have this information readily available on the array node that needs to be reshaped itself, e.g. it can be stored in its metadata. In this case, the path operation function in the custom FastAPI router that processes the /custom/snake endpoint would already have this info in the entry's metadata and wouldn't need to traverse the bluesky container to find the start document. TiledWriter can be made responsible for placing this additional metadata at the array node.

[danielballan]

1. Storing things as far "down" (where Descriptors are "lower" than Run Start) makes sense. I'm not sure I have the full gamut of use cases in mind though, so it may need discussion about whether Descriptors are too far down.
2. See plan_pattern and plan_pattern_args (already in the Run Start) for inspiration.
3. Good point. It could be in the metadata or it could be in the properties. When we get to storing multiple copies of a given dataset (i.e. multiple Data Sources per Node) I could imagine storing "raw" and "pre-remapped" copies, in which case we'd be glad to have the layout stored on each Data Source (properties) rather than on the the Node. In that sense, it's akin to chunking.

[genematx]

Hmm, thank you, Dan! Good point about properties; I haven't thought about them. Would it mean though that if we store both versions of the data under the same node, their shapes would conflict, which would make access from the "standard" endpoints more complicated, if possible at all?

[hyperrealist]

Scanspec paths can be much more general, as in not conforming to a simple grid at all. For instance, a scanspec.specs.Spiral can look something like this:

from scanspec.plot import plot_spec
from scanspec.specs import Fly, Spiral

spec = Fly(Spiral("x", 1, 10, 2.5, "y", 5, 50))
plot_spec(spec)

But I don't expect much of a bloating, as we can re-compute the points from the serialized spec:

{"x_axis": "x",
 "x_centre": 1.0,
 "x_diameter": 10.0,
 "x_step": 2.5,
 "y_axis": "y",
 "y_centre": 5.0,
 "y_diameter": 50.0,
 "type": "Spiral"}

We should perhaps also include scanspec version information in there though.

Also tagging @coretl.

[hyperrealist]

I should clarify that above is one of the more general cases. Obviously, it cannot be reordered into a regular grid. The typical cases that can be ordered this way will also have a similar serialized scanspec, from which I can reconstruct the grid.