Revisit need for tiled/databroker dependency

[thopkins32]

Current workflow:

1. Get next point to try
2. Run Bluesky plan step
3. Read back data from tiled/databroker
4. Pass to digestion function
5. Store outcomes
6. Update models

Key question: Should we remove tiled/databroker dependency and leave it up to digestion functions?

Bad aspects of current approach:

• Assumes specific usage of tiled and databroker
  • primary stream, only
  • loads everything into memory
• Pre-loading all of the data is wasteful, we should let users decide how to fetch their data efficiently
• Can't always tell what keys to actually fetch (e.g. image keys such as "det_image" when the readable name is "det"), leading to fetching everything as a backstop

We may be able to solve these problems in Blop, but it adds complexity.

For instance, we would need to add more arguments to specify what data to actually load and how to load it...but the user probably knows best. I also don't want to create a thin wrapper around databroker or tiled when they already have client APIs...

[danielballan]

Two cents from me, not yet having looked at this closely.

It would make sense to me that blop integrates well with Tiled at the level of building clean examples in the docs—an API to data is useful to avoid getting bogged down in file-based I/O—but it seems unnecessary for it to depend on Tiled. Wrapping the integration with the data access, whether it's Tiled or something else, in a pluggable function with an interface defined by blop makes good sense.

It should certainly not depend on Databroker, which is beginning a (lengthy) deprecation period.

[thopkins32]

"Digestion" converts acquired data -> outcomes configured for the optimization problem. E.g. analyzing an image produced by a detector to calculate average intensity, which the optimization aims to maximize.

It would make sense to me that blop integrates well with Tiled at the level of building clean examples in the docs—an API to data is useful to avoid getting bogged down in file-based I/O—but it seems unnecessary for it to depend on Tiled. Wrapping the integration with the data access, whether it's Tiled or something else, in a pluggable function with an interface defined by blop makes good sense.

Agreed. It will continue to exist in documentation and in our integration tests. We already support pluggable digestion functions, however, they currently assume that the data is fully read in from a data source (tiled/databroker) into a custom data structure that Blop maintains. This makes the digestion function a bit awkward to implement because users have to use Blop's data structure (which is just a dictionary) to access their data instead of the tools they are already familiar with.

The point of this issue is to remove this custom data structure and simply let the user load only the data they need, using the APIs they are already familiar with.

Essentially, we combine steps 3 & 4 above into simply "calling the digestion hook". We can have a simple default hook that uses a Tiled client, but in almost every case, a beamline will implement their own digestion function and fetch data within that function.