Intentionality in (not) bypassing the queueserver

[padraic-shafer]

This is not a package issue per se; it’s a bluesky integration consideration that converges at the queueserver. I want to capture a discussion in progress, and we could move this to a more appropriate venue if someone prefers.

TL;DR: Bypassing the queueserver to control hardware should be discouraged OR an intentional, transparent decision that provide some safeguards.

—-
I’ve been discussing with @dylanmcreynolds and @danielballan the importance of having only one gateway for controlling the state of beamline/endstation during beamtimes. Queueserver naturally fills this role. I think we don’t have way to enforce this restriction currently. Rather than passing plans to queueserver, someone could concurrently access the RunEngine directly at a beamline terminal, or by commanding a movable ophyd device, or by setting an EPICS PV, etc. There are potentially valid reasons for having these multiple pathways, but if left unchecked this is a recipe for chaos for multi-user/multi-client interactions with the beamline.

I don’t propose we try to solve that larger problem here, but there are a few thoughts on the interplay between ophyd, RunEngine, and queueserver that I would like to share and get feedback on.

1. Even simple motor moves should be plans sent to queueserver.
   a. This ensures that motor moves do not bypass the queue or preempt the intentions of the experimenters.
   b. In the context of a bluesky run, ophyd is effectively a middle layer—an implementation detail that I think we would not want users to fiddle with directly.
   c. This concern has particular relevance for controls GUIs where the designer of the client application might be tempted to run scans through queueserver and provide a motor widget that directly accesses ophyd objects.

2. What if I intentionally want to jog motors while a plan is running?
   a. @danielballan noted that it’s a common paradigm to set a detector in continuous capture mode while one or motors are jogged. If the detector is started as a bluesky run, then the motor must be moved by ophyd (or EPICS, or something outside of bluesky).
   b. This mode should be supported. If it ends up being the only notable use case, then perhaps there should be a dedicated escape hatch built into the RunEngine (and the queueserver by extension) to handle this.
   c. Maybe this is just a special case of a flyer plan?

3. Auto-generated GUI (e.g. Typhos) instantiates ophyd objects
   a. We should not destroy the functionality of packages that create GUI widgets for moving motors and setting signals, which do so by inspecting a database of ophyd objects and connecting to them.
   b. We could consider a proxy object that supports this use, while still enqueuing the motor moves. Example An ophyd interface that delegates motor moves to an ophyd motor indirectly by adding that move as a queued plan, and perhaps inspects the queue as part of the readback status.

4. Alignment/troubleshooting vs. data collection
   a. There are times, where using a terminal that directly communicates with hardware or with the RunEngine is simply more convenient. This might be for troubleshooting or for rapid alignments before starting the collection of publication-quality data.
   b. To enable this sensibly, there should probably some mechanism for putting the beamline in a “manual” state—pause the queueserver and pass control to an admin user.

[padraic-shafer]

Tagging several people here for opinions. If there is interest we consider scheduling a community call on this topic.

@tacaswell @mrakitin @dmgav @taxe10 @coretl @callumforrester @rodolakis @prjemian @ZLLentz @klauer @whs92 @clintonroy @ksunden @untzag

[whs92]

We are very interested in this topic and would welcome a call.

[coretl]

2. What if I intentionally want to jog motors while a plan is running?

We are explicitly separating the scanning layer which should only be able to run one plan at a time, from the controls layer which is where the GUI generation, PV live updates and archiving happens. The majority of these jogging moves will be done by beamline staff, who will have read-write access to PVs, so they can use the controls GUIs during a scan. However, we intend to turn the controls GUIs read-only for users, and they occasionally need to jog motors during a plan. For this case we could either turn specific PVs read-write for users, or create a second run engine with some motors in it for this live access. I'm not sure which we will go with at the moment...

[dmgav]

I just wanted to note that the features needed for (1), (2) and (4) already exist in the Queue Server. It was discussed earlier that there should be a separate service for monitoring and direct control of PVs. There was a project started by @untzag to implement the service, but I can not find the repository.

[untzag]

You're thinking of @ksunden's project,

https://github.com/ksunden/bluesky-hwproxy

I only had some design input 😄

For what it's worth we've been using bluesky-hwproxy for more than a year now, it's just perfect for our small-potatoes workflow. It does exactly what @padraic-shafer suggests by making bluesky-protocol compliant objects on the client side---you can just plug it in to anything that expects such an API. We made it to do exactly what you are talking about---keep existing "fancy" GUI features alive in a queueserver world.

importance of having only one gateway for controlling the state of beamline/endstation during beamtimes

I'm not sure exactly why this is so important, but it seems to me that the proxy could be written to support set operations routed via queueserver.

One of the core design features of queueserver is the disposable worker process. Queueserver has no mechanism for interacting with the hardware between plan runs, as far as I know... maybe the newer support for arbitrary ipython stuff changes that story? The important thing about hwproxy, for us, is that it's always online and ready to serve fairly up-to-date hardware state information. It's a lot easier to write clients if you assume that there is always a reliable place to grab current hardware state.

I still think that message caching should be carefully considered as another approach that might work better for larger deployments. If every change to hardware state is captured through the runengine, you should be able to seek backwards through the documents and piece together the current state of the instrument.

[dmgav]

My understanding is that item (3) refers to a web tool for monitoring and/or modifying PVs (scaled-down version of CSS). It involves continuous monitoring of large number of PVs and should not be done in the same process with running plans. This is the reason for https://github.com/ksunden/bluesky-hwproxy. The existing features of the queue server allow to implement workflows that require occasional access to PVs (e.g. to read a single value), but it is not recommended to use it for continuous monitoring.

[dmgav]

Jupyter Console can not be used to interfere with a running plan started by the Queue Server, since IPython kernel can execute only one cell at a time and the console stays unresponsive when the server is executing a plan or some other task. Jupyter Console works only when the server is idle or a plan is paused. Jupyter Console can not connect to a worker using pure Python (which is the default mode).

[ksunden]

To be clear, hwproxy currently is explicitly limited to readonly access. It's goal was to communicate things like limits/current position to control software without having to run a plan (and therefore interrupt a currently running plan). It does not provide a mechanism to set values (though presumably it could, just would require more careful consideration than we needed for our task)

Note that under the hood it is just creating parallel connections to hardware, so if an individual hardware requires that the bluesky object is in process and is not controlled by EPICS/yaq/etc. which allow multiple connections to the underlying device, then it won't work.

In practice, we have a program that knows nothing of bluesky at all that provides the "Engineering Interface" for non scan/data collection setup and initial tuning

There is nothing actually stopping you from using that mid-scan, but it is recommended against. Though if e.g. you notice your shutter is still closed 10 seconds into an hour long scan (in a region without interesting data in the first place), then you can just open the shutter using the other program without interrupting the scan.

[padraic-shafer]

For this case we could either turn specific PVs read-write for users, or create a second run engine with some motors in it for this live access. I'm not sure which we will go with at the moment...

I think the second option (run engine + environment) has an advantage in that it could be generic--R/W access controlled at the ophyd level because it might not be EPICS under the hood.

[padraic-shafer]

importance of having only one gateway for controlling the state of beamline/endstation during beamtimes

I'm not sure exactly why this is so important, but it seems to me that the proxy could be written to support set operations routed via queueserver.

I think that captures the essence. There could be multiple entry points for reading, but requests to mutate the state need to be serialized in some way. It might just be an agreement among a beam time user group to discuss while they are operating the beamline, but user_1 (at home) should probably be prevented from jogging the sample position or energy while user_2 (in hotel room) is running scans. When everyone is in the same room, this is a lot simpler; but we should not assume that is the case.

But, as mentioned above, there are cases where user_1 should be able to jog the motor while user_1 is running a scan. Maybe that's a useful cue, that only one user is in charge at any time. It's a discussion for another day what to do when user falls asleep during an overnight scan and user_2 needs to take charge. :) E.g., QS has a mechanism for unlocking the queue with a password that can be shared amongst the team.

[padraic-shafer]

I'm encouraged by the discussions of hwproxy above that multiple coexisting services could be a useful way to go. This suggests that a higher level coordinating service or gateway is then needed.

[dmgav]

https://github.com/ksunden/bluesky-hwproxy is reading PVs using Ophyd objects. I don't think wrapping each PV read into a plan and calling Run Engine would provide any benifit.

[dmgav]

This suggests that a higher level coordinating service or gateway is then needed.

https://github.com/bluesky/bluesky-httpserver could be extended to coordinate multiple low level services. Now it is simply forwarding REST API requests to the Queue Server, but the original intention was that it will implement higher level API.