Transitioning Arduino Code to AutoPilot

[jmdelahanty]

Hello fellow autopilots!

Our lab is considering moving into the wonderful world of autopilot for running a distributed set of behavior boxes used for training mice. We currently run via Arduinos which works, but they're all controlled by a single MATLAB instance and the boxes cannot be run independently of one another. Autopilot seems like a perfect solution to this problem!

We've ordered a couple Pi's to test with that will arrive in the coming months, so while we wait I figured it'd be a good time to start porting what was on our Arduino's to Autopilot!

Here's a link to the Arduino code that we'd like to port over. The training boxes don't require much of the functionality that's in it (i.e. stimulations, catch trials) so it'll hopefully be simpler.

EDIT: It would actually be cool to have those things enabled in the training in case someone wants to do that in the boxes. Might as well do it anyways...

The way the Arduino runs is that a Python program called bruker_control generates a pseudorandom trial list that follows certain rules. From there, specific arrays are generated for ITI timings, tone durations, and stimulation offset timings. These arrays are transmitted to the Arduino over a serial connection and then the Arduino iterates through the trials until the specified number is complete. The configuration that was used with precise timings is then written in a JSON format to disk so the exact parameters can be rerun again by someone else if they'd like with literally the same trial set and also so the experimental procedure is documented precisely for each run and programmatically accessible by anyone (autopilot inspired!).

I've been confused by trying to use PyDAQmx on my own so we could avoid using Arduinos and run experiments exclusively through Python, but got stuck having a camera record video independently of the routine for sending stimuli to the mouse. I was thinking I could basically use the same structure as the arrays that are sent through to the Arduino when sending things via PyDAQmx and have a while loop that updates new timings and checks against flags like the Arduino does for when to send different stimuli, but it didn't seem like that's how people do it in Python as much or, if it is how people do it, I haven't seen that many examples yet.

How hard do you think it might be, fellow pilots, to port this Arduino functionality to autopilot? Can I use the same code that generates trial structures, use similar configuration files, and store things in directory structures as we currently do?

[sneakers-the-rat]

Hello Jeremy! Can you post some examples of the arrays that you're describing here? maybe give an example play-by-play of "X value used by Y hardware, then Z value used by.."

It should be very possible to write a Task that, given some parameterization like a list of possible ITIs/etc. to generate them like in the bruker_control function (or take whatever it outputs), and the trial timings can be saved in whatever output data you generate from the task, however you'd like it. Shouldn't be too hard!

[jmdelahanty]

not sure what you mean using GPIO to send stimuli, they can do pulses and stuff but i wouldn't like play sound out of them.

I didn't realize how unclear that was lol, I meant send pulses out. Using speakers is something that we need to do though! All we're playing is a higher pitch tone at 9kHz and a lower pitch at 2kHz for the mice. I know that you have examples in your code and docs for how to use it so I will check those out too!

you can sample however you want -- if you want to sample events continuously, see the record arg and the events attr for Digital_In, and then you can save that by just streaming events back to the terminal like any other continuous data

I'm not sure which one is best to do! I think really all we need is timestamps of events starting/stopping which you've already shown here how to do, but I feel like people having the "raw" data sampled out to a file is something they'd want in this lab. Maybe I can do both, that would be neat.

[jmdelahanty]

Something I just realized I needed to ask that's relevant here: Ideally what's running behavior stimulation through the scope is through something like Autopilot. But the machine that runs the scope's software is Windows and there's no Linux distro.

Do you think implementing these things in an Autopilot like fashion is something we can do? Would we invoke these things as a subprocess that goes on while the scope is taking pictures and opencv writes the video to disk?

[sneakers-the-rat]

I'm not sure what you mean here, you mean like you have one computer that's running like scanimage or something on windows? Typically I would expect you would be running the behavioral experiment part on a raspi (raspiOS or similar)? To coordinate with a separate windows machine you could communicate with network messages (zmq), serial, or just with a ttl pulse, but I don't think I really understand the question. Maybe a block diagram of the hardware setup you have in mind?

[jmdelahanty]

Here's what we currently have going:

And here's what I had meant to say:

The way things effectively work right now is basically:

1. Python does the setting up: Gets the scope/scope's software ready, generates trials/ITIs/tone durations/etc and sends everything to the Arduino.
2. The Arduino kicks everything off by sending a Go signal to the microscope. All the data that is written to disk is collected according to the Windows' machine clock that's governing the microscope's software.

The thinking behind that was that it would be best if all the data was recorded by one machine and united to the same software so timestamps would all be the same across each datastream. I messaged with Talmo briefly and he suggested doing an interpolation on every datastream's timestamps to put them into a common time base. He also suggested we gather TTL pulses for every time the camera takes an image to have external timestamps. I used to do that, but I only have 7 DAQ channels available and we're pushing that as it is now. Also, since the camera takes a picture only when the microscope takes a picture, it was decided that gathering that information wasn't necessary (for better or worse). This is the part that would be changing the most.

3. Python collects all the video frames specified by the length of the recording (there's a number of frames needed calculated in video_utils). Once that's reached, Python sends a kill signal to the scope's API and ends the experiment.

[sneakers-the-rat]

Oh yeah totally -- that's how we're using it in a few experiments atm.

tasks are started by the Terminal by sending a 'START' message with a task description to a pilot object, that looks something like this

from autopilot.networking.node import Net_Node

node = Net_Node(...)
node.send(to='pilot_id', key='START', value=task_params)

and then the pilot runs it. See https://github.com/wehr-lab/autopilot/blob/e408c08e76df8d8c930edfb4ba58e31cd1ec4d87/autopilot/core/pilot.py#L271 and https://github.com/wehr-lab/autopilot/blob/e408c08e76df8d8c930edfb4ba58e31cd1ec4d87/autopilot/core/pilot.py#L723

Typically that's done from a Terminal GUI instance, but there's nothing really special about it except that currently the pilot has an unfortunately hardcoded return address for the data as 'T', but that too would be pretty easy to replace (and that is the plan). So you could just send a message using a Net_Node class named 'T' and it would pretty much be the same (though you'd have to mimic its handling of the Subject class to store data. working on that at the moment actually).

re: synchronization, TTL pulses are always useful as redundancy. Another thing you could do is set the main windows machine to be an NTP server and then setup the pi to synchronize its system time off it, which should be ~us precise over a quiet local network.

One thing you could do if you need to double up TTL pulses on one channel is just to invert the polarity of one of the inputs so it goes negative (or attenuate one so you have two pulse heights, idk)