[Bug]: Teleoperation couldn't work after adding both robotic arms in Setup Environment

[zhuo-yoyowz]

Description

I met the issue of not able to teleoperate after setting up the env, adding both robotic arms and 1 camera to a new project. I'm using a Ubuntu 24.04 ARL laptop. Both arms were smoothly added to the project (with calibration, verification, etc.). But when during "Setup environment*, can't perform teleoperation. Seems no teleoperation steps exist in adding robot arms. And

teleoperation couldn't work

Steps to Reproduce

Add new camera
Add new camera

• Select USB Camera. - Set camera name of video input of the model. - Select the camera from the list. - Select resolution and FPS and check preview to verify correct camera - Press Add camera
  Repeat for all the points of view for the robot.

3. Setup environment
   Objective: Define environment for robot

This environment will define your robot and the cameras. This will be used in the dataset to determine the input features of the models.

Configure new environment
Configure new environment

• Press Configure new enviroment - Select previously defined follower. - Select the leader robot. - Press Add
  Add cameras
  Select cameras from list
  Press Add
  Verify
  Verify new environment

• Press Configure new enviroment - A preview will be shown with the robots and camera point of views. - Verify the robots by moving them in real time. - Press Add Environment

Expected Behavior

Perform teleoperation between arms, so as to prepare for the next step of dataset collection

Actual Behavior

No teleoperation between follower and leader arms, although both arms are identified and added to the platform.

Component

UI (React frontend)

Environment

Logs / Error Output

Additional Context

No response

[samet-akcay]

@MarkRedeman @RHeckerIntel any thoughts ?

[RHeckerIntel]

@zhuo-yoyowz Hi there! Could you perhaps add the log of the backend?

[MarkRedeman]

Hi @zhuo-yoyowz thanks for reporting! This is one of our features that hasn't been fully finished yet.
At the moment this screen only shows you the state of both of your robots: if you move one of them then you should see that same movement on the screen.

To start teleoperation (and recording your dataset) you need to go to the dataset tab to first create a new dataset:

Then press "Start recording"

And after filling in the task and clicking "start" you will get the recording + teleoperation screen:

---

In the next few days we will clean up the UI a bit more so that these flows will be more obvious.

[zhuo-yoyowz]

Hi @zhuo-yoyowz thanks for reporting! This is one of our features that hasn't been fully finished yet. At the moment this screen only shows you the state of both of your robots: if you move one of them then you should see that same movement on the screen.

To start teleoperation (and recording your dataset) you need to go to the dataset tab to first create a new dataset:

Then press "Start recording"

And after filling in the task and clicking "start" you will get the recording + teleoperation screen: In the next few days we will clean up the UI a bit more so that these flows will be more obvious.

Hi, thank you for answering! It works for me on teleoperation! However, met new issue in collecting the dataset. Don't know what to do after clicking "accept" when finishing recording an episode. If I clicked "adding episode", it just went back to the previous page and showed "No episodes yet". So when I performed recording one episode, after clicking "accept", what should I do next to actually store the dataset? Hope you could provide guidance here too. Thank you!

[MarkRedeman]

So when I performed recording one episode, after clicking "accept", what should I do next to actually store the dataset? Hope you could provide guidance here too. Thank you!

I think there might be two bugs that could be happening:

1. When you click "Accept" after recording the episode the server needs some time to process the episodes' videos in order to save it. This might take 5 - 30 seconds (depends on how long the video is and your hardware etc)
   The "Accept" button should become disabled during this time and show a small loading spinner. Once it's finished loading you should see the same previous screen again with a "Start episode" button. The intention here is that after you click "Accept" you will reset your environment before starting the new episode.
2. When you go back to the previous page (Clicking the back button next to "Adding episode" in the header) it can take a little while before the UI shows you the newly recorded episodes. If it still does not show after a while then refreshing the page might help.

If it's none of these two then seeing the backend logs will probably help as then there might be an unexpected bug during saving.
You can find the logs at ~/.cache/physicalai/logs (or if you're using Docker they will be under /app/storage - see the docker readme)

[zhuo-yoyowz]

Thank you for the suggested solution above. I retested today. First, I met the issue of not being able to load the already setup env. So I tried to delete it and create a new env for dataset recording, but not able to do it. Then I fetched the recently updated code from the repo, and relaunched everything in my env again. However, now I couldn't open the UI. The error log is attached. Hope you could help. Thank you!

Log_260309.txt

[MarkRedeman]

It looks like the database migration from #295 did not run yet.
Can you try running,

uv src/cli.py migrate

from the backend folder, and then check if the issue persists?

Alternatively you can also run the backend using our run.sh script, this one will first check if there are any database migrations that should be run and start the backend after this is done.

I've got an open PR at #326 that should hopefully resolve some of these configuration/deployment issues.

First, I met the issue of not being able to load the already setup env

Do you mean that you couldn't see the environment on the environments page, or that perhaps the camera was no longer loading?

[zhuo-yoyowz]

Thank you for your suggestion, Mark. Using the run.sh script helped me to migrate the database successfully. Also, I found the issue of not showing recorded episodes on the UI is due to lack of ffmpeg dependency, leading to the error of torchcodec failed to load. After installing ffmpeg, the issue is solved. Now I could run training successfully on my iGPU with my recorded episodes.

[zhuo-yoyowz]

BTW, when recorded episodes for training dataset, I noticed there's no number showing how many episodes have been recorded. If it's possible, maybe adding a count number showing currently how many episodes have been added to the dataset is better?

[MarkRedeman]

BTW, when recorded episodes for training dataset, I noticed there's no number showing how many episodes have been recorded. If it's possible, maybe adding a count number showing currently how many episodes have been added to the dataset is better?

Thanks for the feedback! And yes I totally agree we need to add that. I created #330 yesterday with a list of improvements we really need to make before marking our first "initial release", the tasks on this ticket are mostly meant to improve the initial user experience of using the application.

Thank you for your suggestion, Mark. Using the run.sh script helped me to migrate the database successfully. Also, I found the issue of not showing recorded episodes on the UI is due to lack of ffmpeg dependency, leading to the error of torchcodec failed to load. After installing ffmpeg, the issue is solved. Now I could run training successfully on my iGPU with my recorded episodes.

Ah it sucks that we don't show a proper error message in the UI in that case. I will try to reproduce this on a VM later and update our docs to include it as a prerequisite.