RegoloBot is a project that utilizes a UR5 robot to replicate numbers and operations using math sticks (regoli in italian). This project has been developed using the ROS (Robot Operating System) framework and MoveIt library for robot control.
Ros Noetic: http://wiki.ros.org/noetic/InstallationGazebo: https://classic.gazebosim.org/tutorials?tut=ros_installing&cat=connect_rosMoveit: https://moveit.ros.org/
- Clone this repo:
git clone https://github.com/kodai2199/regolobot.git- Installation of the UR5 robot. You can clone the official UR5 robot repository by running the following command:
git clone https://github.com/ros-industrial/universal_robot -b noetic-develInside the catkin workspace, run the following command to start the simulation:
./src/rapid_start.shThe environment that we created on gazebo in the file regolobot.world simply consists of a table, that rapresents the working area, and a pedestal with the UR5 robot on it. We also added a kinect camera for vision purposes.
Math sticks are an educational tool typically used in the early years of primary school with pedagocial purposes.
There are 10 of them and every color is associated with a different lenght:
We created them using Blender and we converted them in sdf files to integrate them in Gazebo environment. They are saved in the models directory.
Launching the command:
rosrun regolobot simulation_controller.pyit is possible to spawn the desired number of math sticks on the table.
The vision system we set up, utilizes the Gazbos's kinect camera to recognize the math sticks on the table. We trained the neural network with YOLO architecture, using 5000 images, to retrieve the type and bounding box of math sticks, using this convention:
class_index, x1, y1, x2, y2, x3, y3, x4, y4
The classes go from MS01 to MS10 and rapresent the math stick type (therefore color and lenght), while the coordinates are the 4 corners of the bounding box.
There is a server Flask running on one of our pc at this URL that will sends as output the predicted labels.
In the solution.py file, is implemented a back tracking algorithm that outputs a list rapresenting the layout of the (approximate) optimal solution.
We have come up with a system for rapresenting numbers using math sticks that also involves multiplication and addition: when the math sticks are arranged vertically, it indicates addition, while when they are arranged horizontally, it signifies multiplication.
For example, with these inputs:
The number 942 can be rapresented as:
See the regolobot.mkv video for a demonstration of the UR5 disposing the math sticks.