ROSA is an AI Agent designed to interact with ROS-based robotics systems
using natural language queries.
ROSA is an AI agent that can be used to interact with ROS1 and ROS2 systems in order to carry out various tasks. It is built using the open-source Langchain framework, and can be adapted to work with different robots and environments, making it a versatile tool for robotics research and development.
- Support for both ROS1 and ROS2
- Generate system reports using fuzzy templates
- Read, parse, and summarize ROS log files
- Use natural language to run various ROS commands and tools, for example:
- "Give me a list of nodes, categorize them into
navigation
,perception
,control
, andother
" - "Show me a list of topics that have publishers but no subscribers"
- "Set the
/velocity
parameter to10
" - "Echo the
/robot/status
topic" - "What is the message type of the
/robot/status
topic?"
- "Give me a list of nodes, categorize them into
- Control the TurtleSim robot in simulation using ROSA
- Easily adapt ROSA for your robot by adding new tools and prompts
- Use multi-modal models for vision, scene understanding, and more (in-progress)
- Web-based user interface with support for voice commands (in-progress)
- Text and speech modalities for human-robot interaction (in-progress)
ROSA already supports any robot built with ROS1 or ROS2, but we are also working on custom agents for some popular robots. These custom agents go beyond the basic ROSA functionality to provide more advanced capabilities and features.
- Custom Agent for the TurtleSim robot (see turtle_agent)
- Custom Agent for NASA JPL's Open Source Rover
- Custom Agent for the TurtleBot
- Custom Agent for the Spot Robot
This guide provides a quick way to get started with ROSA.
- Python 3.9 or higher
- ROS Noetic (or higher)
Note: ROS Noetic uses Python3.8, but LangChain requires Python3.9 or higher. To use ROSA with ROS Noetic, you will need to create a virtual environment with Python3.9 or higher and install ROSA in that environment.
pip3 install jpl-rosa
from rosa import ROSA
llm = get_your_llm_here()
rosa = ROSA(ros_version=1, llm=llm)
rosa.invoke("Show me a list of topics that have publishers but no subscribers")
We have included a demo that uses ROSA to control the TurtleSim robot in simulation. To run the demo, you will need to have Docker installed on your machine.
The following video shows ROSA reasoning about how to draw a 5-point star, then executing the necessary commands to do so.
turtle_demo.mov
- Clone this repository
- Configure the LLM in
src/turtle_agent/scripts/llm.py
- Run the demo script:
./demo.sh
- Build and start the turtle agent:
catkin build && source devel/setup.bash && roslaunch turtle_agent agent
- Run example queries:
examples
ROSA is designed to be easily adaptable to different robots and environments. To adapt ROSA for your robot, you
can either (1) create a new class that inherits from the ROSA
class, or (2) create a new instance of the ROSA
class
and pass in the necessary parameters. The first option is recommended if you need to make significant changes to the
agent's behavior, while the second option is recommended if you want to use the agent with minimal changes.
In either case, ROSA is adapted by providing it with a new set of tools and/or prompts. The tools are used to interact with the robot and the ROS environment, while the prompts are used to guide the agents behavior.
There are two methods for adding tools to ROSA:
- Pass in a list of @tool functions using the
tools
parameter. - Pass in a list of Python packages containing @tool functions using the
tool_packages
parameter.
The first method is recommended if you have a small number of tools, while the second method is recommended if you have a large number of tools or if you want to organize your tools into separate packages.
Hint: check src/turtle_agent/scripts/turtle_agent.py
for examples on how to use both methods.
To add prompts to ROSA, you need to create a new instance of the RobotSystemPrompts
class and pass it to the ROSA
constructor using the prompts
parameter. The RobotSystemPrompts
class contains the following attributes:
embodiment_and_persona
: Gives the agent a sense of identity and helps it understand its role.about_your_operators
: Provides information about the operators who interact with the robot, which can help the agent understand the context of the interaction.critical_instructions
: Provides critical instructions that the agent should follow to ensure the safety and well-being of the robot and its operators.constraints_and_guardrails
: Gives the robot a sense of its limitations and informs its decision-making process.about_your_environment
: Provides information about the physical and digital environment in which the robot operates.about_your_capabilities
: Describes what the robot can and cannot do, which can help the agent understand its limitations.nuance_and_assumptions
: Provides information about the nuances and assumptions that the agent should consider when interacting with the robot.mission_and_objectives
: Describes the mission and objectives of the robot, which can help the agent understand its purpose and goals.environment_variables
: Provides information about the environment variables that the agent should consider when interacting with the robot. e.g. $ROS_MASTER_URI, or $ROS_IP.
Here is a quick and easy example showing how to add new tools and prompts to ROSA:
from langchain.agents import tool
from rosa import ROSA, RobotSystemPrompts
@tool
def move_forward(distance: float) -> str:
"""
Move the robot forward by the specified distance.
:param distance: The distance to move the robot forward.
"""
# Your code here ...
return f"Moving forward by {distance} units."
prompts = RobotSystemPrompts(
embodiment_and_persona="You are a cool robot that can move forward."
)
llm = get_your_llm_here()
rosa = ROSA(ros_version=1, llm=llm, tools=[move_forward], prompts=prompts)
rosa.invoke("Move forward by 2 units.")
See our CHANGELOG.md for a history of our changes.
See our releases page for our key versioned releases.
Questions about ROSA? Please see our FAQ section in the Wiki.
Interested in contributing to our project? Please see our: CONTRIBUTING.md
For guidance on how to interact with our team, please see our code of conduct located at: CODE_OF_CONDUCT.md
For guidance on our governance approach, including decision-making process and our various roles, please see our governance model at: GOVERNANCE.md
See our: LICENSE
Key points of contact are:
Copyright (c) 2024. Jet Propulsion Laboratory. All rights reserved.