diff --git a/docs/debugging_assistant.md b/docs/debugging_assistant.md index 620dd62d2..3c8cedb8e 100644 --- a/docs/debugging_assistant.md +++ b/docs/debugging_assistant.md @@ -22,7 +22,7 @@ The ROS 2 Debugging Assistant is an interactive tool that helps developers inspe ```sh source setup_shell.sh -streamlit run src/rai/rai/tools/debugging_assistant.py +streamlit run src/rai_core/rai/tools/debugging_assistant.py ``` ## Usage Examples diff --git a/docs/vendors.md b/docs/vendors.md index acee2582a..b83c65a48 100644 --- a/docs/vendors.md +++ b/docs/vendors.md @@ -2,7 +2,9 @@ ## Ollama -For installation see: https://ollama.com/ +For installation see: https://ollama.com/. Then start +[ollama server](https://github.com/ollama/ollama?tab=readme-ov-file#start-ollama) with +`ollama serve` command. ```python from langchain_community.chat_models import ChatOllama @@ -10,6 +12,53 @@ from langchain_community.chat_models import ChatOllama llm = ChatOllama(model='llava') ``` +### Configure ollama wth OpenAI compatible API + +Ollama supports OpenAI compatible APIs (see [details](https://ollama.com/blog/openai-compatibility)). + +> [!TIP] +> Such a setup might be more convenient if you frequently switch between OpenAI API and +> local models. + +To configure ollama through OpenAI API in `rai`: + +1. Add `base_url` to [config.toml](../config.toml) + +```toml +[openai] +simple_model = "llama3.2" +complex_model = "llama3.2" +... +base_url = "http://localhost:11434/v1" +``` + +### Example of setting up vision models with tool calling + +In this example `llama3.2-vision` will be used. + +1. Create a custom ollama `Modelfile` and load the model + +> [!NOTE] +> Such setup is not officially supported by Ollama and it's not guaranteed to be +> working in all cases. + +```shell +ollama pull llama3.2 +echo FROM llama3.2-vision > Modelfile +echo 'TEMPLATE """'"$(ollama show --template llama3.2)"'"""' >> Modelfile +ollama create llama3.2-vision-tools +``` + +3. Configure the model through an OpenAI compatible API in [config.toml](../config.toml) + +```toml +[openai] +simple_model = "llama3.2-vision-tools" +complex_model = "llama3.2-vision-tools" +... +base_url = "http://localhost:11434/v1" +``` + ## OpenAI ```bash diff --git a/src/rai_core/rai/tools/debugging_assistant.py b/src/rai_core/rai/tools/debugging_assistant.py index 54edd4c6e..7693046a7 100644 --- a/src/rai_core/rai/tools/debugging_assistant.py +++ b/src/rai_core/rai/tools/debugging_assistant.py @@ -12,11 +12,13 @@ # See the License for the specific language governing permissions and # limitations under the License. +import rclpy import streamlit as st from langchain_core.messages import AIMessage, HumanMessage, ToolMessage from rai.agents.conversational_agent import create_conversational_agent from rai.agents.integrations.streamlit import get_streamlit_cb, streamlit_invoke +from rai.communication.ros2.connectors import ROS2ARIConnector from rai.tools.ros.cli import ( ros2_action, ros2_interface, @@ -25,15 +27,28 @@ ros2_service, ros2_topic, ) +from rai.tools.ros2.topics import GetROS2ImageTool from rai.utils.model_initialization import get_llm_model @st.cache_resource def initialize_graph(): + rclpy.init() llm = get_llm_model(model_type="complex_model", streaming=True) + + connector = ROS2ARIConnector() + agent = create_conversational_agent( llm, - [ros2_topic, ros2_interface, ros2_node, ros2_service, ros2_action, ros2_param], + [ + ros2_topic, + ros2_interface, + ros2_node, + ros2_service, + ros2_action, + ros2_param, + GetROS2ImageTool(connector=connector), + ], system_prompt="""You are a ROS 2 expert helping a user with their ROS 2 questions. You have access to various tools that allow you to query the ROS 2 system. Be proactive and use the tools to answer questions. Retrieve as much information from the ROS 2 system as possible. """,