Langgraph tool calling with triton

[GGN1994]

Hi team,

Is there any way to do the tool calling using llms hosted locally using triton server?

Thanks
Gokul

[hinthornw]

Assuming you're referring to the prebuilt create_react_agent (since the primary LangGraph framework is agnostic to llms & tool calling ) - the main thing would be getting your LLM to perform tool calling.

If the LLM. you're hosting doesn't support tool calling natively, you'd have to get it to output the json and parse the result as tool calls to then slide that LLM into the create_react_agent abstraction. See the langchain docs here: https://python.langchain.com/docs/concepts/tool_calling/

[GGN1994]

In my scenario, llm will support tool calling in have hosted watt-8b model. I was asking like we have chatollama for ollama , so is there any library for integrating langgraph with llm hosted using nvidia triton framwork

[hinthornw]

There's this: https://api.python.langchain.com/en/latest/llms/langchain_nvidia_trt.llms.TritonTensorRTLLM.html

But I forget if there's a big diff between triton and triton tensor rt