Llama4 Scout and tool calling

[xhejtman]

Hello,

what is proper setup for llama4 Scout to enabel tools? I tried:

'python3', '-m', 'sglang.launch_server', '--model', '/data/hub/models--RedHatAI--Llama-4-Scout-17B-16E-Instruct-FP8-dynamic/snapshots/b23a51649d63c9e87b39bb0e2ebf4e4f5898b0ec', '--host', '0.0.0.0', '--port', '8000', '--served-model-name', 'llama-4-scout-17b-16e-instruct', '--tensor-parallel-size', '2', '--enable-metrics', '--tool-call-parser', 'pythonic', '--sleep-on-idle', '--chat-template', 'llama-4', '--context-length', '131072', '--enable-multimodal', '--hybrid-kvcache-ratio', '0.5', '--attention-backend', 'fa3', '--mem-fraction-static', '0.85', '--mm-attention-backend', 'fa3'

esp. '--chat-template', 'llama-4', '--tool-call-parser', 'pythonic'

but if I request sample call for a tool:

curl "http://10.42.210.126:8000/v1/chat/completions" -H "Content-Type: application/json" -d '{"temperature": 0, "max_tokens": 100, "model": "llama-4-scout-17b-16e-instruct", "tools": [{"type": "function", "function": {"name": "query_weather", "description": "Get weather of an city, the user should supply a city first", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city, e.g. Beijing"}}, "required": ["city"]}}}], "messages": [{"role": "user", "content": "Hows the weather like in Qingdao today"}]}'

I get response:

{
  "id": "00c0ad50b82241fa912cec42f59458c9",
  "object": "chat.completion",
  "created": 1755389060,
  "model": "llama-4-scout-17b-16e-instruct",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "I'm not able to provide real-time weather updates as my last update was in April 2023, and I don't have access to current internet data. However, I can guide you on how to find the most accurate and up-to-date weather information for Qingdao.\n\n1. **Check Online Weather Services**: Websites like AccuWeather, Weather.com, or the official website of the China Meteorological Administration provide current weather conditions and forecasts for locations all over the world, including Qingdao.\n\n2. **Use",
        "reasoning_content": null,
        "tool_calls": null
      },
      "logprobs": null,
      "finish_reason": "length",
      "matched_stop": null
    }
  ],
  "usage": {
    "prompt_tokens": 19,
    "total_tokens": 119,
    "completion_tokens": 100,
    "prompt_tokens_details": null,
    "reasoning_tokens": 0
  },
  "metadata": {
    "weight_version": "default"
  }
}

for gpt-oss the response seems to be ok:

{
  "id": "b0fc9af33bdc4b3393393ca316e51406",
  "object": "chat.completion",
  "created": 1755388966,
  "model": "gpt-oss-120b",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "<|start|>assistant<|channel|>commentary to=functions.query_weather <|constrain|>json<|message|>{\n  \"city\": \"Qingdao\"\n}",
        "reasoning_content": "The user asks: \"Hows the weather like in Qingdao today\". We need to get weather. Use function query_weather with city \"Qingdao\".",
        "tool_calls": []
      },
      "logprobs": null,
      "finish_reason": "tool_calls",
      "matched_stop": null
    }
  ],
  "usage": {
    "prompt_tokens": 144,
    "total_tokens": 206,
    "completion_tokens": 62,
    "prompt_tokens_details": null,
    "reasoning_tokens": 0
  },
  "metadata": {
    "weight_version": "default"
  }
}