Function Calling/Tool use support

[ansarizafar]

It seems Llamafile doesn't support function calling I am trying to use Llama-3.2-1B-Instruct.Q6_K.llamafile with Vercel AI SDK but function calling/Tool use is not working here is my code.

import { createOpenAI as createLlama } from '@ai-sdk/openai';
import { type CoreMessage, streamText, tool } from 'ai';
//import dotenv from 'dotenv';
import { z } from 'zod';
import * as readline from 'node:readline/promises';

//dotenv.config();

const llama = createLlama({
    baseURL: ' http://127.0.0.1:8080/v1',
    apiKey: "sk-no-key-required",
  });

const terminal = readline.createInterface({
  input: process.stdin,
  output: process.stdout,
});

const messages: CoreMessage[] = [];

async function main() {
  while (true) {
    const userInput = await terminal.question('You: ');

    messages.push({ role: 'user', content: userInput });

    const result = streamText({
      model: llama('LLaMA_CPP'),
      messages,
      tools: {
        weather: tool({
          description: 'Get the weather in a location (in Celsius)',
          parameters: z.object({
            location: z
              .string()
              .describe('The location to get the weather for'),
          }),
          execute: async ({ location }) => ({
            location,
            temperature: Math.round((Math.random() * 30 + 5) * 10) / 10, // Random temp between 5°C and 35°C
          }),
        }),
      },
    });

    let fullResponse = '';
    process.stdout.write('\nAssistant: ');
    for await (const delta of result.textStream) {
      fullResponse += delta;
    //  process.stdout.write(delta);
    }
    process.stdout.write('\n\n');

    console.log(await result.toolCalls);
    console.log(await result.toolResults);
    messages.push({ role: 'assistant', content: fullResponse });
  }
}

main().catch(console.error);

[r23j5a]

I can probably relate to it, I'm using Llama-3.2-3B-Instruct.Q6_K.llamafile with Theia IDE https://eclipsesource.com/blogs/2025/03/13/introducing-the-ai-powered-theia-ide/ and its Coder agent just doesn't work.
P.S. llamafile - absolutely awesome, thanks 🙌

[fiorelorenzo]

any news on tool calling support?

[fiorelorenzo]

Actually I found out that they are already working on it.
In fact, by building from source on the new_build_wip branch you get full function calling support (with streaming) by leveraging the updated reference to llama.cpp.

To enable function calling support you have to specify the --jinja flag when starting llamafile.

[aittalam]

Thanks everyone for raising this topic, and thank you @fiorelorenzo for already finding the answer! :-)
That's exactly right, we have been working on supporting features and models from more recent versions of llama.cpp, and finally our work-in-progress branch is in a semi-sharable state. Here's our Christmas update with build instructions and a few models you can start playing with!