AI Function Calling Example fails as model is not precise enough

[mkarg]

As an AI beginner I am really thankful for @tjake's efforts in the Java-AI-area. Unfortunately I noticed that the AI Function Calling Example fails as the used model is not precise enough:

• Most times it works well.
• After several attempts the answer is that transaction T100 or T10005 is not found (apparently the model cuts away the 5 or adds a zero).
• Slightly modifying the question sent into the chat is responded with a (completely correct!) description which method one has to call to get the result, but JLama is not calling the method anymore itself:

  To get the status and date of transaction 'T1005', I will use the 'retrievePaymentStatus' and 'retrievePaymentDate' functions.
   
  First, I will use the 'retrievePaymentStatus' function with the argument 'T1005' to get the status:
   
  retrievePaymentStatus('T1005')
   
  Then, I will use the 'retrievePaymentDate' function with the same argument 'T1005' to get the date:
   
  retrievePaymentDate('T1005')
   
  After getting the status and date, I will display them to you.
  

Can this get fixed? How can I make this work again until a fix exists?

[tjake]

Hi, which model are you using? Often the small quantize ones have trouble with tool calling like this.

[mkarg]

As I said, I am a total newbie wrt AI, just an expect for OpenJDK...

What I did was using this exact code, which works well https://github.com/langchain4j/langchain4j-examples/blob/main/jlama-examples/src/main/java/JlamaAiFunctionCallingExamples.java ... but then added one more tool call:

@Tool("List all transaction IDs")
static List<String> listPayments() {
    return getPaymentData().get("transaction_id");
}

This leads to the effect that the results are not stable anymore, and just slightly modifying asked the question (like "what payment transactions are still open?") leading to the shown effect that the model simply lists the intended steps, but does not actually perform the needed calls.

What I learned from a quick search was that the problem seems to be that the used model (tjake/Mistral-7B-Instruct-v0.3-JQ4) seems to be not precise enough for tool calls, due to Q4. Again, not being an expert, I tend to believe this, looking at the fact that the model does not precisely pick the T1005 transaction but actually it called the tools with T100 and T10005.

[koosty]

As @tjake said, small models is not good enough. You can experiments by downloading a larger model here https://huggingface.co/TheBloke/models?search=mistral

Make sure to use GGUF model e.g.: TheBloke/Mistral-7B-Instruct-v0.2-GGUF
Then use the model:

String model = "TheBloke/Mistral-7B-Instruct-v0.2-GGUF";
String workingDirectory = "./models";

// Downloads the model or just returns the local path if it's already downloaded
File localModelPath = new Downloader(workingDirectory, model).huggingFaceModel();
    
// Loads the quantized model and specified use of quantized memory
AbstractModel m = ModelSupport.loadModel(localModelPath, DType.F32, DType.I8);

The example model has 7B parameters which approximately will used 8GB memory or more.