ollama_openvino produces incorrect output compared to ollama-ipex for AutoGen agent tasks

[arisha07]

Hello,
I replaced ollama-ipex with ollama-openvino in my AutoGen agent workflow (https://microsoft.github.io/autogen/dev//user-guide/agentchat-user-guide/tutorial/agents.html)

Model used on ollama openvino side - llama-3.2-3b-instruct-int4-ov
Model used on ollama ipex side - llama3.2:3b-instruct-q4_1

The example that I have used is on using Roundrobin group chats agents to plan a 3 day trip to Nepal.
I have attached the output from both ollama openvino and ollama ipex. You, will see that the output from ollama_openvino tends to hallucinate, whereas ollama_ipex generates accurate and well-structured response.

I have also attached my autgen-agent script to reproduce.

Thanks !

ollama_openvino_output.txt -- (enable word wrap)

ollama_ipex_output.txt

autogen_agents.py.txt

[zulkifli-halim]

Hi @arisha07

Can you share the link to this model: llama-3.2-3b-instruct-int4-ov? Which OpenVINO version do you use?

[arisha07]

https://modelscope.cn/models/FionaZhao/llama-3.2-3b-instruct-int4-ov/files (Mentioned in the Model Library in README)

I downloaded the OpenVINO GenAI from here - https://storage.openvinotoolkit.org/repositories/openvino_genai/packages/nightly/2025.2.0.0.dev20250320/openvino_genai_windows_2025.2.0.0.dev20250320_x86_64.zip

[zulkifli-halim]

I encountered some issues when running the autogen_agents.py. Can you share the command you used to execute this code?
Can you also share the command you used to download the model?

[arisha07]

I just run it like this - What's the error you are getting ?
python autogen_agents.py

to download -

modelscope download --model FionaZhao/llama-3.2-3b-instruct-int4-ov --local_dir ./llama-3.2-3b-instruct-int4-ov

[zulkifli-halim]

I got this error:
openai.NotFoundError: Error code: 404 - {'error': {'message': 'model "llama-3.2-3b-instruct-int4-ov:v1" not found, try pulling it first', 'type': 'api_error', 'param': None, 'code': None}}

[arisha07]

Do you have that model showing up in ollama list ?

[zulkifli-halim]

I can run the code with this model, llama3.2:3b-instruct-q4_1, without any issues.

May I know how you pull this model llama-3.2-3b-instruct-int4-ov:v1?

[arisha07]

I followed the instructions given in README -

openvino_genai_windows_2025.2.0.0.dev20250320_x86_64\setupvars.bat

modelscope download --model FionaZhao/llama-3.2-3b-instruct-int4-ov --local_dir ./llama-3.2-3b-instruct-int4-ov 
tar -zcvf llama-3.2-3b-instruct-int4-ov.tar.gz llama-3.2-3b-instruct-int4-ov
ollama create  llama-3.2-3b-instruct-int4-ov:v1 -f  Modelfile.txt

contents of Modefile -

FROM llama-3.2-3b-instruct-int4-ov.tar.gz
ModelType "OpenVINO"
InferDevice "GPU"
PARAMETER stop "<｜end▁of▁sentence｜>"
PARAMETER stop "<｜User｜>"
PARAMETER stop "</User|>"
PARAMETER stop "<|end_of_sentence|>"
PARAMETER stop "</｜"
PARAMETER max_new_token 4096
PARAMETER stop_id 151643
PARAMETER stop_id 151647
PARAMETER repeat_penalty 1.5
PARAMETER top_p 0.95
PARAMETER top_k 50
PARAMETER temperature 0.8

[arisha07]

After experimenting with the PARAMETERS in Modelfile.txt I am now able to get correct response (No more hallucination or repeating of words like before) from ollama openvino side using llama-3.2-3b-instruct-int4-ov

This is the final Modelfile.txt that I used -

FROM  llama-3.2-3b-instruct-int4-ov.tar.gz
ModelType "OpenVINO"
InferDevice "GPU"
PARAMETER stop "<｜end▁of▁sentence｜>"
PARAMETER stop "<｜User｜>"
PARAMETER stop "</User|>"
PARAMETER stop "<|end_of_sentence|>"
PARAMETER stop "</｜"
PARAMETER repeat_penalty 1.0
PARAMETER top_p 1.0
PARAMETER temperature 1.0

I have attached the output too -

new_output_ollama_genai.txt