Skip to content

does not support setting top_k #958

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ilya-lavrenov merged 7 commits into from
Apr 8, 2025
Merged

does not support setting top_k #958

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
17c11a2
Improve interface compatibility
zhaohb Apr 1, 2025
5b5413e
del unimportant comments
zhaohb Apr 1, 2025
dfd53d1
Update README.md
zhaohb Apr 2, 2025
9d3f10f
add DeepSeek-R1-Distill-Qwen-14B-int4-ov
zhaohb Apr 3, 2025
f02fec8
Merge branch 'master' into master
zhaohb Apr 3, 2025
e81dcb7
Does not support setting top_k
zhaohb Apr 8, 2025
c1eb3ad
Merge branch 'master' into master
zhaohb Apr 8, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 9 additions & 9 deletions modules/ollama_openvino/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -584,17 +584,17 @@ Getting started with large language models and using the [GenAI](https://github.
We provide two ways to download the executable file of Ollama, one is to download it from Google Drive, and the other is to download it from Baidu Drive:
## Google Driver
### Windows
[Download exe](https://drive.google.com/file/d/1iizO9iLhSJGFUu6BgY3EwOchrCyzImUN/view?usp=drive_link) + [Download OpenVINO GenAI](https://storage.openvinotoolkit.org/repositories/openvino_genai/packages/nightly/2025.2.0.0.dev20250320/openvino_genai_windows_2025.2.0.0.dev20250320_x86_64.zip)
[Download exe](https://drive.google.com/file/d/1Sep1IdGn7mJaE8PCXKYxp_aj1ljiPvpN/view?usp=sharing) + [Download OpenVINO GenAI](https://storage.openvinotoolkit.org/repositories/openvino_genai/packages/nightly/2025.2.0.0.dev20250320/openvino_genai_windows_2025.2.0.0.dev20250320_x86_64.zip)

### Linux(Ubuntu 22.04)
[Download](https://drive.google.com/file/d/1HEyZNNCbWSidKNQl4MRsD8FuwEZtdyew/view?usp=drive_link) + [Donwload OpenVINO GenAI](https://storage.openvinotoolkit.org/repositories/openvino_genai/packages/nightly/2025.2.0.0.dev20250320/openvino_genai_ubuntu22_2025.2.0.0.dev20250320_x86_64.tar.gz)
[Download](https://drive.google.com/file/d/1DdBoEGp_eoyJPbpMGVbEivihYSKrCMGt/view?usp=sharing) + [Donwload OpenVINO GenAI](https://storage.openvinotoolkit.org/repositories/openvino_genai/packages/nightly/2025.2.0.0.dev20250320/openvino_genai_ubuntu22_2025.2.0.0.dev20250320_x86_64.tar.gz)

## 百度云盘
### Windows
[Download exe](https://pan.baidu.com/s/10SQLKaGOYIR3NrIhig1xgQ?pwd=8v8w) + [Download OpenVINO GenAI](https://storage.openvinotoolkit.org/repositories/openvino_genai/packages/nightly/2025.2.0.0.dev20250320/openvino_genai_windows_2025.2.0.0.dev20250320_x86_64.zip)
[Download exe](https://pan.baidu.com/s/1TCH7rYSPr8jQDHLvCeXdLw?pwd=6bk9) + [Download OpenVINO GenAI](https://storage.openvinotoolkit.org/repositories/openvino_genai/packages/nightly/2025.2.0.0.dev20250320/openvino_genai_windows_2025.2.0.0.dev20250320_x86_64.zip)

### Linux(Ubuntu 22.04)
[Download](https://pan.baidu.com/s/1WhzcKRQY_KVLZhy5oa0UIA?pwd=zxgx) + [Donwload OpenVINO GenAI](https://storage.openvinotoolkit.org/repositories/openvino_genai/packages/nightly/2025.2.0.0.dev20250320/openvino_genai_ubuntu22_2025.2.0.0.dev20250320_x86_64.tar.gz)
[Download](https://pan.baidu.com/s/1UVO0ZK4DFTjTwfarQ8LUIw?pwd=pxkd) + [Donwload OpenVINO GenAI](https://storage.openvinotoolkit.org/repositories/openvino_genai/packages/nightly/2025.2.0.0.dev20250320/openvino_genai_ubuntu22_2025.2.0.0.dev20250320_x86_64.tar.gz)

## Docker
### Linux
Expand Down Expand Up @@ -659,19 +659,19 @@ We added two new parameters to Modelfile based on the original parameters:

### Start Ollama

1. First, set GODEGUG=cgocheck=0 env:
1. First, set GODEBUG=cgocheck=0 env:

#### Linux
```shell
export GODEGUG=cgocheck=0
export GODEBUG=cgocheck=0
```

#### Windows
```shell
set GODEBUG=cgocheck=0
```

2. Next, `ollama serve` is used when you want to start ollama (You must use the Ollama compiled by ollama_ov to start the serve):
2. Next, `ollama serve` is used when you want to start ollama (you must use the Ollama compiled by ollama_ov to start the serve):

```shell
ollama serve
Expand Down Expand Up @@ -861,11 +861,11 @@ Then build and run Ollama from the root directory of the repository:

### Running local builds

1. First, set GODEGUG=cgocheck=0 env:
1. First, set GODEBUG=cgocheck=0 env:

#### Linux
```shell
export GODEGUG=cgocheck=0
export GODEBUG=cgocheck=0
```

#### Windows
Expand Down
14 changes: 3 additions & 11 deletions modules/ollama_openvino/examples/modelfile/deepseek_r1_distill_llama/Modelfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,6 @@
FROM DeepSeek-R1-Distill-Llama-8B-int4-ov.tar.gz
ModelType "OpenVINO"
InferDevice "GPU"
PARAMETER stop "<｜end▁of▁sentence｜>"
PARAMETER stop "<｜User｜>"
PARAMETER stop "</User|>"
PARAMETER stop "<|end_of_sentence|>"
PARAMETER stop "</｜"
PARAMETER max_new_token 4096
PARAMETER stop_id 128001
PARAMETER repeat_penalty 1.5
PARAMETER top_p 0.95
PARAMETER top_k 50
PARAMETER temperature 0.8
PARAMETER repeat_penalty 1.0
PARAMETER top_p 1.0
PARAMETER temperature 1.0
15 changes: 3 additions & 12 deletions modules/ollama_openvino/examples/modelfile/deepseek_r1_distill_qwen/Modelfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,6 @@
FROM DeepSeek-R1-Distill-Qwen-7B-int4-ov.tar.gz
ModelType "OpenVINO"
InferDevice "GPU"
PARAMETER stop "<｜end▁of▁sentence｜>"
PARAMETER stop "<｜User｜>"
PARAMETER stop "</User|>"
PARAMETER stop "<|end_of_sentence|>"
PARAMETER stop "</｜"
PARAMETER max_new_token 4096
PARAMETER stop_id 151643
PARAMETER stop_id 151647
PARAMETER repeat_penalty 1.5
PARAMETER top_p 0.95
PARAMETER top_k 50
PARAMETER temperature 0.8
PARAMETER repeat_penalty 1.0
PARAMETER top_p 1.0
PARAMETER temperature 1.0
2 changes: 1 addition & 1 deletion modules/ollama_openvino/genai/genai.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ func SetSamplingParams(samplingparameters *SamplingParams) *C.ov_genai_generatio
C.ov_genai_generation_config_set_max_new_tokens(cConfig, C.size_t(samplingparameters.MaxNewToken))
C.ov_genai_generation_config_set_temperature(cConfig, C.float(samplingparameters.Temp))
C.ov_genai_generation_config_set_top_p(cConfig, C.float(samplingparameters.TopP))
C.ov_genai_generation_config_set_top_k(cConfig, C.size_t(samplingparameters.TopK))
// C.ov_genai_generation_config_set_top_k(cConfig, C.size_t(samplingparameters.TopK))

if samplingparameters.StopIds != nil {
cStopArray := C.malloc(C.size_t(len(samplingparameters.StopIds)) * C.size_t(unsafe.Sizeof(C.int64_t(0))))
Expand Down
Loading