Delayed response from localai

[harshsing2891]

LocalAI version:

Additional context

[harshsing2891]

Like i am using below command and i am getting response from localai after three to four minutes.
curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "ggml-koala-7b-model-q4_0-r2.bin",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'

[lunamidori5]

@harshsing2891 can you please post info on your setup, docker-compose (if using docker), and your models yaml file?

[harshsing2891]

I am using helm chart (latest image) and localai pod is running fine in kubernetes cluster
curl http://local-ai.local-ai.svc.cluster.local:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "ggml-koala-7b-model-q4_0-r2.bin",
"messages": [{"role": "user", "content": "Say this is a test!"}],
"temperature": 0.7
}'

i am getting response of any question i am asking lets say "what is artificial intelligence" but very delayed response ( 3-4 minutes i am getting from local ai

[harshsing2891]

helm command :- helm install local-ai go-skynet/local-ai -f values.yaml

[harshsing2891]

any update on this ?

[localai-bot]

⚠️⚠️⚠️⚠️⚠️

Hi! I'm a bot running with LocalAI ( a crazy experiment of @mudler ) - please beware that I might hallucinate sometimes!

but.... I can also be funny or helpful 😸 and I can provide generally speaking good tips or places where to look after in the documentation or in the code based on what you wrote in the issue.

Don't engage in conversation with me,
I don't support (yet) replying!

⚠️⚠️⚠️⚠️⚠️

I'm sorry to hear that you are experiencing delayed responses from LocalAI. There could be several reasons for this, such as network latency or insufficient resources on your Kubernetes cluster.
To troubleshoot the issue, I would recommend checking the logs of your Kubernetes cluster and the LocalAI container to see if there are any error messages or stack traces that might provide clues. You can also try increasing the number of threads in your .env file to see if that improves performance.
If you are still experiencing issues after troubleshooting, please feel free to open an issue on our GitHub repository and we will do our best to help you resolve it.

Sources:

• https://localai.io/basics/getting_started/index.html
• https://github.com/go-skynet/LocalAI/tree/master/.github/ISSUE_TEMPLATE/bug_report.md
• https://github.com/go-skynet/LocalAI/tree/master/examples/README.md
• https://github.com/go-skynet/LocalAI/tree/master/README.md

[lunamidori5]

any update on this ?

@harshsing2891 Still waiting on the requested files. We need to look at the models yaml file. and if you are using docker, docker compose file <3 If you are running the model raw, please set up a model per - https://localai.io/howtos/easy-model-import-downloaded/ and try again after restarting localai.

[harshsing2891]

Hi
I m using kubernetes cluster and have deployed local ai using helm template.

please find below model.yaml file
name: "gpt4all-j"
description: |
A commercially licensable model based on GPT-J and trained by Nomic AI on the v0 GPT4All dataset.
license: "Apache 2.0"
urls:

• https://gpt4all.io
  config_file: |
  backend: gpt4all-j
  parameters:
  model: ggml-gpt4all-j.bin
  top_k: 80
  temperature: 0.2
  top_p: 0.7
  context_size: 1024
  template:
  completion: "gpt4all-completion"
  chat: gpt4all-chat

files:
- filename: "ggml-gpt4all-j.bin"
sha256: "acd54f6da1cad7c04c48b785178d686c720dcbe549903032a0945f97b1a43d20"
uri: "https://gpt4all.io/models/ggml-gpt4all-j.bin"

prompt_templates:

• name: "gpt4all-completion"
  content: |
  Complete the prompt

  Prompt:

  {{.Input}}

  Response:

• name: "gpt4all-chat"
  content: |
  The prompt below is a question to answer, a task to complete, or a conversation to respond to; decide which and write an appropriate response.

  Prompt:

  {{.Input}}

  Response:

[lunamidori5]

I am not sure why you keep asking for values.yaml We are requesting your models yaml file you made by following the how to, on how to set up a model...