Unable to Use Local LLM with .env Configuration — Fails with "Failed to generate prediction with any model"

[talentam]

Git provider (optional)

Gitlab

System Info (optional)

No response

Issues details

I’m trying to configure the PR Agent to use my local LLM (which is OpenAI-compatible) via the .env file, but the agent still attempts to use default models (gpt-5-2025-08-07, o4-mini) instead of my specified endpoint.

.env Configuration:
`CONFIG__GIT_PROVIDER=gitlab
GITLAB__PERSONAL_ACCESS_TOKEN=glpat-xxxx
GITLAB__SHARED_SECRET=xxx
GITLAB__URL=http://96.0.56.102/
GITLAB__AUTH_TYPE=private_token

OPENAI__KEY=sk-xxxx
OPENAI__API_BASE=http://96.0.78.166:30010/v1
OPENAI__API_TYPE=qwen3-235b`

Docker Command:
docker run -d --name pr-agent -p 8080:3000 --env-file .env gitlab_pr_agent:latest

Observed Error:
{"text": "Failed to generate code suggestions for PR, error: Failed to generate prediction with any model of ['gpt-5-2025-08-07', 'o4-mini']\n"

Despite setting OPENAI__API_BASE, OPENAI__API_TYPE and OPENAI__KEY, the system appears to ignore these and falls back to hardcoded model names

Questions:
Is there an additional configuration needed to override the default model list?
Does OPENAI__API_TYPE correctly map to the model name used in requests, or is there a separate MODEL environment variable?
How can I ensure the agent uses my OpenAI-compatible endpoint instead of defaulting to OpenAI’s model names?

[qodo-merge-for-open-source]

💡 Auto-generated documentation-based answer:

Based on your configuration, there are a few issues to address:

1. Model Configuration

The error shows PR-Agent is trying to use default models (gpt-5-2025-08-07, o4-mini) instead of your local LLM. You need to explicitly set the model name using environment variables:

CONFIG.MODEL=ollama/qwen3-235b
CONFIG.FALLBACK_MODELS=["ollama/qwen3-235b"]

Note: The OPENAI__API_TYPE variable you're using doesn't control the model name. You need to use CONFIG.MODEL instead.

2. Ollama Configuration

For local models via Ollama (or similar OpenAI-compatible endpoints), you should configure:

CONFIG.MODEL=ollama/qwen3-235b
CONFIG.FALLBACK_MODELS=["ollama/qwen3-235b"]
CONFIG.CUSTOM_MODEL_MAX_TOKENS=128000
OLLAMA.API_BASE=http://96.0.78.166:30010

3. Complete .env File

Your corrected .env file should look like:

CONFIG__GIT_PROVIDER=gitlab
GITLAB__PERSONAL_ACCESS_TOKEN=glpat-xxxx
GITLAB__SHARED_SECRET=xxx
GITLAB__URL=http://96.0.56.102/
GITLAB__AUTH_TYPE=private_token

CONFIG__MODEL=ollama/qwen3-235b
CONFIG__FALLBACK_MODELS=["ollama/qwen3-235b"]
CONFIG__CUSTOM_MODEL_MAX_TOKENS=128000
OLLAMA__API_BASE=http://96.0.78.166:30010

Important Notes:

• Environment variables use double underscores (__) to separate sections and keys
• The OPENAI__KEY may not be needed if your local endpoint doesn't require authentication, but you can keep it if your setup requires it
• For local/open-source models, be aware that they may not perform as well as commercial models (GPT-5, Claude, Gemini) for complex code analysis tasks. They're suitable for experimentation but may not be production-ready for comprehensive PR analysis.

Additional Configuration:

If you need to adjust other settings for your local model, you can also set:

CONFIG__DUPLICATE_EXAMPLES=true  # Helps the model generate structured output

Relevant Sources:

• https://qodo-merge-docs.qodo.ai//usage-guide/changing_a_model#ollama
• https://qodo-merge-docs.qodo.ai//usage-guide/changing_a_model#openai-like-api
• https://qodo-merge-docs.qodo.ai//usage-guide/changing_a_model#custom-models
• https://qodo-merge-docs.qodo.ai//usage-guide/automations_and_usage#utilizing-environment-variables
• https://qodo-merge-docs.qodo.ai//installation/locally#i-get-an-error-when-running-the-docker-image-what-should-i-do