How to use pr-agent with self-hosted litellm proxy?

[FatGhosta]

For some reason, I have to use pr-agent with self-hosted litellm proxy.
However, I fail to find the configuration instructions in the usage guide.

I attempted the following configuration, but it didn't work.Moreover, I've confirmed that my network connection is working properly.

[litellm_proxy]
custom_llm_provider ="openai"
api_base = "http://xxx.xxx.xxx.xxx:14000"
api_key = "sk-xxxx"

[config]
model="litellm_proxy/gemini-2.0-flash"  ## gemini-2.0-flash is defined in my litellm proxy
model_turbo="litellm_proxy/gemini-2.0-flash"
fallback_models=["litellm_proxy/gemini-2.0-flash"]

[FatGhosta]

I dig into the source code of the pr-agent and made the following configuration changes, but it still doesn't work:

[ollama]
api_base = "http://xxx.xxx.xxx.xxx:14000"  ## it makes litellm.api_base to be assigned
deployment_id = "litellm_proxy/gemini-2.0-flash"

[config]
model="litellm_proxy/gemini-2.0-flash"
fallback_models=["litellm_proxy/gemini-2.0-flash"]
response_language="zh-CN"

[litellm]
extra_headers = '{"Authorization": "Bearer sk-xxxx"}'   ## maybe it is a wrong way to set LITELLM_PROXY_API_KEY ?

[mrT23]

i dont know what is "litellm_proxy"

maybe you mean "openai proxy" ?

anyway, thoroughly review this page:
https://qodo-merge-docs.qodo.ai/usage-guide/changing_a_model/#custom-models

you cannot invent categories like [litellm_proxy] in the configuration. they won't be used

you can give additional parameters as env variables, and open a PR to better support them in configuration

[FatGhosta]

Litellm Proxy: https://docs.litellm.ai/docs/proxy/docker_quick_start
In fact, I attempted to configure it correctly by referring to https://docs.litellm.ai/docs/providers/litellm_proxy.
As a result, I obtained the following configuration.
However, it still didn't function as expected:

[ollama]
api_base = "http://xxx.xxx.xxx.xxx:24000"   ## This assignment sets the value of litellm.api_base
deployment_id = "gemini-2.0-flash"

[groq]
key = "sk-xxxx"   ## This assignment sets the value of litellm.api_key. It's a bit of a workaround.

[config]
model="litellm_proxy/gemini-2.0-flash"
fallback_models=["litellm_proxy/gpt-4o"]
response_language="zh-CN"

[litellm]
extra_headers = '{"Authorization": "Bearer sk-xxxx"}'

By the way, I successfully called my Litellm proxy using Python code within the GitLab CI pipeline.

url = 'http://xxx.xxx.xxx.xxx:24000/chat/completions'
    headers = {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer sk-xxxx'
    }
    data = {
        "model": "gemini-2.0-flash",
        "messages": [
            {
                "role": "user",
                "content": "what llm are you"
            }
        ]
    }

    try:
        response = requests.post(url, headers=headers, json=data)
        response.raise_for_status()
        print(response.json())
    except requests.exceptions.HTTPError as http_err:
        print(f"HTTP error occurred: {http_err}")
    except requests.exceptions.RequestException as req_err:
        print(f"Request error occurred: {req_err}")

[FatGhosta]

Finally, I found out what the problem was. It was that the custom_model_max_tokens parameter was not configured in my configuration file. However, in the code flow, this exception thrown in get_max_tokens was not caught, which really confused me.

Here is my final version of the configuration file, just for reference:

[ollama]
api_base = "http://xxx.xxx.xxx.xxx:24000"   ## This assignment sets the value of litellm.api_base
deployment_id = "gemini-2.0-flash"

[config]
model="litellm_proxy/gemini-2.0-flash"
fallback_models=["litellm_proxy/gpt-4o"]
response_language="zh-CN"
custom_model_max_tokens=1048576

[litellm]
extra_headers = '{"Authorization": "Bearer sk-xxxx"}'