feat(generic-openai): passthrough sampling and extra request params

[Souravrajvi0]

Pull Request Type

• ✨ feat (New feature)
• 🐛 fix (Bug fix)
• ♻️ refactor (Code refactoring without changing behavior)
• 💄 style (UI style changes)
• 🔨 chore (Build, CI, maintenance)
• 📝 docs (Documentation updates)

Relevant Issues

resolves #6043

Description

The Generic OpenAI provider (generic-openai) only forwarded model, messages, temperature, and max_tokens to OpenAI-compatible backends. Standard Chat Completions sampling params and provider-specific extensions (e.g. repetition_penalty, xtc_probability) were silently dropped.

Per maintainer feedback, this is now configured via a JSON file instead of ENV vars:

storage/config/llm/<provider>.json

{
  "top_p": 0.9,
  "repetition_penalty": 1.1,
  "xtc_probability": 0.1,
  "xtc_threshold": 0.1,
  "models": {
    "my-local-model": {
      "top_p": 0.95,
      "repetition_penalty": 1.2
    }
  }
}

• Arbitrary / unlisted keys are passed through as-is (not a fixed OpenAI-only schema)
• Optional models.<modelId> overrides merge on top of provider defaults
• Applied to chat + streaming completions and agent function-call paths
• AnythingLLM-owned fields (model, messages, stream, tools, etc.) cannot be overridden
• Missing/invalid config → no extras (safe no-op)

Shared helper: server/utils/helpers/llmProviderConfig.js (getLLMProviderRequestParams) so other providers can adopt the same pattern later.

Visuals (if applicable)

N/A

Additional Information

• Example + docs: server/storage/config/llm/generic-openai.json.example, README.md
• Unit tests: server/__tests__/utils/AiProviders/genericOpenAi/index.test.js (9 tests)
• User JSON configs are gitignored (storage/config/llm/*.json)

Local testing performed:

yarn lint:ci
yarn test
yarn test server/__tests__/utils/AiProviders/genericOpenAi/index.test.js

Results:

• yarn lint:ci — pass
• yarn test — 246/246 tests pass (29 suites)
• Targeted suite — 9/9 tests pass

Developer Validations

• I ran yarn lint from the root of the repo & committed changes
• Relevant documentation has been updated (if applicable)
• I have tested my code functionality
• Docker build succeeds locally

[aikidovation]

You guys are awesome. Looking forward to release. Thank you

[timothycarambat]

Instead of doing this through an infinitely long ENV we wanted to support this via a storage/config/llm/<provider>.json that just has all the keys you wish to apply and we can spread the eligible ones on inference for chat/agents.

This would be easier to support long term since you could just add and type in any value as every provider has different configs and error states that even have side effects to the model selection too. It might even make sense to have model id subkeys in this config so that the same config can be tuned per provider/model selection.

[aikidovation]

+1 on moving this to storage/config/llm/.json — and the per-model subkey idea is genuinely useful beyond just this issue (we occasionally A/B different local models against the same workspace and having per-model tuning live in one place would be nice).

One thing I’d like to confirm before it’s built, since it determines whether this actually solves the problem for us: will the new config format still support arbitrary/unlisted parameters, not just a curated set of named fields (top_p, frequency_penalty, etc.)?

The concrete case that pushed us to file the original issue was repetition_penalty and xtc_probability/xtc_threshold — sampler params exposed by our local inference server (oMLX, MLX-based) that aren’t part of the standard OpenAI Chat Completions spec at all. We hit a real repetition-loop degeneration in production that these params fixed, but had no way to pass them through the Generic OpenAI provider, and ended up bypassing AnythingLLM’s chat API entirely to call our inference server directly instead — which works, but means we lose being able to browse/interact with that workflow’s output through AnythingLLM itself.

If the new config file ships as a fixed schema of named OpenAI-spec fields, it’ll fix the issue for standard params but not for cases like ours. If it keeps something equivalent to the original PR’s GENERIC_OPEN_AI_EXTRA_PARAMS escape hatch (freeform JSON merged into the request body), that covers both — happy to test against a real oMLX deployment once there’s something to try.

[Souravrajvi0]

Thanks for the direction @timothycarambat — updated this PR to use storage/config/llm/<provider>.json instead of the ENV-based approach.

What changed

• New shared loader: getLLMProviderRequestParams(provider, model) in server/utils/helpers/llmProviderConfig.js
• Generic OpenAI chat + agent paths spread those params at inference time
• Config supports arbitrary/unlisted keys (so repetition_penalty, xtc_probability, etc. all pass through)
• Optional per-model overrides via a models map, e.g.:

{
  "top_p": 0.9,
  "repetition_penalty": 1.1,
  "xtc_probability": 0.1,
  "models": {
    "my-local-model": { "repetition_penalty": 1.2 }
  }
}

Copy server/storage/config/llm/generic-openai.json.example → generic-openai.json to try it. Happy to adjust the schema if you'd prefer a different shape for model subkeys.

[Souravrajvi0]

@aikidovation yes — freeform / unlisted keys are supported. There is no curated allowlist of OpenAI-only fields.

Whatever you put in storage/config/llm/generic-openai.json (except structural models and AnythingLLM-owned keys like model/messages/stream/tools) is spread into the outbound chat/agent request body as-is. So your oMLX case works directly:

{
  "repetition_penalty": 1.1,
  "xtc_probability": 0.1,
  "xtc_threshold": 0.1,
  "models": {
    "your-omlx-model-id": {
      "repetition_penalty": 1.2
    }
  }
}

Copy server/storage/config/llm/generic-openai.json.example → generic-openai.json under your storage dir and restart/reload as usual. Would love a smoke test against your real oMLX deployment if you get a chance.

[aikidovation]

Great, @Souravrajvi0, that resolves my concern. Looking forward to the fix!

[aikidovation]

Ran the smoke test you asked for — and then some. Built this branch (8b1aabd) into a local image and tested it two ways against a real self-hosted oMLX deployment (OpenAI-compatible endpoint, Gemma model):

1. Isolated smoke test. Separate container/volume, with a proxy standing in for the LLM backend so I could capture the literal outgoing request body. Dropped this in as storage/config/llm/generic-openai.json:

{
  "repetition_penalty": 1.15,
  "xtc_probability": 0.5,
  "xtc_threshold": 0.1,
  "top_p": 0.95,
  "models": {
    "my-model-id": { "seed": 42 }
  }
}

The captured request showed all four top-level keys passed through verbatim (including xtc_probability/xtc_threshold, which aren't part of the OpenAI spec at all), the per-model seed override merged in correctly, and AnythingLLM's own owned fields (model/messages/temperature/max_tokens) were untouched. Exactly the freeform passthrough behavior described here and in the thread on #6043.

2. Production stress test. Swapped this build into my actual production AnythingLLM instance (backed up first) and reverted a pipeline I'd previously had to bypass AnythingLLM entirely for, specifically because of the gap this PR fixes (repetition_penalty/xtc_* never reaching the model regardless of config, causing a recurring degenerate-repetition failure mode in long list-formatting completions). With this PR's config file in place, ran that pipeline for real — 6 full generations back-to-back against a real, large candidate set, going through AnythingLLM's chat API this time instead of calling the backend directly. Zero repetition/formatting failures across all 6 runs. That's the actual behavior this PR needs to deliver in practice, not just pass a request through — confirmed.

One unrelated build note in case it trips anyone else up testing locally: docker/Dockerfile's Node 18.x install via deb.nodesource.com currently 403s (Node 18 is EOL, looks like nodesource pulled that repo). Had to swap in the official nodejs.org v20 tarball to get a local build working — not a bug in this PR, just current upstream infra rot.

Happy to answer questions about the setup if useful for review.

[Souravrajvi0]

@aikidovation thank you for the thorough verification — especially the production stress test with the real oMLX + Gemma path. Really glad the freeform passthrough + per-model merge behaved as expected and unblocked that pipeline.

Noted on the Node 18 nodesource 403 during local Docker builds; agreed that’s upstream/EOL unrelated to this change.

[aikidovation]

Following up since this went quiet for a month — noticed it was unassigned on 2026-08-26 with no comment, and today's v1.16.1 release doesn't include it either.

For context on where things stood: as of the last exchange here (2026-07-28), this had been verified two ways against a real production oMLX + Gemma deployment — an isolated request-capture test confirming the freeform passthrough + per-model merge work exactly as designed, and a live production stress test (6 back-to-back generations through AnythingLLM's actual chat API) with zero regressions, replacing a workaround we'd been forced into (bypassing AnythingLLM's chat API entirely) specifically because of the gap this fixes.

Is there anything blocking review at this point, or is this queued behind other work? Happy to re-verify against a newer base if a rebase is the holdup. This is still genuinely needed on our side (see #6043) and it'd be a shame for fully-verified work to stall out with no explanation.