Feature/configurable max tokens

[dcox761]

*Issue #227 *

Description of changes:

1. Add DEFAULT_MAX_TOKENS environment variable in setting.py (default: 2048 to preserve existing behaviour)
2. Use it as the schema default for max_tokens
3. Compute effective_max_tokens in _parse_request that prefers max_completion_tokens over max_tokens, eliminating duplicate logic:

effective_max_tokens = (
    chat_request.max_completion_tokens
    if chat_request.max_completion_tokens is not None
    else chat_request.max_tokens
)
inference_config = {"maxTokens": effective_max_tokens}

# Example: raise default for reasoning models
export DEFAULT_MAX_TOKENS=16384

Files: src/api/setting.py, src/api/schema.py, src/api/models/bedrock.py
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[zxkane]

Thanks for the PR! The refactoring to consolidate effective_max_tokens and eliminate duplicate logic is a nice improvement.

The main concern is around the default value for max_tokens. Per the Bedrock InferenceConfiguration docs, maxTokens is optional and defaults to the model's maximum when omitted. The OpenAI API also treats max_tokens as optional. Hardcoding 2048 artificially caps every model's output — for example, Claude Sonnet 4 supports 8192 and Opus 4 supports 32768 output tokens.

Rather than making the hardcoded 2048 configurable, the better fix would be to change the default to None and only include maxTokens in the Bedrock request when explicitly set by the client. This aligns with both the OpenAI API and Bedrock API behavior.

See inline comments for details.

[zxkane]

A couple of concerns here:

1. Should the default really be 2048?

Per the Bedrock InferenceConfiguration docs, maxTokens is optional — when omitted, it defaults to the maximum allowed value for the model. The OpenAI API also treats max_tokens as optional (defaults to model max).

Hardcoding 2048 artificially caps every model's output. For example, Claude Sonnet 4's max output is 8192 tokens and Opus 4 supports up to 32768. Users who don't explicitly set max_tokens would get truncated responses without realizing why.

A better default would be None, which lets Bedrock use the model's native max — consistent with both the OpenAI API and the Bedrock API:

# Let Bedrock use the model's native max output tokens when not specified
max_tokens: int | None = None

Then in _parse_request, only include maxTokens in inference_config when it's not None.

2. Unguarded int() conversion

If a user sets DEFAULT_MAX_TOKENS=abc or DEFAULT_MAX_TOKENS= (empty string), the application will crash at import time with an unhelpful ValueError. This is the only int() cast in the settings module. Consider wrapping with try/except and validating >= 1.

[zxkane]

Related to the comment on setting.py — since maxTokens is optional in the Bedrock Converse API (defaults to model max), and max_tokens is also optional in the OpenAI API, the default here should arguably be None rather than DEFAULT_MAX_TOKENS (2048).

Also, other numeric fields in this model use Field() with validation constraints (e.g., temperature, top_p). Consider adding the same for consistency:

max_tokens: int | None = Field(default=None, ge=1)
max_completion_tokens: int | None = Field(default=None, ge=1)

This would give users clear Pydantic validation errors for invalid values (e.g., max_tokens: 0 or max_tokens: -1) instead of opaque Bedrock API errors.

[zxkane]

The refactoring to consolidate effective_max_tokens is a nice improvement. A couple of things to note:

1. Behavior change: falsy check → is not None check

The old code in the Claude reasoning block used if chat_request.max_completion_tokens (falsy — treats 0 as False). The new code uses is not None (treats 0 as a valid value). This is more correct, but it's a subtle behavior change that could surface as a regression if any client sends max_completion_tokens: 0.

2. effective_max_tokens can be None

If max_tokens defaults to None (as suggested above) and the client doesn't send max_completion_tokens either, effective_max_tokens will be None. In that case, we should omit maxTokens from inference_config entirely so Bedrock uses the model default, rather than sending maxTokens: None which would cause a ParamValidationError:

effective_max_tokens = (
    chat_request.max_completion_tokens
    if chat_request.max_completion_tokens is not None
    else chat_request.max_tokens
)
inference_config = {}
if effective_max_tokens is not None:
    inference_config["maxTokens"] = effective_max_tokens

[zxkane]

Minor: inference_config["maxTokens"] = effective_max_tokens on line 832 is now redundant — it was already set to the same value on line 785. In the old code this re-assignment was needed because a different local max_tokens was computed here, but after your refactoring both use effective_max_tokens. Could be removed for clarity.