[APIServer] Dynamic default values for workers and max-concurrency based on platform

[Copilot]

Motivation

--workers and --max-concurrency were hardcoded to 1 and 512 respectively. On NVIDIA GPUs with large max_num_seqs, a single worker becomes a bottleneck. Defaults should scale with the workload on capable hardware.

Modifications

• fastdeploy/entrypoints/openai/utils.py:
  • Changed --workers and --max-concurrency defaults to None
  • Added resolve_workers_and_concurrency(args) that resolves defaults based on platform:
    • NVIDIA GPU (CUDA): workers = ceil(max_num_seqs / 64), max_concurrency = workers * 512
    • Other platforms: workers = 1, max_concurrency = workers * 512
  • User-provided values are never overridden (only None is resolved)
• fastdeploy/entrypoints/openai/api_server.py:
  • Calls resolve_workers_and_concurrency(args) after parse_args(), before any usage
  • Added max_concurrency to startup logging

Usage or Command

Behavior is automatic. Explicitly passing --workers or --max-concurrency still overrides the computed defaults:

# Auto-resolved based on platform and max_num_seqs
fastdeploy serve --model my_model

# Explicit override still works
fastdeploy serve --model my_model --workers 4 --max-concurrency 2048

Accuracy Tests

No model output changes — this only affects API server worker/concurrency configuration.

Checklist

• Add at least a tag in the PR title.
• Format your code, run pre-commit before commit.
• Add unit tests. No unit tests added — logic depends on current_platform.is_cuda() which requires GPU hardware. The resolution function is straightforward and exercised on every server startup.
• Provide accuracy results. N/A — no model output changes.
• If the current PR is submitting to the release branch, make sure the PR has been submitted to the develop branch, then cherry-pick it to the release branch with the [Cherry-Pick] PR tag.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[paddle-bot]

Thanks for your contribution!

[PaddlePaddle-bot]

🤖 AI Code Review | 2026-04-20 11:18 CST

📋 Review 摘要

PR 概述：根据平台动态计算 --workers 和 --max-concurrency 的默认值，NVIDIA GPU 上按 max_num_seqs 自动伸缩 worker 数
变更范围：entrypoints/openai/ (api_server.py, utils.py)
影响面 Tag：APIServer

问题

级别	文件	概述
🟡 建议	utils.py:398	魔法数字 64/512 缺乏解释，建议抽取为命名常量
🟡 建议	utils.py:400	max_concurrency 计算逻辑在两个分支中重复，可简化
❓ 疑问	utils.py:394	or 8 vs if is None 语义差异

总体评价

整体实现逻辑清晰，用户显式传值不会被覆盖的设计合理。主要建议是将魔法数字抽取为常量并简化重复的分支逻辑，提高可维护性。

[PaddlePaddle-bot]

🟡 建议 魔法数字 64 和 512 缺乏解释

这里的 64 和 512 是关键的调优参数，但没有注释说明为什么选择这些值。建议抽取为命名常量并添加注释说明选择依据，方便后续维护和调优：

# 每个 worker 处理的最大序列数（经验值，平衡 worker 数与负载）
_SEQS_PER_WORKER = 64
# 每个 worker 的并发连接上限
_CONCURRENCY_PER_WORKER = 512

[PaddlePaddle-bot]

🟡 建议 max_concurrency 计算逻辑在两个分支中完全相同，可以简化

CUDA 和非 CUDA 分支中 max_concurrency 的计算逻辑是相同的（workers * 512），可以提取到 if/else 之后，减少代码重复：

if current_platform.is_cuda():
    if args.workers is None:
        args.workers = math.ceil(max_num_seqs / 64)
else:
    if args.workers is None:
        args.workers = 1

if args.max_concurrency is None:
    args.max_concurrency = args.workers * 512

[PaddlePaddle-bot]

❓ 疑问 使用 or 8 而非 if is None 的意图

getattr(args, "max_num_seqs", None) or 8 会在 max_num_seqs 为 0 或其他 falsy 值时也 fallback 到 8。虽然 max_num_seqs=0 在实践中不太可能出现，但如果只是想处理 None 的情况，使用显式的 if ... is None 更清晰：

max_num_seqs = getattr(args, "max_num_seqs", None)
if max_num_seqs is None:
    max_num_seqs = 8