Skip to content

Commit b2cf70e

Browse files
ilmarkovMarkov Ilya
andauthored
[CI] BugFix Eval Small Models Distributed test for DiffusionGemma (#47980)
Signed-off-by: Markov Ilya <markovilya19@gmail.com> Co-authored-by: Markov Ilya <markovilya19@gmail.com>
1 parent d1f1d86 commit b2cf70e

2 files changed

Lines changed: 6 additions & 0 deletions

File tree

tests/evals/gsm8k/configs/DiffusionGemma-26B-A4B-it-FP8-dynamic.yaml

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,10 @@ num_questions: 1319
44
num_fewshot: 5
55
startup_max_wait_seconds: 1200
66
use_chat_completions: true
7+
# Diffusion models use a fixed internal temperature schedule and do not
8+
# support per-request temperature or seed overrides.
9+
temperature: 1.0
10+
seed: null
711
server_args: >-
812
--enforce-eager
913
--max-model-len 4096

tests/evals/gsm8k/test_gsm8k_correctness.py

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -69,6 +69,8 @@ def run_gsm8k_eval(eval_config: dict, server_url: str) -> dict:
6969
use_chat_completions=eval_config.get("use_chat_completions", False),
7070
host=host,
7171
port=port,
72+
temperature=eval_config.get("temperature", 0.0),
73+
seed=eval_config.get("seed", 42),
7274
request_timeout_seconds=request_timeout_seconds,
7375
gen_prefix=eval_config.get("gen_prefix", ""),
7476
)

0 commit comments

Comments
 (0)