Skip to content

Commit 295ab07

Browse files
committed
fix(api/task): guard empty results list in process_results
For generate_until / generate_visual_cot tasks, when a sample's generation failed upstream (model raised, retries exhausted, etc.) the results list can be empty. The existing isinstance check then falls through to `results[0]` on the list-of-list branch and raises IndexError — which aborts the full postprocess loop for that task, not just the missing sample. Add a leading `results and` so an empty list falls through to the else branch that does `[res.strip() for res in results]` (empty output) — downstream task-level process_results then receives an empty list and can decide how to score the missing sample without taking down the whole task.
1 parent 4caa4a6 commit 295ab07

1 file changed

Lines changed: 4 additions & 1 deletion

File tree

lmms_eval/api/task.py

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1576,7 +1576,10 @@ def construct_requests(self, doc_id: int, ctx: str, **kwargs) -> Union[List[Inst
15761576
@retry(stop=(stop_after_attempt(5) | stop_after_delay(1200)), wait=wait_fixed(2))
15771577
def process_results(self, doc, results, full_docs=None):
15781578
if self.OUTPUT_TYPE in ("generate_until", "generate_visual_cot"):
1579-
if isinstance(results, list) and isinstance(results[0], list):
1579+
# Guard empty results so results[0] below does not IndexError for
1580+
# samples whose generation failed. Downstream process_results then
1581+
# receives an empty list and can decide how to score the miss.
1582+
if results and isinstance(results, list) and isinstance(results[0], list):
15801583
results = [res.strip() for res in results[0]]
15811584
else:
15821585
results = [res.strip() for res in results]

0 commit comments

Comments
 (0)