Skip to content

[Fix] Support spawn workers in LiveCodeBench evaluation - #2603

Open
tandede wants to merge 2 commits into
open-compass:mainfrom
tandede:fix/livecodebench-spawn-worker
Open

[Fix] Support spawn workers in LiveCodeBench evaluation#2603
tandede wants to merge 2 commits into
open-compass:mainfrom
tandede:fix/livecodebench-spawn-worker

Conversation

@tandede

@tandede tandede commented Aug 20, 2026

Copy link
Copy Markdown

Motivation

codegen_check_correctness defines its multiprocessing target as a nested function. Process start methods that serialize the target, such as spawn, therefore fail before evaluating any generation with AttributeError: Can't pickle local object. This affects callers that select spawn explicitly and platforms where it is the default.

Modification

  • Move the LiveCodeBench subprocess entry point to module scope so it can be serialized by spawn.
  • Preserve the existing timeout, memory-limit argument, result, and metadata behavior.
  • Keep the existing mock-based memory-limit tests on their required fork context and add a regression test that launches the worker through a real spawn process.

No public API or behavior changes for existing fork users.

Local verification:

  • python -m pytest -q tests/openicl tests/prompt --disable-warnings (67 passed, 2 subtests passed)
  • python -m pytest -q tests/partitioners tests/summarizers tests/evaluator tests/tasks tests/tools tests/utils --disable-warnings (107 passed, 15 subtests passed)
  • python -m pytest -q tests/datasets/test_livecodebench_evaluator.py tests/datasets/test_code_execution.py tests/datasets/test_humaneval.py tests/datasets/test_taco.py --disable-warnings (17 passed, 1 platform-specific skip)
  • flake8, isort, yapf, and codespell pre-commit hooks passed for the changed files.

BC-breaking (Optional)

No.

Checklist

Before PR:

  • Pre-commit or other linting tools are used to fix the potential lint issues.
  • Bug fixes are fully covered by unit tests, the case that causes the bug is added in the unit tests.
  • The modification is covered by complete unit tests.
  • Documentation is not required because this is an internal portability fix with no public API change.

After PR:

  • If the modification has potential influence on downstream or other related projects, this PR should be tested with those projects.
  • CLA has been signed and all committers have signed the CLA in this PR.

@tandede
tandede marked this pull request as ready for review August 20, 2026 08:35
@tandede

tandede commented Aug 30, 2026

Copy link
Copy Markdown
Author

Hi maintainers, I’m following up on this PR in case it was missed. Both documentation builds have passed, and the LiveCodeBench spawn-worker support fix is ready for review. Please let me know if any changes or additional validation would be helpful. Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants