Skip to content

SlurmEngine: ignore submit history from a different engine in get_task_termination_info - #309

Merged
albertz merged 1 commit into
masterfrom
albert-slurm-task-info-fix
Aug 19, 2026
Merged

SlurmEngine: ignore submit history from a different engine in get_task_termination_info#309
albertz merged 1 commit into
masterfrom
albert-slurm-task-info-fix

Conversation

@albertz

@albertz albertz commented Aug 19, 2026

Copy link
Copy Markdown
Member

E.g. after switching engine from local engine to Slurm, you get this error:

[2026-08-19 19:08:59,634] ERROR: Exception in thread <_MainThread(MainThread, started 22535308648512)>:
EXCEPTION
Traceback (most recent call last):
  File ".../tools/sisyphus/sisyphus/tools.py", line 304, in default_handle_exception_interrupt_main_thread.<locals>.wrapped_func
    line: return func(*args, **kwargs)
    locals:
      func = <local> <function sisyphus.manager.Manager.run>
  File ".../tools/sisyphus/sisyphus/manager.py", line 664, in Manager.run
    line: self.run_jobs()
  File ".../tools/sisyphus/sisyphus/manager.py", line 463, in Manager.run_jobs
    line: self.thread_pool.map(f, self.jobs.get(gs.STATE_RUNNABLE, []))
  File ".../python3.12/multiprocessing/pool.py", line 367, in Pool.map
    line: return self._map_async(func, iterable, mapstar, chunksize).get()
  File ".../python3.12/multiprocessing/pool.py", line 774, in ApplyResult.get
    line: raise self._value
  File ".../python3.12/multiprocessing/pool.py", line 125, in worker
    line: result = (True, func(*args, **kwds))
  File ".../python3.12/multiprocessing/pool.py", line 48, in mapstar
    line: return list(map(*args))
  File ".../tools/sisyphus/sisyphus/manager.py", line 456, in Manager.run_jobs.<locals>.f
    line: self.job_engine.submit(task)
    locals:
      task = <local> <Task 'run' job=Job<work/i6_experiments/users/dorian_koch/jobs/RoverJob/RoverJob.u8KLovkc8FMy>>
  File ".../tools/sisyphus/sisyphus/engine.py", line 181, in EngineBase.submit
    line: rqmt = self.get_rqmt(task, task_id)
  File ".../tools/sisyphus/sisyphus/engine.py", line 110, in EngineBase.get_rqmt
    line: termination_info = self.get_task_termination_info(task, task_id, last_rqmt)
    locals:
      last_rqmt = <local> {'cpu': 1, 'mem': 12.0, 'time': 1.0, 'engine': 'short', 'sbatch_args': [],
                           'engine_info': 'login23-4.hpc.itc.rwth-aachen.de', 'engine_name': 'local',
                           'completed_fraction': None}, len = 8
  File ".../tools/sisyphus/sisyphus/engine.py", line 338, in EngineSelector.get_task_termination_info
    line: return self.get_used_engine_by_rqmt(submit_info).get_task_termination_info(task, task_id, submit_info)
    locals:
      submit_info = <local> {'cpu': 1, 'mem': 12.0, 'time': 1.0, 'engine': 'short', 'sbatch_args': [],
                             'engine_info': 'login23-4.hpc.itc.rwth-aachen.de', 'engine_name': 'local',
                             'completed_fraction': None}, len = 8
  File ".../tools/sisyphus/sisyphus/simple_linux_utility_for_resource_management_engine.py", line 330,
       in SimpleLinuxUtilityForResourceManagementEngine.get_task_termination_info
    line: job_id = next(
              (job_id for task_ids, job_id in submit_info.get("engine_info", []) if task_id in task_ids),
              None,
          )
    locals:
      task_id = <local> 1
ValueError: not enough values to unpack (expected 2, got 1)

This PR fixes this.

@albertz
albertz merged commit 21ba684 into master Aug 19, 2026
4 checks passed
@albertz
albertz deleted the albert-slurm-task-info-fix branch August 19, 2026 18:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants