Skip to content

Commit 790e1bd

Browse files
committed
Extract task execution to TaskRunner
At its core a process is a function that accepts an input, executes a task and returns an asynchronous result. TaskProcessor implements this functionality together with the dataflow interface of the process, so that a task can only be executed by feeding values through the process input channels. Move the execution of a task to a separate `TaskRunner` class: - the resolution of the task inputs and outputs - the store directory and cache lookup - the delegation to the executor, including task arrays - the task error strategy and retries TaskProcessor keeps the dataflow interface -- the operator, the process state, the output binding and the publishing -- as well as the process scoped services shared by all task executions, such as the name, the config, the executor and the task environment. The processor creates a runner with `taskCompleted` as the completion handler, so that the runner does not depend on the dataflow interface: a task can be executed by invoking `TaskRunner.submit()` directly, without the process being part of a dataflow network (see the test 'should execute a task without a dataflow network'). Note that `resumeOrDie` is now synchronized on the runner rather than on the processor, so it no longer excludes `bindOutputs` and `sendPoisonPill`. These methods do not share mutable state, the coarse lock was incidental. Assisted-by: Claude Opus 5 (1M context) Signed-off-by: Ben Sherman <bentshermann@gmail.com>
1 parent 2b2fce1 commit 790e1bd

9 files changed

Lines changed: 1361 additions & 1139 deletions

File tree

docs/developer/diagrams/nextflow.processor.mmd

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,13 @@ classDiagram
1212
operator : DataflowProcessor
1313
taskBody : BodyDef
1414
}
15-
TaskProcessor --> TaskRun : invokeTask
16-
TaskProcessor --> PublishDir : finalizeTask
15+
TaskProcessor --* TaskRunner
16+
TaskProcessor --> PublishDir : taskCompleted
17+
18+
class TaskRunner {
19+
processor : TaskProcessor
20+
}
21+
TaskRunner --> TaskRun : submit
1722

1823
class TaskRun {
1924
config : TaskConfig

docs/developer/nextflow.processor.mdx

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -20,8 +20,13 @@ classDiagram
2020
operator : DataflowProcessor
2121
taskBody : BodyDef
2222
}
23-
TaskProcessor --> TaskRun : invokeTask
24-
TaskProcessor --> PublishDir : finalizeTask
23+
TaskProcessor --* TaskRunner
24+
TaskProcessor --> PublishDir : taskCompleted
25+
26+
class TaskRunner {
27+
processor : TaskProcessor
28+
}
29+
TaskRunner --> TaskRun : submit
2530
2631
class TaskRun {
2732
config : TaskConfig

modules/nextflow/src/main/groovy/nextflow/processor/TaskPollingMonitor.groovy

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -677,7 +677,7 @@ class TaskPollingMonitor implements TaskMonitor {
677677
if (evict(handler)) {
678678
handler.decProcessForks()
679679
}
680-
fault = handler.task.processor.resumeOrDie(handler?.task, error, handler.getTraceRecord())
680+
fault = handler.task.processor.getRunner().resumeOrDie(handler?.task, error, handler.getTraceRecord())
681681
log.trace "Task fault (1): $fault"
682682
}
683683
finally {
@@ -743,7 +743,7 @@ class TaskPollingMonitor implements TaskMonitor {
743743

744744
protected void finalizeTask( TaskHandler handler ) {
745745
// finalize the task execution
746-
final fault = handler.task.processor.finalizeTask(handler)
746+
final fault = handler.task.processor.getRunner().finalizeTask(handler)
747747

748748
// notify task completion
749749
session.notifyTaskComplete(handler)

0 commit comments

Comments
 (0)