In the docs it says:
Note that the change of working directory (and possibly the copy) only happens when the infra is called for submitting the decorated function. Depending on your code, this may not be at the very beginning of your execution.
Addressing this limitation would have a lot of impact in making exca more flexible to computing pipelines with sequential steps that require very different computing requirements, and where the caching of intermediate steps is desirable. Is there an accepted way of facilitating such functionality with exca? If not, I have an ideas for a workarounds to address this, I am curious whether this has been thought about already or might work out of the box in the current implementation.
My idea is to first submit an infra task from the outermost code execution point with cluster=local, essentially facilitating a process-wide initial workdir cache (so the source code is uninterrupted by local changes) but not yet allocating resources. Inside this process, different elements of the code would submit new infra tasks with cluster=slurm, and with specific resource allocations for specific functions that need more computing power. Would this work? Does the current implementation support nested tasks in this way?
In the docs it says:
Addressing this limitation would have a lot of impact in making exca more flexible to computing pipelines with sequential steps that require very different computing requirements, and where the caching of intermediate steps is desirable. Is there an accepted way of facilitating such functionality with exca? If not, I have an ideas for a workarounds to address this, I am curious whether this has been thought about already or might work out of the box in the current implementation.
My idea is to first submit an infra task from the outermost code execution point with cluster=local, essentially facilitating a process-wide initial workdir cache (so the source code is uninterrupted by local changes) but not yet allocating resources. Inside this process, different elements of the code would submit new infra tasks with cluster=slurm, and with specific resource allocations for specific functions that need more computing power. Would this work? Does the current implementation support nested tasks in this way?