task cancelation problem

[haidi-ustc]

Hi, covalent team,
I'm encountering some issues with task cancellation, and I was wondering if you could help me resolve them. Below is my code. I execute the do_something task, I try to simulate the task or computation output using Python's sleep and subprocess.run.
The current script works fine when I execute ct.cancel(disp_id), but if I remove this line from the script and try to cancel the tasks from a new command line terminal using ct.cancel(disp_id), the tasks cannot be cancelled, especially the subprocess.run task continues running indefinitely.

import os
import covalent as ct
import time
from random import randint
import subprocess
local = ct.executor.LocalExecutor()

@ct.electron 
def read_structure(path,filename):
    pwd=os.getcwd()
    os.chdir(path)
    with open(filename,'r') as f:
         lines=f.readlines()
    
    with open('a.vasp','w') as f:
         f.writelines(lines)
    os.chdir(pwd)
    return lines

@ct.electron(executor=local)
def do_something(path,struct):
    # this will be a time consuming task via call the external program 
    pwd=os.getcwd()
    os.chdir(path)
    time.sleep(randint(30,50))   # simulate job by sleep 

    #simulate the output file 
    #subprocess.run("while true; do echo 1 ; sleep 1s; done > 1.txt ",shell=True)
    os.chdir(pwd)
    return struct


@ct.lattice
def workflow(path,filename):
    struct = read_structure (path,filename)
    return do_something(path,struct)

content="""
B8
1.0
2.000000 0.000000 0.000000
0.000000 2.600000 0.000000
0.000000 0.000000 18.355599
B
8
direct
0.000000 0.333000 0.591405 B
0.000000 0.667000 0.408595 B
0.000000 0.667000 0.591405 B
0.000000 0.333000 0.408595 B
0.500000 0.500000 0.564166 B
0.500000 0.500000 0.435834 B
0.500000 0.000000 0.527937 B
0.500000 0.000000 0.472063 B
""" 
with open("POSCAR",'w') as f:
     f.write(content)

pwd=os.getcwd()
disp_id=ct.dispatch(workflow)(path=pwd,filename=os.path.abspath('./POSCAR'))
result = ct.get_result(disp_id)
print(result)
ct.cancel(disp_id)

[kessler-frost]

Hey @haidi-ustc so sorry to have missed your discussion. So, one thing to note here is that the LocalExecutor does not currently support task cancellation. But the DaskExecutor (which is the default) does support it.

Or rather I should say that if a task has already started then there isn't a way to make it stop (i.e. cancelling it) in case of LocalExecutor but there is if you use the DaskExecutor.

I am wondering though, is there a reason why you're not using the default executor to run that task?

[kessler-frost]

That said, it seems like using shell=True is also problematic. It spawns a separate shell and tells it to run your command - this prevents the propagation of the "cancel" or "shutdown" command to that shell. It is also in general advised to not use the shell=True argument and let it remain False unless there's a very specific reason for that.