Developer Recommended Best Practices

[cmoore24-24]

Hello!

I was hoping to open a post asking about what the best practices are for setting up a successful (modern) TaskVine workflow.

It's been a while since I brought this up in a meeting, but I do still occasionally run into an issue of TaskVine jobs slowing down quite a bit towards the end of the workflow (i.e. the final tasks on the progress bar take much longer than any previous, despite the output files being of these last tasks being not substantially different than those previous). I could easily believe that this is due to user error, so I wanted to post my current TaskVine "settings" and hopefully get some feedback on them!

I currently open each of my TaskVine applications with the following:

    m = DaskVine(
        [9101, 9200],
        name=f"{os.environ['USER']}-hgg",
        run_info_path=f"/project01/ndcms/{os.environ['USER']}/vine-run-info/",
        run_info_template=f'{template_name}',
    )

    m.tune("temp-replica-count", 3)
    m.tune("worker-source-max-transfers", 1000)
    m.tune("immediate-recovery", 0)
    m.tune("max-retrievals", -1)
    m.tune("transient-error-interval", 1)

And call compute with the following:

    dask.compute(
            tasks,
            scheduler=m.get,
            worker_transfers=True,
            resources={"cores": 1},
            task_mode="function-calls",
            lib_resources={'cores': 1, 'slots': 1},
            prune_depth=2, 
            optimize_graph=True,
        )

And for reference, I am currently on ndcctools version 7.15.8. Thank you for the thoughts, and let me know what you think!

[btovar]

I think the -1 is ok here, as the issue is at the end of the workflow, thus unlikely to cause workers' disconnection.
One thing to consider is that perhaps these tasks have been running for a very long time making it seem that they are tasks that started running at the end of the workflow. It could also be that they are running on a particularly slow worker. Do you have logs we can look at?

[cmoore24-24]

Yes, I have a few different sets of logs! Since they're on the larger side, I hope it's okay if I point you to their location on the CRC.

This first set of logs is from a run that had 12000 cores and was able to complete with only a few recovery tasks (I think less than 10 total): /project01/ndcms/cmoore24/keep_logs/nolepton_JetHT_KEEP_LOGS/vine-logs

The second set is from a job that had a similarly large input but had all of its workers evicted at some point and had to entirely recover: /project01/ndcms/cmoore24/keep_logs/nolepton_qcd_KEEP_LOGS/vine-logs

The third is a job that went relatively normally, barring the slowdown at the end: /project01/ndcms/cmoore24/keep_logs/nolepton_ttboosted_KEEP_LOGS/vine-logs

Hopefully one or all of these are helpful in some way! In the meantime, I'll give the suggested tune changes a shot to see if they help behavior.

As for the cause being long-running tasks, I believe we've seen that there are tasks that run long and stall when we've looked into this previously. The problem is that I haven't been able to successfully associate a long-running task with a particular chunk in order to figure out the problem. When I've tried running one chunk at a time in the past to see if there's any one that causes problems, I've found that individually they all run in similar timeframes. It's only in aggregate that this ending hangtime occurs, so I'm not sure what I'm doing that causes these very long tasks.

[btovar]

Ok, got them, looking at them now.