Prevent simultaneous task failures multiple archive calls

[arhall0]

If multiple tasks fail simultaneously, they will all attempt to call beeflow.wf_manager.resources.wf_update.archive_workflow. This will succeed for the first task that makes the call, but then lead to an endless loop of calls for the other tasks that error since the workflow folder will no longer exist in .beeflow/workflows.

This solves that problem by checking the workflow state does not begin with Archived when attempting to archive a failed workflow. db.workflows.get_workflow_state seems to be the only way to do this once a workflow has been archived.

I put this check where a failed task would call archive_fail_workflow, but potentially this could be in archive_workflow directly.

Needs a test before removing WIP.

[arhall0]

I adjusted the solution based on discussion. Since the behavior of a failing task failing and archiving the entire workflow may change, I added a more general check to archive_workflow that prevents archiving if the workflow is already in an Archived state and logs a warning about this. I think this is a more robust fix as it should not need to be changed if task failing behavior changes.

[pagrubel]

Nice I tried a workflow of cat-grep-tar with a 2nd branch starting with running the cat step without a proper input file and all looks good. One branch fails and the workflow is still archived. Here is the dag. I think I may put the workflow in our data directory. I'll add a readme for it.

[pagrubel]

So the workflow that caused the error doesn't cause the error anymore, however, the above workflow continued but the final status was not in the archive so because a task failed on one branch the other branch continued to run but the workflow was archived too early. I've added the example. This workflow should be archived and removed from the ~/.beeflow/workflows directory. It was archived early and the status is incorrect. If you do a beeflow query on it, it looks fine because it is accessing the ~/.beeflow/workflow directory.

[arhall0]

To solve that problem we need to think about how we want this to work at the moment.

There's a couple things to consider:

• Should a failed task ever archive the workflow
• What determines if a workflow has the state "Archived/Failed" (if we want to use this at all)

For the first, it seems like the answer is no. Perhaps we may want to add some way of specifying this in the future; where failure of some very important task kills the workflow entirely.

For the second, we could use the "Archived/Failed" state if any task fails (setting a flag) or just not use this state. I'm not sure which is more clear to a user. If you do beeflow query you would see tasks had failed, in either case. But if you did beeflow list then it could be confusing. Using "Archived/Failed" may lead them to believe the entire workflow failed. Using just "Archive" may hide that some tasks failed.

I think I personally lean towards not using "Archived/Failed".

[pagrubel]

To solve that problem we need to think about how we want this to work at the moment.

There's a couple things to consider:

• Should a failed task ever archive the workflow
• What determines if a workflow has the state "Archived/Failed" (if we want to use this at all)

For the first, it seems like the answer is no. Perhaps we may want to add some way of specifying this in the future; where failure of some very important task kills the workflow entirely.

For the second, we could use the "Archived/Failed" state if any task fails (setting a flag) or just not use this state. I'm not sure which is more clear to a user. If you do beeflow query you would see tasks had failed, in either case. But if you did beeflow list then it could be confusing. Using "Archived/Failed" may lead them to believe the entire workflow failed. Using just "Archive" may hide that some tasks failed.

I think I personally lean towards not using "Archived/Failed".
Sounds fine.

1. We should think about implementing a way to do your first point, I guess this can be an issue or discussion. At that time we may want to re-introduce the Archived/Failed status.
2. We intend to add more information to query and list.

[arhall0]

I noticed that if there is a build fail for a task, it also fails/archives the workflow. I think this case would have the same issues as the topic of this PR.

It seems like this should be handled the same as any task failure; I'm not sure why it is being handled differently at the moment. Is this correct?

[pagrubel]

I’ll have to look at this one. The problem is: I think it’s difficult to tell when a build fails. Let me see if I can simulate one. From: Aaron R Hall ***@***.***> Reply-To: lanl/BEE ***@***.***> Date: Wednesday, February 19, 2025 at 9:19 AM To: lanl/BEE ***@***.***> Cc: "Grubel, Patricia A" ***@***.***>, Review requested ***@***.***> Subject: [EXTERNAL] Re: [lanl/BEE] Prevent simultaneous task failures multiple archive calls (PR #1008) I noticed that if there is a build fail for a task, it also fails/archives the workflow. I think this case would have the same issues as the topic of this PR. It seems like this should be handled the same as any task failure; I'm not sure why it is being handled differently at the moment. Is this correct?

[pagrubel]

Apparently, Build_FAIL is not being handle correctly at all
beeflow query 0f
Running
clamr--BUILD_FAIL
ffmpeg--WAITING

ffmpeg depends on clamr so the entire workflow should fail.

[pagrubel]

I'm thinking a container failure is fairly catastrophic to a workflow. Maybe we should stop the workflow. As we work on the builder code we can make other decisions

[arhall0]

I think this is ready for review again. With the latest commit, a failed task will not fail the workflow (BUILD_FAIL is not being changed by this PR however). I checked the different workflows we've discussed on this PR are working as expected.

Should make issues about the following that came up during review, but are outside the scope of this PR:

1. End workflow state should be Archived, Archived/Failed, or Archived/Partial-Fail depending on state of final task nodes
2. If a task ends in BUILD_FAIL, the entire workflow should end; including fixing a bug where BUILD_FAIL doesn't even fail dependent tasks.