[One Workflow] Orphaned scheduled task root cause fix + self-healing

[VladimirFilonov]

Summary

closes: https://github.com/elastic/security-team/issues/17420

Root cause fix

Fix orphaned workflow:scheduled Task Manager tasks after workflow deletion that caused Workflow <id> not found errors in workflowsExecutionEngine (observed on observability serverless).

Root cause: unscheduleWorkflowTasks used taskManager.fetch to find tasks before bulkRemove. Task Manager schedules with refresh: false, so a task could be missing from search at delete time while still being created or claimable later. Unschedule then no-op’d, leaving a task that ran against a deleted or tombstoned workflow.

Fix:

• Single-workflow unschedule (syncSchedulerAfterSave, etc.): call removeIfExists with the deterministic id workflow:<workflowId>:scheduled — no fetch.
• Bulk unschedule (soft/hard delete, disableAllWorkflows): add bulkUnscheduleWorkflowTasks that bulkRemoves by those ids; treat 404 as success; log per-workflow warnings on other failures without failing the batch.
• Thin unschedule_workflow_tasks wrapper delegates to bulkUnscheduleWorkflowTasks and drops the unused logger parameter from call sites.

Self-healing for already affected envs

• add self-healing behavior for workflow:scheduled runs when the workflow document is missing
• return shouldDeleteTask: true from the scheduled task runner so Task Manager deletes orphaned recurring tasks
• add a focused unit test to verify the missing-workflow path returns delete intent and skips execution

[kibanamachine]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 901a1b0
• Build duration: 73 mins

Failed CI Steps

• FTR Configs #54
• FTR Configs #128
• FTR Configs #143
• FTR Configs #202

Metrics [docs]

✅ unchanged

History

• 💛 Build #449337 was flaky e48f742

[h88]

LGTM
One question: self-healing deletes when getWorkflow returns null, which includes soft-deleted tombstones (not just hard-deleted). Is that intentional?

[VladimirFilonov]

LGTM One question: self-healing deletes when getWorkflow returns null, which includes soft-deleted tombstones (not just hard-deleted). Is that intentional?

Yes - on any edit (restore if its possible), task will be rescheduled