Fix workflow consumer greylist removal to prevent workflows stuck in "Pending" status

[Copilot]

The workflow consumer was not properly removing workflows from the greylist when they couldn't be processed, causing workflows to get permanently stuck in "Pending" status. The issue occurred when workflows were queued by the RunnablePoller but had their status changed to non-Runnable before the consumer processed them.

Root Cause

In WorkflowConsumer.ProcessItem(), the greylist removal was only executed when workflow.Status == WorkflowStatus.Runnable:

if (workflow.Status == WorkflowStatus.Runnable)
{
    try
    {
        result = await _executor.Execute(workflow, cancellationToken);
    }
    finally
    {
        // ... other code ...
        _greylist.Remove($"wf:{itemId}"); // Only removed if status was Runnable
    }
}
// No greylist removal if status was NOT Runnable!

This created a scenario where:

1. RunnablePoller finds a workflow and adds it to greylist
2. RunnablePoller queues the workflow for processing
3. Before the consumer processes it, the workflow status changes (suspended, completed, etc.)
4. Consumer skips processing since status is not Runnable
5. Workflow remains greylisted forever, preventing future re-queuing

Solution

Moved greylist removal to always execute regardless of workflow status:

try {
    // Process workflow if runnable
    if (workflow.Status == WorkflowStatus.Runnable) {
        // Execute workflow
    } else {
        Logger.LogDebug("Workflow {ItemId} is not runnable, status: {Status}", itemId, workflow.Status);
    }
} finally {
    // Always remove from greylist regardless of workflow status
    Logger.LogDebug("Removing workflow {ItemId} from greylist", itemId);
    _greylist.Remove($"wf:{itemId}");
    // ... other cleanup ...
}

Additional Improvements

• Added debug logging for non-runnable workflows to improve troubleshooting
• Enhanced error handling with workflow ID logging
• Added comprehensive documentation explaining the consumer's role
• Improved exception logging for better debugging

This fix ensures workflows are never permanently stuck in the greylist while maintaining all existing functionality for runnable workflows.

Fixes #1381.

---

💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

[Copilot]

@danielgerlag 👋 This repository doesn't have Copilot instructions. With Copilot instructions, I can understand the repository better, work faster and produce higher quality PRs.

I can generate a .github/copilot-instructions.md file for you automatically. Click here to open a pre-filled issue and assign it to me. I'll write the instructions, and then tag you for review.