feat: cordon should allow in-progress pipelines to drain

[Reiers]

Problem

When a node is cordoned (unschedulable=true), the Harmony scheduler stops picking up all new tasks, including the next stage for sectors already mid-pipeline on that machine.

Example: a sector finishes SDR on a cordoned node → TreeD won't be scheduled, even though the cache files are on this machine and no other node can do the work. The pipeline is stuck.

The only task with a workaround is Finalize, which has SchedulingOverrides for batch tasks.

Expected Behavior

Cordon should mean "stop accepting new work, but finish what you started." Specifically:

• New pipelines (new SDR tasks, new CC sectors via IAmBored) → blocked ✅ (works today)
• Running tasks → allowed to complete ✅ (works today)
• Next stage for in-progress pipelines on this node → should be allowed ❌ (broken today)

Current Workaround

Disable early-stage tasks individually (EnableSealSDR = false, etc.) while keeping later stages enabled. This allows in-progress pipelines to drain but is manual and error-prone.

Root Cause

In pollerTryAllWork(), when schedulable=false:

• All handlers without SchedulingOverrides are skipped
• followWorkInDB() is also skipped (the continue on line ~299)
• The SealPoller still creates tasks in DB, but the Harmony poller won't claim them

Possible Approach

Extend the SchedulingOverrides pattern: when cordoned, allow scheduling a task if the sector's data already resides on this machine (i.e., an earlier pipeline stage was completed here). This could be implemented by:

1. Checking if any related pipeline task was previously completed by this node (via harmony_task_history)
2. Or checking if the sector's storage paths are on this machine
3. Or adding a pipeline-aware "drain mode" flag that's distinct from full cordon

Related: pipeline-aware scheduling / anti-starvation improvements.

[Shekelme]

That would be an incredibly useful improvement.

[snadrus]

This current design allows your machine to be shut-down at the earliest non-destructive time so that you can get your updates done and get running again quickly. Once you restart the process, the work will be picked-up again promptly.

Meanwhile if you have other machines that can do the work, they will continue the work starting with the next task.

[Reiers]

Closing as won't fix (by design).

Cordon is intentionally simple: it blocks all new task scheduling so the node can be shut down at the earliest non-destructive time. Adding exceptions for location-bound pipeline tasks was explored in #1026, but the team consensus is that this leads to scope creep — operator architectures vary too much for one-size-fits-all drain logic.

The right solution is operator awareness. Opened #1030 with a dedicated Node Maintenance & Cordoning guide that covers:

• How cordoning affects location-bound pipeline tasks (TreeD, TreeRC, SyntheticProofs, Finalize)
• Recommended workflows for quick maintenance, extended downtime, and node decommissioning
• How to avoid losing SDR work by planning maintenance around pipeline state

tl;dr: Check pipeline state before cordoning. For long maintenance, either wait for pipelines to clear first or attach storage to another node.

[Shekelme]

It is unfortunate that a much more convenient alternative to the current "Cordon" mechanism has been rejected. I'm really, really sorry.