Skip to content

feat: add depends_on condition: completed for one-off programs - #26

Merged
ademidoff merged 2 commits into
mainfrom
feat/depends-on-completed
Aug 16, 2026
Merged

feat: add depends_on condition: completed for one-off programs#26
ademidoff merged 2 commits into
mainfrom
feat/depends-on-completed

Conversation

@ademidoff

Copy link
Copy Markdown
Owner

Closes #25.

started waits for the process, healthy waits for readiness. Neither expresses "wait for the work to finish", which is what a migration, an init task or a seed job is depended on for. This adds the third condition.

programs:
  migrate:
    command: /usr/bin/migrate --up
    autorestart: never

  api:
    command: /usr/bin/api
    depends_on:
      migrate:
        condition: completed

Semantics

As proposed in the issue:

  • Satisfied by exit status 0, which is the same line autorestart: unexpected draws.
  • Latched, for the daemon's lifetime. The condition stays satisfied while the task sits in EXITED, which is what the current behaviour gets wrong.
  • Cleared when the program runs again, so sctl restart migrate re-runs the work and latches again on success. Dependents already up are left alone.
  • A task that cannot complete drives its dependents to the existing "will never come up" path rather than an indefinite wait.
  • Reload replacing the task clears the latch and runs the new definition; dependents already running stay running.

One decision the issue did not cover: sctl stop on a completed task does not clear the latch. Stopping a program that has already exited does not undo the work it did.

Implementation

The latch lives on Process (internal/process/process.go), set in the monitor when a run nobody asked to stop exits 0, cleared in Start(). It is set before the EXITED state change, because that change is what wakes the dependents — setting it afterwards would have them look and find nothing. Reload replaces the Process, so the latch clears there without any extra bookkeeping.

blockedByDependency (internal/server/reconcile.go) became condition-aware, which it could not be while it walked the dependency graph: the graph carries ordering, the config carries conditions. It now also treats an unsuccessful exit under autorestart: never as final, which is the second way a task fails to complete — FATAL was already covered.

Validation rejects autorestart: always on a program something waits to complete, naming both programs. autorestart: unexpected is the natural pairing and is accepted.

Two pre-existing waits this had to fix

Both are reachable today with any one-off, and sctl restart migrate — which the issue requires to re-run and re-latch — hits both:

  • StopProcess waited for STOPPED specifically. The reconciler deliberately leaves EXITED alone, so stopping a program that had already exited timed out after 30s.
  • StartProcess waited for RUNNING. A task that finishes inside startsecs never reaches it, so a successful run was reported as a failure after 30s.

StopProcess now accepts any settled state, and StartProcess accepts a completed run. Both are covered by tests that measure the elapsed time, so a regression shows up as a timeout rather than as a slow pass.

Tests

Unit tests cover the four cases the issue lists, plus the latch's placement relative to the state change, a failed exit not latching, and the two waits above.

probes/completed-latch.sh runs the whole shape in a container. --started is the control, and reproduces both of today's failures:

t=2s  migrate=RUNNING  api=RUNNING     <- ran alongside the task, not after it
t=5s  migrate=EXITED   api=RUNNING

  sctl stop api ... ok
  sctl start api ... REFUSED (30s)     <- and now it cannot come back

Why api is not running: dependency migrate is EXITED

against the same configuration with condition: completed:

t=4s  migrate=RUNNING  api=STOPPED     <- did not start alongside the work
t=5s  migrate=EXITED   api=RUNNING     <- started once the work was done

  sctl stop api ... ok
  sctl start api ... ok (1s)           <- the completion outlived the run

Documentation

README.md gains a Waiting for a one-off to finish subsection under Dependency Management, and the Best practices section from #10 is rewritten: it told people not to do this, and now points at the supported way of doing it.

started waits for the process and healthy waits for readiness. Neither
expresses "wait for the work to finish", which is what a migration, an
init task or a seed job is depended on for. Naming one in depends_on
today fails two ways: with condition: started the dependent runs
alongside the task rather than after it, and once the task has exited a
dependent can never start at all, so one that crashes an hour later
cannot come back.

A program latches as completed when a run it was not asked to stop exits
with status 0. The latch is set before the EXITED state change, since
that change is what wakes the dependents, and it survives the program
sitting in EXITED. Running the program again clears it, so sctl restart
re-runs the work and latches again on success; reload replaces the
process outright, so a redefined task runs on its own account.

A task that cannot complete drives its dependents to the existing "will
never come up" path rather than an indefinite wait: FATAL was already
covered, and an unsuccessful exit under autorestart: never is now
covered too. autorestart: always on a program something waits to
complete is rejected at startup, naming both programs.

Two waits had to be fixed for the re-run flow to work at all, both
reachable today with any one-off:

- stopping a program that had already exited waited for STOPPED
  specifically, which the reconciler never moves it to, so the request
  timed out after 30s;
- starting a program that finished its work inside startsecs waited for
  RUNNING, which it never reaches, so a successful run was reported as
  a failure.

probes/completed-latch.sh reproduces both failures against
condition: started and shows the latch holding across a dependent
restart, which is the part a unit test shows least convincingly.

Closes #25
Stopping a program that had already exited reported success without
releasing it. EXITED is not running, but it is not settled either: the
monitor may be sitting in a restart backoff that only Stop() cancels.
The reconciler skipped every stopped state, so a stopped program spawned
a fresh run one backoff after the stop was reported as done, and a
reload that replaced it discarded the Process object while that run was
still queued, leaving a child nothing tracked, nothing could stop and
nothing would reap.

The reconciler now releases anything that is not already STOPPED when
its desired state is STOPPED, which cancels the pending restart. That is
the root of what the previous commit worked around by accepting EXITED
and FATAL as "stopped", so those predicates go back to waiting for
STOPPED specifically, and now get it.

A stop no longer reports that a program cannot start. awaitState's
"a dependency of this will never come up" shortcut applied to stop
requests too, so stopping a program whose dependency happened to be
stopped failed with "process api cannot start: dependency db is stopped
and is not set to start" while the stop was proceeding normally, and
sctl restart propagated it instead of starting anything. Dependencies
decide what may start, never what may stop.

Also widen the window TestReconcile_WaitsForAOneOffToComplete asserts
RUNNING in, keep the probe's closing diagnostics from aborting under
set -e when the daemon is the thing that died, and note in the README
that a task with autostart: false holds its dependents back across a
daemon restart until it is started by hand.
@ademidoff
ademidoff merged commit ddf444f into main Aug 16, 2026
2 checks passed
@ademidoff
ademidoff deleted the feat/depends-on-completed branch August 16, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add condition: completed so one-off programs can have dependents

1 participant