Support provisional restart upgrades under external process supervisors

[ausimian]

Summary

Support OTP restart_emulator transitions when the release is supervised by systemd, Docker, Kubernetes, or another external process supervisor rather than heart.

This is part of completing the stock Mix launcher architecture: Castle should preserve OTP's release-state semantics while adapting the restart handshake so an external supervisor remains responsible for starting the new BEAM process.

Verified OTP behaviour

On OTP 28.3, plain restart_emulator does use OTP's provisional restart preparation path. After the relup evaluator returns restart_emulator, release_handler calls prepare_restart_new_emulator/7, which:

• marks the target release current, then persists it as tmp_current in RELEASES;
• writes releases/new_start_erl.data naming the target ERTS/release version;
• calls heart:set_cmd/1 with the classic OTP start command;
• only after that succeeds does the outer handler call init:reboot().

restart_new_emulator is different because it first creates a hybrid temporary release (new ERTS/kernel/stdlib/SASL + old remaining applications) and continues the rest of the upgrade after reboot. It is not distinguished by exclusive use of new_start_erl.data, tmp_current, or heart; plain restart_emulator uses those mechanisms too.

The actual compatibility gap

A normal Mix release supervised by systemd/Docker/etc. usually does not run with -heart.

heart:set_cmd/1 fails with badarg if the registered heart process does not exist. Therefore a plain restart_emulator transition currently fails during OTP's restart preparation and never reaches the intended init:reboot().

Classic OTP also assumes a start_prg such as $ROOT/bin/start that accepts the generated start-data file; a Mix release instead owns bin/<release> and selects RELEASE_VSN itself.

We should adapt these assumptions without replacing the Mix launcher or reimplementing OTP's RELEASES state machine.

Proposed architecture

1. Preserve OTP's provisional release state

Continue to use plain restart_emulator for full-restart transitions. Let release_handler own:

• tmp_current state;
• RELEASES persistence/reconciliation;
• new_start_erl.data generation;
• make_permanent/1 / start_erl.data semantics.

Do not introduce Castle-owned edits of RELEASES or replace OTP's transactional state model unless this approach proves infeasible.

2. Bridge the heart:set_cmd/1 handshake — real heart, defanged

Run real heart, configured so it is operationally inert, and let the external supervisor remain the sole restart authority. This keeps us entirely on documented OTP interfaces.

Setting	Purpose
-heart, via ELIXIR_ERL_OPTIONS in env.sh	the heart process exists, so heart:set_cmd/1 returns ok instead of raising badarg
HEART_NO_KILL=TRUE	heart does not SIGKILL a node that misses heartbeats — but see the correction below, it does not keep the node alive
HEART_BEAT_TIMEOUT=65535	heart's documented maximum, which is what actually makes the timeout unreachable
HEART_COMMAND unset	nothing happens on an unexpected death when no temporary command is set
$ROOT/bin/start, shipped by Forecastle and inert (exit 0)	neutralises the temporary command release_handler installs

release_handler then completes its normal preparation and calls init:reboot(); the BEAM exits; the supervisor starts bin/<release> again; the env hook selects the provisional version.

Correction — HEART_NO_KILL does not mean the node survives a heartbeat timeout. An earlier revision of this section said "heart never kills a node that misses heartbeats", measured by SIGSTOPing a beam and observing it still in state T after heart had run the command. That measurement was wrong, because a stopped process cannot act on its own shutdown. The node dies the moment it resumes:

heart: heart-beat time-out, no activity for 15 seconds
heart: Executed ".../probe.sh t" -> 0. Terminating.
Kernel pid terminated (heart) ({port_terminated,{heart,loop,...}})

HEART_NO_KILL suppresses the SIGKILL, but the port program terminates once it has run the command, the Erlang heart process then exits {port_terminated, …}, and heart is a kernel process linked to init (new_kernelpid/3) — so init halts the node anyway.

So -heart gives a stalled node a way to die that a deployment without heart did not have. That is what HEART_BEAT_TIMEOUT=65535 is for: at the 60-second default this is reachable by an ordinary long stall, and at heart's documented maximum it is not.

The inert bin/start is still mandatory, and for the reason originally given — HEART_NO_KILL suppresses the kill but not the command, so heart runs system(cmd) on a timeout. A bin/start that launched the target would start it while the old node was still up, in the window before init gets round to halting it. Doing nothing in that script is the whole point, and the HEART_NO_KILL documentation assumes as much — "useful if the command executed by heart takes care of this".

Also measured, and the reason bin/start must exist rather than merely being named: heart does execute the temporary command on init:reboot() with HEART_COMMAND unset, receiving the data file as $1.

Keep it at the default start_prg path. The default is {no_check, $ROOT/bin/start} and needs no configuration; setting {sasl, start_prg} explicitly is {do_check, _} and would mean injecting :sasl config, which Forecastle deliberately no longer does.

Rejected alternative — impersonating heart. A Castle process registered as heart answering {Caller, set_cmd, Cmd} with {heart, ok} also works, and was the earlier recommendation here. It depends on an OTP-internal message protocol behind the public API, and heart:wait/0 is a receive with no after clause — so a shim that took the message and died before replying would leave release_handler blocked inside handle_call indefinitely, wedging every subsequent release operation on the node. Real heart avoids both. Keep the shim only as a fallback if running -heart proves impossible in some supported deployment.

Rejected alternative — heart as the restarter. Real heart plus a bin/start that actually starts the target, alongside a supervisor with Restart=, gives two independent authorities starting the service. Sharing one implementation between the two entry points does not help: the problem is duplicated authority, not duplicated code.

3. Use the stock Mix launcher for the provisional boot

After the BEAM exits, the external supervisor starts the standard bin/<release> again.

The Castle/Forecastle release env hook should:

1. detect a valid one-shot releases/new_start_erl.data;
2. atomically consume/rename it;
3. extract the target release version;
4. set RELEASE_VSN;
5. re-exec the untouched Mix launcher so it recomputes REL_VSN_DIR from that version.

On subsequent starts, with no provisional marker, the stock launcher uses ordinary releases/start_erl.data and therefore boots the permanent release.

Rollback semantics

Desired transition:

A permanent
  -> install B with restart_emulator
  -> OTP persists B as tmp_current + writes new_start_erl.data
  -> BEAM exits
  -> external supervisor restarts
  -> env hook boots B provisionally
  -> B is current in memory, A remains permanent

If B crashes before commit:

no one-shot provisional marker
  -> external supervisor starts normally
  -> start_erl.data still names A
  -> A boots again

If B is accepted:

Castle.commit(B)
  -> release_handler.make_permanent(B)
  -> start_erl.data now names B

Failure handling

A failed restart preparation may already have written new_start_erl.data before heart:set_cmd/1 fails. release_handler can reconcile tmp_current back to unpacked when SASL restarts on the old script, but the launcher hook must not blindly consume a stale marker from a failed transition.

The implementation therefore needs an explicit validity/one-shot strategy coupling the Castle restart preparation and Forecastle env hook.

Acceptance criteria

• Plain restart_emulator works without requiring heart to own process restart.
• OTP remains authoritative for RELEASES, tmp_current, new_start_erl.data, and make_permanent/1 semantics.
• The standard Mix bin/<release> script is not replaced or patched.
• External supervisor restart boots the provisional target through the env-hook/re-exec path.
• A second restart before commit rolls back to the previous permanent release.
• Castle.commit/1 makes the target permanent and ordinary starts select it afterward.
• Failed restart preparation cannot leave a stale marker that is later mistaken for a valid provisional boot.
• heart is operationally inert: a heart-beat timeout neither kills the node nor starts anything, and an unexpected death starts nothing.
• The heart configuration and the inert bin/start are integration-tested across all supported OTP versions. heart's behaviour lives in a C port program rather than in heart.erl, so it is not verifiable by source review and has to be measured per OTP release.
• Verified under each supported supervisor — systemd, Docker restart policies, runit — that the supervisor is the only thing that starts the replacement, and that -heart in foreground mode causes no trouble. Mix recommends heart only for daemon* and does not say why.

Related work

• ausimian/forecastle#4: generates explicit hot vs restart_emulator transitions.
• ausimian/forecastle#10: launcher-side new_start_erl.data selection/re-exec hook.
• ausimian/forecastle#3: established the invariant that the Mix-generated launcher remains stock.

[ausimian]

Scope note (1.0.0 planning)

This issue is part of completing the stock-launcher architecture, not a separate
restart feature. Recording why, because the framing matters for how it gets
implemented.

forecastle#3's PR description claimed "version selection needs no code at all". That
holds after commit, because releases/start_erl.data names the permanent version.
It does not hold for a restart_emulator transition, which is exactly the gap this
issue describes: OTP writes the provisional target to releases/new_start_erl.data
and deliberately leaves start_erl.data pointing at the old permanent version, so a
crash before commit rolls back. So this is the missing piece of "keep the stock Mix
launcher", not an addition on top of it.

The approach in the issue body is right and should be followed as written —
especially the note that assigning RELEASE_VSN after the launcher has computed
REL_VSN_DIR is not enough, and the hook has to re-exec.

Split across the two repos:

• Here (Castle): the marker's atomic one-shot consumption and commit/1 making
  the provisional version permanent. start_erl.data — what an ordinary restart
  reads — is only updated by make_permanent, which is precisely what provides the
  rollback property.
• Forecastle: the env.sh hook that reads the marker, sets RELEASE_VSN and
  re-execs the stock launcher — forecastle#10.

The two are atomic: the marker semantics and the hook that consumes them are
useless apart, so they land together. Once forecastle#6 removes the boot-time
configuration expansion, this marker check is the env hook's only remaining job on
a normal start.

Also depends on forecastle#4 for
a relup that actually contains restart_emulator.

[ausimian]

Warning

Substantially wrong — do not implement from this comment. Its central claim, that
new_start_erl.data/tmp_current/heart belong to restart_new_emulator alone, is
false: prepare_restart_new_emulator/7 has two call sites and plain restart_emulator
is the second, so both paths use all three. The comparison table below is wrong on those
rows, and "option 2" is void. Corrected in the replies below; the settled design is in
the issue body, §2. What survives here: the heart:set_cmd/1 blocker, the unchecked
start_prg default, and start_erl.data being written only by make_permanent.

---

OTP restart-transition mechanics, verified from source

Recording this on the issue because it was established during the 1.0.0 work and
otherwise lives nowhere: an implementer starting fresh would have to re-derive it,
and one of the findings invalidates the premise this issue was written on.

All line references are sasl-4.3/src/release_handler.erl from OTP 28.3. Everything
below marked verified was read in that source or measured; everything marked
unverified is explicitly not established.

There are two restart instructions and they behave completely differently

Verified, handle_call({install_release, ...}) at :1312-1337:

{restart_emulator, CurrentVsn, Descr} ->
    gen_server:reply(From, {ok, CurrentVsn, Descr}),
    init:reboot();
{restart_new_emulator, CurrentVsn, Descr} ->
    gen_server:reply(From, {continue_after_restart, CurrentVsn, Descr}),
    init:reboot();

	restart_emulator	restart_new_emulator
Where in the relup script	at the end	at the start
When it runs	after the upgrade is applied in the running VM	before anything is applied
Reply to install_release/1	{ok, Vsn, Descr} — indistinguishable from a hot upgrade	{continue_after_restart, Vsn, Descr}
Release status left behind	current	tmp_current
Writes new_start_erl.data	no	yes
Touches heart	no	yes

new_start_erl.data is only involved in restart_new_emulator. It is written by
prepare_restart_new_emulator/7 (:2428-2454), which is reached only from the
[restart_new_emulator|_Script] branch at :1704. Plain restart_emulator never
goes near it.

That matters because this issue's scope note, and the 1.0.0 plan, both assume the
provisional-version marker is the mechanism for restart upgrades generally. It is
not — it is the mechanism for emulator upgrades specifically.

start_erl.data is written only by make_permanent

Verified: the only writer is set_permanent_files/5 (:2082-2090), called from
do_make_permanent/2 (:1976) and one other permanent-making path (:2058).
Nothing in the install path touches it.

This is what gives the rollback property, and it is worth stating precisely
because it is the useful half: install restart-based B while A is permanent, and
start_erl.data still names A. If B fails before commit, the next start reads
start_erl.data, boots A, and the deployment is back where it was without anyone
intervening.

The blocker: OTP's own restart plumbing does not work in a Mix release

This is the finding that changes the design, and it is why this issue cannot be
implemented as written.

prepare_restart_new_emulator/6 (:2436-2454) does, in order:

1. sets the target current, then tmp_current, and writes RELEASES
2. writes releases/new_start_erl.data
3. resolves the start program, then on unix:

   case heart:set_cmd(StartP ++ " " ++ DataFile) of
       ok -> ok;
       Error -> throw({error, {'heart:set_cmd() error', Error}})
   end

So OTP's intended consumer of new_start_erl.data is heart, not the
launcher. It hands heart a reboot command of <start_prg> <path to new_start_erl.data>.

Two things break that on a Mix release:

• start_prg defaults to $ROOT/bin/start, and is not checked. init/1 at
  :1268-1272 yields {no_check, filename:join([Root, "bin", "start"])} unless
  {sasl, start_prg} is configured, and check_start_prg({_, StartPrg}, _)
  (:2459) returns it unexamined. A Mix release has bin/<name> and bin/castle;
  it has no bin/start. So the command handed to heart names a file that does not
  exist, and nothing notices at install time.

• Without heart running, heart:set_cmd/1 raises rather than returning an
  error. heart:set_cmd/1 (kernel/src/heart.erl:210-212) is
  ?MODULE ! {self(), set_cmd, Cmd}, wait(), and a send to an unregistered name
  is a badarg. Measured on OTP 28.3 with no -heart:

  heart pid: undefined
  set_cmd: {'EXIT',{badarg,[{erlang,send,[heart,{<0.10.0>,set_cmd,...}],...},
                            {heart,set_cmd,1,[{file,"heart.erl"},{line,211}]},
  

  The case above only handles a returned error, so the exit propagates. It is
  caught by catch do_install_release(...) at :1314 and lands in the
  {'EXIT', Reason} clause at :1338, which logs and replies {error, Reason}.

So a restart_new_emulator install on a Mix release with no heart fails, and
fails after step 1 and step 2 have already happened. It leaves
new_start_erl.data on disk and tmp_current persisted in RELEASES, then
reports an error. That is a partially-applied install with durable state, which is
the shape AGENTS.md requires everything to refuse ahead of — and it is
:release_handler doing it, inside the point of no return, so Castle cannot
prevent it by checking earlier. It can only refuse to ask for such an upgrade.

What this means for this issue and forecastle#10

The plan's design — env hook reads new_start_erl.data, sets RELEASE_VSN,
re-execs the stock launcher — is a reasonable way to consume the marker, and it
is still needed, because heart's route is unusable. But it is not sufficient on
its own: the install that should write the marker fails before the reboot ever
happens. Anything built on the consumption side alone will not be reachable.

The options, none of them yet chosen:

1. Set {sasl, start_prg} to something real that Forecastle ships, and require
   heart. Follows OTP's design, but makes heart mandatory and puts a second
   restart mechanism next to the supervisor the deployment already has.
2. Keep restart transitions to plain restart_emulator and never generate
   restart_new_emulator. No heart, no marker, no tmp_current — but the reply
   is {ok, ...} and then the node reboots, and since start_erl.data still names
   the old permanent version, the upgrade is silently lost on the way back up
   unless something sets the version for the next boot. That is precisely what the
   env hook could do, if it had a marker — which this path does not produce. So
   this option needs Castle to write its own marker rather than consume OTP's.
3. Refuse restart_new_emulator outright, with a message saying why, and
   support only the hot path until one of the above is built. Honest, and small.

Option 2 looks closest to the architecture — no heart, stock launcher, one marker
Castle owns end to end — but it means this issue is not "consume
new_start_erl.data", it is "decide what the marker is and who writes it".
forecastle#4 is where the two instructions get chosen between, so it should land
first and its choice constrains this.

Not established

• Whether a current (not tmp_current) release is reconciled on the next boot.
  init/1 (:1259-1265) reads RELEASES and transforms it; no status
  reconciliation was found, which suggests a plain restart_emulator reboot
  comes back with the launcher having booted the old release via
  start_erl.data while RELEASES still calls the new one current. That
  inconsistency is inferred from absence of code and must be measured before
  anything is built on it.
• What new_emulator_upgrade/2 does with tmp_current on the way back up. It is
  referenced from Castle's AGENTS.md for running/1's progress marker but was
  not read for this.
• Whether heart is usable at all under the external supervisors this issue
  targets, or whether it conflicts with them.

Incidental, and already acted on

init/1 matches consult(...) against {ok, [Term]} (:1260), so a RELEASES
file of two syntactically valid terms is rejected and the empty record synthesised.
That is why the refusal added in #25 asks for a file :release_handler accepts
rather than one that is readable or parses. Same source read; noted here because it
is the same function.

[ausimian]

Correction to the OTP restart-mechanics finding

The new source-reading comment found a real blocker, but one of its central conclusions is incorrect. I re-checked the exact OTP 28.3 source (lib/sasl/src/release_handler.erl), and plain restart_emulator does use the same provisional restart preparation path that writes new_start_erl.data, persists tmp_current, and talks to heart.

Plain restart_emulator does use the provisional marker path

In do_install_release/3, after the ordinary relup script is evaluated, the plain restart_emulator branch is:

restart_emulator ->
    mon_nodes(false),
    prepare_restart_new_emulator(StartPrg, RootDir,
                                 RelDir, Vsn, Release,
                                 NReleases, Masters),
    {restart_emulator, CurrentVsn, Descr};

So the table in the previous comment is wrong on three important points. Plain restart_emulator does not merely leave the release current; it calls prepare_restart_new_emulator/7. That helper:

NReleases1 = set_status(Vsn, current, Releases),
NReleases2 = set_status(Vsn, tmp_current, NReleases1),
write_releases(RelDir, NReleases2, Masters),
...
DataFile = write_new_start_erl(Data, RelDir, Masters),
...
heart:set_cmd(StartP ++ " " ++ DataFile)

Therefore, for plain restart_emulator OTP 28.3 does:

• persist the target as tmp_current;
• write releases/new_start_erl.data;
• call heart:set_cmd/1;
• return {restart_emulator, ...} to the outer handler only after that preparation succeeds.

The outer handle_call then replies {ok, ...} and calls init:reboot():

{restart_emulator, CurrentVsn, Descr} ->
    gen_server:reply(From, {ok, CurrentVsn, Descr}),
    init:reboot();

restart_new_emulator differs because OTP first constructs a hybrid temporary release containing the new ERTS/kernel/stdlib/SASL and old versions of the remaining applications, then continues the rest of the relup after that reboot. It is not distinguished by whether new_start_erl.data/tmp_current/heart are used; both restart paths use that machinery.

This also matches OTP's upgrade documentation: a handwritten relup containing only restart_emulator is the "brutal" alternative which restarts into the new versions wholesale, instead of the two-stage restart_new_emulator flow.

The valid new finding: heart:set_cmd/1 is a blocker without -heart

The previous comment is correct that this still breaks on a normal externally-supervised Mix release. heart:set_cmd/1 sends to the registered heart process, and the heart module explicitly documents that its functions fail with badarg when heart is not started.

So the actual flow today is:

release_handler.install_release(B)
  -> evaluate relup -> restart_emulator
  -> persist B as tmp_current
  -> write new_start_erl.data
  -> heart:set_cmd(...)
  -> no heart: badarg

and release_handler never reaches its normal init:reboot() branch.

One correction to the previous comment's failure-state description: the default install_release/1 error action is restart, so the caught exception is replied to the caller and followed by init:restart(). When SASL/release_handler reinitializes, transform_release/3 compares init:script_id() with the persisted tmp_current; if the node is still actually running the old release, the target is written back as unpacked. So the RELEASES state is not simply left permanently as tmp_current. However, new_start_erl.data has already been written and is not the thing that performs this reconciliation, so a stale marker remains something our hook must never consume blindly after a failed preparation.

Recommendation

I would keep the design of this issue — restart_emulator + OTP's tmp_current/new_start_erl.data semantics + a stock-Mix-launcher env hook — and solve the narrower heart:set_cmd handshake problem. I would not replace OTP's release-state machine with Castle-owned RELEASES manipulation.

The most promising external-supervisor adapter to prototype is a very small, temporary heart protocol shim:

1. Castle.install/1 detects that real heart is not running.

2. Immediately before invoking a restart transition, Castle starts a tightly-scoped process registered as heart.

3. It implements only the message used by heart:set_cmd/1:

   {caller, set_cmd, command}
       -> send caller {:heart, :ok}
   

4. release_handler then performs its normal, unmodified preparation:

   • persists tmp_current;
   • writes new_start_erl.data;
   • heart:set_cmd/1 receives :ok from the shim;
   • returns {restart_emulator, ...};
   • calls init:reboot().

5. The whole BEAM, including the shim, exits.

6. systemd/Docker/Kubernetes restarts the normal Mix launcher.

7. The Forecastle env hook consumes the one-shot new_start_erl.data, sets RELEASE_VSN, and re-execs the stock launcher so the provisional target boots.

8. If that target later crashes before commit, there is no new one-shot marker and ordinary start_erl.data still selects the previous permanent release.

This shim deliberately does not execute the command passed to heart:set_cmd; the external supervisor remains the only process-restart mechanism.

This does rely on the internal message protocol used by the public heart:set_cmd/1 API, so it must be treated as a compatibility adapter rather than assumed stable. Before accepting it, add integration tests across every supported OTP release (at least the current OTP 27/28/29 support matrix) which prove:

• plain [restart_emulator] reaches init:reboot() without real heart;
• RELEASES contains the expected provisional state before reboot;
• new_start_erl.data names the target release;
• the external-supervisor restart boots the target through the stock Mix launcher;
• a second restart before commit rolls back to the prior permanent version;
• make_permanent/1 makes subsequent ordinary starts select the target;
• a failed restart preparation cannot leave a stale marker that the env hook will later consume.

If the shim proves too brittle across OTP versions, the fallback is to run real heart with a deliberately inert restart command and let the external supervisor remain authoritative. I would prefer the shim if it tests cleanly, because requiring -heart introduces a second watchdog/restart mechanism solely to satisfy this one release-handler handshake.

Consequence for related issues

• forecastle#4: continue generating plain restart_emulator for full-restart transitions. There is still no reason for Castle's normal restart policy to use the two-stage restart_new_emulator path.
• forecastle#10: the new_start_erl.data consumption/re-exec hook is still the right launcher-side design. It just needs the restart preparation side here to make heart:set_cmd succeed first.
• This issue should gain an explicit implementation item for the heart handshake adapter; it should not be rewritten around a Castle-owned replacement marker unless the OTP-backed approach proves infeasible.

[ausimian]

Conceded: the correction above is right, and my comment's central claim was wrong

Re-checked all of it against OTP 28.3. The correction stands on every point I can verify, and the error was mine.

prepare_restart_new_emulator/7 has two call sites, :1719 and :1749, and plain restart_emulator is the second one:

restart_emulator ->
    mon_nodes(false),
    prepare_restart_new_emulator(StartPrg, RootDir,
                                 RelDir, Vsn, Release,
                                 NReleases, Masters),
    {restart_emulator, CurrentVsn, Descr};

So both restart transitions write new_start_erl.data, persist tmp_current, and call heart:set_cmd/1. My claim that the marker belongs to restart_new_emulator alone was false, and the three rows of my comparison table that rested on it — marker, status, heart — were wrong.

How I got it wrong is worth recording, because it is avoidable: I grepped for the function, the output was truncated by a head -25 at :1719, and I asserted "reached only from the [restart_new_emulator|_Script] branch" from that partial listing without enumerating the call sites. A conclusion of the form "only X" needs the complete set, and I did not have it.

Two further corrections, both verified:

• The default error action is restart — install_release(Vsn) -> call({install_release, Vsn, restart, []}) at :471-472. So the {'EXIT', Reason} clause at :1338-1348 replies {error, Reason} and then calls init:restart(). I described the failure as merely returning an error; the node also restarts.
• tmp_current is reconciled, not stranded — transform_release/3 at :2489-2492 maps any tmp_current release back to unpacked on a boot whose init:script_id() is not a tmp version, and rewrites RELEASES. My "partially-applied install with durable state" overstated it: that half self-heals.

And so my "option 2" is void. I proposed generating plain restart_emulator to avoid heart entirely. It does not avoid heart — it takes exactly the same preparation path. There is no restart transition through install_release/1 that does not call heart:set_cmd/1.

What survives

• The heart blocker itself, which is the finding that matters. start_prg defaults to {no_check, $ROOT/bin/start} (:1268-1272), is returned unexamined by check_start_prg/2 (:2459), and does not exist in a Mix release; and without heart running, heart:set_cmd/1 raises badarg rather than returning an error — measured on OTP 28.3, heart pid: undefined, {'EXIT',{badarg,[{erlang,send,[heart,...]}]}}.
• start_erl.data is written only by make_permanent (set_permanent_files/5 at :2082-2090, called from :1976 and :2058), which is what gives the rollback property.
• A stale new_start_erl.data outlives a failed preparation. The correction above makes this point and it is the right one to carry forward: transform_release/3 reconciles the release record and does not touch the marker file, so the hook must never consume one blindly. With ErrorAction = restart the node restarts immediately after the failure, so the hook meets that stale marker on the very next boot — which makes this the first thing the consumption side has to get right, not an edge case.

One detail for the shim, verified

The protocol is right and the shim is implementable: set_cmd/1 is ?MODULE ! {self(), set_cmd, Cmd}, wait(), and wait() is

wait() ->
    receive
        {?MODULE, Res} -> Res
    end.

So a process registered as heart replying {heart, ok} satisfies it.

But note there is no after clause. A shim that receives the message and then dies before replying leaves release_handler blocked in that receive forever — inside handle_call, so the gen_server is wedged and every subsequent release operation on the node blocks with it. That is a worse failure than the one being fixed. Whatever is registered under that name must reply unconditionally and before it can plausibly exit; and the test list should include a case where the shim is killed mid-handshake.

On the recommendation

I have no objection to the shim in principle, and agree it should not be replaced by a Castle-owned release-state machine. Worth weighing alongside it, since it uses only documented configuration and no internal protocol: {sasl, start_prg} pointing at a script Forecastle ships, with real -heart. Its cost is exactly the one the correction identifies — a second watchdog next to the supervisor the deployment already has — so the trade is "internal protocol dependency" against "an extra restart mechanism", and the answer depends on how the supported deployments are actually supervised. The -heart route has the advantage that wait()'s missing timeout stops being our problem.

Either way the integration tests listed above are the right gate, and the OTP 27/28/29 matrix is the right scope: the shim depends on an internal message shape, and prepare_restart_new_emulator/7's two call sites are exactly the kind of thing that can be refactored between releases.

[ausimian]

Agreed design: real defanged heart, inert bin/start, systemd as sole restart authority

This supersedes the heart-protocol shim recommended earlier in this thread. Two of us converged on it independently; the HEART_NO_KILL subtlety below came from the review of it and is the part that makes the shape non-obvious.

The design

release_handler ──► heart:set_cmd(...) ──► real OTP heart acknowledges
                                                   │
                                            init:reboot() ──► A exits
                                                   │
                                          systemd observes, ExecStart
                                                   │
                                        stock Mix launcher ──► env.sh hook
                                                   │
                                    reads new_start_erl.data ──► boots B

Component	Job
-heart (via ELIXIR_ERL_OPTIONS in env.sh)	make heart:set_cmd/1 return ok instead of raising. Nothing else.
HEART_NO_KILL=TRUE	heart never kills a node that misses heartbeats
HEART_COMMAND unset	nothing happens on an unexpected death when no temporary command is set
$ROOT/bin/start, inert	neutralise the temporary command release_handler installs
systemd	the only restart authority
env.sh hook	consume the marker, set RELEASE_VSN, exec the stock launcher

All documented OTP interfaces. No dependency on the internal {self(), set_cmd, Cmd} / {heart, ok} protocol, and no exposure to heart:wait/0's missing after clause — a fake heart that died mid-handshake would have wedged release_handler's gen_server permanently, which was the shim's worst failure mode.

Mix ships this idiom commented out in its own generated env.sh (release.init.ex:66-75), though it sets HEART_COMMAND to the launcher and gates on daemon*. We deliberately do neither.

HEART_NO_KILL is safe only because the command is inert — measured

This is the important finding and it inverts the obvious reading. HEART_NO_KILL suppresses the kill, not the command: heart still runs system(cmd) on a heartbeat timeout. So with a bin/start that actually launched B, a timeout occurring after set_cmd would start B while A was still running, precisely because we told heart not to kill A.

Verified on OTP 28.3 rather than inferred. HEART_NO_KILL=TRUE, HEART_BEAT_TIMEOUT=11, the beam SIGSTOPed so heartbeats stop while the process stays alive:

heart: heart-beat time-out, no activity for 15 seconds
heart: Executed ".../heart-probe.sh timeout-path" -> 0. Terminating.

$ ps -o pid=,stat= -p 58737
58737 T          ← A still alive, suspended

heart declined to kill A and ran the command anyway. The HEART_NO_KILL doc says as much once read carefully — "useful if the command executed by heart takes care of this" — i.e. it assumes the command handles the old VM.

So the inert bin/start is not belt-and-braces. It is what makes HEART_NO_KILL safe instead of actively dangerous, and the two settings only make sense together.

Also verified, since it was previously only inferred: heart does execute the temporary command on init:reboot() with HEART_COMMAND unset.

heart: Erlang has closed.
heart: Executed ".../heart-probe.sh reboot-path" -> 0. Terminating.

with the data file arriving as $1. So bin/start is genuinely invoked on the normal path — it is load-bearing, not decorative, and it must exist and exit 0.

Incidental: HEART_BEAT_TIMEOUT must satisfy 10 < X <= 65535. A value of exactly 10 is silently ignored and the 60s default used, which cost one inconclusive test.

bin/start

#!/bin/sh
# Castle releases are externally supervised: the supervisor owns restarting.
#
# release_handler composes "$ROOT/bin/start <data file>" and installs it as
# heart's temporary reboot command for any restart transition. heart really does
# execute it - on init:reboot(), and also on a heart-beat timeout, where
# HEART_NO_KILL=TRUE means the old VM is still alive. Starting anything here
# would therefore risk two live nodes. Doing nothing is the whole point.
#
# The provisional version is selected by the env.sh fragment on the next start.
exit 0

Left at the default start_prg path rather than configured through {sasl, start_prg}: the default is {no_check, $ROOT/bin/start} (:1268-1272) and needs no configuration, whereas setting it explicitly is {do_check, _} and would mean injecting :sasl config, which Forecastle no longer does by design.

Stale marker handling — one correction and one residual gap

The correlation-marker proposal is better than what I suggested before (cross-checking the marker's version against RELEASES from shell, which is genuinely unpleasant). Castle arms castle-restart-pending before a restart install and clears it on failure; the hook requires both files, so OTP's marker stays OTP-owned and a stale one cannot become a boot instruction on its own.

Correction to my earlier claim in this thread. I said ErrorAction = restart means a failed install "restarts the node straight into a stale marker". That is wrong: init:restart/0 restarts the runtime inside the same OS process, so env.sh does not re-run. Measured — the OS pid is unchanged across the restart, and the -eval boot argument is simply re-evaluated:

os pid before: 59634
os pid before: 59634      ← same process, restarted runtime

So the stale marker is not consumed immediately. It persists on disk until the next genuine OS-level restart, which makes the window wider in time but less immediate than I described. The defence is still required; the urgency was overstated.

Residual gap worth stating rather than papering over. A hard kill (SIGKILL, power loss) between arming castle-restart-pending and completing the install leaves both markers, and the hook would boot B. The damage is bounded: B is unpacked with its configuration materialised, so this is a cold boot of B rather than a hot upgrade, and start_erl.data still names A, so the next restart returns to A. Bounded is not the same as closed, and it deserves a test rather than an assumption.

Ranking

1. Real defanged heart + inert bin/start + systemd sole authority + env-hook selection — no undocumented interfaces, and heart is operationally harmless in every path measured above.
2. heart-protocol shim + systemd + env hook — works, but depends on an internal message shape and carries the gen_server wedge risk.
3. Avoid: real heart + a bin/start that actually starts B + systemd Restart=. Two independent authorities starting the service. Sharing one implementation between the two entry points does not help — the problem is not duplicated code, it is duplicated authority.

Still to prototype

• The full path end to end under systemd, Docker restart policies and runit.
• -heart in foreground mode: Mix recommends heart only for daemon* and does not say why. With HEART_NO_KILL and an inert command the stated hazards look neutralised, but that should be demonstrated rather than argued.
• The Castle-armed marker across a hard kill, per the gap above.
• The OTP 27/28/29 matrix. The measurements above are OTP 28.3 only, and heart's behaviour is in a C port program rather than in heart.erl, so it is not visible to source review.

[ausimian]

Coupling: forecastle#4's auto refusal has to be flipped as part of this issue

Recording so it is not orphaned. forecastle#4 ships upgrade-strategy selection with auto refusing any transition that requires an emulator restart, because nothing can install one until this issue lands: release_handler calls heart:set_cmd/1 during restart preparation and it raises badarg with no heart process, so the install fails before init:reboot(). Emitting such a relup by default would have replaced previously-working hot behaviour with a plan that fails at install time.

That refusal is one named predicate with a comment, not a version sniff, and it is meant to be temporary. Completing this issue includes flipping it so auto announces the restart and proceeds — the announcement is already written and tested in forecastle#4, it is simply gated.

Suggest adding to this issue's acceptance criteria: auto in mix forecastle.relup no longer refuses restart transitions, and the restart-strategy end-to-end test drives one through to a committed upgrade. A temporary gate with nothing pointing back at it is how it becomes permanent by accident.

Note the sequencing this preserves: the plan had forecastle#4 landing before this issue, and that still holds — the refusal is what keeps it honest rather than forcing a reshuffle.