diff --git a/AGENTS.md b/AGENTS.md index c1a9662..d962072 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -12,9 +12,9 @@ build-time dependency — Forecastle is not intended to be taken directly. - **`pre_assemble/1`** — checks and stages any `relup`, and adds a `:preboot` boot script that starts `:sasl`, `:compiler`, `:elixir` and `:castle`. -- **`post_assemble/1`** — writes `bin/castle`, appends the Castle hook to the - generated `env.sh`, and copies the `.rel` file and any staged `relup` into the - release. +- **`post_assemble/1`** — writes `bin/castle` and `bin/start`, appends the Castle + hook to the generated `env.sh`, and copies the `.rel` file and any staged + `relup` into the release. **Nothing here touches configuration, and nothing new may.** Forecastle used to intercept all of it: `:runtime_config_path` set to `false`, a substitute @@ -176,23 +176,390 @@ around it composes instead: prevent. An answer is only good for the call that acts on it, which is why the check is `Castle.Commands.ensure_upgradable/2` and belongs to the operations. Do not reintroduce one here, in any shape, and do not add one to make a refusal - arrive sooner: `Castle.install/1` materialises the target's configuration - before it refuses, so a refused install starts a peer first. That is harmless — - the peer writes only into the target's version directory, never to the running - system or to any release record, and it is idempotent. + arrive sooner. There is nothing left for it to buy: `Castle.install/1` used to + materialise the target's configuration ahead of the operation, so a refused + install started a peer first, and that composition has since been folded into + `Castle.Commands.install/5` — behind the record check and behind the + pending-marker refusal — precisely because materialising ends in a rename onto + the target's `sys.config` and so is not the harmless idempotent work it was + described as here. A refused install now configures nothing. `bin/castle upgradable` asks the question on its own, for an operator who wants to know where a system stands without staging anything. It is a plain command like `releases`, gates nothing, and says nothing when the answer is yes. - The `env.sh` fragment expands no configuration and applies none of the - launcher's defaults; on every start after the first it does nothing at all. It - is appended, so a project's own `rel/env.sh.eex` survives and runs first, and - it is where [#10](https://github.com/ausimian/forecastle/issues/10) will - consume the provisional restart marker. -- Version selection needs no code at all *after commit*: `release_handler` writes - the committed version to `releases/start_erl.data`, which is where the standard - launcher already reads `RELEASE_VSN` from. It does not hold for a transition - that restarts the emulator, which is #10. + launcher's defaults. It is appended, so a project's own `rel/env.sh.eex` + survives and runs first, and everything in it is gated on + `$RELEASE_COMMAND` naming a command that starts the system — so an `eval`, an + `rpc` or a `remote` reaches none of it. That gate is not tidiness: `bin/castle` + drives every command through `rpc`, so a fragment that ran for those would + consume the provisional marker while an install was still waiting for the + reboot. Three things are in it: the `RELEASES` bootstrap above, and the two + [#10](https://github.com/ausimian/forecastle/issues/10) added, below. +- **The provisional version, after a transition that restarted the emulator.** + `release_handler` writes the installed version to + `releases/new_start_erl.data` and deliberately leaves + `releases/start_erl.data` — where the stock launcher reads `RELEASE_VSN` from — + naming the version that is still permanent. That is the rollback property, and + only `make_permanent/1` ever writes that file. So version selection needs no + code at all *after commit*, and needs this before it. + + **Two markers are required, and both are consumed.** OTP's is written *before* + the reboot and nothing removes it, so on its own it is not evidence that a + reboot was asked for: a preparation that failed after writing it leaves a file + naming a version that was never installed. Castle arms + `releases/castle-restart-pending` with the same version immediately before + asking for such an install and clears it if the install failed, so the pair is + what says a reboot happened — and they have to agree on the version. See + castle#14 for the arming side. + + **Agreeing on a version is not enough, and making the pair an install + *attempt's* is Castle's end of the protocol rather than this one's.** Castle + clears any `new_start_erl.data` an earlier attempt left *before* it arms a + marker, publishes the marker exclusively so that two attempts cannot share it, + and records the attempt on a second line so that no install removes another's. + What that buys this fragment is that OTP's file being there means this + attempt's own preparation wrote it. **Only the first line of the marker is read + here** — it is the version, and everything after it is Castle's bookkeeping. Do + not teach this script to parse the rest. + + **What is atomic is the claim, not the pair.** The `mv` is one operation, so + whichever start wins it is the only one that can act on the pair. OTP's marker + is then read and removed in separate steps, and no POSIX operation moves two + files together — so the two are *not* consumed as a unit, and an earlier + version of this note, Castle's `AGENTS.md` and this release's own notes all + said they were. What makes it safe + is the order: the pending marker goes first, so a start killed anywhere after + the rename leaves no marker behind, and the next start reads `start_erl.data` + and boots the version that was permanent. The selection is lost, never + duplicated, and never applied to a version nothing installed — which is the + direction this whole design fails in. `Forecastle.EnvScriptTest` covers that + interruption by planting what it leaves behind. + + Both are gone before the launcher is re-exec'd, which is what stops the second + pass recurring. The claim is named per process — `castle-restart-consumed.$$` — + so two starts racing for the marker cannot read each other's, and one left by a + start that died between the rename and the read is replaced rather than + trusted. + + **It re-execs `$RELEASE_ROOT/bin/` rather than assigning `RELEASE_VSN`.** + By the time the launcher sources `env.sh` it has already computed + `REL_VSN_DIR`, and everything after that hangs off it — `RELEASE_VM_ARGS`, + `RELEASE_REMOTE_VM_ARGS`, `RELEASE_SYS_CONFIG`, the boot script, the `elixir` + launcher itself. Assigning the version in place boots the old version's + everything under the new version's name. A sourced file still sees the + caller's positional parameters, so `exec … "$@"` loses nothing about how the + launcher was invoked, and the launcher path is interpolated at assembly time + the way `bin/castle` interpolates it rather than taken from `$0`. + + The version is validated before any of that: non-empty, no path separator, and + a version directory that has an `env.sh` and a `start.boot` in it — the first + because the next pass sources it and the second because it is what boots. A + pair that does not settle warns and boots the permanent version; no pair at all + says nothing and does nothing. + + **This block comes first in the fragment, ahead of everything else it does, and + that ordering is load bearing. Do not move it.** The re-exec means the fragment + is read twice, so anything decided before the selection is decided about the + version being *replaced* — and exported to the pass that boots. It shipped the + other way round, with `heart` ahead of the selection, and that is exactly what + went wrong: pass 1 probed the permanent version's `vm.args`, found no `-heart` + and exported `ELIXIR_ERL_OPTIONS=-heart`; pass 2 probed the target's, measured + the inherited flag beside the target's own, and so declined to append a third — + but nothing removes what pass 1 exported, so a target carrying its own `-heart` + booted with two and hung. Measuring on both passes and appending on neither is + not a fix for that; there being only one pass that decides anything is. + The same ordering is what makes the probe's emulator unambiguous, since the + release whose `elixir` names it is then the selected one. + + A consequence worth stating: a target release whose `env.sh` is not Forecastle's + gets no `heart` configuration at all, because the pass that would have set it + execs away first. That is right — such a release has no Castle upgrade path into + it in the first place — but it is a behaviour change from the order this had, and + it is the reason not to "helpfully" export anything else across the re-exec. + + `bin/ version` still reports the version in `start_erl.data`, so during a + provisional boot it names the *previous* one while the node runs the installed + one. That is right rather than a defect: it prints "the version to be booted", + which for an uncommitted release is the rollback target. Ask the node if you + want to know what is running. +- **`heart`, run deliberately defanged.** `release_handler` calls + `heart:set_cmd/1` while preparing *any* transition that restarts the emulator — + `prepare_restart_new_emulator/7`, which both restart instructions go through — + and with no `heart` process that raises `badarg`, so the install failed before + `init:reboot()` on exactly the externally supervised release this library is + for. Running the real heart satisfies the handshake through documented + interfaces only. The alternative, a Castle process registered as `heart` + answering the internal `{Caller, set_cmd, Cmd}` protocol, was refused: + `heart:wait/0` is a `receive` with no `after`, so a shim that took the message + and died would wedge `release_handler`'s gen_server for the life of the node. + + heart must then do nothing, because the supervisor is the only thing that + starts this release and two authorities starting one service is worse than the + problem being solved. `HEART_COMMAND` is not set. `HEART_NO_KILL=TRUE`. + `HEART_BEAT_TIMEOUT` is raised to heart's documented maximum, 65535. And + `$ROOT/bin/start` — the next entry — does nothing. + + **`HEART_NO_KILL` and the inert `bin/start` are a mandatory pair.** Measured on + OTP 28.3, with the beam `SIGSTOP`ed so heartbeats stop while the process stays + alive: heart declines to kill it and **runs the command anyway**. So a + `bin/start` that really started the release would start it beside a live node, + precisely because heart was told not to kill that node. The documentation says + as much once read carefully — *"useful if the command executed by heart takes + care of this"*. + + **`HEART_BEAT_TIMEOUT` is not decoration either, and it is the one place + castle#14's agreed design was incomplete.** `HEART_NO_KILL` does not make a + heart-beat time-out harmless. The port program terminates once it has run the command; the Erlang + `heart` process then exits `{port_terminated, …}`; heart is a *kernel* process, + linked to `init` by `new_kernelpid/3`; and `init:terminate/3` halts the node + when a kernel pid dies. Measured, again on OTP 28.3: the node survived while + suspended and died the moment it resumed, with + `Kernel pid terminated (heart) ({port_terminated, …})` and a crash dump. So + `-heart` at the 60s default introduces a way for a stalled node to be killed + that a deployment without it did not have, and heart is here to satisfy a + handshake rather than to watch anything. The maximum timeout is what keeps that + unreachable. + + **All three are assigned, and `HEART_COMMAND` is unset — they used to be + defaulted, and that was a hole.** `HEART_NO_KILL="${HEART_NO_KILL:-TRUE}"` and + friends left a deployment that already had these in its environment with + active watchdog behaviour, while this file, the README and the release notes + all said it had none: an inherited `HEART_COMMAND` is a second restart + authority beside the supervisor, `HEART_NO_KILL=FALSE` restores the kill, and a + shorter `HEART_BEAT_TIMEOUT` restores the death-by-time-out that the paragraph + above exists to keep unreachable. There is no opting out of this while the hook + is in use. heart is here to make `heart:set_cmd/1` return `ok` and for nothing + else. + + **It overrides rather than refuses, and says so.** An operator who set one of + these has a configuration conflict, not an emergency, and a failed boot is a + worse answer than the conflict — but a setting that silently stops taking + effect is worse than either, so each value actually being displaced is named on + standard error along with what replaced it and why. A deployment that set none + of them — every ordinary one — says nothing at all, which is why the checks are + on the *value* rather than on the variable being set: `HEART_NO_KILL=TRUE` + already agrees, and an empty `HEART_COMMAND` displaces nothing. + + **A variable set to nothing is a value, and `${VAR:-default}` cannot see + that.** It treats set-and-empty as absent, which is what these were written + with, so `HEART_NO_KILL=` and `HEART_BEAT_TIMEOUT=` were displaced in silence — + neither is the value that gets assigned, so both were being overridden while + the promise above is to name every value that stops taking effect. The two + assigned ones therefore use `${VAR-default}`, without the colon: unset takes + the default and says nothing, set-and-empty compares unequal and is reported as + `[]`. An empty `HEART_COMMAND` stays silent, and that is a *different* rule + rather than the same one — unsetting a variable that was already empty changes + nothing heart can read, so there is nothing to tell an operator. Do not + "consistently" collapse the two back together. + + Note for whoever writes the next test here: an empty value in `System.cmd/3`'s + `:env` **removes** the variable, the same as `nil` does, so the case that named + this state could not be arranged that way and the test that claimed to was + really testing the unset one. `Forecastle.EnvScriptTest` prefixes shell + assignments to the command for the empty ones instead. + + Measured, and worth having written down because it makes the `HEART_COMMAND` + case sharper than "an unexpected death would start something": heart runs the + command on an **orderly** halt as well. On OTP 28.4 with `HEART_COMMAND` set, + a clean `halt()` produced *"Erlang has closed. Executed … -> 0. + Terminating."* So an inherited `HEART_COMMAND` turns every `bin/ stop` + into a restart. The same measurement is what makes the e2e assertion possible: + `heart:get_cmd/0` reports the port program's command, and the port program + takes its initial value from the environment, so an inherited command comes + back out of `heart:get_cmd/0` and the existing "has no command of its own" test + becomes the discriminator once the suite starts with one set. + + `-heart` is appended to `ELIXIR_ERL_OPTIONS` **only if the emulator is not going + to be given one anyway**. That guard is load bearing, not hygiene: two `-heart` + flags make `init:get_argument(heart)` answer `{ok, [[], []]}`, which + `heart:check_start_heart/0` has no clause for — a `case_clause` at + `heart.erl:348` — and the boot hangs with nothing printed; measured. What the + guard is for is a deployment supplying its own flag — in its `vm.args` or in one + of the variables `erlexec` reads. It used to have to cover the fragment's own + flag as well, because this block ran ahead of the provisional selection and so on + both passes of a re-exec, and it could not: see that block for what that cost. + This block now runs once per boot, on the pass that boots. + + **The guard asks `erlexec` what the argument vector came out as. It does not + read the environment and decide, and three versions of it that did were each + refuted by a narrower counterexample than the last.** Write the series down, + because the lesson is the series rather than any one entry in it: + + 1. `case " $ELIXIR_ERL_OPTIONS " in *" -heart "*` — a match bounded by literal + spaces. Mix's generated `elixir` expands that variable **unquoted** (`set -- + … $ELIXIR_ERL_OPTIONS $ERL "$@"`), so what reaches the emulator is its fields + under `$IFS`: space, tab and newline. `-heart-noshell` carries a live + flag this could not see. + 2. Two loops splitting the fields of `ELIXIR_ERL_OPTIONS`, `ERL_AFLAGS`, + `ERL_FLAGS` and `ERL_ZFLAGS`. `erlexec` prepends the second and appends the + third and fourth to the command line it builds, so all four reach + `init:get_argument(heart)` — but it also applies **shell-style quoting and + backslash escaping** to the three it reads itself, so `ERL_AFLAGS="'-heart'"`, + `ERL_FLAGS='"-heart"'` and `ERL_ZFLAGS='-he\art'` all arrive as `-heart` + while carrying no `-heart` substring at all. Measured. + 3. The same, plus a literal scan of `vm.args` — the launcher passes it as + `-args_file`, so a project's own `rel/vm.args.eex` is a fifth source. Wrong in + three further ways at once: the escapes above apply to the file too, + `erlexec` treats `#` as a comment there *except inside a quoted value*, and it + follows a nested `-args_file` that no shell can see through. + + **Every one of those shipped with a unit test that agreed with it**, because the + test mirrored the same model — see `Forecastle.EnvScriptTest`'s note on its own + counter. There was no reason to think the series had ended, so it was not + continued. + + `erl -emu_args_exit` prints the argument vector `erlexec` assembled, one + argument per line, and **exits without starting a VM**. So the fragment runs + that with the start's own environment, its own unquoted `$ELIXIR_ERL_OPTIONS` and + its own `-args_file`, in the order Mix's `elixir` would put them, and looks for a + line that is exactly `-heart`. That is not an approximation of the union of the + sources — it *is* the union, including `ERL_OTP_FLAGS`, which the fragment + can neither name nor needs to. + + Six properties of that, each load bearing: + + - **The emulator asked is the one the launcher will run, and the fragment is told + which that is by the file that decides it.** The launcher execs + `$REL_VSN_DIR/elixir`, and the only thing in that script choosing an emulator + is its `ERTS_BIN`: `mix release` rewrites Mix's own `ERTS_BIN="$ERTS_BIN"` into + `ERTS_BIN="$SCRIPT_PATH"/../../erts-/bin/` when the release brought an + ERTS, and leaves it alone when it did not — in which case `ERTS_BIN` is empty + and `erl` comes off `PATH`. That script's `SCRIPT_PATH` is the directory it is + in, which is `REL_VSN_DIR`, so the fragment reads the assignment out of the + same file the exec will read it out of and resolves it against `REL_VSN_DIR`. + Anything that does not come out executable falls back to `erl`, which is what + an un-rewritten assignment means anyway. **This is the resolution, not a model + of it, and that is the point** — the same rule as asking `erlexec` rather than + parsing the environment. + + It used to expand a glob over the release root's `erts-*` directories and keep + the last executable it found. That is wrong the moment the root holds more than + one of them, which is what unpacking an ERTS-changing release leaves behind, + and the last one a glob yields is the *lexicographically* last — not even the + newest, since `erts-9.9` sorts after `erts-16.2`. `ERL_OTP_FLAGS` is + named for the OTP version of whichever binary answers, and an args file may be + written for one emulator and not another, so asking the wrong one is a question + about a different deployment rather than a near miss. `Forecastle.AssemblyTest` + refutes the glob in the shipped file, and the fragment's own prose deliberately + does not repeat its spelling so that the refutation is about the code. + + - **It cannot hang, so it needs no timeout.** A deployment already carrying two + `-heart` flags — the boot this guard exists to prevent — makes it print two + lines and exit 0. + - **`-emu_args_exit` is undocumented** — present in the `erlexec` binary in OTP + 27, 28 and 29, absent from the usage string and the documentation. Relying on + it is acceptable *because it fails safe*, and the fragment makes that true + rather than assuming it: the probe carries `-boot /nonexistent/…`, so an + `erlexec` that passed the flag through would fail on a boot file that cannot + exist instead of starting a node, and the output is believed only if it + contains a `-root` line, which every vector `erlexec` builds has. Both + failures land in the same branch as an unreadable `-args_file`: add nothing, + and say so on standard error. + - **The probe always runs on a start, and there is deliberately no gate in front + of it.** There was one: a `case` over the four flag variables and a `read` loop + over the args file, looking for `heart`, `args_file`, a quote, a backslash or a + glob character, so that an ordinary deployment forked nothing. **It was removed + on purpose and it must not come back.** It was the last thing in the fragment + reasoning about the *text* of those values rather than measuring them, and its + soundness rested on a claim about `build_args_from_string()` in `erlexec.c` + removing nothing but quotes and backslashes — a claim about a C state machine, + of exactly the kind that was wrong three times above. And it left a hole it + could not close, which is the next point. + + What every start now pays is **one `fork`+`exec` of a C program that exits + without booting an emulator** — about 11ms, measured — and it is paid once per + node start, because the whole fragment is gated on `$RELEASE_COMMAND` naming + `start`, `start_iex`, `daemon` or `daemon_iex`. An `eval` or an `rpc` reaches + none of it, which is what keeps the fork off `bin/castle`'s path. + - **All six sources are covered, `ERL_OTP_FLAGS` included.** That is the + hole the gate could not close: POSIX sh cannot enumerate environment variable + *names*, and `env` and `export -p` both need a fork of their own to look for a + variable whose name carries the emulator's OTP major — one that `erlexec`'s own + source calls "intentionally undocumented and intended for OTP internal use + only". So a deployment setting only that variable never tripped the gate, + received a second `-heart`, and hung the boot having printed nothing. The probe + sees it because `erlexec` reads it, and `Forecastle.EnvScriptTest` pins that + with a case deriving the variable's name from the running OTP major. + - **The one condition left is a fact about the filesystem, not a judgement about + contents.** `-args_file` is passed only when the path exists, because `erlexec` + refuses an args file it cannot open and exits non-zero — which would make every + start with no readable args file report the measurement as impossible and + decline to add the flag. **It is defensive rather than load-bearing**, and an + earlier note here overstated it as covering "a release with no `vm.args`, a + supported shape". A stock build does not produce that: `mix release` always + renders `rel/vm.args.eex` into `releases//vm.args`, so the launcher's own + `${RELEASE_VM_ARGS:-…}` default always resolves to a file that exists. What is + reachable is a deployment exporting `RELEASE_VM_ARGS` to a missing path, or a + hand-deleted `vm.args` — both starts the launcher fails on moments later + anyway. So it costs nothing and spares a doomed start a confusing warning, + rather than holding up the common case. + Every variable `erlexec` reads still reaches the probe, `ERL_OTP_FLAGS` + among them, so nothing goes unmeasured; only a file that is not there goes + unmentioned. A path that exists and cannot be read is still handed over and + still reported, because that one is worth knowing about. + + The coverage follows the same rule: **do not validate any of this with a shell + counter**, whatever shape it takes, because what such a counter measures is + somebody's model of `erlexec`. `Forecastle.EnvScriptTest` counts by asking a real + `erlexec` with `-emu_args_exit`, over a matrix of quoted, double-quoted, + backslash-escaped and whitespace-separated values in each variable and in + `vm.args`, plus `ERL_OTP_FLAGS` under its real name, a nested args file, a + commented flag, and an emulator that does not know the flag. It also records + every invocation of the release's own `erl`, which is the only way to see whether + the fragment probed and what it probed with — so an ordinary start asserts one + invocation rather than none, and a missing `vm.args` asserts one carrying no + `-args_file`. Cases asserting *no* invocation of it mean one of three things, + and the distinction matters: a `$RELEASE_COMMAND` that does not start the + system probed nothing at all, while the `include_erts: false` shape and a + version directory with no `elixir` probed `PATH`'s emulator instead — which is + what the launcher would run. A second emulator installed under another `erts-*` + writes to its own log, which is the only way to see *which* one answered; that + is what pins the resolution against the glob it replaced, and it is also how a + provisional start is held to asking the target's. + + `Forecastle.RestartUpgradeTest` is the real boot, and the fixture's own + `rel/vm.args.eex` carries the flag spelled **`-he\art`**. That is the point of + it: `erlexec` unescapes it and the booted node answers + `init:get_argument(heart) == {ok, [[]]}`, so the only way for the fragment to see + it is to ask. `ERL_AFLAGS` carries `-env CASTLE_TAB_PROBE tabbed` behind tabs — + `erlexec` splits that variable on tabs as readily as on spaces, measured — so the + node answering with the value says the tabs really were separators, and the value + carrying no `heart` of its own says that asking about an unremarkable flag + variable does not manufacture a second flag. The suite additionally asserts + `ELIXIR_ERL_OPTIONS` is **unset** in the running node, which is what says the + fragment added nothing beside a flag no reading of that file could have found. + + `Forecastle.Fixture` scrubs all four variables, `ERL_OTP_FLAGS`, and both + `RELEASE_*VM_ARGS`. The three `ERL_*FLAGS` matter more than `ELIXIR_ERL_OPTIONS` + does — they are read by `erlexec` rather than by anything Mix generates — and + `ERL_OTP_FLAGS` is a fifth of that kind, appended at run time because its + name carries the emulator's OTP major and cannot be written into a module + attribute. The `RELEASE_*VM_ARGS` pair matters differently again: those do not + add a flag, they point the launcher at another project's args file entirely. + + **What holds this to the environment rather than to the text of the script is + `Forecastle.EnvScriptTest`**, which sources the fragment in a release-shaped + directory and reports what it left behind. It exists because the assembly-level + test that used to be the whole of the coverage asserted the `${VAR:-default}` + expressions were *present*, and so passed against exactly the defect above. Any + future claim about the effective heart configuration belongs there; the + assembly suite keeps only the claim that the configuration is in the release's + `env.sh` and that it assigns. + + Incidental, measured: heart prints `heart_beat_kill_pid = ` on every + start, and `heart_beat_timeout = ` when the variable is set. Two lines of + noise on the release's own output, not a fault. +- **`$ROOT/bin/start`, inert.** `install_start_program/1` writes it, and + `priv/start.sh.eex` is the whole of it: a comment and `exit 0`. It is at the + *default* `start_prg` path deliberately — `init/1` yields + `{no_check, filename:join([Root, "bin", "start"])}` when `{sasl, start_prg}` is + unset, and `check_start_prg/2` returns that unexamined, so it needs no + configuration; naming a path of our own means `{do_check, _}` and injecting + `:sasl` configuration into the release, which is the interception #6 removed. + Measured, since it was previously only inferred: heart *does* run this command + on `init:reboot()` with `HEART_COMMAND` unset, receiving the data file as `$1`. + So it has to exist and exit 0; it is load bearing rather than decorative. ## Relup generation and upgrade strategy @@ -243,8 +610,8 @@ chosen it. That is the gap `auto` exists to close, and it closes it twice: `restart_new_emulator` itself, and then `auto` refuses the relup rather than writing it, because the two-stage transition is not supported at all. `restart_emulator` from an appup is the same transition `auto` would have - chosen for itself, so it is settled the same way `auto`'s own choice is (see - the temporary refusal below). `--hot` refuses both. + chosen for itself, so it is settled the same way `auto`'s own choice is — named + in the one announcement below. `--hot` refuses both. **A restart relup is written directly, never through `:systools`.** That is the only way to be certain which instruction lands: `make_relup/4`'s own @@ -314,112 +681,81 @@ compare.** Verified against OTP 28.3, `sasl-4.3`: tests"), so **call it**; do not reimplement the matching, and do not compare strings, or `auto` and the `:systools` run a moment later will disagree. -**`auto` refuses a restart edge unconditionally, and that is temporary.** Castle -cannot *complete* a restart transition yet: `heart:set_cmd/1` raises `badarg` -with no `heart` process, so the install fails before the reboot, and the reboot -would return on the old permanent version anyway. `auto` is the no-switch -default, so emitting a restart edge from it means a routine invocation producing -a relup that cannot be installed — worse than refusing and saying why. `--hot` -and `--restart` are unaffected: both are explicit requests. - -**There is no flag, predicate or environment variable that turns the refusal into -an announcement, and adding one would be wrong.** An earlier revision of this -branch had one — `restart_transitions_installable?/0`, a private function -returning a literal `false`, consulted by `refuse_chosen_restarts!/1` and -`settle_restarts!/2` — and it is gone. Elixir 1.20's type inference proves the -`true` branch of both call sites dead and `mix compile --warnings-as-errors` -fails on them, which is what took CI red on all six 1.20 cells. Every way of -hiding that from inference is worse than deleting the branches: reading -application env or a switch makes the gate user-flippable, and a user who flips -it gets precisely the uninstallable relup the gate exists to prevent; a module -attribute folds to the same literal; there is no suppression pragma. The -warnings were also correct — nothing reached those branches, and nothing tested -them. Do not reintroduce a gate. - -**What [castle#14](https://github.com/ausimian/castle/issues/14) (with -[#10](https://github.com/ausimian/forecastle/issues/10)) has to add.** Not a -flip: the announcement does not exist, and the code that decides *when* to speak -has to move. In `lib/mix/tasks/forecastle.relup.ex`: - -- **`refuse_chosen_restarts!/1` goes away, and with it the pre-generation - refusal.** It sits before generation only because, while the outcome is a - refusal whatever the hot remainder turns out to be, a `:systools` error from - that remainder would otherwise stand in front of the real reason. Once the run - can proceed there is nothing to settle early — and settling early is what made - a run contradict itself once already (see *One verdict per invocation* below). -- **`settle_restarts!/2` becomes the only verdict, and its second clause - announces instead of refusing.** That clause already receives both lists: - `chosen`, the edges classification decided must restart, and `found`, the - one-stage restarts `appup_restarts!/1` returned from the generated relup. Both - are the same kind of transition, so they are named together, in one - `Mix.shell().info/1`, built the way `refuse_restarts!/2` builds its message — - `describe_causes/3` with a phrase of its own, over the same - `describe_edges/1` and `describe_restarts/1`. `describe_causes/3` takes that - phrase as an argument for exactly this reason; it has a single caller today. - Say that each restart is one `restart_emulator`, so `install_release/1` - replies `{ok, Vsn, Descr}` rather than `{continue_after_restart, Vsn, Descr}` - and the emulator then reboots, and name `--hot` and `--restart` as the two - ways to choose otherwise. -- **The `([], [])` clause does not move and does not gain a sibling.** It is the - all-hot line, and it is only true after generation. -- **`refuse_restarts!/2` and `describe_remedies/2` go with the refusal.** Once - nothing refuses they have no caller, and keeping them against a hypothetical - regression is the same mistake as keeping a gate. What the announcement reuses - is `describe_causes/3`, `describe_edges/1` and `describe_restarts/1` — the - last of which `--hot`'s `refuse_hot_restarts!/1` and `appup_restarts!/1` share - and which is not going anywhere. -- **`appup_restarts!/1`'s refusal of the two-stage `restart_new_emulator` must - not become conditional.** It is gated on nothing today and it stays that way: - Castle is built for the one-stage instruction, and that is independent of - whether a restart transition can be completed. - -The tests that have to change, in `test/forecastle/relup_test.exs`: - -- **The mixed-restart assertions invert.** "settles the restart it chose without - generating the rest of the relup" and "refuses the restart it chose rather - than a systools error from the rest" both pin today's *single*-cause - behaviour: a non-zero exit, one cause named, `refute output =~ "an appup asks - for the emulator to be restarted"`, and no relup on disk. Once the run - proceeds, all of that inverts — zero exit, a relup written, and **both** causes - in the one announcement. Their comments say as much. -- **The all-hot refutation stays exactly as it is.** `refute_all_hot/1` matters - *more* afterwards, not less: a *successful* run printing both the all-hot line - and a restart is the contradiction that shipped once. -- **The announcement needs coverage of its own.** Nothing executes it today, so - do not assume the wording works: assert it for a classified restart edge, for - an appup-supplied one, and for a plan carrying both. -- **The merge tests keep working unchanged.** What changes is that - `plan_transitions!/5` stops being the *only* way to reach the merge, so the - task-level cases can assert the merged relup too. +**`auto` announces a restart edge; it used to refuse one.** The refusal existed +because Castle could not *complete* such a transition — `heart:set_cmd/1` raised +`badarg` with no `heart` process — and `auto` is the no-switch default, so +emitting a restart edge meant a routine invocation producing a relup that could +not be installed. [castle#14](https://github.com/ausimian/castle/issues/14) and +[#10](https://github.com/ausimian/forecastle/issues/10) closed that, and +`settle_restarts!/2` is now the whole of the verdict: `([], [])` is the all-hot +line and everything else is one `Mix.shell().info/1` naming both kinds. + +**A history worth keeping, because it constrains how the announcement is +written.** An earlier revision of this work had a gate — +`restart_transitions_installable?/0`, a private function returning a literal +`false`, consulted from two call sites. Elixir 1.20's type inference proved the +`true` branch of both dead and `mix compile --warnings-as-errors` failed on them, +which took CI red on all six 1.20 cells. That is why the refusal was deleted +rather than made conditional, and why nothing here may grow a predicate the +compiler can fold: reading application env or a switch makes it user-flippable, a +module attribute folds to the same literal, and there is no suppression pragma. +The lesson generalises past this feature — a feature gate whose branches nothing +reaches is a compile error waiting for the next Elixir. + +**`appup_restarts!/1`'s refusal of the two-stage `restart_new_emulator` is gated +on nothing and stays that way.** It never was about whether a restart could be +completed. Castle is built for the one-stage instruction, and the two-stage one +reboots into a temporary hybrid release whose version directory has no `env.sh`, +no `elixir` and no `vm.args` — so there is nothing for the launcher to boot and +Castle arms no marker for it. **One verdict per invocation.** A restart reaches an `auto` run two ways — `auto` classified the edge as one, or an appup named `restart_emulator` itself — and only -the first is knowable before the relup exists. - -The all-hot line therefore lives only in `settle_restarts!/2`'s `([], [])` clause, -which runs after generation. Announcing from classification alone says every -transition is hot and then reports a restart; that shipped once, and once the -announcement lands it would be a *successful* run printing both. Do not add a -second announcement anywhere else. `relup_test.exs` asserts the all-hot line is -absent from every `auto` case that ends in a restart. - -While restarts are uninstallable, though, a classified restart edge makes the -outcome a refusal *whatever* generation does — so generating first only lets a -`:systools` failure on the hot remainder stand in front of the real reason, which -a user would fix in order to be told about the restart anyway. Hence the -pre-generation refusal. Its accepted cost: a mixed plan names the classified edge -and stays silent about an appup-supplied restart in the hot half until the first -one is gone. - -Because of that refusal the split-and-merge — the path that puts hand-written -restart entries and generated hot ones into one relup — is unreachable through -the task. It is still the defining behaviour of `auto`, so -`plan_transitions!/5` is `@doc false`-public and `relup_test.exs` drives it -in process. Keep the settling **outside** that function rather than inside it; -that is what keeps the merge testable, and it is where the announcement above -will have to go. What must stay inside is the unconditional refusal of -`restart_new_emulator` (`appup_restarts!/1`), which is not conditional on -anything: it returns the one-stage restarts and raises on the two-stage ones. +the first is knowable before the relup exists. So nothing is settled until after +generation: the all-hot line and the restart announcement both live in +`settle_restarts!/2`, which runs there. Announcing from classification alone says +every transition is hot and then reports a restart; that shipped once, and now +that such a run *succeeds* it would print both from a successful invocation. Do +not add a second announcement anywhere else. `relup_test.exs` asserts the +all-hot line is absent from every `auto` case that ends in a restart — +`refute_all_hot/1` matters more now, not less. + +**Where the earlier plan for this change was wrong, so it is not re-derived.** +This file used to carry a step-by-step account of what castle#14 had to add here, +and one item of it did not survive contact. It said the two mixed-restart tests +both inverted to "zero exit, a relup written, and both causes in the one +announcement". That is true of the first — now *"names both kinds of restart in +one announcement"* — and false of the second: that test deliberately removes +`sample.appup` so the hot half cannot be generated at all, and once the +pre-generation refusal is gone, generation runs first and the `:systools` error +*is* the verdict. It is now *"reports a systools error from the hot half rather +than announcing anything"*, and what it pins is the new ordering rather than the +old one. There is nothing left to announce about a relup that was not produced. + +Everything else in that plan held: `refuse_chosen_restarts!/1`, +`refuse_restarts!/2` and `describe_remedies/2` are gone with the refusal; +`describe_causes/3`, `describe_edges/1` and `describe_restarts/1` are what the +announcement reuses; and `plan_transitions!/5` stays `@doc false`-public even +though the task reaches the merge now, because what the merge tests assert is a +term rather than anything the task prints. + +The announcement is covered in the three shapes it can take — a restart +classification chose, one an appup asked for by name, and a plan carrying both — +and each of those cases now asserts the *relup* as well as the wording, because +an announcement is a claim about a file and the file is the thing that gets +installed. + +The split-and-merge — the path that puts hand-written restart entries and +generated hot ones into one relup — is reachable through the task now, and the +mixed-restart cases in `relup_test.exs` go through it. `plan_transitions!/5` +stays `@doc false`-public anyway, and `relup_test.exs` still drives it in +process, because what those tests assert is the *shape* of the merged plan: which +from-version keeps which script, in which direction. That is a term, and reading +it back off disk through the task would say less. Keep the settling **outside** +that function rather than inside it; that is what keeps the merge callable. What +must stay inside is the unconditional refusal of `restart_new_emulator` +(`appup_restarts!/1`), which is not conditional on anything: it returns the +one-stage restarts and raises on the two-stage ones. **The merge tests have to pin the direction, not just the shape.** Both sections of the fixture's mixed relup carry the same from-versions, the same restart @@ -452,6 +788,7 @@ changing the fixture. | `lib/mix/tasks/forecastle.relup.ex` | `mix forecastle.relup` — chooses an upgrade strategy per transition, and writes the relup | | `priv/castle.sh.eex` | EEx template for `bin/castle`, the release management CLI | | `priv/env.sh.eex` | EEx template for the fragment appended to the release's `env.sh` | +| `priv/start.sh.eex` | EEx template for `bin/start`, the inert program heart is handed | | `test/fixtures/sample` | A real application, assembled by the test suite into a real release | | `test/fixtures/sample/dep` | An application the relup never mentions, whose version moves with the sample's unless `SAMPLE_DEP_VSN` pins it | | `test/support` | The workspace the fixture is built in, the case template for tests that build it, and the helpers that drive one once it is built | @@ -483,14 +820,75 @@ directory to start from a clean slate. | `test/forecastle_test.exs` | The step functions, against a synthetic `Mix.Release` | | `test/forecastle/assembly_test.exs` | The assembled tree, including `bin/` being byte-identical to the launcher plain Mix produces | | `test/forecastle/castle_cli_test.exs` | `bin/castle` as a shell script, against a launcher stub that records its arguments | +| `test/forecastle/env_script_test.exs` | The `env.sh` fragment as a shell script, sourced in a release-shaped directory with a launcher stub: the heart environment it leaves behind, and which provisional version each state of the two markers selects | | `test/forecastle/configuration_test.exs` | A release that names its own runtime configuration file and declares providers whose init arguments are not keyword lists — assembled, and booted through `bin/ eval` | | `test/forecastle/relup_test.exs` | `mix forecastle.relup` as a command, against three assembled releases: argument handling, exit status, and all three upgrade strategies | | `test/forecastle/upgrade_test.exs` | Booting a release and hot-upgrading it, including the code path of an application the relup does not load, tagged `:e2e` | +| `test/forecastle/restart_upgrade_test.exs` | The same shape through an emulator restart: the OS pid changes, an uncommitted release rolls back when killed, and a commit makes it what an ordinary start boots. Tagged `:e2e` | The `:e2e` suite is excluded by default and included by `mix precommit`. Run it on its own with `mix test --include e2e`. It needs no epmd daemon: the fixture configures distribution without one. +`restart_upgrade_test.exs` is the hot suite's opposite where it counts — +`refute provisional.os_pid == booted.os_pid` against the hot suite's +`assert installed.os_pid == booted.os_pid` — and it has one thing no other suite +does: **it is the supervisor.** Nothing in the release restarts it after +`init:reboot()`, deliberately, so `Forecastle.Deployment.install_supervised!/3` +runs `bin/castle install` in a task, waits for the old *operating system process* +to go, and starts the release again. Waiting on the process rather than on the +node matters: a node that has stopped answering rpc is not necessarily one that +has exited, and starting the replacement while the old beam still holds the +distribution port is a name clash rather than a boot. + +**It also runs the whole transition on a hostile environment**, and that is not +incidental colour: `HEART_COMMAND`, `HEART_NO_KILL=FALSE` and an 11-second +`HEART_BEAT_TIMEOUT` are in the deployment's environment for the first start, so +a fragment that only *defaulted* them would have this suite exercising a release +with a live watchdog beside its supervisor. What is asserted is what the node +says it was started with — `{nil, "TRUE", "65535"}` — plus the warning naming +each displaced value, and, on the one start with a clean environment, silence. + +That first start also inherits `ELIXIR_ERL_OPTIONS` with a **tab-separated** +`-heart` in it, which no e2e start used to set at all: the fixture scrubs the +variable, so every boot here had the fragment *assigning* it rather than finding +one, and the case where it has to recognise an inherited flag was covered only by +a unit test asserting the exact string `-heart`. Two `-heart` flags hang the boot, +so the suite failing to start is the regression. The value carries +`-env CASTLE_TAB_PROBE tabbed` behind the tab so that the node answering with +`"tabbed"` says the tabs were field separators, and +`:init.get_argument(heart) == {ok, [[]]}` says the emulator got one flag. + +**Which is why `Forecastle.Deployment.start!/2` puts a deadline on the launcher.** +That regression does not fail, it *hangs*, and it hangs inside `daemon` rather +than after it: `env.sh` runs the preboot VM synchronously on a first start and +that VM inherits the same options, so the whole suite stops there. +`System.cmd/3` has no deadline and `setup_all` has no ExUnit timeout, so the +deadline is the only thing turning it into a named failure. Measured by putting +the old guard back — the run had to be killed. Do not remove it as +belt-and-braces. + +`env_script_test.exs` is the other half of that, and the division between the two +is worth keeping: this suite proves the effective configuration survives a real +boot and a real reboot, and that one proves each individual override and each +marker state, cheaply and without a `mix release`. Neither replaces the other, +and **no claim about the effective heart configuration belongs in +`assembly_test.exs`** — asserting the text of the fragment is what let the +defaulted version pass. + +Three of its assertions are load bearing in ways the obvious ones are not. +`RELEASE_VM_ARGS`, which `runtime.exs` reads with `fetch_env!`, is derived by the +launcher from `REL_VSN_DIR` *after* `env.sh` is sourced — so the provisional +node reporting the *new* version's `vm.args` is what says the fragment re-execs +rather than assigning `RELEASE_VSN` in place. `SAMPLE_GREETING` changes between +the first boot and the provisional one, so the provisional node answering the +second value is what says the version's own config providers ran again over the +`sys.config` Castle materialised — which is also the only coverage of that +interaction anywhere. And the rollback half comes *before* the commit half, +because the only way to see a provisional release roll back is to kill one; the +second install then goes through the same transition again from the release that +came back. + `configuration_test.exs` is where the two #6 defects are pinned, and it is deliberately behavioural: there is nothing left in `lib/` to unit-test, so what it asserts is that the release carries the file the project asked for and that @@ -518,32 +916,38 @@ returns its argument, which is what makes the second observable at all. `test/forecastle/upgrade_test.exs` covers that shape with `:sample_dep`, whose version changes and whose appup asks for nothing. - **Root scripts do not update through a hot upgrade.** `release_handler` - extracts with `keep_old_files`, so `bin/` and `bin/castle` are - whatever the deployment was first built with. New files do appear, so a - migrating deployment gains `bin/castle`, but keeps its old launcher. Mix's own - launcher has always behaved this way; do not add a dispatcher to work around - it without deciding that question for `bin/` too. -- **The emulator-restart transitions cannot be performed.** A relup that asks for - one can be generated — `mix forecastle.relup --restart` — and `relup_test.exs` - covers it being generated, accepted by `verify_relup!/2` and copied into the - release. The transition itself fails: `release_handler` calls `heart:set_cmd/1` - while preparing the reboot, which raises `badarg` where there is no `heart` - process, so the install fails before anything reboots; and the reboot would - come back up on whichever version `releases/start_erl.data` names, which only - `make_permanent` writes, while `release_handler` leaves the installed version - in `releases/new_start_erl.data`. `bin/castle install` handles the reply and - each branch of that handling is tested against a launcher stub, but nothing - performs the upgrade end to end until - [castle#14](https://github.com/ausimian/castle/issues/14) and - [#10](https://github.com/ausimian/forecastle/issues/10). The `:e2e` suite - covers the hot-upgrade path only, and any test of a restart relup should - assert on the relup and the assembled tree rather than on a completed upgrade. - This is what `auto`'s temporary refusal above exists for; `--restart` is the - explicit way to build one anyway. + extracts with `keep_old_files`, so `bin/`, `bin/castle` and + `bin/start` are whatever the deployment was first built with. New files do + appear, so a migrating deployment gains `bin/castle` and `bin/start`, but keeps + its old launcher. Mix's own launcher has always behaved this way; do not add a + dispatcher to work around it without deciding that question for + `bin/` too. + + What this costs the restart path is worth stating: the heart configuration and + the provisional-version selection are in `releases//env.sh`, which a hot + upgrade *does* replace, so a deployment can take one hot upgrade to this + release and a restart transition after it. It cannot take a restart transition + as the *first* upgrade from an older deployment — the running node came up on + the old version's `env.sh`, so there is no `heart` process, and the install + fails in `heart:set_cmd/1` before anything reboots. +- **An emulator restart needs something outside the release to start it again.** + That is the design rather than a gap: `bin/start` is inert, `HEART_COMMAND` is + unset, and the supervisor is the only restart authority. A deployment run by + hand from a shell therefore stays down after such an upgrade until somebody + starts it — and comes back on the installed version when they do, because the + markers are still waiting. Only systemd has been exercised in anger; Docker + restart policies and runit are the same shape but unmeasured, and `:e2e` stands + in for a supervisor by starting the release itself. - **`restart_new_emulator` is not supported and is refused, not generated.** - Adding it is its own piece of work: the provisional boot would have to come up - and *resume* an upgrade through `new_emulator_upgrade/2`, which is strictly - more than coming up on a provisional version. + Adding it is its own piece of work, and for two reasons rather than one. The + provisional boot would have to come up and *resume* an upgrade through + `new_emulator_upgrade/2`, which is strictly more than coming up on a provisional + version — and the version `release_handler` writes into `new_start_erl.data` for + it is the temporary hybrid release, `__new_emulator__`, whose version + directory `new_emulator_make_hybrid_boot/6` gives a `start.boot` and a + `sys.config` and none of the launcher's own files. There is no `env.sh` to + source, no `elixir` to run and no `vm.args` to read, so the fragment could not + select it even if it wanted to; Castle arms no marker for that transition. - **An unpacked release holds two `.rel` files, and that is normal.** `release_handler:do_unpack_release/4` copies `releases/-.rel` into `releases//` unconditionally, "for backwards compatibility reasons with diff --git a/README.md b/README.md index e8786ed..8828c3e 100644 --- a/README.md +++ b/README.md @@ -102,18 +102,53 @@ In the post-assembly step: - A `bin/castle` command is added, providing the commands that manage releases. The standard `bin/` launcher that Mix generates is left untouched. - - The generated `env.sh` is extended with a hook. On the **first** start of a + - A `bin/start` is added, and it does nothing at all. `release_handler` + composes `$ROOT/bin/start ` and installs it as `heart`'s temporary + reboot command while preparing an emulator restart, and `heart` really does + run it. A Castle release is restarted by its supervisor rather than by + `heart`, so the one correct thing for that script to do is exit 0. + - The generated `env.sh` is extended with a hook, and everything in it runs + only for the commands that start the system. On the **first** start of a deployment it creates `releases/RELEASES`, which is what lets the system manage its own releases — a short-lived VM, once, and only while that file is absent. The release root has to be writable for it to succeed; if it is not, the start still proceeds, with a warning, and `bin/castle unpack` and `bin/castle install` will later refuse — each reading the running system's own release records as it acts — rather than upgrade a system that cannot - record what it is running. Every start after the first does nothing at all. - The - hook is also where the provisional version marker left by a relup that - restarts the emulator will be consumed. Any `env.sh` the project supplies - through `rel/env.sh.eex` is preserved, and runs first. + record what it is running. + + Every start also runs OTP's `heart`, deliberately configured to do nothing: + `HEART_NO_KILL`, no `HEART_COMMAND`, a beat timeout at heart's documented + maximum, and the inert `bin/start` above. It is there for one reason — + `release_handler` calls `heart:set_cmd/1` while preparing an emulator + restart, and that raises where no `heart` process exists. + + If your deployment already asks for `-heart` — in `rel/vm.args.eex`, in + `ELIXIR_ERL_OPTIONS`, in one of `ERL_AFLAGS`, `ERL_FLAGS` and `ERL_ZFLAGS`, + or in `ERL_OTP_FLAGS` — that is fine, and nothing is added beside it: + two of the flag make `init:get_argument(heart)` answer `{ok, [[], []]}`, + which heart's own startup check has no clause for, so the boot would hang + having printed nothing. The hook settles it by asking `erl` what argument + list it would build, so quoting and escaping in those values are read the way + `erl` reads them, and all six places a flag can come from are covered at + once. It asks on every start, which costs one short-lived `erl` that exits + without booting anything; commands that do not start the system — `eval`, + `rpc`, `remote` — ask nothing. + + Those three are **assigned**, and `HEART_COMMAND` is **unset**, rather than + defaulted — so a deployment that already has any of them in its environment + still gets a heart that does nothing. There is no opting out of that while + this hook is in use: your supervisor owning the restart is what the rest of + it depends on. A start that displaces one of your settings says so on + standard error, naming what it displaced, rather than failing the boot over a + configuration conflict or losing the setting silently. A deployment that sets + none of them says nothing at all. + + And a start that follows such a restart selects the version that was + installed. See *Upgrades that restart the emulator* below. + + Any `env.sh` the project supplies through `rel/env.sh.eex` is preserved, and + runs first. - The generated _name.rel_ is copied into the `releases` folder as _name-vsn.rel_, which is where `release_handler` looks for it when unpacking a tarball. - Any checked `relup` is written into the version path of the release. @@ -142,7 +177,8 @@ Moving the running system from one version to the next is done through `bin/cast # Unpack myapp-0.1.1.tar.gz, which you have placed in myapp/releases. > myapp/bin/castle unpack 0.1.1 -# Make 0.1.1 the version that is running now, without restarting the VM. +# Make 0.1.1 the version that is running now. Whether the VM is restarted is a +# property of the relup rather than of this command. > myapp/bin/castle install 0.1.1 # Make it the version that runs on restart too. With no version given, this @@ -159,9 +195,38 @@ record OTP made up out of its boot script. Each asks the system itself, as it acts, rather than trusting an answer given earlier; the refusal names the remedy, which is a restart. -Version selection on restart needs nothing from `Forecastle`: OTP's -`release_handler` records the committed version in `releases/start_erl.data`, -which is exactly where the standard launcher reads it from. +Version selection on restart needs nothing from `Forecastle` once a version has +been committed: OTP's `release_handler` records the committed version in +`releases/start_erl.data`, which is exactly where the standard launcher reads it +from. + +### Upgrades that restart the emulator + +`bin/castle install` is the same command whichever kind of transition the relup +describes, and it exits 0 only once the version it installed is the one running — +across a reboot, if there is one. What differs is what has to be in place around +it. + +**Your supervisor owns the restart.** `release_handler` calls `init:reboot()`, +the operating system process exits, and nothing inside the release starts it +again: `bin/start` is inert and `HEART_COMMAND` is unset — unset by the hook on +every start, even where the environment supplies one — because two things +starting one service is worse than the problem being solved. Run the release +under systemd, a Docker restart policy, Kubernetes or runit. A release started by +hand from a shell will simply stay down until you start it again. + +**Until you commit, a restart takes you back.** `release_handler` writes the +installed version to `releases/new_start_erl.data` and leaves +`releases/start_erl.data` naming the version that is still permanent — only +`bin/castle commit` writes that file. So a provisional release that crashes +before it is committed is followed by an ordinary start of the version you were +on, with nobody intervening. `bin/ version` reports that version too, +because what it prints is the version *to be booted*; ask the running system if +you want to know what is running. + +**Only the one-stage `restart_emulator` is supported.** `mix forecastle.relup` +never generates the two-stage `restart_new_emulator` and refuses it wherever it +finds one; see below. ## The Appup Compiler @@ -290,20 +355,13 @@ own is missing. A transition it judged hot and `systools` then could not generat is a failure, so that the default never quietly ships something other than the upgrade it decided on. -> **`auto` currently refuses a restart transition.** Castle can install a relup -> that restarts the emulator but cannot yet complete the transition (see below), -> so rather than write an upgrade plan that is known not to install, `auto` exits -> non-zero and names the edge that forced the restart and why. It names it before -> generating anything: an edge classification found to need a restart decides the -> run on its own, so a `systools` error from the transitions that were still going -> to be hot cannot be reported in its place. A `restart_emulator` an appup asked -> for by name is refused just as much - the same transition arrived at another -> way - but only becomes visible once there is a script to look at, so it is -> reported after generation. A relup with both kinds in it therefore names the -> classified edge, and the appup's own restart is reported by the run that follows -> once that edge is gone. This is temporary; when the refusal is lifted the run -> proceeds and both kinds are named in one announcement. `--restart` is the -> deliberate override for anyone who wants the relup anyway. +The announcement names every edge that will restart and why — both the ones +classification chose and any `restart_emulator` an appup asked for by name, in one +message, since they are the same transition arrived at two ways. It also says +what that means for reading the install back: `install_release/1` replies +`{ok, Vsn, Descr}` for such a transition, indistinguishably from a completed hot +upgrade, and the emulator then reboots. `--hot` and `--restart` are the ways to +insist on something else. **`--hot`** requires a genuine hot upgrade of every transition, and exits non-zero, having written nothing, if one cannot be: a missing appup entry, an @@ -341,12 +399,9 @@ its own whenever the ERTS version differs between two releases, so a default that simply generated a relup would ship the two-stage transition without anybody having chosen it. -Note that a restart transition can be *generated* but not yet *performed*. -`release_handler` calls `heart:set_cmd/1` while preparing the reboot, which fails -where there is no `heart` process, so the install fails before anything reboots; -and the reboot would come back up on whichever version -`releases/start_erl.data` names, which nothing writes until the release is -committed. Until [castle#14](https://github.com/ausimian/castle/issues/14) and -[#10](https://github.com/ausimian/forecastle/issues/10) land, treat a restart -relup as something to generate and inspect rather than to deploy - and that is -why `auto` refuses to produce one, while `--restart` still will. +Performing a one-stage restart transition takes two things the release now +carries: a `heart` process, because `release_handler` calls `heart:set_cmd/1` +while preparing the reboot, and something to select the installed version on the +way back up, because the reboot would otherwise come back on whichever version +`releases/start_erl.data` names. Both are in the `env.sh` hook; see +*Upgrades that restart the emulator* above for what your supervisor has to do. diff --git a/RELEASE.md b/RELEASE.md index efb9f1b..5ad4c72 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -93,18 +93,17 @@ that asked for it has gone. `bin/castle releases` is how to find out where the system actually got to. - A relup that restarts the emulator can now be built - see - `mix forecastle.relup --restart` below - but the transition it describes - **cannot be performed**: the install fails while preparing the reboot, and - nothing would bring the reboot back up on the version that was installed - anyway. That is - [castle#14](https://github.com/ausimian/castle/issues/14) and - [#10](https://github.com/ausimian/forecastle/issues/10). What is covered is - the hot-upgrade path, by the `:e2e` suite, which now installs *and* confirms; - and every branch of the shell logic - inconclusive, confirmed, failed, timed - out - against a launcher stub, which remains the only place the restart shape - can be simulated. A continuation that fails and rolls back is left to the - end-to-end coverage those two bring. + That polling is what carries an emulator restart, and it is the reason the + reply cannot be trusted: `release_handler` answers `{ok, Vsn, Descr}` for a + `restart_emulator` transition exactly as it does for a completed hot upgrade, + and then reboots, so the reply may not even survive long enough to arrive. + `install` treats a lost connection as settling nothing and keeps asking until + the version it installed answers - across the reboot, and across the cold boot + after it. Both paths are covered end to end by the `:e2e` suite, and every + branch of the shell logic - inconclusive, confirmed, failed, timed out - against + a launcher stub as well. A continuation that fails and rolls back on the way up + belongs to `restart_new_emulator`, which is not supported; see *Known + limitations*. - `mix forecastle.relup` now takes an upgrade strategy, because whether a transition can be hot is a property of the edge between two releases rather than of either release. `--hot` requires a genuine hot upgrade and fails, @@ -163,38 +162,162 @@ something other than the upgrade it decided on; ask for the restart with `--restart`. - **While a restart transition cannot be performed, `auto` refuses to write one.** - A relup that restarts the emulator can be built and packaged, but the transition - it describes fails on install - see the known limitation below - so `auto` - exits non-zero rather than produce an upgrade plan that is known not to - install. The message names the edge that forced the restart and why, whether - that is the ERTS change or the application whose move no appup covers, and - points at `--restart` as the deliberate override. The same applies to a - `restart_emulator` an appup asked for by name during an `auto` run: how the - relup came by the instruction makes no difference to whether it can be - installed. - A run says which transitions restart - or that none of them do - exactly once, - and the two kinds are settled at the point each becomes knowable. An edge - classification found to need a restart is refused before anything is generated, - because that already decides the run: nothing the remaining transitions turn out - to be can change it, so generating them can only fail in a way that reports - something other than the reason the run is failing. An appup that asks for the - restart by name is invisible until `systools` has produced a script, so it is - settled after generation. A relup with both kinds in it therefore names the - classified edge and generates nothing; anything an appup in the rest of it asks - for is reported by the run that follows, once that edge is gone. - - This is temporary and will be lifted, at which point `auto` will announce the - restart it chose instead of refusing - and then, since the run proceeds, both - kinds are named in the one announcement. Lifting it is work still to do rather - than a switch to throw: there is no flag, environment variable or build option - that turns the refusal off, and no announcement waiting behind one. `--hot` - and `--restart` are unaffected in either direction; both are explicit - requests, and it is fine for `--restart` to produce a relup that cannot yet be - deployed. + and the announcement names both ways a restart can arrive: the edges `auto` + classified, with the reason for each, and any `restart_emulator` an appup asked + for by name. It also says what that means for reading the install back, since + `install_release/1` replies `{ok, Vsn, Descr}` for such a transition, + indistinguishably from a completed hot upgrade, and the emulator then reboots. + Both kinds are settled after generation, because only one of them is knowable + before it: an appup's own instruction is invisible until `systools` has produced + a script. `--hot` and `--restart` remain the ways to insist on something else. +- The release now selects a provisional version after an upgrade that restarted + the emulator, which is what makes such an upgrade work on a deployment + supervised by systemd, Docker, Kubernetes or runit. + + `release_handler` writes the version it installed to + `releases/new_start_erl.data` and deliberately leaves + `releases/start_erl.data` - which is where the stock launcher reads + `RELEASE_VSN` from - naming the version that is still permanent. That is the + rollback property, and it is worth keeping: a provisional release that dies + before `bin/castle commit` is followed by an ordinary start of the version that + was permanent before, with nobody intervening. What it costs is that something + has to select the installed version on the boot after the reboot, and the + `env.sh` fragment is now that something. + + It requires *two* markers, not one, and it re-execs the launcher rather than + assigning `RELEASE_VSN` in place. Two markers, because `new_start_erl.data` is + written before the reboot and never removed, so on its own it is not evidence + that a reboot was asked for: Castle arms a marker of its own beside it and + clears it if the install failed, and the fragment requires both files and + requires them to name one version. A re-exec, because by the time the launcher + sources `env.sh` it has already resolved the version directory, and everything + it goes on to use - the boot script, `vm.args`, `sys.config`, the `elixir` + launcher itself - hangs off that. With no valid pair the fragment does nothing + at all and the stock launcher reads `start_erl.data` exactly as it always did. + + The selection comes before anything else the hook configures, and that is + load-bearing rather than tidy. Re-exec'ing means the hook is read again, so + anything decided beforehand is decided about the version being *replaced* - and + exported to the pass that boots. `heart` is the case that shows it: the two + versions can carry different `vm.args`, so whether the emulator is already + getting a `-heart` has to be asked about the one that will actually start. + Everything after the selection is therefore settled once, by that pass. + + What is atomic is the *claim*, and it is worth being exact about because the + rest follows from it. The fragment takes Castle's marker by renaming it, which + is one operation, so exactly one start can act on the pair however many are + racing. OTP's file is then read and removed in further steps, and no POSIX + operation moves two files together - so the pair is not consumed as a unit, and + the order is what makes that safe: the marker goes first, so a start killed + part way through leaves no marker behind, and the next start reads + `start_erl.data` and boots the version that was permanent. A provisional + selection can therefore be lost by an ill-timed kill. It cannot be applied + twice, and it cannot be applied to a version that was never installed. +- The release now runs OTP's `heart`, deliberately configured to do nothing, on + the commands that start the system. + + This is not a watchdog and is not offered as one. `release_handler` calls + `heart:set_cmd/1` while preparing *any* transition that restarts the emulator, + and with no `heart` process that raises `badarg` - so the install failed before + anything rebooted, on exactly the externally supervised deployment this library + is for. The handshake has to be satisfied, and running the real `heart` + satisfies it through documented interfaces only. + + `heart` is then kept out of the way. `HEART_COMMAND` is not set, so an + unexpected death starts nothing; `HEART_NO_KILL=TRUE`, so a node that misses + heartbeats is not killed; `HEART_BEAT_TIMEOUT` is raised to heart's documented + maximum, because `HEART_NO_KILL` alone does not make a heart-beat time-out + harmless - the port program exits once it has run its command, and `heart` is a + kernel process, so `init` halts the node when it goes; and `$ROOT/bin/start`, + the path `release_handler` composes into heart's temporary command and does not + check, is shipped and does nothing at all. That last one is not belt and + braces: `HEART_NO_KILL` suppresses the kill but *not* the command, so a + `bin/start` that really started the release could start a second node beside a + live one. The external supervisor remains the only thing that starts this + release. `-heart` is added to `ELIXIR_ERL_OPTIONS` only when the emulator is not + going to get one anyway: two of them make `init:get_argument(heart)` answer + `{ok, [[], []]}`, which heart's own startup check has no clause for, and the + boot hangs having printed nothing. + + **Whether the emulator is already going to get one is measured rather than + guessed at, and there is more than one way for it to arrive.** + `ELIXIR_ERL_OPTIONS` is the variable Mix's generated `elixir` expands; `erl` + itself prepends `ERL_AFLAGS` and appends `ERL_FLAGS` and then `ERL_ZFLAGS` to + its effective command line; and the launcher passes `vm.args` as `-args_file`, + so a project's own `rel/vm.args.eex` carries flags too - and `erl` follows a + nested `-args_file` out of it. A flag arriving by any of those routes reaches + `init:get_argument/1` exactly as one on the command line does, and a deployment + that had one used to receive it plus the appended one, which is the boot hang + above. + + It is recognised however it is written, which is why the hook does not read + these itself: `erl` applies shell-style quoting and backslash escaping to + everything it takes from the environment and from an args file, so `'-heart'`, + `"-heart"` and `-he\art` all arrive as `-heart` without containing the word. + The hook therefore asks `erl` - with the start's own environment and args file, + and without starting a VM - what argument list it would build, and adds a flag + only if that list has none. **It asks on every start**, whether or not anything + in the environment looks like it could carry a flag, which costs one + `fork`+`exec` of a C program that exits without booting an emulator - about + 11ms, once per node start. Nothing is asked for an `eval`, an `rpc` or a + `remote`: the whole hook runs only for the commands that start the system. + + `ERL_OTP_FLAGS` is covered along with the rest. It is undocumented and + described by OTP's own source as for internal use, but `erl` reads it, so a + `-heart` there is a `-heart` the emulator gets - and because asking is + unconditional, one set only there is detected like any other. + + The `erl` that is asked is the one the launcher is going to run, and the hook is + told which that is by the file that decides it: Mix's generated + `releases//elixir` resolves the emulator through an `ERTS_BIN` it rewrites + at build time, so the hook reads that assignment out of the same file the + launcher will, and falls back to `erl` on `PATH` - which is what an + un-rewritten `ERTS_BIN` means - for a release built with `include_erts: false`. + That matters because a release root holds more than one `erts-*` as soon as an + ERTS-changing release has been unpacked into it, and `ERL_OTP_FLAGS` is + named for the OTP version of whichever emulator answers. + + Where the question cannot be answered - an args file `erl` refuses to read, for + instance - the hook adds nothing and says so on standard error. That direction + is deliberate: adding a flag that turns out to be a second one hangs the boot + in silence, while adding none makes an upgrade that restarts the emulator fail + loudly with the system still running. A `vm.args` that is simply *absent* is not + such a case: the file is passed to `erl` only when it exists, so a release + shipping none is asked about without it rather than reported unmeasurable. + + All three variables are **assigned**, and `HEART_COMMAND` is **unset**, rather + than defaulted - so a deployment that already has any of them in its + environment gets the defanged heart anyway. That is deliberate and it is not + negotiable while this hook is in use: the supervisor owning the restart is the + contract the rest of this depends on, and an inherited `HEART_COMMAND`, a + `HEART_NO_KILL` of anything but `TRUE`, or a shorter `HEART_BEAT_TIMEOUT` each + break it - the last one by giving a stalled node a way to be killed that a + release without `-heart` has not got. + A start that overrides one of them says so on standard error, naming the value + it displaced and why, because a setting that silently stops taking effect is + worse than one that is refused - and refusing is what it does *not* do: a + conflicting variable is a configuration mistake, not a reason to fail a boot. A + deployment that sets none of them, which is the ordinary case, says nothing. A + variable that is *set to nothing* counts as a value for the two that are + assigned, and is reported as `[]`: neither an empty `HEART_NO_KILL` nor an empty + `HEART_BEAT_TIMEOUT` is the value that replaces it. An empty `HEART_COMMAND` + stays silent, because unsetting a variable that was already empty changes + nothing heart can read. - A test suite. It assembles a real release from a fixture application and, in - the `:e2e` suite, boots it and performs a hot upgrade. + the `:e2e` suite, boots it and upgrades it - once hot, asserting that the + operating system pid does not change, and once through an emulator restart, + asserting that it does, that an uncommitted provisional release rolls back when + it is killed, and that committing makes it the version an ordinary start boots. + The restart suite runs the whole transition on a deployment whose environment + already carries a `HEART_COMMAND`, a `HEART_NO_KILL` of `FALSE` and an + 11-second beat timeout, and whose `vm.args` supplies a `-heart` spelled + `-he\art`, and asks the running node what it was actually started with. That + start is given a deadline, because a boot handed two `-heart` flags hangs rather + than fails and would otherwise stop the suite for as long as whatever ran it + would wait. The `env.sh` hook is also run directly, over a release-shaped directory, + so that what it selects and what environment it leaves behind are asserted by + observation rather than by reading the script. ### Changed @@ -230,10 +353,10 @@ configuration half is gone outright. What remains is `releases/RELEASES`, and the fragment now creates it only when the release has not got one — the first start of a deployment, and no start after it. It is still appended after any - `env.sh` the project supplied, and it is still where - [#10](https://github.com/ausimian/forecastle/issues/10) will consume the - provisional restart marker that a relup restarting the emulator leaves in - `releases/new_start_erl.data`. + `env.sh` the project supplied. It also configures `heart` and selects a + provisional version after an emulator restart — see *Added* — and everything in + it is now gated on a command that starts the system, so an `eval`, an `rpc` or + a `remote` reaches none of it. A release therefore starts as quickly as a plain Mix release every time bar the first, and a start that used to fail because configuration could not be @@ -326,15 +449,16 @@ `systools_relup:check_for_emulator_restart/5` inserts the two-stage `restart_new_emulator` on its own whenever the ERTS version differs, warning only that it changed - so the relup carried a transition nobody had chosen, - which replies `{continue_after_restart, Vsn, Descr}` and which Castle cannot - install. `auto` now decides this case for itself, as a one-stage restart - transition, which while such a transition cannot be performed means the - generation is refused. `--restart` generates it. **Materially changed.** + which replies `{continue_after_restart, Vsn, Descr}` and which Castle does + not support. `auto` now decides this case for itself, as a one-stage restart + transition, and announces it. `--restart` generates the same thing on + request. **Materially changed.** - **An appup that names an emulator restart itself** was passed straight - through, and the relup was written with the restart in it. `auto` now refuses - it, `--hot` refuses it, and `--restart` - which reads no appup at all - makes - the transition a `restart_emulator` by its own choosing. **Materially - changed.** + through, and the relup was written with the restart in it without anybody + being told. `auto` now announces a one-stage `restart_emulator` and refuses + the two-stage `restart_new_emulator`; `--hot` refuses both; and `--restart` - + which reads no appup at all - makes the transition a `restart_emulator` by + its own choosing. **Materially changed.** So the two cases that changed are the two that used to write a relup carrying an emulator restart; the other two are a hot upgrade that is still a hot @@ -520,6 +644,16 @@ This applies to `bin/castle` too: once installed, later changes to it will not reach an existing deployment through a hot upgrade. That is the same property Mix's own `bin/` has always had. +`bin/start` is new, so it does appear, and both the heart configuration and the +provisional-version selection live in the version directory's `env.sh`, which a +hot upgrade does replace. So a deployment that takes one hot upgrade to this +release can take a restart transition after it. What it cannot do is take a +restart transition *as* the first upgrade from an older deployment: the node is +running from the old version's `env.sh`, so it has no `heart` process, and +`release_handler` calls `heart:set_cmd/1` while preparing the reboot - which +raises, and the install fails before anything reboots. Get to this release with a +hot upgrade or a redeploy first. + Configuration is decided per version directory, so a deployment part way through this migration is coherent rather than confused: the version it is running keeps its `build.config`, and a restart back into it is still expanded the old way — by @@ -530,30 +664,25 @@ the new version reads the old file. ### Known limitations -- **A transition that restarts the emulator can be generated, but not - performed.** `mix forecastle.relup --restart` writes the relup, assembly - packages it, and `bin/castle install` handles the reply, each branch of that - handling tested against a stub - but the transition itself does not complete. - `release_handler` calls `heart:set_cmd/1` while preparing the reboot, which - fails where there is no `heart` process, so the install fails before anything - reboots; and even past that, the reboot comes back on whichever version - `releases/start_erl.data` names, which only `commit` writes, while - `release_handler` leaves the installed version in - `releases/new_start_erl.data` instead. - [castle#14](https://github.com/ausimian/castle/issues/14) and - [#10](https://github.com/ausimian/forecastle/issues/10) are what close that. - Until they land, treat a restart relup as something to generate and inspect - rather than to deploy - which is why `auto` refuses to produce one, and why - `--restart`, an explicit request, still will. +- **An emulator restart needs an external supervisor, and the release will not + restart itself.** The reboot is the point at which something outside the + release has to start it again: `bin/start` is inert on purpose, `HEART_COMMAND` + is unset, and nothing else in the release is watching. A deployment run by hand + from a shell, rather than under systemd, Docker, Kubernetes or runit, therefore + stays down after such an upgrade until somebody starts it - and the version it + comes back on is the one that was installed, because the markers are still + there waiting to be consumed. - **`restart_new_emulator` is not supported.** The two-stage transition - a hybrid temporary release, a reboot into it, and the rest of the relup applied on the way up - is refused wherever it turns up rather than generated. An ERTS change, which is what `systools` would otherwise insert it for, is taken out of - `systools`' hands and treated as a one-stage restart transition, which under - `auto` today means the generation is refused and under `--restart` means a - `restart_emulator` relup. Supporting the two-stage transition properly is its - own piece of work: the provisional boot would have to come up and *resume* an - upgrade, which is strictly more than coming up on a provisional version. + `systools`' hands and treated as a one-stage restart transition instead. + Supporting the two-stage transition properly is its own piece of work, and not + only because the provisional boot would have to come up and *resume* an upgrade: + the version `release_handler` writes into `new_start_erl.data` for it is the + temporary hybrid release, whose version directory holds a boot script and a + configuration and none of the launcher's own files, so there is nothing there + for a launcher to boot. Castle arms no marker for it for that reason. - **A system that cannot write `releases/RELEASES` cannot be upgraded.** The release creates it on its first start; where that fails — a read-only release root is the usual reason — the start warns, the system runs perfectly well, and diff --git a/lib/forecastle.ex b/lib/forecastle.ex index 44bd338..e54969b 100644 --- a/lib/forecastle.ex +++ b/lib/forecastle.ex @@ -37,6 +37,7 @@ defmodule Forecastle do def post_assemble(%Mix.Release{} = release) do release |> tap(&install_castle_cli/1) + |> tap(&install_start_program/1) |> tap(&extend_env_script/1) |> tap(©_relfile/1) |> tap(©_relup/1) @@ -98,6 +99,28 @@ defmodule Forecastle do end end + # `bin/start` is the path `release_handler` hands to `heart:set_cmd/1` while + # preparing an emulator restart, and it does nothing. The script itself says + # why; what belongs here is why it is at *this* path. + # + # `init/1` resolves the start program as `{do_check, Configured}` when + # `{sasl, start_prg}` is set and `{no_check, filename:join([Root, "bin", + # "start"])}` otherwise, and `check_start_prg/2` returns the second + # unexamined. So the default needs no configuration at all, whereas naming a + # path of our own would mean injecting `:sasl` application configuration into + # the release - which is exactly the interception + # [#6](https://github.com/ausimian/forecastle/issues/6) removed and which + # nothing here may reintroduce. `Root` is `code:root_dir()`, which for a Mix + # release that brought its own ERTS is the release root; a release that did not + # is refused by Castle's ERTS guard long before any of this. + defp install_start_program(%Mix.Release{path: path} = release) do + if unix_executables?(release) do + start = Path.join([path, "bin", "start"]) + File.write!(start, render("start.sh.eex", release)) + File.chmod!(start, 0o755) + end + end + defp extend_env_script(%Mix.Release{version_path: vp} = release) do env_sh = Path.join(vp, "env.sh") diff --git a/lib/mix/tasks/forecastle.relup.ex b/lib/mix/tasks/forecastle.relup.ex index 982b134..3553a0e 100644 --- a/lib/mix/tasks/forecastle.relup.ex +++ b/lib/mix/tasks/forecastle.relup.ex @@ -78,37 +78,22 @@ defmodule Mix.Tasks.Forecastle.Relup do could not generate is a failure, so that `auto` never silently ships something other than the upgrade it decided on; ask for the restart with `--restart`. - ### `auto` refuses a restart, for now - - Castle can install a relup that restarts the emulator but cannot yet *complete* - the transition, so while that holds `auto` refuses rather than writes a restart - transition. It exits non-zero, having written nothing, naming the edge that - forced the restart and why. `auto` is what a run with no switches gets, and a - default that quietly produces an upgrade plan which cannot be installed is - worse than one that stops and says so. - - That covers a `restart_emulator` an appup asked for by name just as much as one - `auto` chose for itself - the same transition arrived at another way. Which of - the two a run reports is decided by when it can be known: - - - An edge *classification* found to need a restart settles the run on its own, - because the outcome is a refusal whatever the rest of the relup turns out to - be. So it is refused before anything is generated, and a hot remainder - `:systools` cannot produce - a missing entry in an appup the project owns - - cannot report its own error in place of a refusal that was already known. - - An appup that asks for the restart by name is invisible until there is a - script to look at, so it is settled after generation, together with whatever - classification chose. A run therefore has one verdict, and cannot announce - an all-hot relup and a restart in the same breath. - - While this refusal stands, then, a relup with both kinds in it names the - classified edge and generates nothing: anything an appup in the hot remainder - asks for is reported by the run that follows, once that edge is gone. When the - refusal becomes an announcement the run proceeds, and both kinds are named in - the one announcement. - - This is temporary, and `--hot` and `--restart` are unaffected. `--restart` is - the deliberate override for anyone who wants the relup anyway. + ### `auto` announces a restart + + A restart transition is a legitimate outcome of `auto`, not a failure, so a run + that produces one writes the relup and says so. What it says names every edge + that will restart and why, whether `auto` classified it or an appup asked for + `restart_emulator` by name - the same transition arrived at two ways, and the + run has one verdict about it either way. + + Both kinds are settled after generation, because only one of them is knowable + before it: an appup that names the instruction is invisible until `:systools` + has produced a script. Announcing from classification alone is how a run came + to say that every transition was a hot upgrade and then report a restart in the + same breath. + + `--hot` and `--restart` remain the ways to insist. `--hot` fails on a + transition that would restart; `--restart` makes every transition one. ### `--hot` @@ -217,14 +202,13 @@ defmodule Mix.Tasks.Forecastle.Relup do # `auto` is what merges hand-written restart entries into the same relup as the # ones `:systools` generated, and it is where an edge could be dropped, or # attached to the wrong direction, or one strategy applied to the whole relup. - # While a restart transition cannot be performed `auto` refuses to emit one at - # all (see `refuse_chosen_restarts!/1`), so the merge is not reachable through - # the task - the refusal deliberately sits outside the merge rather than inside - # it, which is what keeps this callable, and is where the announcement that - # replaces it will have to go. + # It is reachable through the task now that a restart transition can be + # installed, and it stays reachable on its own: the announcement deliberately + # sits outside the merge rather than inside it, so that the merge can be driven + # without a shell and asserted on as a term. # # What `auto` settles is not settled here: this returns the merged relup, and - # nothing is said about the strategy or refused on account of it. + # nothing is said about the strategy on account of it. @spec plan_transitions!(binary(), [binary()], [binary()], [binary()], [binary()]) :: {charlist(), list(), list()} def plan_transitions!(target_path, hot_ups, hot_downs, restart_ups, restart_downs) do @@ -406,26 +390,24 @@ defmodule Mix.Tasks.Forecastle.Relup do |> refuse_hot_restarts!() end - # Two things decide what `auto` does, and they become knowable at different - # times, so they are settled at different points - in the order in which the - # answer is certain. + # Two things decide what `auto` does, and only one of them is knowable before + # the relup exists - so nothing is said until both are. # # A restart classification chose is knowable from the two `.rel` files and the - # appups, and while such a transition cannot be performed it decides the whole - # run: the outcome is a refusal whatever the hot remainder turns out to be, and - # so the refusal comes first. Generating first meant a hot remainder - # `:systools` could not produce reported that error instead, hiding a refusal - # that was already known and that fixing the error would not avoid. + # appups. An appup that asks for the emulator to be restarted by name is not: + # it is invisible until `:systools` has produced a script. Announcing after + # classification alone is how a run said every transition was a hot upgrade and + # then reported a restart in the same breath. So generation comes first and the + # two kinds are settled together, which is what makes one verdict per + # invocation possible at all. # - # Where the run may proceed - today only because classification chose nothing - - # the relup is generated before anything is said about the strategy, because - # classification is only half of what decides it: an appup can ask for the - # emulator to be restarted by name, and that is not visible until `:systools` - # has produced a script. Announcing after classification alone is how a run - # said every transition was a hot upgrade and then reported a restart in the - # same breath - and, once a restart transition can be written, would have said - # both in a run that succeeded. So the two kinds are settled together there, - # and there is one verdict per invocation either way. + # This used to refuse a classified restart *before* generating, because while + # such a transition could not be installed the classification was already the + # whole answer and a `:systools` error from the hot remainder would have stood + # in front of it. It can be installed now + # ([castle#14](https://github.com/ausimian/castle/issues/14) and + # [#10](https://github.com/ausimian/forecastle/issues/10)), so there is nothing + # to settle early and the ordering above is the only one that can be right. defp plan!(:auto, target, froms, ups, downs) do {restart_ups, hot_ups} = split_edges(:up, ups, froms, target) {restart_downs, hot_downs} = split_edges(:down, downs, froms, target) @@ -434,8 +416,6 @@ defmodule Mix.Tasks.Forecastle.Relup do label_edges("upgrade from", restart_ups, froms) ++ label_edges("downgrade to", restart_downs, froms) - refuse_chosen_restarts!(chosen) - {plan, appup_restarts} = plan_transitions( target, @@ -775,10 +755,14 @@ defmodule Mix.Tasks.Forecastle.Relup do # `{continue_after_restart, ...}`, and Castle is built for the one-stage one. # The one-stage ones are handed back, because a plain `restart_emulator` from an # appup is the transition `auto` would have generated for itself, and is - # settled together with `auto`'s own choices - announced, or refused while such - # a transition cannot be performed. Getting there by way of an appup rather - # than by classification makes no difference to whether the relup can be - # installed, and it must not make a difference to what the run says either. + # settled together with `auto`'s own choices, in the one announcement. Getting + # there by way of an appup rather than by classification makes no difference to + # the transition, and it must not make a difference to what the run says + # either. + # + # The two-stage refusal is gated on nothing and stays that way. It is not about + # whether a restart can be completed - the one-stage one can - but about which + # instruction Castle is built for, which no amount of restart support changes. defp appup_restarts!(plan) do case Enum.split_with(found_restarts(plan), &(elem(&1, 2) == :restart_new_emulator)) do {[], one_stage} -> @@ -840,87 +824,34 @@ defmodule Mix.Tasks.Forecastle.Relup do end) end - ## While a restart transition cannot be performed - - # Why `auto` refuses a restart transition at all, rather than writing one. - # - # A restart relup can be generated and packaged, but the transition it - # describes cannot yet be completed. `release_handler` calls `heart:set_cmd/1` - # while preparing the reboot - from `prepare_restart_new_emulator/7`, which - # both restart instructions go through - and that raises `badarg` where there - # is no `heart` process, so the install fails before anything reboots. Past - # that, the reboot would come back on the old permanent version anyway, because - # only `commit` writes `releases/start_erl.data`. - # [castle#14](https://github.com/ausimian/castle/issues/14) and - # [#10](https://github.com/ausimian/forecastle/issues/10) close those halves. - # - # `auto` is the strategy a run with no switches gets. Emitting a restart - # transition from it would mean a routine invocation producing an upgrade plan - # that is known not to install - worse than one that refuses and says why. So - # it refuses, naming the edge, the reason, and `--restart` as the override. - # `--hot` and `--restart` are unaffected either way: both are explicit - # requests, and it is fine for `--restart` to produce a relup that cannot yet - # be deployed. - # - # The refusal is unconditional, and there is no switch or predicate that turns - # it into an announcement: an announcement whose branches nothing reaches is a - # claim the code cannot support, and one a deployment could flip is exactly the - # uninstallable relup the refusal exists to prevent. AGENTS.md sets out what - # castle#14 has to add here instead. - - # The transitions classification chose to restart, refused before the relup is - # generated, because that is already the whole answer: nothing `:systools` goes - # on to produce for the transitions that are still hot can make this run - # succeed, so generating them can only fail in a way that reports something - # other than the reason the run is failing. - # - # It is deliberately the *only* thing settled here. An appup that asks for a - # restart by name is not knowable yet, so nothing is said about an all-hot - # classification: that is settled after generation, where it can be true. - defp refuse_chosen_restarts!([]), do: :ok - defp refuse_chosen_restarts!(chosen), do: refuse_restarts!(chosen, []) - # One verdict per invocation, arrived at once the relup exists: the transitions # `auto` classified as restarts and the emulator restarts an appup asked for by - # name are the same kind of transition, so they are settled together. Saying - # every transition is hot before the second kind has been looked for is how a - # run came to contradict itself. + # name are the same kind of transition, so they are settled together and named + # in one message. Saying every transition is hot before the second kind has been + # looked for is how a run came to contradict itself, and the all-hot line lives + # in the first clause here for exactly that reason - it is only true after + # generation. # - # While a restart transition cannot be performed, `chosen` is empty here - a - # non-empty one was refused before generation - so what this settles is the - # all-hot relup and the restart only an appup could have asked for. Both lists - # arrive together, and the second clause becomes an announcement rather than a - # refusal, only once castle#14 lands; the parameter is kept because it is what - # the verdict is about, not because anything reaches it with a value today. + # This is where `auto` refused, while a restart transition could not be + # completed. It announces now, and what it says is what the operator has to know + # to read the install back: a `restart_emulator` is replied to with + # `{ok, Vsn, Descr}`, indistinguishably from a hot upgrade, and the emulator + # then reboots. defp settle_restarts!([], []) do Mix.shell().info("auto: every transition in this relup is a hot upgrade.") end - defp settle_restarts!(chosen, found), do: refuse_restarts!(chosen, found) - - defp refuse_restarts!(chosen, found) do - Mix.raise( - describe_causes("auto would make a restart transition of ", chosen, found) <> - ". A restart transition cannot yet be performed: release_handler calls " <> - "heart:set_cmd/1 while preparing the reboot, which fails where there is no heart " <> - "process, so the install fails before rebooting - and the reboot would come back " <> - "on the old permanent version even if it did not. castle#14 and forecastle#10 " <> - "close that. Rather than write an upgrade plan that cannot be installed, auto " <> - "refuses. Pass --restart to generate it anyway, which is the deliberate override; " <> - "pass --hot to fail on the transition itself instead; or " <> - describe_remedies(chosen, found) <> "." + defp settle_restarts!(chosen, found) do + Mix.shell().info( + describe_causes("auto made a restart transition of ", chosen, found) <> + ". Each of those is a single restart_emulator, so install_release/1 replies " <> + "{ok, Vsn, Descr} rather than {continue_after_restart, Vsn, Descr} and the " <> + "emulator then reboots; the system comes back on the version that was installed, " <> + "which stays provisional until it is committed. Pass --hot to fail on such a " <> + "transition instead, or --restart to make every transition in the relup one." ) end - defp describe_remedies(chosen, found) do - [ - {chosen, "take the change that forced the restart out of this release"}, - {found, "take the restart instruction out of the appup"} - ] - |> Enum.reject(fn {causes, _remedy} -> causes == [] end) - |> Enum.map_join(" and ", fn {_causes, remedy} -> remedy end) - end - ## Writing the relup # A relup file is an encoding comment and a single term followed by a period, diff --git a/priv/env.sh.eex b/priv/env.sh.eex index b7f69c8..b18a70d 100644 --- a/priv/env.sh.eex +++ b/priv/env.sh.eex @@ -1,7 +1,9 @@ # --- Forecastle: Castle integration (generated, do not edit) --- # -# Nothing to do on a normal start. +# Three things, all of them only for the commands that start the system, and all +# of them only because release_handler's own restart plumbing assumes a classic +# OTP release rather than a Mix one. # # This fragment used to run a preboot VM on every start, daemon and eval, to # expand build.config into sys.config and to create releases/RELEASES. The @@ -10,23 +12,495 @@ # for a release Forecastle was never involved in - which is also why the # launcher's own defaults no longer have to be applied here, since it assigns # them itself before anything reads them. -# -# What is left is a one-shot bootstrap, and it has to be here rather than in -# bin/castle. release_handler reads releases/RELEASES once, in its init, and -# when the file is missing it builds a release record out of the boot script's -# name and version instead - a record with no application versions and no ERTS -# version in it. Nothing after that boot can replace it: the first operation -# that changes anything writes the in-memory record straight back over whatever -# the file holds. So before the system starts is the only moment the file can be -# created usefully, and the test below is what keeps this to the first start of -# a deployment and out of the way of every start after it. -# -# Failure is a warning rather than a refusal. The system does not need this file -# to boot, a read-only release root is an ordinary way to run one, and bin/castle -# declines to unpack or install into a system whose records say it started -# without one - which is a better place to insist than a start that used to work. case $RELEASE_COMMAND in start|start_iex|daemon|daemon_iex) + # 1. the provisional version, after a transition that restarted the emulator. + # + # release_handler writes the target to releases/new_start_erl.data and + # deliberately leaves releases/start_erl.data - which is where the stock + # launcher reads RELEASE_VSN from - naming the version that is still + # permanent. That is the rollback property: a crash before Castle.commit/1 + # brings the previous release back with nobody intervening. Only + # make_permanent/1 ever writes start_erl.data. + # + # So on the start that follows such a reboot, the version to boot is the one + # in new_start_erl.data, and selecting it is this fragment's job. Two files + # are required, not one. new_start_erl.data on its own is not evidence of + # anything: it is written *before* the reboot, nothing ever removes it, and a + # preparation that failed after writing it leaves a file naming a version + # that was never installed - release_handler reconciles the release record + # and not the marker. Castle arms releases/castle-restart-pending with the + # same version immediately before asking for such an install, and clears it + # if the install fails, so the pair is what says a reboot was really asked + # for. They have to agree on the version too. + # + # **This is first, and everything below it depends on that.** The rest of + # this fragment configures the boot, and on a start that follows a + # restart-install the version that is going to boot is not the one the + # launcher resolved before it sourced this file. So the version is settled + # here, by re-exec'ing, and every decision after it is taken once, by the + # pass that really boots, with RELEASE_VSN and REL_VSN_DIR naming the version + # it is about. + # + # It used to be the other way round, and the heart block below was wrong + # twice over for it. Pass 1 probed the *permanent* version's vm.args, found + # no -heart and exported ELIXIR_ERL_OPTIONS=-heart; pass 2 then probed the + # target's vm.args with that variable inherited, measured the two flags + # together and so declined to append a third - but nothing removes what pass 1 + # exported, so a target carrying its own -heart booted with two. That is + # init:get_argument(heart) == {ok, [[], []]}, the case_clause in heart's own + # check_start_heart/0, and a boot that hangs having printed nothing. Measuring + # twice and appending once was never a fix for that; deciding after the + # version is settled is, because there is then only one pass that decides. + # + # Settling it first also answers which emulator the probe should ask, which + # was the other half of the same defect: once the version is chosen, the + # release's own erl is that version's and there is nothing left to + # disambiguate. + # + # The pair belongs to one install attempt rather than to a version, and that + # is Castle's end of the protocol rather than this one's: it clears any + # new_start_erl.data an earlier attempt left before it arms a marker, so the + # file being here means this attempt's own preparation wrote it. What this + # side owes is only that the pair is consumed once. Only the first line of + # the marker is read, and it is the version; anything after it is Castle's + # bookkeeping and none of this script's business. + # + # **What is atomic is the claim, not the pair.** The rename below is atomic, + # so whichever start wins it is the only one that can act on the pair - but + # OTP's file is then read and removed in separate steps, and no POSIX + # operation moves two files together. The order is what makes that safe: the + # marker goes first, so an interruption anywhere after the rename leaves no + # marker, and the next start reads start_erl.data and boots the version that + # was permanent. The selection is lost, never duplicated, and never applied + # to a version nothing installed. Do not write that both files are consumed + # atomically; they are not. + # + # The claim is named per process, so two starts racing cannot read each + # other's, and anything left over - a claim from a start that died between + # the rename and the read - is named so that it cannot be mistaken for either + # marker, and is replaced rather than trusted. + # + # Nothing at all happens without the pair: no marker, or two that disagree, + # and the stock launcher goes on to read start_erl.data exactly as it does + # today. + castle_pending="$RELEASE_ROOT/releases/castle-restart-pending" + castle_provisional="$RELEASE_ROOT/releases/new_start_erl.data" + castle_claim="$RELEASE_ROOT/releases/castle-restart-consumed.$$" + + if mv "$castle_pending" "$castle_claim" 2>/dev/null; then + # **Every removal here is guarded, and the guard is not tidiness.** The + # launcher sources this file under `set -e` (bin/ sets it on its second + # line), and `rm -f` still exits non-zero for a path it cannot unlink - a + # directory at the name, most obviously, which `-f` does not cover. Unguarded, + # such a path would end the *launcher* here, having already claimed the + # pending marker with the `mv` above: no warning, no fallback, no boot. The + # service would simply stay down, which is worse than any wrong version. + # + # So a removal that fails is treated as this start having found the pair + # unsettled. It falls through to the mismatch path below, which warns and + # leaves the stock launcher to read start_erl.data - the permanent version, + # which is the safe direction and the one the rollback property rests on. The + # leftover is then something an operator can see and remove, rather than + # something that stopped the system without saying so. + castle_armed=$(head -n 1 "$castle_claim" 2>/dev/null) || castle_armed="" + rm -f "$castle_claim" 2>/dev/null || castle_armed="" + + castle_target=$(head -n 1 "$castle_provisional" 2>/dev/null | cut -d' ' -f2) || + castle_target="" + rm -f "$castle_provisional" 2>/dev/null || castle_target="" + + # A version names a directory under releases/ and is exported into the + # environment of the VM, so an empty one or one carrying a separator is + # refused rather than resolved. Held in a second variable so that the + # warning below can still say what was actually in the file. + case $castle_armed in + "" | */*) + castle_usable="" + ;; + + *) + castle_usable="$castle_armed" + ;; + esac + + # The two files have to name one version, and that version has to have the + # launcher's own furniture in it: env.sh is sourced from it on the next + # pass and start.boot is what the VM boots, so a version directory without + # them is not something to re-exec into. + if [ -n "$castle_usable" ] && [ "$castle_usable" = "$castle_target" ] && + [ -f "$RELEASE_ROOT/releases/$castle_usable/env.sh" ] && + [ -f "$RELEASE_ROOT/releases/$castle_usable/start.boot" ]; then + # Re-exec'd rather than assigned. The launcher has already resolved + # REL_VSN_DIR from the version it read out of start_erl.data by the time + # it sources this file, and every path it goes on to use - the boot + # script, vm.args, sys.config, the elixir launcher itself - hangs off + # that. Setting RELEASE_VSN here and returning would boot the old + # version's everything under the new version's name. The launcher is + # exec'd with the arguments it was given, which a sourced file still + # sees, so nothing about how it was invoked is lost. + # + # It cannot recur: the marker was claimed by the rename above, before + # anything else happened, so the next pass finds nothing to consume and + # falls through to the rest of this fragment. + # + # Nothing is exported for the next pass beyond the version, and that is + # deliberate - the heart configuration is the pass that boots to decide. + # A target release whose env.sh is not Forecastle's therefore gets no + # heart configuration from this pass either, and that is correct: it is a + # release Castle has no upgrade path into anyway. + RELEASE_VSN="$castle_usable" + export RELEASE_VSN + exec "$RELEASE_ROOT/bin/<%= release.name %>" "$@" + else + echo "warning: this system was restarted to install a release, and the" \ + "version to boot could not be settled: Castle armed" \ + "[$castle_armed] and release_handler recorded [$castle_target]." \ + "Booting the version releases/start_erl.data names instead, which" \ + "is the one that was permanent before. bin/castle releases will" \ + "say where the system got to." >&2 + fi + fi + + # 2. heart, deliberately defanged. + # + # release_handler calls heart:set_cmd/1 while preparing *any* transition that + # restarts the emulator - prepare_restart_new_emulator/7, which both restart + # instructions go through - and with no heart process that raises badarg, so + # the install fails before init:reboot(). A Mix release supervised by systemd, + # Docker or runit normally runs without -heart, so the handshake has to be + # satisfied. Running the real heart satisfies it through documented + # interfaces only. + # + # heart must then do nothing. The supervisor is the only thing that starts + # this release, and two authorities starting one service is worse than the + # problem being solved: + # + # * HEART_COMMAND is unset, so an unexpected death starts nothing. + # * HEART_NO_KILL stops heart killing a node that misses heartbeats. It + # does *not* stop heart running the command - measured on OTP 28.3 with + # the beam SIGSTOPed, where heart declined to kill it and ran the command + # anyway - which is why $RELEASE_ROOT/bin/start is inert. The two settings + # are a pair; neither is safe without the other. + # * HEART_BEAT_TIMEOUT is pushed to heart's documented maximum, which is + # what keeps heart from having any effect at all. HEART_NO_KILL is not + # enough on its own: the port program terminates once it has run the + # command, the Erlang heart process is a *kernel* process, and init halts + # the node when one of those dies - measured, "Kernel pid terminated + # (heart) ({port_terminated, ...})". So a stalled node still goes down at + # the default 60s, which is not something -heart was added here to + # introduce. + # + # All three are *assigned*, and HEART_COMMAND is unset rather than merely + # left unassigned. Defaulting them - which is what this did first - meant a + # deployment that had HEART_COMMAND in its environment, or HEART_NO_KILL set + # to anything else, or a shorter timeout, kept active watchdog behaviour + # while every document about this release said it had none. There is no + # opting out of that while using this hook: heart is here to make + # heart:set_cmd/1 return ok and for nothing else, and the supervisor owns + # restarting. + # + # But overriding is not refusing. An operator who set one of these has a + # configuration conflict, not an emergency, and turning it into a failed boot + # is worse than the conflict - so each value that is actually being overridden + # says so on standard error, naming what it was and why it stopped taking + # effect. A deployment that set none of them, which is every ordinary one, + # says nothing at all. The tests on either side of that are what hold it: + # a start with all three inherited, and a start with none. + # + # Each check is on the value rather than on the variable being set, so + # HEART_NO_KILL=TRUE displaces nothing and is not reported. + # + # **A variable that is set to nothing is a value, and the two assigned ones + # report it.** ${VAR:-default} treats set-and-empty as absent, which is what + # these were written with, so HEART_NO_KILL= and HEART_BEAT_TIMEOUT= were + # displaced in silence - neither of them is the value that gets assigned, and + # the promise is to name every value that stops taking effect. ${VAR-default}, + # without the colon, is the distinction: unset takes the default and says + # nothing, set-and-empty compares unequal and is reported as []. An empty + # HEART_COMMAND stays silent on purpose, and that is a different rule rather + # than the same one: unsetting a variable that was already empty changes + # nothing heart can read, so there is nothing an operator needs told. + # + # -heart is added only if it is not already there: two of them make + # init:get_argument(heart) answer {ok, [[], []]}, which heart's own + # check_start_heart/0 has no clause for - a case_clause at heart.erl:348 - and + # the boot hangs before anything is printed. What that guards against is a + # deployment supplying its own - in its vm.args, or in one of the variables + # erlexec reads - because this block runs once per boot: the provisional + # selection above re-execs before reaching here, so the pass that runs this is + # the pass that boots. It was not always so, and an inherited -heart of this + # fragment's own making is what the ordering above exists to prevent. + # + # **So the question is asked of erlexec rather than answered here, and that is + # the whole design of this block.** Three previous versions of it read the + # environment and modelled what the emulator would make of it, and each was + # refuted by a narrower counterexample than the last: a match bounded by + # literal spaces missed "-heart-noshell"; splitting the fields missed + # ERL_AFLAGS="'-heart'", ERL_FLAGS='"-heart"' and ERL_ZFLAGS='-he\art', all + # three of which erlexec unquotes into -heart; and a literal scan of vm.args + # missed both of those *and* a nested -args_file. Every one of those versions + # shipped with a unit test that agreed with it, because the test mirrored the + # same model. There is no reason to think the series had ended, so it is not + # continued: `erl -emu_args_exit` prints the argument vector erlexec has + # assembled and exits without starting a VM, and this looks for -heart in it. + # + # **Six sources reach that vector, and asking erlexec covers all six at once.** + # ELIXIR_ERL_OPTIONS is expanded, unquoted, by Mix's generated `elixir` on its + # way to `erl`. erlexec itself prepends ERL_OTP_FLAGS and ERL_AFLAGS and + # appends ERL_FLAGS and then ERL_ZFLAGS, applying shell-style quoting and + # backslash escaping to each. And the launcher passes --vm-args to `elixir`, + # which becomes erl's -args_file, whose contents erlexec splices in with the + # same quoting plus # comments - and follows a nested -args_file out of. Each of + # those puts -heart into init:get_argument(heart) on its own; measured, one at a + # time. The probe is given the same environment, the same unquoted + # ELIXIR_ERL_OPTIONS and the same args file, in the same order, and so it is + # not an approximation of the union - it is the union. + # + # ${RELEASE_VM_ARGS:-$REL_VSN_DIR/vm.args} is the effective args file and both + # halves matter. The launcher sets RELEASE_VM_ARGS *after* it sources this file + # - release.init.ex sources env.sh and then defaults RELEASE_VM_ARGS ten lines + # later - so the variable is set here only when the deployment or the project's + # own env.sh set it, which is exactly when the default would be wrong. + # REL_VSN_DIR is set before the source, so the fallback always resolves - and + # it resolves to the version that is going to boot, because the selection above + # has already happened. + # + # **-emu_args_exit is undocumented, and it is relied on because it fails safe + # rather than because it is guaranteed.** It is in the erlexec binary in OTP + # 27, 28 and 29 and in neither the usage string nor the documentation. If it + # ever goes away, erlexec passes it through as an ordinary flag and would start + # a VM instead of printing anything - so the probe also carries a -boot naming + # a file that cannot exist, which makes that case exit non-zero at once instead + # of booting a node or hanging, and the -root test below refuses to read an + # answer out of output that is not an argument vector. Both failures land in + # the same branch as an unreadable args file: add nothing, and say so. + # + # **It cannot hang, which is why there is no timeout.** The hazard a timeout + # would have bounded is a deployment that already carries two -heart flags of + # its own - which boots into the case_clause above, but which the probe simply + # reports as two lines and exits 0 on. Nothing here starts an emulator. + # + # **The probe always runs, and there is deliberately no gate in front of it.** + # There was one: a `case` over the four variables and a `read` loop over the + # args file, looking for `heart`, `args_file`, a quote, a backslash or a glob + # character, so that an ordinary deployment forked nothing. It was removed on + # purpose, and it should not come back. + # + # Two reasons. The gate was the last thing here that reasoned about the *text* + # of those values rather than measuring them, and its soundness rested on a + # claim about build_args_from_string() in erlexec.c removing nothing but + # quotes and backslashes - a claim about a C state machine, of exactly the + # kind that was wrong three times before this probe existed: matching -heart + # between literal spaces missed tabs and newlines, then field splitting missed + # quoting and escaping, then none of it saw an args file at all. A gate is the + # same bet with the same shape. + # + # And it left a hole it could not close. ERL_OTP_FLAGS reaches erlexec + # and carries -heart like the rest, but POSIX sh cannot enumerate variable + # *names* without forking, so a deployment setting only that one never tripped + # the gate and hung anyway. The probe sees it, because erlexec reads it. + # + # What it costs is one fork+exec of a C program that exits without starting an + # emulator - about 11ms - once per node start, since this whole fragment runs + # only for start/start_iex/daemon/daemon_iex and never for eval or rpc. That + # is nothing against a start measured in seconds, and it buys a fragment with + # no model of erlexec left in it. + castle_heart="" + castle_vm_args="${RELEASE_VM_ARGS:-$REL_VSN_DIR/vm.args}" + + # **The emulator that is asked is the one the launcher will run, and it is + # taken from the file that decides which that is.** The launcher execs + # "$REL_VSN_DIR/elixir", and the only thing in that script choosing an + # emulator is its ERTS_BIN: Mix rewrites the stock `ERTS_BIN="$ERTS_BIN"` into + # `ERTS_BIN="$SCRIPT_PATH"/../../erts-/bin/` when the release brought an + # ERTS of its own, and leaves it alone when it did not - in which case ERTS_BIN + # is empty, `erl` comes off PATH, and that shape has to keep working. That + # script's SCRIPT_PATH is the directory it is in, which is REL_VSN_DIR, so + # reading the assignment out of the file and resolving it against REL_VSN_DIR + # names the binary the exec will reach rather than a binary like it. This is + # not a model of the launcher's resolution: the last ERTS_BIN= line in that + # file *is* the resolution, and this reads it out of the same file the exec + # will. Anything that does not come out executable falls back to `erl`, which + # is what an un-rewritten assignment means anyway - and is also the answer if + # the version directory is itself a symlink, since the launcher resolves + # SCRIPT_PATH through readlink and this does not. + # + # It used to expand a glob over the erts-* directories in the release root and + # keep the last executable it found, and that is wrong as soon as the root + # holds more than one of them - which is what unpacking an ERTS-changing + # release leaves behind. The last one a glob yields is the lexicographically + # last, which is not even the newest: erts-9.9 sorts after erts-16.2. And the + # glob's spelling is deliberately not repeated here, so that the assembly suite + # can refute it in the shipped file without matching this paragraph. + # + # ERL_OTP_FLAGS is named for the OTP version of whichever binary + # answers, and an args file may be written for one emulator and not another, so + # asking the wrong one is not a near miss - it is a question about a different + # deployment. + castle_erl="erl" + castle_erts_bin=$(sed -n 's|^ERTS_BIN="$SCRIPT_PATH"||p' \ + "$REL_VSN_DIR/elixir" 2>/dev/null) || + castle_erts_bin="" + + if [ -n "$castle_erts_bin" ] && [ -x "$REL_VSN_DIR${castle_erts_bin}erl" ]; then + castle_erl="$REL_VSN_DIR${castle_erts_bin}erl" + fi + + # ELIXIR_ERL_OPTIONS is expanded *unquoted* and before -args_file, because + # that is what Mix's `elixir` does with it - `set -- "$ERTS_BIN$ERL_EXEC" + # -noshell $ELIXIR_ERL_OPTIONS $ERL "$@"`, with the --vm-args pair at the + # front of "$@" - so this is the launcher's own expansion rather than a + # rendering of it, down to an -extra in that variable swallowing the args + # file exactly as it would on the real start. + # + # -emu_args_exit comes first so that such an -extra cannot hide it, and the + # -boot comes second for the same reason plus one of its own: init takes the + # *first* -boot it is given - measured, against a second one naming a script + # that does exist - so nothing in the args file can make this probe boot. + # ERL_CRASH_DUMP_SECONDS=0 is what keeps the degraded case from dropping an + # erl_crash.dump in whatever directory the operator started the release + # from. Standard input is /dev/null and standard error is discarded: this is + # a question asked on the way to a boot, and erlexec's own diagnostics about + # the args file belong to the launcher's attempt rather than to this one. + # -args_file is passed only when the path is there. That is not a return of + # the gate: the gate judged the *contents* of values, which is the modelling + # this replaced, whereas this is a fact about the filesystem. erlexec refuses + # an args file it cannot open and exits non-zero, and this keeps that from + # being read as "the measurement is impossible, so add nothing". + # + # **It is defensive rather than load-bearing, and the distinction is worth + # keeping straight.** An earlier version of this comment called a release with + # no vm.args "a supported shape". It is not one that a stock build produces: + # `mix release` always renders rel/vm.args.eex into releases//vm.args, so + # the launcher's own ${RELEASE_VM_ARGS:-...} default always resolves to a file + # that exists. What is reachable is a deployment exporting RELEASE_VM_ARGS to a + # path that is not there, or a hand-deleted vm.args - and both are starts the + # launcher fails on moments later with a message of erlexec's own. So this + # costs nothing and removes a confusing warning from a start that was already + # doomed; it is not holding up the common case. + # + # Every variable erlexec reads still reaches the probe - the four above and + # ERL_OTP_FLAGS, none of which this fragment has to name - so nothing + # goes unmeasured; only a file that is not there goes unmentioned. A path that + # exists and cannot be read is still handed over, and is still reported, + # because that one is worth knowing about. A dangling symlink is -e-false and + # skipped, which is the same case: the start fails either way. + castle_args_file="" + if [ -e "$castle_vm_args" ]; then + castle_args_file="$castle_vm_args" + fi + + castle_argv=$(ERL_CRASH_DUMP_SECONDS=0 "$castle_erl" \ + -emu_args_exit -noinput \ + -boot /nonexistent/castle-heart-probe \ + ${ELIXIR_ERL_OPTIONS-} \ + ${castle_args_file:+-args_file "$castle_args_file"} \ + 2>/dev/null &2 + fi + + if [ -z "$castle_heart" ]; then + ELIXIR_ERL_OPTIONS="${ELIXIR_ERL_OPTIONS:+$ELIXIR_ERL_OPTIONS }-heart" + export ELIXIR_ERL_OPTIONS + fi + + if [ -n "${HEART_COMMAND-}" ]; then + echo "warning: unsetting HEART_COMMAND=[$HEART_COMMAND] for this start." \ + "This release runs OTP's heart only so that release_handler's" \ + "heart:set_cmd/1 succeeds while it prepares an emulator restart;" \ + "restarting the system belongs to whatever supervises it. Because" \ + "HEART_NO_KILL is set, heart would run that command beside a node it" \ + "had been told not to kill. Remove HEART_COMMAND from the" \ + "environment to silence this." >&2 + fi + unset HEART_COMMAND + + if [ "${HEART_NO_KILL-TRUE}" != "TRUE" ]; then + echo "warning: overriding HEART_NO_KILL=[$HEART_NO_KILL] with TRUE for" \ + "this start. heart is here to satisfy a handshake rather than to" \ + "watch anything, and killing a node that missed a heartbeat is a" \ + "failure mode a release without -heart does not have. Remove" \ + "HEART_NO_KILL from the environment to silence this." >&2 + fi + HEART_NO_KILL=TRUE + export HEART_NO_KILL + + if [ "${HEART_BEAT_TIMEOUT-65535}" != "65535" ]; then + echo "warning: overriding HEART_BEAT_TIMEOUT=[$HEART_BEAT_TIMEOUT] with" \ + "65535, heart's documented maximum, for this start. HEART_NO_KILL" \ + "does not make a heart-beat time-out harmless: heart is a kernel" \ + "process, so init halts the node when its port program exits. A" \ + "shorter timeout is therefore a way for a stalled node to be killed" \ + "that this release does not otherwise have. Remove" \ + "HEART_BEAT_TIMEOUT from the environment to silence this." >&2 + fi + HEART_BEAT_TIMEOUT=65535 + export HEART_BEAT_TIMEOUT + + # 3. releases/RELEASES, once, before the system starts. + # + # It has to be here rather than in bin/castle. release_handler reads the file + # once, in its init, and when it is missing it builds a release record out of + # the boot script's name and version instead - a record with no application + # versions and no ERTS version in it. Nothing after that boot can replace it: + # the first operation that changes anything writes the in-memory record + # straight back over whatever the file holds. So before the system starts is + # the only moment the file can be created usefully, and the test below is + # what keeps this to the first start of a deployment and out of the way of + # every start after it. + # + # Failure is a warning rather than a refusal. The system does not need this + # file to boot, a read-only release root is an ordinary way to run one, and + # bin/castle declines to unpack or install into a system whose records say it + # started without one - which is a better place to insist than a start that + # used to work. if [ ! -f "$RELEASE_ROOT/releases/RELEASES" ]; then # Nothing about the release reaches the expression: no path is # interpolated into it and no directory is changed to put the file where diff --git a/priv/start.sh.eex b/priv/start.sh.eex new file mode 100644 index 0000000..bd8162e --- /dev/null +++ b/priv/start.sh.eex @@ -0,0 +1,22 @@ +#!/bin/sh +# Castle releases are externally supervised: the supervisor owns restarting, and +# it is the only thing that starts this release. This script exists so that +# nothing else does. +# +# release_handler composes "$ROOT/bin/start " and installs it as +# heart's temporary reboot command while preparing any transition that restarts +# the emulator (prepare_restart_new_emulator/7, which both restart instructions +# go through). $ROOT is code:root_dir(), which on a Mix release that brought its +# own ERTS is the release root, and the path is the unchecked default - +# {no_check, filename:join([Root, "bin", "start"])} - so nothing configures it +# and nothing verifies it either. +# +# heart really does run it. Measured on OTP 28.3: on init:reboot() with +# HEART_COMMAND unset, and again on a heart-beat time-out, where HEART_NO_KILL +# means the old VM is still alive. Starting the release here would therefore risk +# two live nodes, which is exactly what setting HEART_NO_KILL asks for. Doing +# nothing is the whole point. +# +# The version to boot is selected by the env.sh fragment on the next start, from +# the pair of markers release_handler and Castle leave behind. +exit 0 diff --git a/test/fixtures/sample/mix.exs b/test/fixtures/sample/mix.exs index 84566c3..a251892 100644 --- a/test/fixtures/sample/mix.exs +++ b/test/fixtures/sample/mix.exs @@ -26,7 +26,12 @@ defmodule Sample.MixProject do [ # TEMPORARY for the 1.0.0 cycle: the fixture needs Castle's in-progress API. # Flip back to {:castle, "~> 1.0"} before publishing. - {:castle, github: "ausimian/castle", branch: "release/1.0.0"}, + # + # Pinned to Castle's issue branch while castle#14 and #10 are open: the two + # halves of an emulator-restart upgrade are useless apart, so the fixture + # has to resolve against the Castle side of it. Back to release/1.0.0 once + # both have merged. + {:castle, github: "ausimian/castle", branch: "issue/14-restart"}, {:forecastle, path: System.get_env("FORECASTLE_PATH", @forecastle), override: true}, # An application the relup never mentions, versioned in step with this # one, so that "release_handler knows its version changed" is observable diff --git a/test/fixtures/sample/rel/vm.args.eex b/test/fixtures/sample/rel/vm.args.eex index 89ef05a..5dd6746 100644 --- a/test/fixtures/sample/rel/vm.args.eex +++ b/test/fixtures/sample/rel/vm.args.eex @@ -1,3 +1,44 @@ ## Distribution without epmd, so that the test suite needs no epmd daemon. -start_epmd false -erl_epmd_port <%= System.get_env("SAMPLE_EPMD_PORT", "24601") %> + +## And, for the suites that ask for it at assembly time, a -heart. It is the e2e +## half of a guard test rather than something this fixture needs. +## +## The launcher passes this file to the emulator as -args_file, so this is a real +## source of -heart alongside ELIXIR_ERL_OPTIONS and the flag variables erlexec +## reads. A fragment that misses it appends a second flag, +## init:get_argument(heart) answers {ok, [[], []]}, heart's own +## check_start_heart/0 has no clause for that, and the boot hangs having printed +## nothing - so every e2e start of such a release would time out rather than fail. +## +## **`Forecastle.RestartUpgradeTest` asks for it spelled `-he\art`, and that is +## the point of it.** erlexec's args-file parser removes the backslash and hands +## the emulator -heart; measured, and `init:get_argument(heart)` on the booted node +## is `{ok, [[]]}`, which that suite asserts. So the only way for the env.sh +## fragment to see this flag is to ask erlexec what the argument vector came out +## as. Every reading of the environment that castle#14/forecastle#10 tried before +## that - a match between spaces, a split into fields, a literal scan of this +## file - passes straight over it, appends a second, and hangs the boot. +## +## **It is opt-in rather than always here because -heart is not quiet.** heart +## prints `heart_beat_kill_pid = ` on every VM started with this args file and +## a pair of lines on the way out, and `eval` uses this file too - so a -heart here +## by default lands in the middle of the output `Forecastle.ConfigurationTest` +## asserts on exactly. A suite that wants one passes it to `assemble!/1`. +## +## One consequence worth stating rather than leaving to be rediscovered: every +## start of every release assembled from this fixture asks the emulator once, +## whether or not SAMPLE_HEART was set, because the fragment has no gate in front +## of its probe. That is one fork of erlexec, which exits without starting a VM. +## There used to be a gate, and these comments saying "heart" is what kept it from +## ruling this file out; it was removed because it reasoned about the text of +## values instead of measuring them, and because it could not see +## ERL_OTP_FLAGS at all. So these comments no longer have that job, and the +## backslash in the flag below still has the one stated above: erlexec unescapes +## it, and nothing reading this file can. +## +## Which means: do not "tidy" the spelling into a plain -heart, and do not add +## -heart to ELIXIR_ERL_OPTIONS or one of the ERL_*FLAGS in a suite that boots a +## release assembled with this, because two of them is the hang. +<%= System.get_env("SAMPLE_HEART", "") %> diff --git a/test/forecastle/assembly_test.exs b/test/forecastle/assembly_test.exs index 3f2bfba..bf874f9 100644 --- a/test/forecastle/assembly_test.exs +++ b/test/forecastle/assembly_test.exs @@ -67,6 +67,49 @@ defmodule Forecastle.AssemblyTest do end end + describe "bin/start" do + test "is installed, executable, and does nothing", %{forecastle: forecastle} do + # The path release_handler composes into heart's temporary reboot command, + # and returns unexamined: init/1 yields {no_check, $ROOT/bin/start} when + # {sasl, start_prg} is unset, and check_start_prg/2 does not look at it. So + # it has to exist, and it has to do nothing - heart really does run it, both + # on init:reboot() and on a heart-beat time-out, where HEART_NO_KILL means + # the old VM is still alive. + start = Path.join(forecastle, "bin/start") + + assert File.exists?(start) + assert Bitwise.band(File.stat!(start).mode, 0o111) != 0 + assert {"", 0} = System.cmd(start, ["releases/new_start_erl.data"], cd: forecastle) + end + + test "starts nothing", %{forecastle: forecastle} do + # Asserted on the script rather than on its behaviour, because "it did not + # start the release" is not observable from a single run of something that + # exits 0. What must never appear here is an invocation of the launcher: a + # bin/start that started the release would be a second restart authority + # beside the supervisor, and with HEART_NO_KILL it could start one node + # beside another that is still alive. + start = File.read!(Path.join(forecastle, "bin/start")) + + refute start =~ "bin/sample" + refute start =~ "exec" + assert start =~ "exit 0" + end + + test "is not installed by a plain Mix release", %{mix: mix} do + refute File.exists?(Path.join(mix, "bin/start")) + end + + test "ships in the release tarball", %{forecastle: forecastle} do + # Which is what puts it into a deployment that migrates by hot upgrade: + # release_handler extracts with keep_old_files, so a new file appears. + tarball = Path.join(forecastle, "sample-#{@vsn}.tar.gz") + + assert {:ok, entries} = :erl_tar.table(to_charlist(tarball), [:compressed]) + assert ~c"bin/start" in entries + end + end + describe "env.sh" do setup %{forecastle: forecastle} do {:ok, env_sh: File.read!(Path.join(forecastle, "releases/#{@vsn}/env.sh"))} @@ -115,16 +158,272 @@ defmodule Forecastle.AssemblyTest do test "only runs for the commands that start the system", %{env_sh: env_sh} do # Not eval, which the configuration expansion needed and this does not: an - # eval VM manages no releases. + # eval VM manages no releases - and, since #10, must not consume the + # provisional marker or start a heart process either. bin/castle drives + # every command through `rpc`, so a fragment that ran for those would + # consume the marker while an install was still waiting for the reboot. assert env_sh =~ ~r/case \$RELEASE_COMMAND in\n\s+start\|start_iex\|daemon\|daemon_iex\)/ end + test "runs heart, and assigns the whole of its configuration", %{env_sh: env_sh} do + # heart exists only so that heart:set_cmd/1 returns ok instead of raising + # badarg while release_handler prepares a reboot. Everything else about it + # is switched off: no HEART_COMMAND, no kill, and a beat timeout at the + # documented maximum, because HEART_NO_KILL does not make a time-out + # harmless - the port program exits after running its command, heart is a + # kernel process, and init halts the node when one of those dies. + # + # **What the fragment does to the environment is asserted in + # `Forecastle.EnvScriptTest`, by running it.** This test used to be the + # whole of the coverage and it asserted that the `${VAR:-default}` + # expressions were *present* - which is exactly how a deployment with + # HEART_COMMAND in its environment kept an active watchdog while this + # passed. What is left here is the assembly-level claim: the heart + # configuration is in the release's own env.sh, and it assigns. + assert env_sh =~ ~s(ELIXIR_ERL_OPTIONS:+$ELIXIR_ERL_OPTIONS }-heart) + assert env_sh =~ "unset HEART_COMMAND" + assert env_sh =~ "HEART_NO_KILL=TRUE" + assert env_sh =~ "HEART_BEAT_TIMEOUT=65535" + + # And the defect refuted by shape: a default is a value a deployment may + # override, and there is nothing here it may override. Mix ships the + # HEART_COMMAND assignment commented out in its own generated env.sh, + # pointing at the launcher, which is the one thing that must never be + # assigned here. + refute env_sh =~ ~s(HEART_NO_KILL="${HEART_NO_KILL:-) + refute env_sh =~ ~s(HEART_BEAT_TIMEOUT="${HEART_BEAT_TIMEOUT:-) + refute env_sh =~ ~s(HEART_COMMAND="$) + refute env_sh =~ "export HEART_COMMAND" + + # Nor may either of them be *read* with a colon, anywhere. The two are + # compared against the value that replaces them so that a deployment which + # already agrees is not warned at, and `${VAR:-default}` cannot make that + # comparison: it treats a variable set to nothing as absent, so an empty + # value is displaced in silence. `${VAR-default}` is the form that tells + # unset from set-and-empty. + refute env_sh =~ ~s(${HEART_NO_KILL:-) + refute env_sh =~ ~s(${HEART_BEAT_TIMEOUT:-) + end + + test "adds -heart only once", %{env_sh: env_sh} do + # Load bearing rather than hygiene: two -heart flags make + # init:get_argument(heart) answer {ok, [[], []]}, which heart's own startup + # check has no clause for, and the boot hangs with nothing printed. What the + # guard is for is a deployment supplying its own flag - in its vm.args, or in + # one of the variables erlexec reads. The fragment's *own* flag used to be a + # second source of one, because the heart block ran ahead of the provisional + # selection and so on both passes of a re-exec; the order that stops that is + # asserted below. + # + # The assembly-level claim is that the guard in the shipped env.sh *asks the + # emulator* rather than reading the environment and deciding for itself, and + # that it asks with everything the start will be given. Which values it + # recognises is `Forecastle.EnvScriptTest`'s, by running it against a real + # erlexec, and whether a real boot survives an inherited flag is + # `Forecastle.RestartUpgradeTest`'s. + # + # `erl -emu_args_exit` prints the argument vector erlexec assembled and exits + # without starting a VM, so it covers all six sources at once - the command + # line, ERL_OTP_FLAGS, ERL_AFLAGS, ERL_FLAGS, ERL_ZFLAGS and every + # -args_file followed out of vm.args - with erlexec's own quoting, escaping + # and comment handling. + # + # And that it reads that answer with the boundary `init` applies to it: + # everything after -extra is the application's arguments rather than emulator + # flags, so a plain argument spelled -heart is printed by -emu_args_exit and + # is not a flag. Counting it would suppress the one that should be added, + # which fails the opposite way to a duplicate - the release boots happily + # without heart and the damage surfaces at heart:set_cmd/1 much later. + assert env_sh =~ "-emu_args_exit" + assert env_sh =~ "/^-extra$/ { exit 1 }" + assert env_sh =~ "/^-heart$/ { found = 1; exit 0 }" + + # Asked of the emulator the launcher will run, and told which that is by the + # file that decides it: the launcher execs `$REL_VSN_DIR/elixir`, and the + # ERTS_BIN line Mix rewrites in that script is the only thing in it choosing + # an emulator. Reading the assignment out of the same file the exec will read + # it out of is not a model of the resolution - it is the resolution. + # ERL_OTP_FLAGS is named for the answering binary's OTP version, so + # asking a different one would be asking about a different deployment. + assert env_sh =~ ~s(sed -n 's|^ERTS_BIN="$SCRIPT_PATH"||p') + assert env_sh =~ ~s("$REL_VSN_DIR/elixir" 2>/dev/null) + assert env_sh =~ ~s([ -x "$REL_VSN_DIR${castle_erts_bin}erl" ]) + + # And it used to glob the release root for erts-* and keep the last match, + # which is the lexicographically last and so not even the newest. That is + # refuted rather than merely superseded: a root holds several erts-* the + # moment an ERTS-changing release is unpacked, and it reads as the obvious + # spelling. Both the loop and the pattern, because either alone would be a + # partial revert. The prose in the fragment deliberately does not repeat the + # pattern, so this refutation is about the code. + refute env_sh =~ "for castle_candidate in" + refute env_sh =~ ~s|"$RELEASE_ROOT"/erts-*/bin/erl| + + # `erl` from PATH is the fallback, which is what a release built with + # `include_erts: false` needs: Mix leaves ERTS_BIN empty there, so the + # launcher runs PATH's emulator and the probe has to as well. + assert env_sh =~ ~s(castle_erl="erl") + + # And asked with what the start will be given. ELIXIR_ERL_OPTIONS is expanded + # *unquoted* and before -args_file, because that is what Mix's `elixir` does + # with it, so an -extra in that variable swallows the args file here exactly + # as it would on the boot. + assert env_sh =~ "${ELIXIR_ERL_OPTIONS-} \\\n" + refute env_sh =~ ~s("${ELIXIR_ERL_OPTIONS-}" \\\n) + assert env_sh =~ ~s(${castle_args_file:+-args_file "$castle_args_file"}) + + # The args file is the effective one, and the default is spelled with + # REL_VSN_DIR because the launcher does not set RELEASE_VM_ARGS until *after* + # it has sourced this file. + assert env_sh =~ ~s(${RELEASE_VM_ARGS:-$REL_VSN_DIR/vm.args}) + + # The probe must never boot anything, and the -boot it carries is what makes + # that true even if -emu_args_exit - which is undocumented - ever stops being + # recognised: erlexec would pass it through, and the emulator would then fail + # on a boot file that cannot exist instead of starting a node. The -root test + # is the other half, refusing to read an answer out of output that is not an + # argument vector. + assert env_sh =~ "-boot /nonexistent/castle-heart-probe" + assert env_sh =~ ~s(grep -q '^-root$') + assert env_sh =~ "ERL_CRASH_DUMP_SECONDS=0" + + # And there is nothing in front of it. The fragment used to gate the probe on + # a `case` over the four flag variables and a `read` loop over the args file, + # so that an ordinary start forked nothing; that is refuted here rather than + # merely absent, because it reads as the obvious optimisation and would come + # back. It judged the *text* of those values - the modelling asking erlexec + # replaced - and it could not see ERL_OTP_FLAGS at all, since POSIX sh + # cannot enumerate variable names without a fork of its own. What the probe + # costs, and that it really does run on an ordinary start, is + # `Forecastle.EnvScriptTest`'s; that it has no gate to run behind is here. + refute env_sh =~ "castle_ask" + refute env_sh =~ ~S(*heart* | *args_file*) + refute env_sh =~ ~s(while IFS= read -r castle_line) + + # The one condition that is left, and it is about the filesystem rather than + # about contents: erlexec refuses an args file it cannot open, so a release + # that ships no vm.args is asked about without one instead of being reported + # unmeasurable. The `-e` test is deliberately not an `-f`/`-r` pair - anything + # that exists is handed over and answered for. + assert env_sh =~ ~s|if [ -e "$castle_vm_args" ]; then| + + # No timeout, no polling, no killing: the probe cannot hang, because it + # starts no emulator. A deployment that already carries two -heart flags - + # which is the boot this whole block exists to prevent - makes it print two + # lines and exit 0. + refute env_sh =~ "sleep" + + # And the three shapes this has been wrong in, refuted rather than merely + # superseded, because each of them reads as the obvious spelling and each + # shipped with a test in this repository that agreed with it. + # + # A match bounded by literal spaces: the launcher expands these variables + # unquoted, so tabs and newlines separate fields as surely as spaces do. + refute env_sh =~ ~s(case " ${ELIXIR_ERL_OPTIONS:-} " in) + refute env_sh =~ ~s(*" -heart "*) + + # A split into fields, compared against the literal flag: erlexec unquotes + # what it reads out of the environment, so `'-heart'`, `"-heart"` and + # `-he\art` are all -heart to the emulator and none of them to that. + refute env_sh =~ ~s|[ "$castle_opt" = "-heart" ]| + refute env_sh =~ ~s(for castle_opt in ${ELIXIR_ERL_OPTIONS:-}; do) + + # And a literal scan of the args file, which misses the same escapes, misses + # a nested -args_file, and cannot tell a commented flag from a live one + # without a comment rule erlexec does not share. + refute env_sh =~ ~s|for castle_opt in $(cat "$castle_vm_args")| + refute env_sh =~ "sed 's/#" + end + + test "selects a provisional version from two markers, and consumes them", + %{env_sh: env_sh} do + # new_start_erl.data is written before the reboot and never removed, so on + # its own it is not evidence that a reboot was asked for. Castle's marker is + # the other half, and the two have to name one version. + # + # Which selection each state produces is asserted in + # `Forecastle.EnvScriptTest`, by running the fragment over a release-shaped + # directory; what is here is that the two names are the ones Castle and + # release_handler write, and that the marker is claimed by rename. + assert env_sh =~ "releases/castle-restart-pending" + assert env_sh =~ "releases/new_start_erl.data" + assert env_sh =~ ~s(mv "$castle_pending" "$castle_claim") + assert env_sh =~ ~s([ "$castle_usable" = "$castle_target" ]) + assert env_sh =~ ~s(rm -f "$castle_provisional") + + # The claim is per process, so two starts racing for the marker cannot read + # each other's - only one of them wins the rename, and the loser must not + # find a file the winner is still working through. + assert env_sh =~ ~s(castle-restart-consumed.$$) + end + + test "refuses a version that could name something other than a release", + %{env_sh: env_sh} do + # The version comes out of a file and is used to build a path and exported + # into the VM's environment, so an empty one or one carrying a separator is + # refused rather than resolved. Kept apart from the value that was read, so + # that the warning can still say what was actually in the file. + assert env_sh =~ ~s(case $castle_armed in) + assert env_sh =~ ~s("" | */*) + assert env_sh =~ ~s([ -f "$RELEASE_ROOT/releases/$castle_usable/env.sh" ]) + assert env_sh =~ ~s([ -f "$RELEASE_ROOT/releases/$castle_usable/start.boot" ]) + end + + test "re-execs the launcher rather than assigning the version", %{env_sh: env_sh} do + # The launcher has already resolved REL_VSN_DIR by the time it sources this, + # and vm.args, sys.config, the boot script and the elixir launcher all hang + # off it. Assigning RELEASE_VSN here and returning boots the old version's + # everything under the new version's name. + assert env_sh =~ ~s(exec "$RELEASE_ROOT/bin/sample" "$@") + + # And the consumption comes first, which is what stops the second pass + # recurring. + claim = :binary.match(env_sh, ~s(mv "$castle_pending")) |> elem(0) + reexec = :binary.match(env_sh, ~s(exec "$RELEASE_ROOT/bin/sample")) |> elem(0) + + assert claim < reexec + end + + test "settles the version before it configures the boot", %{env_sh: env_sh} do + # The re-exec means the fragment is read again, so anything it decides ahead + # of the selection it decides about the version being *replaced*. heart is + # the decision that made that fatal: the earlier order probed the permanent + # version's args, found no flag and exported ELIXIR_ERL_OPTIONS=-heart, and + # the second pass then counted that beside the target's own. Two flags is + # init:get_argument(heart) == {ok, [[], []]}, which heart's startup check has + # no clause for, and a boot that hangs having printed nothing. + # + # The same order is what makes the probe's emulator unambiguous, since the + # release whose `elixir` names it is the selected one. + # + # Which flag count each order produces is `Forecastle.EnvScriptTest`'s, over + # a release-shaped directory whose two versions carry different vm.args - the + # case that suite did not have, which is why this shipped. What is asserted + # here is the order itself, in the file that ships, and against the code + # rather than the prose either side of it. + reexec = :binary.match(env_sh, ~s(exec "$RELEASE_ROOT/bin/sample" "$@")) |> elem(0) + probe = :binary.match(env_sh, "castle_argv=$(ERL_CRASH_DUMP_SECONDS=0") |> elem(0) + + heart = + :binary.match(env_sh, ~s(ELIXIR_ERL_OPTIONS:+$ELIXIR_ERL_OPTIONS }-heart)) |> elem(0) + + assert reexec < probe + assert reexec < heart + end + test "warns rather than refuses when it cannot create the file", %{env_sh: env_sh} do # A system does not need the file in order to boot, and a release root # nothing may write to is an ordinary way to run one. bin/castle is where # the consequence is refused, not the start. + # + # The refutation is anchored to a *shell* exit - a line whose whole content + # is `exit 1` - rather than to the substring. A bare `=~ "exit 1"` also + # matched `{ exit 1 }` inside the awk program that reads the probe's output, + # where it ends an awk pass and has nothing to do with the launcher. What + # this test is about is the fragment never taking the start down, so it has + # to say that rather than something that happens to be spelled like it. assert env_sh =~ "warning: could not create" - refute env_sh =~ "exit 1" + refute env_sh =~ ~r/^\s*exit 1\s*$/m end test "comes after the project's own customization", %{env_sh: env_sh} do diff --git a/test/forecastle/castle_cli_test.exs b/test/forecastle/castle_cli_test.exs index 8826644..92e0e1a 100644 --- a/test/forecastle/castle_cli_test.exs +++ b/test/forecastle/castle_cli_test.exs @@ -32,8 +32,35 @@ defmodule Forecastle.CastleCliTest do end setup %{release: release} do - root = Path.join(System.tmp_dir!(), "castle-cli-#{System.unique_integer([:positive])}") + # The operating system pid as well as a unique integer, and the root removed + # before it is created as well as after. + # + # `System.unique_integer/1` is unique within a *VM run* and no further than + # that: a fresh VM starts its counter at very nearly the same place as the + # last one - measured, three consecutive runs handing out 2690, 2693 and + # 2694 - so the names recur across runs. And `File.mkdir_p!/1` succeeds on a + # directory that is already there. So a run killed part-way - a timeout, a + # signal, an interrupted matrix cell - leaves its `argv` behind, and a later + # run's `setup` can adopt it. Eleven assertions in this file read + # `File.exists?(record)` to say the launcher was *not* reached, and a stale + # file answers yes. + # + # It does not reproduce on demand, because it needs the earlier run to have + # died *and* the counter to land on the same value - which is what makes it + # worth removing by construction rather than chasing. + # + # `on_exit` cannot cover this, since the case being defended against is + # precisely the one where `on_exit` did not run. The `rm_rf!` is what makes + # the directory this run's own; the pid is what stops two VMs running at once + # from choosing the same name and deleting each other's. + root = + Path.join( + System.tmp_dir!(), + "castle-cli-#{System.pid()}-#{System.unique_integer([:positive])}" + ) + bin = Path.join(root, "bin") + File.rm_rf!(root) File.mkdir_p!(bin) on_exit(fn -> File.rm_rf!(root) end) diff --git a/test/forecastle/env_script_test.exs b/test/forecastle/env_script_test.exs new file mode 100644 index 0000000..2ab52a8 --- /dev/null +++ b/test/forecastle/env_script_test.exs @@ -0,0 +1,1239 @@ +defmodule Forecastle.EnvScriptTest do + @moduledoc """ + Runs the `env.sh` fragment as a shell script, in a directory laid out like a + release, and asserts on what it *did*. + + This is the companion to the `env.sh` assertions in `Forecastle.AssemblyTest`, + and the division between them is deliberate. That suite says the fragment is + appended to the release's own `env.sh` and that it is gated on a command which + starts the system - facts about assembly, which need a real release. This one + says what the fragment does when it runs, which nothing about the text of it + can establish. + + That distinction is not academic. The heart configuration was first written as + a set of `${VAR:-default}` expressions, and the test that covered it asserted + those expressions were *present* - so it passed while a deployment that had + `HEART_COMMAND` in its environment kept an active watchdog, which is the one + thing this release says it has not got. What the fragment has to be held to is + the environment it leaves behind, and the only way to see that is to source it + and look. + + The sandbox is a release-shaped directory with a stub `bin/sample`, because the + fragment's other half *execs* the launcher and what it hands over - the + version, and that it handed over at all - is the whole of what there is to + observe. `Forecastle.CastleCliTest` drives `bin/castle` against a launcher stub + for the same reason. + + The fragment is rendered from `priv/env.sh.eex` rather than read out of an + assembled release, so these tests need no `mix release` and stay async; + `Forecastle.AssemblyTest` is what pins that this template is what ships. + + Two things about the sandbox exist for the heart guard in particular, and both + are answers to the same three-times-repeated defect - a guard that modelled what + `erlexec` would make of the environment, and a test that mirrored the model. + + The flag count is obtained by asking a real emulator with `erl -emu_args_exit`, + which prints the argument vector `erlexec` assembled and exits without starting + a VM. Nothing here parses a variable or an args file. See `reporting/0`. + + And the sandbox holds a release's own `erts-16.2/bin/erl` - a script that records + every invocation and then hands over to the real emulator - named by an `elixir` + carrying the one line of Mix's generated launcher that decides which emulator it + runs. That is how the fragment resolves its probe, so `probes/2` is how a test + sees whether the fragment asked *and what it asked with*, neither of which the + resulting environment says. It asks on every start, deliberately - there used to + be a gate, and the cases below that changed when it went say so one at a time; + and `legacy_erl/1` replaces it with one that strips `-emu_args_exit`, which is + the only way to exercise what the fragment does when an emulator does not know + the flag it relies on. A second emulator installed under another `erts-*` writes + to its own log, which is the only way to see *which* one answered. + + The stub `bin/sample` sources the fragment again, as the real launcher sources + `env.sh` again after the re-exec, so a provisional start is observed as two + passes rather than as the fact of one. That matters more than it sounds: the + version is settled before anything else is decided, so on such a start every + decision the fragment makes belongs to the second pass. + """ + + use ExUnit.Case, async: true + + @moduletag :tmp_dir + + @vsn "0.1.0" + @next "0.1.1" + + # What the stub launcher prints when the fragment execs it. Nothing else in the + # fragment writes to standard output, so its presence is the discriminator + # between a start that selected a provisional version and one that did not. + @exec "exec:" + + # The emulator of the VM these tests run in, by the path `code:root_dir()` + # names rather than by a PATH lookup, so that both the fragment's probe and this + # suite's own reporting ask exactly one binary. + @erl Path.join([to_string(:code.root_dir()), "bin", "erl"]) + + # The ERTS directory the sandbox's releases name, as a release that brought its + # own has one. Its version is not the running emulator's and does not have to + # be: nothing here reads a version out of the directory name, which is the + # point - the fragment is told which directory by the release's own `elixir` + # rather than left to work it out from the root. + @erts "erts-16.2" + + setup %{tmp_dir: root} do + File.mkdir_p!(Path.join(root, "bin")) + + # Two version directories with the launcher's own furniture in them. The + # fragment refuses to exec into a version that has no `env.sh` and no + # `start.boot`, because the next pass sources the first and the VM boots the + # second. + for vsn <- [@vsn, @next] do + dir = Path.join([root, "releases", vsn]) + File.mkdir_p!(dir) + File.write!(Path.join(dir, "env.sh"), "# nothing\n") + File.write!(Path.join(dir, "start.boot"), "") + + # The launcher passes this file to the emulator as -args_file, so it is a + # source of -heart exactly as the flag variables are. This one is the + # ordinary deployment's: inert, so the probe reads it, finds no flag, and + # the fragment adds one. The cases that are about it write over this. + File.write!(Path.join(dir, "vm.args"), "## nothing\n-start_epmd false\n") + + # The one line of Mix's generated `elixir` that decides which emulator the + # launcher runs, and where the fragment reads that decision from. Both + # versions name the same ERTS, as every release that did not change one + # does; the case that is about a root holding several writes over this. + elixir(root, vsn, @erts) + end + + # The release's own emulator, as a release that brought its ERTS has it, and + # the only way to see whether the fragment probed at all: it records the + # invocation and then hands over to the real thing, so a probe is measured + # rather than reconstructed. The fragment resolves it out of the `elixir` + # written above, and this suite's own reporting deliberately does *not* go + # through it - otherwise every run would look like a probe. + probing_erl(root) + + # Present so that the RELEASES bootstrap - the fragment's third part, which + # starts a VM - is skipped. It is not what any of this is about. + File.write!(Path.join(root, "releases/RELEASES"), "[].\n") + File.write!(Path.join(root, "releases/start_erl.data"), "16.0 #{@vsn}\n") + + # The launcher, as much of it as the fragment's re-exec depends on. It says + # which version it was handed - the discriminator for a provisional start - + # and then does what `bin/` does with it: resolves REL_VSN_DIR from the + # RELEASE_VSN the previous pass exported, and sources the fragment again. + # + # **Sourcing it is the whole point.** Without that, a provisional start is + # observable only by what the exec printed, and every decision the second pass + # makes - which is now all of them, since the version is settled before + # anything else - would be invisible. The pass that boots is the pass under + # test; this is what lets a test see it. + File.write!(Path.join(root, "bin/sample"), """ + #!/bin/sh + echo "#{@exec}$RELEASE_VSN" + REL_VSN_DIR="$RELEASE_ROOT/releases/$RELEASE_VSN" + . "$RELEASE_ROOT/run.sh" + """) + + File.chmod!(Path.join(root, "bin/sample"), 0o755) + + {:ok, root: root} + end + + describe "heart, with nothing inherited" do + test "assigns the whole of it", %{root: root} do + # The ordinary deployment. heart runs, because release_handler calls + # heart:set_cmd/1 while preparing a reboot and that raises badarg with no + # heart process - and it is given nothing to do. + run = start(root) + + assert run.env["ELIXIR_ERL_OPTIONS"] =~ "-heart" + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.env["HEART_COMMAND"] == "" + assert run.env["HEART_NO_KILL"] == "TRUE" + assert run.env["HEART_BEAT_TIMEOUT"] == "65535" + end + + test "says nothing at all", %{root: root} do + # The normal case has to be silent. A warning on every start of every + # deployment is a warning nobody reads. + assert start(root).stderr == "" + end + + test "asks the emulator anyway", %{root: root} do + # The cost of the fix on an ordinary start, and it is stated as a property + # rather than left implied: no flag variable is set and the vm.args is plain + # flags and comments, so there is nothing here that *could* carry a -heart, + # and the emulator is asked all the same. + # + # There used to be a gate - a `case` over the four flag variables and a + # `read` loop over the args file - so that a start like this forked nothing, + # and this test asserted exactly that. It went because it was the last thing + # in the fragment reasoning about the *text* of those values instead of + # measuring them, and because it could not see ERL_OTP_FLAGS at all. + # So what every start now pays is this: one invocation, of a C program that + # exits without booting an emulator. It is deliberate, and it is asserted + # rather than tolerated. + # + # Recorded by the release's own erl, which is the binary the fragment + # resolves and the only place a probe could come from. + assert start(root).env["CASTLE_HEART_FLAGS"] == "1" + assert [_probe] = probes(root) + end + end + + # **The fragment does not read these variables to decide anything; it asks + # erlexec what the argument vector came out as.** That is the fix, and the + # matrix below is why it had to be: erlexec applies shell-style quoting and + # backslash escaping to each of the flag variables it reads, so `'-heart'`, + # `"-heart"` and `-he\art` all arrive at the emulator as -heart while carrying no + # `-heart` substring for anything to match on. Three successive versions of this + # guard modelled the parsing - a match between literal spaces, then a split into + # fields, then a literal scan of the args file - and each shipped with a test in + # this file that mirrored the same model and agreed with it. + # + # ELIXIR_ERL_OPTIONS is deliberately in the same table and behaves *differently*, + # which is the sharpest thing here. It is expanded by Mix's generated `elixir`, + # so the shell splits it into fields and the quotes survive into the token: + # erlexec unquotes what it reads out of the environment and does not touch the + # command line, so ELIXIR_ERL_OPTIONS="'-heart'" leaves the emulator with no + # heart at all and the fragment has to add one. No model of "the four flag + # variables" gets that right, because they are not four of a kind. + describe "heart, inherited through a variable erlexec reads" do + for source <- ~w(ERL_AFLAGS ERL_FLAGS ERL_ZFLAGS), + {shape, value} <- [ + {"plain", "-heart"}, + {"single quoted", "'-heart'"}, + {"double quoted", ~s("-heart")}, + {"backslash escaped", "-he\\art"}, + {"tab separated", "-heart\t-noshell"}, + {"newline separated", "-noshell\n-heart"}, + {"padded with whitespace", " -heart \t -noshell "} + ] do + test "is not added a second time for a #{shape} #{source}", %{root: root} do + source = unquote(source) + run = start(root, [{source, unquote(value)}]) + + # One flag reaching the emulator, from the variable that already had it, + # and nothing appended to ELIXIR_ERL_OPTIONS - which the fragment must + # leave alone rather than assign, since assigning it is what makes two. + # The inherited value is not asserted on: one of these carries a newline, + # and a value with a newline in it cannot be reported a line at a time. + assert run.env["CASTLE_HEART_FLAGS"] == "1", "#{source} = #{inspect(unquote(value))}" + assert run.env["ELIXIR_ERL_OPTIONS"] == "" + end + end + + test "is not added a second time for an ERL_OTP_FLAGS", %{root: root} do + # **The fifth variable, and the one the fragment's gate could not see.** + # erlexec prepends ERL_OTP_FLAGS exactly as it prepends ERL_AFLAGS, so + # a -heart in it reaches init:get_argument(heart) like any other - but its + # name carries the emulator's OTP major, and a POSIX shell looking for it + # would have to enumerate variable names, which `env` and `export -p` can + # only do by forking. So the gated fragment never probed for a deployment + # that set only this one: it appended a second flag and the boot hung having + # printed nothing. Removing the gate is what closes that, and this is the + # property it was removed for. + # + # The name is derived from the running emulator rather than written out. A + # hardcoded ERL_OTP28_FLAGS is inert on four of the six cells this repository + # is verified on, and a test that sets an inert variable measures nothing + # while passing. + run = start(root, [{otp_flags(), "-heart"}]) + + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.env["ELIXIR_ERL_OPTIONS"] == "" + assert run.stderr == "" + end + + for source <- ~w(ELIXIR_ERL_OPTIONS ERL_AFLAGS ERL_FLAGS ERL_ZFLAGS) do + test "is still added when #{source} carries other flags but no -heart", + %{root: root} do + # The other direction, and what stops the fix from being "never add one". + # A deployment with unrelated flags in these variables still needs the + # heart the restart transition depends on - and the emulator is asked about + # them even though nothing in the value could become a -heart, because + # measuring is unconditional now. The count is what changed here; the + # outcome is the same one the gated version reached without asking. + source = unquote(source) + run = start(root, [{source, "-kernel shell_history enabled"}]) + + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.env["ELIXIR_ERL_OPTIONS"] =~ "-heart" + assert [_probe] = probes(root) + end + end + + for {shape, value} <- [ + {"plain", "-heart"}, + {"tab separated", "-heart\t-noshell"}, + {"newline separated", "-noshell\n-heart\n"}, + {"padded with whitespace", " \t -heart \n -noshell "} + ] do + test "is not added a second time for a #{shape} ELIXIR_ERL_OPTIONS", + %{root: root} do + # The variable Mix's `elixir` expands, unquoted, so what reaches the + # emulator is its fields under $IFS - space, tab and newline, all three. + # Only the flag count is asserted for the values carrying a newline, + # because a value with one in it cannot be reported a line at a time - and + # the count is the property anyway: it is what init:get_argument(heart) + # gets built from. + run = start(root, [{"ELIXIR_ERL_OPTIONS", unquote(value)}]) + + assert run.env["CASTLE_HEART_FLAGS"] == "1" + end + end + + for {shape, value} <- [ + {"single quoted", "'-heart'"}, + {"double quoted", ~s("-heart")}, + {"backslash escaped", "-he\\art"} + ] do + test "is added beside a #{shape} ELIXIR_ERL_OPTIONS, which is not one", + %{root: root} do + # And the case that says this is a measurement rather than a rule about + # four look-alike variables. The shell removes nothing when it expands a + # variable's value, and erlexec unquotes only what it reads out of the + # environment itself - so this token reaches the emulator with its quotes + # or its backslash intact and is not the heart flag. A guard that treated + # these variables alike would leave the release with no heart and a + # restart install failing at heart:set_cmd/1. + run = start(root, [{"ELIXIR_ERL_OPTIONS", unquote(value)}]) + + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.env["ELIXIR_ERL_OPTIONS"] == unquote(value) <> " -heart" + end + end + + test "counts one across two sources that each carry a -heart", %{root: root} do + # A deployment that had already broken its own boot, in which case the + # fragment's job is only not to make it worse: it adds nothing, and + # ELIXIR_ERL_OPTIONS is left exactly as it arrived. The two flags are the + # deployment's, and nothing here can un-break that - but the probe is not + # troubled by it either, which is what makes a timeout unnecessary: it prints + # two lines and exits, where a boot would hang. + run = start(root, [{"ELIXIR_ERL_OPTIONS", "-heart"}, {"ERL_AFLAGS", "-heart"}]) + + assert run.env["CASTLE_HEART_FLAGS"] == "2" + assert run.env["ELIXIR_ERL_OPTIONS"] == "-heart" + end + end + + # The source that is not an environment variable at all: the launcher passes + # --vm-args to Mix's `elixir`, which becomes erl's -args_file, and a project may + # legitimately put -heart in its own rel/vm.args.eex. erlexec reads that file + # with the same quoting it applies to the variables, plus # comments, and follows + # a nested -args_file out of it - so a literal scan of the file is wrong in three + # separate directions, and the probe is right in all of them by not being a scan. + describe "heart, inherited through vm.args" do + for {shape, contents} <- [ + {"plain", "-heart\n-noshell\n"}, + {"single quoted", "'-heart'\n"}, + {"double quoted", ~s("-heart"\n)}, + {"backslash escaped", "-he\\art\n"}, + {"quoted-hash-preceded", ~s(-setcookie "a#b" -heart\n)} + ] do + test "is not added a second time when vm.args carries a #{shape} -heart", + %{root: root} do + # The quoted-hash case is the one that refutes comment stripping: erlexec + # does not treat that # as starting a comment - measured, the cookie + # arrives as "a#b" - so the -heart after it is live, and a scan that + # stripped from the first hash would drop it and hang the boot. + vm_args(root, unquote(contents)) + + run = start(root) + + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.env["ELIXIR_ERL_OPTIONS"] == "" + assert run.stderr == "" + end + end + + test "is still added when a plain argument after -extra is spelled -heart", + %{root: root} do + # The boundary `init` applies and `-emu_args_exit` does not. Everything after + # -extra goes to the application rather than being parsed as an emulator + # flag, so `init:get_argument(heart)` answers `error` here - measured - while + # the probe's output still carries a `-heart` line for it. Counting that line + # would suppress the flag that *should* be added. + # + # Note which way this fails, because it is the opposite of every other case + # in this describe block and that is why it outlived them: those would add a + # second flag and hang the boot, this adds none, the release boots happily + # without heart, and the damage surfaces much later as heart:set_cmd/1 + # refusing a restart transition on a system that looks entirely healthy. + vm_args(root, "-noinput\n-extra\n-heart\n") + + run = start(root) + + assert run.env["ELIXIR_ERL_OPTIONS"] == "-heart" + assert run.stderr == "" + end + + test "is not added a second time for one a nested args file supplies", + %{root: root} do + # erlexec follows a nested -args_file, so the flag is live and there is no + # second one to add. The fragment does not follow it and does not have to: + # what it asks is what the vector came out as, and erlexec built the vector. + # This case used to produce a warning and no heart, because the fragment + # could see the nesting and not through it. + nested = Path.join(root, "nested.vm.args") + File.write!(nested, "-heart\n") + vm_args(root, "-args_file #{nested}\n") + + run = start(root) + + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.env["ELIXIR_ERL_OPTIONS"] == "" + assert run.stderr == "" + end + + test "is added when a nested args file supplies no heart", %{root: root} do + # The other side of following it, and the reason "assume a nested file has + # one" was not good enough either: it silently took heart away from every + # deployment whose vm.args happened to include another file. + nested = Path.join(root, "nested.vm.args") + File.write!(nested, "-start_epmd false\n") + vm_args(root, "-args_file #{nested}\n") + + run = start(root) + + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.env["ELIXIR_ERL_OPTIONS"] == "-heart" + assert run.stderr == "" + end + + test "is added when the only -heart is inside a comment", %{root: root} do + # erlexec treats # as starting a comment in an args file, so the emulator + # really gets no -heart here and one has to be added. The previous fragment + # counted this as present on purpose, because it could not tell a commented + # flag from a live one without a comment-stripping rule that would have been + # wrong about the quoted hash above. Asking erlexec needs no such trade: it + # is the thing that decides what a comment is. + vm_args(root, "## an example: -heart\n-noshell\n") + + run = start(root) + + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.env["ELIXIR_ERL_OPTIONS"] == "-heart" + assert run.stderr == "" + end + + test "is still added when vm.args carries other flags but no -heart", + %{root: root} do + # The ordinary deployment, whose vm.args is full of distribution settings and + # says nothing about heart. The probe reads it and finds nothing, so a flag + # is added - which is the outcome the gated version reached without asking. + # The count is what moved; the answer did not. + vm_args(root, "-start_epmd false\n-erl_epmd_port 24601\n") + + run = start(root) + + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.env["ELIXIR_ERL_OPTIONS"] == "-heart" + assert run.stderr == "" + assert [_probe] = probes(root) + end + + test "is read from the file RELEASE_VM_ARGS names, not from the default", + %{root: root} do + # RELEASE_VM_ARGS is documented as settable before the release is invoked or + # inside env.sh, and the launcher only defaults it *after* it has sourced + # this fragment - so when it is set here, it is set by the deployment and the + # default would be the wrong file. The default is left inert, so a fragment + # asking about it instead would append a second flag. + elsewhere = Path.join(root, "custom.vm.args") + File.write!(elsewhere, "-heart\n") + + run = start(root, [{"RELEASE_VM_ARGS", elsewhere}]) + + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.env["ELIXIR_ERL_OPTIONS"] == "" + + # And the probe was told which file, rather than being left to default it + # the way the launcher will after this returns. + assert [probe] = probes(root) + assert probe =~ "-args_file #{elsewhere}" + end + + test "adds one when the vm.args path does not exist at all", %{root: root} do + # **The one thing about the measurement that is still conditional, and it is + # a fact about the filesystem rather than a judgement about contents.** + # erlexec refuses an args file it cannot open and exits non-zero, so handing + # over a path that is not there would report the measurement as impossible + # and decline to add a flag. So the question is asked without the file, and a + # flag is added. + # + # The condition is defensive, not load-bearing: `mix release` always renders + # rel/vm.args.eex, so a stock build's launcher default always resolves to a + # file that exists. What this covers is a RELEASE_VM_ARGS pointing at a + # missing path, or a hand-deleted vm.args - starts the launcher fails on + # moments later anyway. It is pinned here because it is free and because the + # refutation below is what makes it observable at all. + # + # The reported count is `?` rather than a number because this suite's own + # reporting passes the missing path unconditionally and so cannot be + # answered. That is the launcher's position, not the fragment's. + run = start(root, [{"RELEASE_VM_ARGS", Path.join(root, "nope.vm.args")}]) + + assert run.env["ELIXIR_ERL_OPTIONS"] == "-heart" + assert run.env["CASTLE_HEART_FLAGS"] == "?" + assert run.stderr == "" + + # Asked, and asked *without* an args file. Both halves are the discriminator: + # the first says the fragment measured rather than assuming an absent file + # carries nothing, and the second says the existence check is what keeps that + # measurement answerable. Without it this start would take the unmeasurable + # branch and leave the release with no heart. + assert [probe] = probes(root) + refute probe =~ "-args_file" + end + end + + # Which emulator answers, which is a question the resulting environment never + # says anything about - and one the fragment got wrong. It used to glob + # "$RELEASE_ROOT"/erts-*/bin/erl and keep the last match, which is fine while a + # release root holds one ERTS and wrong the moment an ERTS-changing release is + # unpacked into it. ERL_OTP_FLAGS is named for the OTP version of + # whichever binary answers, so a probe that asked a different one would be + # answering about a different deployment. + describe "the emulator the probe asks" do + test "is the one the selected version's own launcher will run", %{root: root} do + # **The discriminator: the lexicographically last erts-* is deliberately not + # the one the release names.** `erts-9.9` sorts after `erts-16.2` - '1' comes + # before '9' - so this is also the case that shows the old glob was not even + # "the newest one", which is the reading that makes it sound harmless. + # + # The decoy is a working emulator writing to its own log, so the start + # succeeds either way and the only thing that differs is which log has an + # entry in it. A test that arranged for the wrong choice to *fail* would pass + # for the wrong reason. + probing_erl(root, "erts-9.9", "decoys") + + run = start(root) + + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert [_probe] = probes(root) + assert probes(root, "decoys") == [] + end + + test "is PATH's when the release brought no ERTS of its own", %{root: root} do + # `include_erts: false` leaves Mix's own `ERTS_BIN="$ERTS_BIN"` in place, so + # ERTS_BIN is empty and the launcher runs whatever `erl` is on PATH. The + # probe has to do the same - this is the shape the fallback exists for, and + # it has to keep working rather than merely not crash. + # + # The release's own erl is still sitting in the root, and its log staying + # empty is what says the fragment read the file instead of globbing. The + # measurement itself still lands: PATH's emulator answers, so a -heart is + # added and nothing is reported as unmeasurable. + no_erts_elixir(root, @vsn) + + run = start(root) + + assert run.env["ELIXIR_ERL_OPTIONS"] == "-heart" + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.stderr == "" + assert probes(root) == [] + end + + test "is PATH's when the version directory has no elixir at all", %{root: root} do + # Not a shape a release has - Mix always writes one - but the fragment reads + # a file to answer this and so has to answer when the file is not there. It + # falls back rather than reporting the measurement impossible: `erl` from + # PATH is what an empty ERTS_BIN means, and a start that has lost its + # `elixir` is one the launcher fails moments later anyway. + File.rm!(Path.join([root, "releases", @vsn, "elixir"])) + + run = start(root) + + assert run.env["ELIXIR_ERL_OPTIONS"] == "-heart" + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.stderr == "" + assert probes(root) == [] + end + end + + # The two ways the measurement can fail, and they take the same branch: add + # nothing and say so. Appending a flag that turns out to be a second one hangs + # the boot with nothing printed, while adding none leaves heart:set_cmd/1 raising + # badarg during a restart install - which fails loudly with the system still + # running. The two are not symmetric, so the unmeasurable case takes the one an + # operator can see. + describe "heart, when the emulator cannot be asked" do + test "adds nothing, and says why, when the args file cannot be read", + %{root: root} do + # A directory at the path, rather than a mode, because root and some + # filesystems ignore modes and this has to be the state it says it is. + # erlexec refuses to read it, exits non-zero and prints no argument vector. + path = Path.join(root, "opaque.vm.args") + File.mkdir!(path) + + run = start(root, [{"RELEASE_VM_ARGS", path}]) + + assert run.env["ELIXIR_ERL_OPTIONS"] == "" + assert run.stderr =~ "not adding -heart" + assert run.stderr =~ "could not be measured" + assert run.stderr =~ "would hang having printed nothing" + assert [_probe] = probes(root) + end + + test "adds nothing when -emu_args_exit is not recognised", %{root: root} do + # The flag is undocumented, and this is the reason relying on it is + # acceptable: an emulator that no longer knows it degrades to *this*, not to + # a wrong answer. The probe carries a -boot naming a file that cannot exist, + # so such an emulator exits at once instead of starting a node, and the -root + # test refuses to read an answer out of output that is not an argument + # vector. Arranged by an erl that strips the flag before handing over. + legacy_erl(root) + vm_args(root, "-heart\n") + + run = start(root) + + assert run.env["ELIXIR_ERL_OPTIONS"] == "" + assert run.stderr =~ "could not be measured" + assert [_probe] = probes(root) + + # Nothing was booted and nothing was left behind by the attempt: a crash + # dump in the directory an operator started the release from would be a + # regression of its own. + assert Path.wildcard(Path.join(root, "**/erl_crash.dump")) == [] + end + end + + describe "heart, with a deployment's own settings inherited" do + # The environment that made the defaulted version wrong: a HEART_COMMAND to + # restart with, heart's killing turned back on, and a timeout short enough + # for it to bite. + @hostile [ + {"HEART_COMMAND", "/usr/local/bin/restart-me"}, + {"HEART_NO_KILL", "FALSE"}, + {"HEART_BEAT_TIMEOUT", "11"} + ] + + test "defangs heart anyway", %{root: root} do + # Asserted on the effective environment, which is the only thing heart + # reads. The contract is that the external supervisor is the only thing + # that starts a replacement, and a deployment does not get to opt out of + # that while using this hook. + run = start(root, @hostile) + + assert run.env["HEART_COMMAND"] == "" + assert run.env["HEART_NO_KILL"] == "TRUE" + assert run.env["HEART_BEAT_TIMEOUT"] == "65535" + end + + test "says what it overrode, and why, on standard error", %{root: root} do + # Not a refusal: an operator who set these has a configuration conflict, + # not an emergency, and a failed boot is worse than the conflict. But the + # setting has stopped taking effect, so silence would be losing it. + run = start(root, @hostile) + + assert run.stderr =~ "unsetting HEART_COMMAND=[/usr/local/bin/restart-me]" + assert run.stderr =~ "restarting the system belongs to whatever supervises it" + + assert run.stderr =~ "overriding HEART_NO_KILL=[FALSE] with TRUE" + assert run.stderr =~ "overriding HEART_BEAT_TIMEOUT=[11] with 65535" + + # And nothing about it reaches standard output, which on a start belongs to + # the release. + refute run.stdout =~ "warning" + end + + test "is silent about a setting that agrees with it", %{root: root} do + # Assigning is not the same as complaining. A deployment that set the + # values this hook would have chosen has no conflict to be told about. + run = start(root, [{"HEART_NO_KILL", "TRUE"}, {"HEART_BEAT_TIMEOUT", "65535"}]) + + assert run.stderr == "" + assert run.env["HEART_NO_KILL"] == "TRUE" + assert run.env["HEART_BEAT_TIMEOUT"] == "65535" + end + + test "leaves an empty HEART_COMMAND alone to complain about", %{root: root} do + # HEART_COMMAND= is a variable that is set and means nothing, so unsetting + # it changes nothing an operator would notice. Warning about it would be + # noise on a deployment that has no conflict. + # + # Set and empty, which is not the same thing as unset and is what this + # test used to arrange by accident: an empty value in `System.cmd/3`'s + # `:env` *removes* the variable, so the case it named was never run. See + # `run/3`. + run = start(root, [{"HEART_COMMAND", ""}]) + + assert run.stderr == "" + assert run.env["HEART_COMMAND"] == "" + end + + test "says what it overrode when the inherited value is empty", %{root: root} do + # The two assigned values have no such exemption. An empty HEART_NO_KILL is + # not TRUE and an empty HEART_BEAT_TIMEOUT is not 65535, so both are being + # displaced, and the promise is to name every value that stops taking + # effect. `${VAR:-default}` cannot keep that promise, because it treats a + # variable set to nothing as absent - which is what these were written with, + # and why this went unsaid. + run = start(root, [{"HEART_NO_KILL", ""}, {"HEART_BEAT_TIMEOUT", ""}]) + + assert run.stderr =~ "overriding HEART_NO_KILL=[] with TRUE" + assert run.stderr =~ "overriding HEART_BEAT_TIMEOUT=[] with 65535" + + assert run.env["HEART_NO_KILL"] == "TRUE" + assert run.env["HEART_BEAT_TIMEOUT"] == "65535" + end + end + + describe "a command that does not start the system" do + test "reaches none of it", %{root: root} do + # bin/castle drives every command through `rpc`, so a fragment that ran for + # those would start a heart process in a VM that manages nothing - and, + # worse, consume the provisional marker while an install was still waiting + # for the reboot. + arm(root, @next) + + run = run(root, "eval", @hostile) + + assert run.env["HEART_COMMAND"] == "/usr/local/bin/restart-me" + assert run.env["HEART_NO_KILL"] == "FALSE" + assert run.env["ELIXIR_ERL_OPTIONS"] == "" + refute run.stdout =~ @exec + assert armed?(root) + assert provisional?(root) + + # And no emulator was asked anything, which is the sharper half now that the + # heart guard asks on *every* start: `bin/castle` reaches every command + # through `rpc`, so a fragment that ran here would put a fork of erl in front + # of every release-management call. This is the assertion that keeps the + # probe's cost bounded to a node start, and it is the reason the removed gate + # was not the only thing standing between the fork and the common path. + assert probes(root) == [] + end + end + + describe "the provisional version" do + test "is selected from a pair that agrees, by exec'ing the launcher", + %{root: root} do + # release_handler wrote the target to new_start_erl.data and left + # start_erl.data naming the version that is still permanent, so the stock + # launcher on its own would boot the old one. An exec rather than an + # assignment, because by the time env.sh is sourced the launcher has + # already resolved REL_VSN_DIR and everything hangs off it. + arm(root, @next) + + run = start(root) + + assert run.stdout =~ "#{@exec}#{@next}" + assert run.stderr == "" + end + + test "consumes both markers before the exec", %{root: root} do + # One-shot. The second pass has to find nothing, or a provisional boot + # would select itself again for ever and the rollback would be unreachable. + arm(root, @next) + + assert start(root).stdout =~ @exec + refute armed?(root) + refute provisional?(root) + assert claims(root) == [] + end + + test "still boots when a marker cannot be removed", %{root: root} do + # A removal that fails must not take the launcher down with it. The launcher + # sources this fragment under `set -e`, and `rm -f` exits non-zero for a path + # it cannot unlink - a directory at the name being the easy case, since `-f` + # does not cover one. Unguarded, that ended the launcher here, *after* the + # `mv` had already claimed the pending marker: no warning, no fallback, and + # no boot at all. A service that stays down is worse than one on the wrong + # version, and the wrong version was never the risk here. + # + # So the pair is treated as unsettled and the fragment falls through to the + # mismatch path: it warns, execs nothing, and leaves the stock launcher to + # read start_erl.data - the permanent version, which is the safe direction + # and the one the rollback property rests on. + # + # **The discriminator is the exit status**, not the output. Under `set -e` an + # unguarded `rm` ends the sourcing shell where it stands, so the fragment + # neither warns nor returns: status is non-zero and the warning is absent. + # Both assertions below fail against the unguarded version, and the status is + # the one that says the launcher would have died rather than merely said + # something different. + arm(root, @next) + File.rm!(provisional(root)) + File.mkdir_p!(provisional(root)) + + run = start(root) + + assert run.status == 0, "the launcher was taken down while sourcing env.sh" + assert run.stderr =~ "the version to boot could not be settled" + refute run.stdout =~ @exec + end + + test "is not selected from Castle's marker alone", %{root: root} do + # What a hard restart between the arming and release_handler's own write + # leaves: Castle said a reboot was coming and the preparation never got to + # write its half. There is nothing to boot, so the permanent version is + # what starts - and the marker is consumed, so the next start is an + # ordinary one. + File.write!(pending(root), "#{@next}\nsome-attempt\n") + + run = start(root) + + refute run.stdout =~ @exec + assert run.stderr =~ "the version to boot could not be settled" + assert run.stderr =~ "Castle armed [#{@next}] and release_handler recorded []" + refute armed?(root) + end + + test "is not selected from OTP's marker alone", %{root: root} do + # The stale half a failed preparation leaves behind, which nothing removes. + # On its own it is not a boot instruction and it is not evidence of + # anything, so this is silent: no marker, nothing to say. It is Castle that + # clears the file, on the way into the next install that needs to. + File.write!(provisional(root), "16.0 #{@next}\n") + + run = start(root) + + refute run.stdout =~ @exec + assert run.stderr == "" + assert provisional?(root) + end + + test "is not selected by the start after an interrupted consumption", + %{root: root} do + # The claim is atomic and the pair is not: OTP's file is read and removed + # after the rename, so a start killed in between leaves a claim behind and + # OTP's file with it. What must not happen is the next start acting on what + # is left. The marker is gone, so it does not - the selection is lost, and + # a lost selection is a boot of the version that was permanent. + File.write!(Path.join(root, "releases/castle-restart-consumed.999999"), "#{@next}\nx\n") + File.write!(provisional(root), "16.0 #{@next}\n") + + run = start(root) + + refute run.stdout =~ @exec + assert run.stderr == "" + end + + test "is not selected from a pair that disagrees", %{root: root} do + # Both files are still consumed, and the warning names both values: what + # they are is the only diagnosis there is, and leaving either behind would + # have the next start make the same mistake. + File.write!(pending(root), "#{@next}\nsome-attempt\n") + File.write!(provisional(root), "16.0 #{@vsn}\n") + + run = start(root) + + refute run.stdout =~ @exec + assert run.stderr =~ "Castle armed [#{@next}] and release_handler recorded [#{@vsn}]" + refute armed?(root) + refute provisional?(root) + end + + test "is not selected from a version that could name something else", + %{root: root} do + # The version comes out of a file, is used to build a path and is exported + # into the environment of a VM, so one carrying a separator is refused + # rather than resolved. + File.write!(pending(root), "../#{@next}\nsome-attempt\n") + File.write!(provisional(root), "16.0 ../#{@next}\n") + + run = start(root) + + refute run.stdout =~ @exec + assert run.stderr =~ "could not be settled" + end + + test "is not selected from a version directory with nothing to boot", + %{root: root} do + # A version the two markers agree on, unpacked far enough to be named and + # not far enough to be started. env.sh is sourced from it on the next pass + # and start.boot is what the VM boots. + File.rm!(Path.join([root, "releases", @next, "start.boot"])) + arm(root, @next) + + run = start(root) + + refute run.stdout =~ @exec + assert run.stderr =~ "could not be settled" + end + end + + # **Everything the fragment configures, it configures for the version that is + # going to boot.** The selection re-execs, so the fragment is read twice on a + # provisional start - and the boot is the second pass's. Anything decided before + # the re-exec is decided about the version the launcher was *invoked* as, which + # on this start is the one that is being replaced. + # + # That is not a tidiness argument. It shipped the other way round, with the heart + # block ahead of the selection, and the two halves of what that costs are below. + # The suite had no case where the two versions differed, which is why it shipped. + describe "a provisional start decides for the version it boots" do + test "counts the target's -heart rather than inheriting one for it", + %{root: root} do + # **The bug, and the case the suite was missing.** With the heart block + # ahead of the selection: pass 1 probed the *permanent* version's vm.args, + # found no -heart, and exported ELIXIR_ERL_OPTIONS=-heart; pass 2 then probed + # the target's, saw its own flag beside the inherited one, measured two and + # so declined to append a third - but nothing removes what pass 1 exported. + # The target booted with two, which is init:get_argument(heart) == + # {ok, [[], []]} and a hang with nothing printed. + # + # So the target carries a -heart and the permanent version does not, which is + # the shape that separates the two orders. There is nothing for the fragment + # to do here: one flag is already coming, and it must add none. + vm_args(root, "-heart\n", @next) + arm(root, @next) + + run = start(root) + + assert run.stdout =~ "#{@exec}#{@next}" + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert run.env["ELIXIR_ERL_OPTIONS"] == "" + assert run.stderr == "" + + # Asked once, and asked about the target. Two probes is the other face of + # the same defect: the first of them was a question about a version that is + # not going to boot, and its answer is what got exported. + assert [probe] = probes(root) + assert probe =~ "-args_file #{Path.join([root, "releases", @next, "vm.args"])}" + end + + test "adds one for a target that has none, whatever the permanent version had", + %{root: root} do + # The other direction, and the one that says the permanent version's args are + # not consulted at all rather than merely consulted early. Its vm.args has a + # -heart, the target's does not, and the target needs one added. + vm_args(root, "-heart\n") + arm(root, @next) + + run = start(root) + + assert run.stdout =~ "#{@exec}#{@next}" + assert run.env["ELIXIR_ERL_OPTIONS"] == "-heart" + assert run.env["CASTLE_HEART_FLAGS"] == "1" + assert [_probe] = probes(root) + end + + test "asks the target's own emulator, not the root's last erts-*", + %{root: root} do + # **The two defects meeting, and the one case that separates both of them at + # once.** An install that changes ERTS leaves a second erts-* in the root, + # and it is a provisional start that boots the version which brought it. The + # version being replaced names `erts-9.9` and the target names `erts-16.2`, + # so the wrong answer is the same either way round: the glob this replaced + # keeps the lexicographically last match, which is `erts-9.9`, and the order + # this replaced asked before the target was chosen, which is also `erts-9.9`. + probing_erl(root, "erts-9.9", "decoys") + elixir(root, @vsn, "erts-9.9") + arm(root, @next) + + run = start(root) + + assert run.stdout =~ "#{@exec}#{@next}" + assert run.env["CASTLE_HEART_FLAGS"] == "1" + + # The target's emulator answered, and the one belonging to the version being + # replaced was not asked at all. + assert [_probe] = probes(root) + assert probes(root, "decoys") == [] + end + + test "reports an overridden heart setting once", %{root: root} do + # The warnings belong to the boot, and there is one boot. This used to hold + # for a different reason - pass 1 warned and exported the corrected values, + # so pass 2 found nothing left to override - and it has to keep holding now + # that the block runs only on the pass that boots. Either way an operator + # sees each conflict named once. + arm(root, @next) + + run = start(root, [{"HEART_COMMAND", "/usr/local/bin/restart-me"}]) + + assert run.stdout =~ "#{@exec}#{@next}" + assert run.env["HEART_COMMAND"] == "" + assert [_before, _after] = String.split(run.stderr, "unsetting HEART_COMMAND") + end + end + + ## The sandbox + + # Sources the fragment with `$RELEASE_COMMAND` set to one that starts the + # system, and reports what it did: what the launcher printed if it was exec'd, + # what went to standard error, and the environment as the fragment left it. + defp start(root, env \\ []), do: run(root, "start", env) + + defp run(root, command, env) do + script = Path.join(root, "run.sh") + out = Path.join(root, "stdout") + err = Path.join(root, "stderr") + + # `set -e`, because the launcher has it: `bin/` is `#!/bin/sh` followed + # by `set -e`, and it *sources* this fragment, so every command in here runs + # under it. Without it the sandbox is more forgiving than the boot path - a + # top-level non-zero exit would take a real launcher down and pass here - and + # the fragment has several commands whose failure is ordinary: a `grep -q` that + # finds nothing, a command substitution around an emulator that refused an args + # file. Each of those has to be written so that it cannot end the start. + File.write!(script, "set -e\n" <> fragment() <> reporting()) + + {empty, set} = Enum.split_with(environment(root, command, env), &(elem(&1, 1) == "")) + + # Redirected inside the shell rather than merged by System.cmd/3, because + # which stream a warning went to is one of the things being asserted. + # + # A variable that has to arrive *set and empty* cannot go through + # `System.cmd/3`'s `:env`, and this is measured rather than assumed: an empty + # value there removes the variable, the same as `nil` does. So the empty ones + # are shell assignments prefixed to the command instead, which is the one + # place the distinction can be made - and it has to be made, because + # `${VAR:-default}` cannot tell the two apart and that is the defect these + # tests are here for. Without `exec`, so that they prefix an ordinary command + # and are exported to it by the rules for one. + assignments = Enum.map_join(empty, "", fn {name, _} -> "#{name}= " end) + + # `cd: root` so that anything an emulator drops in its working directory - a + # crash dump, most of all - lands inside the sandbox where a test can look for + # it, rather than in whatever directory the suite was started from. + {_, status} = + System.cmd("sh", ["-c", ~s(#{assignments}sh "#{script}" > "#{out}" 2> "#{err}")], + env: set, + cd: root + ) + + stdout = File.read!(out) + + %{ + status: status, + stdout: stdout, + stderr: File.read!(err), + env: reported(stdout) + } + end + + # Everything the launcher exports before it sources env.sh that the fragment + # reads, and nothing else: an inherited variable has to arrive through `env` + # rather than out of the test runner's own environment. + # + # The three ERL_*FLAGS are cleared for the same reason ELIXIR_ERL_OPTIONS is. + # They matter more, not less: they are read by erlexec rather than by anything + # Mix generates, so a developer or a CI image with one of them set would leak a + # -heart into every case here and make the fragment's own additions invisible. + # + # `otp_flags()` is in the list for that reason and no other. It is a fifth + # variable erlexec reads, so it leaks the same way - and it is the one this + # environment could not have named while the fragment's gate existed, since the + # gate's whole limitation was that a shell cannot enumerate variable names. + # Cleared here and set by the one test that is about it. + defp environment(root, command, env) do + [ + {"RELEASE_COMMAND", command}, + {"RELEASE_ROOT", root}, + {"REL_VSN_DIR", Path.join([root, "releases", @vsn])}, + {"RELEASE_VSN", @vsn}, + {"ELIXIR_ERL_OPTIONS", nil}, + {"ERL_AFLAGS", nil}, + {"ERL_FLAGS", nil}, + {"ERL_ZFLAGS", nil}, + {otp_flags(), nil}, + {"RELEASE_VM_ARGS", nil}, + {"HEART_COMMAND", nil}, + {"HEART_NO_KILL", nil}, + {"HEART_BEAT_TIMEOUT", nil} + ] ++ env + end + + # ERL_OTP_FLAGS for the emulator the fragment will probe. That is the + # same one this suite reports with - the sandbox's erts-*/bin/erl hands over to + # `@erl` - so one name covers both, and it is resolved at run time rather than + # compiled in, because the major differs across the cells this is verified on. + defp otp_flags, do: "ERL_OTP#{:erlang.system_info(:otp_release)}_FLAGS" + + defp fragment do + :forecastle + |> :code.priv_dir() + |> Path.join("env.sh.eex") + |> EEx.eval_file(release: %Mix.Release{name: :sample}) + end + + # Appended to the fragment, so it only runs when the fragment did *not* exec + # the launcher. `${VAR-}` rather than `${VAR:-}`, because a + # variable set to nothing and a variable that is not set are different answers + # and one of them is what unsetting HEART_COMMAND has to produce. + # + # CASTLE_HEART_FLAGS is not one of the variables: it is how many -heart flags + # the emulator would be given, which is the property the fragment's check + # exists for and the only one that survives a value with a tab or a newline in + # it, since those cannot be reported a line at a time. Two of them is + # init:get_argument(heart) == {ok, [[], []]}, which hangs the boot. + # + # **It is counted by asking an emulator, and it used to be counted by a shell + # loop.** That loop is how three successive defects survived: the guard modelled + # what erlexec would make of the environment, this counter modelled the same + # thing the same way, and so they agreed with each other and with the bug. It + # matched between spaces while the guard did; it split into fields while the + # guard did; it read vm.args literally while the guard did. Every version of it + # was refuted by the same counterexample that refuted the guard, one round late. + # + # So there is no parsing here at all. `erl -emu_args_exit` prints the argument + # vector erlexec assembled - out of the command line, ERL_OTP_FLAGS, + # ERL_AFLAGS, ERL_FLAGS, ERL_ZFLAGS and every -args_file it followed, with all of + # erlexec's quoting, escaping and comment handling applied - one argument per + # line, and exits without starting a VM. A line that is exactly `-heart` is a + # -heart the emulator would get. **Do not replace this with a shell counter + # again**, whatever shape it takes: what it would be counting is somebody's model + # of erlexec, which is the thing under test. + # + # ELIXIR_ERL_OPTIONS is expanded unquoted and placed before -args_file because + # that is where Mix's generated `elixir` puts them, so this is the vector the + # start would really have produced rather than a rearrangement of it. + # + # The emulator is asked by its absolute path, deliberately *not* through the + # release's own erts-*/bin/erl - that one records the fact of being called, so + # that a test can tell whether the fragment probed, and this reporting must not + # look like a probe. + # + # `?` rather than a number when the answer is not an argument vector, which + # happens when erlexec refuses the args file. A count of nothing must not be + # reportable as a count of zero. + defp reporting do + """ + + for castle_var in ELIXIR_ERL_OPTIONS ERL_AFLAGS ERL_FLAGS ERL_ZFLAGS \\ + HEART_COMMAND HEART_NO_KILL HEART_BEAT_TIMEOUT; do + eval "castle_val=\\${$castle_var-}" + printf 'env %s=%s\\n' "$castle_var" "$castle_val" + done + + castle_report_argv=$(ERL_CRASH_DUMP_SECONDS=0 "#{@erl}" \\ + -emu_args_exit -noinput \\ + -boot /nonexistent/castle-heart-report \\ + ${ELIXIR_ERL_OPTIONS-} \\ + -args_file "${RELEASE_VM_ARGS:-$REL_VSN_DIR/vm.args}" \\ + 2>/dev/null line <- String.split(stdout, "\n"), into: %{} do + [name, value] = String.split(line, "=", parts: 2) + {name, value} + end + end + + # A pair that agrees, as Castle and release_handler leave it: the marker with + # the version on its first line and the attempt on its second, and OTP's + # ` `. + defp arm(root, vsn) do + File.write!(pending(root), "#{vsn}\n1234-5678-1\n") + File.write!(provisional(root), "16.0 #{vsn}\n") + end + + # Replaces the inert vm.args the setup wrote for a version, which is the file + # the launcher will pass as -args_file when it boots that one. The default is + # the version start_erl.data names, because that is the one an ordinary start + # boots; a provisional start's target is named explicitly. + defp vm_args(root, contents, vsn \\ @vsn) do + File.write!(Path.join([root, "releases", vsn, "vm.args"]), contents) + end + + # The one line of Mix's generated `elixir` that the fragment reads to find the + # emulator the launcher will run. `mix release` rewrites the stock + # `ERTS_BIN="$ERTS_BIN"` into this when the release brought an ERTS of its own, + # and leaves it alone when it did not - which `no_erts_elixir/2` is for. The + # rest of that script is a few hundred lines of argument parsing that nothing + # here reads, so only the assignment is written. + # + # `$SCRIPT_PATH` is left as the literal it is in the real file: the fragment + # resolves it against REL_VSN_DIR rather than expanding it, so writing anything + # else here would be testing a file Mix does not generate. + defp elixir(root, vsn, erts) do + File.write!( + Path.join([root, "releases", vsn, "elixir"]), + ~s|ERTS_BIN=\nERTS_BIN="$SCRIPT_PATH"/../../#{erts}/bin/\n| + ) + end + + # The same file as a release with `include_erts: false` has it, which is Mix's + # own two lines untouched. ERTS_BIN comes out empty, so the launcher runs `erl` + # from PATH and so must the probe. + defp no_erts_elixir(root, vsn) do + File.write!( + Path.join([root, "releases", vsn, "elixir"]), + ~s|ERTS_BIN=\nERTS_BIN="$ERTS_BIN"\n| + ) + end + + # The release's own emulator, at the path Mix's generated `elixir` resolves + # ERTS_BIN to and the fragment resolves its probe through. It records that it was + # called, with its arguments, and then hands over to the real one - so whether + # the fragment probed, how often, and what it asked are observable, which nothing + # about the resulting environment says. + # + # A recording wrapper rather than a stub answer, because a stub would make every + # case about the wrapper's idea of erlexec instead of about erlexec. + defp probing_erl(root, erts \\ @erts, log \\ "probes") do + install_erl(root, erts, log, ~s|exec "#{@erl}" "$@"\n|) + end + + # The same, for an emulator that does not know -emu_args_exit: it strips the flag + # and hands the rest over, which is what a future OTP that dropped it would + # effectively do - erlexec would pass the unknown flag through and try to boot. + # That is the case the fragment has to degrade safely on rather than answer + # wrongly, and there is no other way to arrange it. + defp legacy_erl(root) do + install_erl(root, @erts, "probes", """ + castle_n=$# + castle_i=0 + while [ $castle_i -lt $castle_n ]; do + case $1 in + -emu_args_exit) ;; + *) set -- "$@" "$1" ;; + esac + shift + castle_i=$((castle_i + 1)) + done + exec "#{@erl}" "$@" + """) + end + + # The log is a parameter so that a second emulator in the same root can be told + # apart from the release's own. Which binary answered is not visible in the + # arguments - it is the same probe either way - so the only way to see it is for + # each to write somewhere different. + defp install_erl(root, erts, log, body) do + path = Path.join([root, erts, "bin", "erl"]) + File.mkdir_p!(Path.dirname(path)) + + File.write!(path, """ + #!/bin/sh + echo "probe: $*" >> "#{Path.join(root, log)}" + #{body} + """) + + File.chmod!(path, 0o755) + end + + # One entry per invocation of the erl that writes to this log, each the + # arguments it was given. Split on the marker rather than on newlines, because + # an argument may carry one. + defp probes(root, log \\ "probes") do + case File.read(Path.join(root, log)) do + {:ok, contents} -> String.split(contents, "probe: ", trim: true) + {:error, :enoent} -> [] + end + end + + defp pending(root), do: Path.join(root, "releases/castle-restart-pending") + defp provisional(root), do: Path.join(root, "releases/new_start_erl.data") + + defp armed?(root), do: File.exists?(pending(root)) + defp provisional?(root), do: File.exists?(provisional(root)) + defp claims(root), do: Path.wildcard(Path.join(root, "releases/castle-restart-consumed*")) +end diff --git a/test/forecastle/relup_test.exs b/test/forecastle/relup_test.exs index a2cb2b6..5fd2a92 100644 --- a/test/forecastle/relup_test.exs +++ b/test/forecastle/relup_test.exs @@ -24,11 +24,11 @@ defmodule Forecastle.RelupTest do the target of the tests about appup-supplied instructions, and of the split-and-merge test, where it supplies a hot edge to sit beside a restart one. - Two things here are driven in process rather than as a command, for the same - reason: the split-and-merge, which the refusal in front of it puts out of the - task's reach, and publication, whose promise - that a failed run leaves the - previous relup whole - can only be tested from inside the window in which it - could be broken. + Two things here are driven in process rather than as a command: the + split-and-merge, so that the merged plan can be asserted on as a term rather + than through whatever the task prints, and publication, whose promise - that a + failed run leaves the previous relup whole - can only be tested from inside the + window in which it could be broken. """ use Forecastle.ReleaseCase @@ -110,8 +110,8 @@ defmodule Forecastle.RelupTest do # Backwards: 0.1.0's appup says nothing about coming from 0.1.1, which is # what a project that has not written the instructions yet looks like. # `--hot`, because the dependency's appup at 0.1.0 says nothing about it - # either, so `auto` would classify this edge a restart and refuse it before - # `:systools` was asked for anything. + # either, so `auto` would classify this edge a restart, write it and + # succeed - and it is the failure that is under test here. {output, status} = relup(["--target", rel(ctx.from, @from), "--fromto", rel(ctx.to, @to), "--hot"], @from) @@ -283,36 +283,47 @@ defmodule Forecastle.RelupTest do :file.consult(to_charlist(ctx.relup)) end - test "refuses an edge it could only restart, naming the application and the gap", ctx do + test "announces an edge it can only restart, naming the application and the gap", ctx do # The dependency's appup now says nothing about coming from 0.1.0 in either # direction, which is what a dependency that never wrote instructions for # this project's transitions looks like. There is no hot upgrade to be had, - # and a restart transition cannot yet be performed, so `auto` refuses - # rather than write an upgrade plan that will not install. + # so `auto` writes the restart and says so - which is what it used to refuse + # to do, while nothing could complete such a transition. set_dep_appup!(ctx.to, @to, [], []) - {output, status} = relup(upgrade(ctx) ++ ["--outdir", @outdir], @to) + output = relup!(upgrade(ctx) ++ ["--outdir", @outdir], @to) - assert status != 0, "a restart transition was written by auto:\n\n#{output}" refute_all_hot(output) - assert output =~ "auto would make a restart transition of the upgrade from #{@from}" + assert output =~ "auto made a restart transition of the upgrade from #{@from}" assert output =~ "the downgrade to #{@from}" assert output =~ ":sample_dep is a dependency and changed from #{@from} to #{@to}" assert output =~ "sample_dep.appup has no upgrade instructions from #{@from}" assert output =~ "sample_dep.appup has no downgrade instructions to #{@from}" - # The override is named, because refusing without saying what to do instead - # would leave a pipeline with no way forward. - assert output =~ "Pass --restart to generate it anyway" - refute File.exists?(ctx.relup), "a refused relup was written anyway" + # Which instruction, and therefore what install_release/1 replies, because + # that reply is what an operator or a CI check reads back. + assert output =~ "single restart_emulator" + assert output =~ "{ok, Vsn, Descr} rather than {continue_after_restart, Vsn, Descr}" + assert output =~ "provisional until it is committed" + + # And the two ways to insist on something else, so that a pipeline which + # wanted neither has somewhere to go. + assert output =~ "Pass --hot to fail on such a transition instead" + assert output =~ "--restart to make every transition in the relup one" + + assert {:ok, + [ + {@to_vsn, [{@from_vsn, [], [:restart_emulator]}], + [{@from_vsn, [], [:restart_emulator]}]} + ]} = :file.consult(to_charlist(ctx.relup)) end test "says so when the application has no appup at all", ctx do remove_appup!(ctx.to, @to, "sample_dep") - {output, status} = relup(upgrade(ctx) ++ ["--outdir", @outdir], @to) + output = relup!(upgrade(ctx) ++ ["--outdir", @outdir], @to) - assert status != 0, "a restart transition was written by auto:\n\n#{output}" + refute_all_hot(output) assert output =~ "there is no appup at " assert output =~ "sample_dep-#{@to}/ebin/sample_dep.appup" end @@ -322,36 +333,50 @@ defmodule Forecastle.RelupTest do # need not be in the other. Here the dependency can be upgraded from 0.1.0 # but not downgraded back to it, so the only correct answer is a hot # upgrade and a restart the other way - and it is the *downgrade*, alone, - # that the refusal names. Classifying the edge once and using the answer - # for both directions would name both. + # that the announcement names. Classifying the edge once and using the + # answer for both directions would name both. set_dep_appup!(ctx.to, @to, [{~c"#{@from}", []}], []) - {output, status} = relup(upgrade(ctx) ++ ["--outdir", @outdir], @to) + output = relup!(upgrade(ctx) ++ ["--outdir", @outdir], @to) - assert status != 0, "a restart transition was written by auto:\n\n#{output}" refute_all_hot(output) - assert output =~ "auto would make a restart transition of the downgrade to #{@from}" + assert output =~ "auto made a restart transition of the downgrade to #{@from}" assert output =~ "sample_dep.appup has no downgrade instructions to #{@from}" refute output =~ "upgrade from #{@from}" + + # And the relup carries the split, which is the half an announcement cannot + # be trusted for: one direction generated from the appups and the other a + # single instruction. + assert {:ok, [{@to_vsn, [{@from_vsn, [], up}], [{@from_vsn, [], down}]}]} = + :file.consult(to_charlist(ctx.relup)) + + assert down == [:restart_emulator] + refute :restart_emulator in up + assert {:load_object_code, {:sample, @to_vsn, [Sample.Counter]}} in up end - test "refuses a transition that changed ERTS, whatever the appups say", ctx do + test "restarts a transition that changed ERTS, whatever the appups say", ctx do # An ERTS change is not a hot upgrade under any policy and no appup could # make it one, so it is not put to the appups at all - the dependency's # appup covers its own move here, and the ERTS change is the whole reason. # The decision is also taken before :systools is asked for anything, which # inserts restart_new_emulator for an ERTS change: the two-stage - # transition, which nothing about the strategy asked for. + # transition, which nothing about the strategy asked for. What lands is the + # one-stage instruction, written by hand. change_erts!(ctx.from, @from, "0.0.0") - {output, status} = relup(upgrade(ctx) ++ ["--outdir", @outdir], @to) + output = relup!(upgrade(ctx) ++ ["--outdir", @outdir], @to) - assert status != 0, "an ERTS change was written as a restart transition:\n\n#{output}" refute_all_hot(output) - assert output =~ "auto would make a restart transition of the upgrade from #{@from}" + assert output =~ "auto made a restart transition of the upgrade from #{@from}" assert output =~ "ERTS changed from 0.0.0 to" refute output =~ ":sample_dep" - refute File.exists?(ctx.relup), "a refused relup was written anyway" + + assert {:ok, + [ + {@to_vsn, [{@from_vsn, [], [:restart_emulator]}], + [{@from_vsn, [], [:restart_emulator]}]} + ]} = :file.consult(to_charlist(ctx.relup)) end test "generates from the appups when only the project's own applications moved", ctx do @@ -384,66 +409,74 @@ defmodule Forecastle.RelupTest do refute File.exists?(ctx.relup), "a refused relup was written anyway" end - test "refuses an appup that asks for the one-stage emulator restart", ctx do - # The same refusal by another route. A restart_emulator an appup asked for - # by name is the transition `auto` would have chosen for itself, and it is - # just as uninstallable, so how the relup came by it makes no difference. + test "announces an appup that asks for the one-stage emulator restart", ctx do + # The same announcement by another route. A restart_emulator an appup asked + # for by name is the transition `auto` would have chosen for itself, so how + # the relup came by it makes no difference to what the run says. # # Classification cannot see this one: nothing but :sample moves between # these two releases, so every edge is classified hot and the restart only # exists once :systools has produced a script. That is what the all-hot # refutation is here for - announcing the verdict from classification alone - # had this run report a hot relup and a restart in the same breath, and - # would have it report both from a run that succeeded once a restart - # transition can be written. + # had this run report a hot relup and a restart in the same breath, and now + # that the run succeeds it would report both from a successful one. add_appup_instruction!(ctx.hot, @hot, :restart_emulator) - {output, status} = relup(project_only(ctx) ++ ["--outdir", @outdir], @hot) + output = relup!(project_only(ctx) ++ ["--outdir", @outdir], @hot) - assert status != 0, "an appup-supplied restart was accepted by auto:\n\n#{output}" refute_all_hot(output) assert output =~ "an appup asks for the emulator to be restarted" assert output =~ "restart_emulator on the upgrade from #{@from}" - assert output =~ "--restart" - refute File.exists?(ctx.relup), "a refused relup was written anyway" + assert output =~ "single restart_emulator" + + assert {:ok, [{@hot_vsn, [{@from_vsn, [], up}], _down}]} = + :file.consult(to_charlist(ctx.relup)) + + assert :restart_emulator in up end - test "settles the restart it chose without generating the rest of the relup", ctx do - # Both kinds of restart in one run. 0.1.1 is an edge `auto` classifies as a - # restart - the dependency moves between it and 0.1.2, whose copy of the - # dependency's appup covers nothing - while 0.1.0 is a hot edge whose appup - # then asks for the emulator to be restarted by name. + test "names both kinds of restart in one announcement", ctx do + # Both kinds in one run. 0.1.1 is an edge `auto` classifies as a restart - + # the dependency moves between it and 0.1.2, whose copy of the dependency's + # appup covers nothing - while 0.1.0 is a hot edge whose appup then asks for + # the emulator to be restarted by name. # - # While a classified restart is refused, that classification is already the - # whole answer: the run fails whatever the hot half turns out to be. So it - # is settled first and the hot half is never generated, which is why the - # appup's own restart goes unmentioned - it is reported by the run that - # follows, once the classified edge is gone. This test used to assert both - # in one refusal, which is what put generation in front of a decision that - # had already been made; when the refusal becomes an announcement the run - # proceeds and the two are named together again. + # One verdict per invocation means both are named together, and that is only + # possible after generation: the appup's own instruction is invisible until + # there is a script to look at. This test used to assert the opposite - a + # refusal naming the classified edge alone, because that refusal was settled + # before generation while nothing could complete a restart transition. add_appup_instruction!(ctx.hot, @hot, :restart_emulator) - {output, status} = - relup(project_only(ctx) ++ ["--upfrom", rel(ctx.to, @to), "--outdir", @outdir], @hot) + output = + relup!(project_only(ctx) ++ ["--upfrom", rel(ctx.to, @to), "--outdir", @outdir], @hot) - assert status != 0, "a relup with two kinds of restart in it was accepted:\n\n#{output}" refute_all_hot(output) - assert output =~ "auto would make a restart transition of the upgrade from #{@to}" + assert output =~ "auto made a restart transition of the upgrade from #{@to}" assert output =~ "sample_dep.appup has no upgrade instructions from #{@to}" - refute output =~ "an appup asks for the emulator to be restarted" - refute File.exists?(ctx.relup), "a refused relup was written anyway" + assert output =~ "an appup asks for the emulator to be restarted" + assert output =~ "restart_emulator on the upgrade from #{@from}" + + # And both are in the one relup: the hand-written entry for the classified + # edge, and the generated one carrying what the appup asked for. + assert {:ok, [{@hot_vsn, [{@from_vsn, [], hot_up}, {@to_vsn, [], restart_up}], _down}]} = + :file.consult(to_charlist(ctx.relup)) + + assert restart_up == [:restart_emulator] + assert :restart_emulator in hot_up + assert {:load_object_code, {:sample, @hot_vsn, [Sample.Counter]}} in hot_up end - test "refuses the restart it chose rather than a systools error from the rest", ctx do + test "reports a systools error from the hot half rather than announcing anything", ctx do # The same mixed plan, with a hot half that cannot be generated at all. # :sample is the project's own application, so classification never reads # its appup and the 0.1.0 edge is still classified hot - and :systools then # has nothing to generate that transition from. remove_appup!(ctx.hot, @hot, "sample") - # The control, which is what makes the refutation below mean anything: the - # hot half of that plan, on its own, really does fail in :systools. + # The control: the hot half of that plan, on its own, really does fail in + # :systools, which is what makes the assertions below about the mixed run + # mean something. {control, status} = relup(project_only(ctx) ++ ["--outdir", @outdir], @hot) assert status != 0, "the hot half was generated without an appup:\n\n#{control}" @@ -452,32 +485,31 @@ defmodule Forecastle.RelupTest do {output, status} = relup(project_only(ctx) ++ ["--upfrom", rel(ctx.to, @to), "--outdir", @outdir], @hot) - # The restart on the 0.1.1 edge is the reason this run cannot succeed, and - # it is what the user is shown. Generating first reported the systools error - # instead - an error about the other half of the relup, in front of a - # refusal that was already known - and fixing it only uncovered this - # refusal on the next run. + # This is the ordering that changed with the refusal. It used to settle the + # classified restart *first*, because while such a transition could not be + # installed the classification was already the whole answer and a systools + # error from the other half would only have stood in front of it. Now the + # run can succeed, so generation comes first and its failure is the verdict: + # there is nothing to announce about a relup that was not produced. assert status != 0, "a mixed relup with an ungeneratable half was accepted:\n\n#{output}" - refute_all_hot(output) - assert output =~ "auto would make a restart transition of the upgrade from #{@to}" - assert output =~ "sample_dep.appup has no upgrade instructions from #{@to}" + assert output =~ "sample-#{@hot}/ebin/sample.appup" - refute output =~ "sample-#{@hot}/ebin/sample.appup", - "a systools error preempted the restart refusal:\n\n#{output}" + refute output =~ "auto made a restart transition", + "a run that generated nothing announced a strategy anyway:\n\n#{output}" - refute File.exists?(ctx.relup), "a refused relup was written anyway" + refute_all_hot(output) + refute File.exists?(ctx.relup), "a failed run wrote a relup anyway" end end describe "merging hot and restart transitions into one relup" do # `auto` is the only strategy that splits a relup's edges by class and merges - # the two kinds back together, and while a restart transition cannot be - # performed it refuses to emit one - so the merge is not reachable through the - # task. It is still the defining behaviour, and the one place an edge could be + # the two kinds back together, and it is the one place an edge could be # dropped, or attached to the wrong direction, or one class's script applied - # to the whole relup. So it is driven directly, in process, past the refusal - # that sits in front of it. This is the test that has to keep working when - # that refusal becomes an announcement. + # to the whole relup. The task reaches it now that a restart transition can be + # installed - the mixed-restart cases above go through it - but it is still + # driven directly here, because what these assert is the *shape* of the merged + # plan, which is a term rather than anything the task prints. # # 0.1.2 is the target: 0.1.0 is a hot edge to it (only :sample moved, since # this assembly pins the dependency) and 0.1.1 is declared a restart edge, @@ -690,11 +722,9 @@ defmodule Forecastle.RelupTest do # this covers is Forecastle.verify_relup!/2 accepting a hand-written # restart plan and post-assembly copying it into the version directory. # - # It does not cover the transition. Nothing can perform an - # emulator-restart upgrade yet - castle#14 and - # [#10](https://github.com/ausimian/forecastle/issues/10) are what make - # the reboot come back on the installed version - so this asserts on the - # relup and the assembled tree and stops there. + # It does not cover the transition itself. `restart_upgrade_test.exs` does + # that, end to end and against a real supervised release, so this asserts + # on the relup and the assembled tree and stops there. relup!(upgrade(ctx) ++ ["--restart"], @to) staged = @@ -713,8 +743,8 @@ defmodule Forecastle.RelupTest do # answer that - an appup can ask for the emulator to be restarted by name, and # nothing knows until :systools has produced a script - so this is asserted # against every `auto` case that ends in a restart, whichever of the two ways - # the restart arrived. It has to keep holding when the refusal becomes an - # announcement: then it is a *successful* run that must not say both. + # the restart arrived. It matters *more* now that those runs succeed: what it + # forbids is a successful run saying both. defp refute_all_hot(output) do refute output =~ "every transition in this relup is a hot upgrade", "auto announced an all-hot relup for a transition it restarts:\n\n#{output}" diff --git a/test/forecastle/restart_upgrade_test.exs b/test/forecastle/restart_upgrade_test.exs new file mode 100644 index 0000000..a4e9039 --- /dev/null +++ b/test/forecastle/restart_upgrade_test.exs @@ -0,0 +1,509 @@ +defmodule Forecastle.RestartUpgradeTest do + @moduledoc """ + Boots a real release under the stock Mix launcher and upgrades it through a + transition that restarts the emulator. + + The sibling of `Forecastle.UpgradeTest`, and its opposite where it counts: that + suite asserts the operating system pid does *not* change across the install, + and this one asserts that it does. Everything else about the shape is the same, + and deliberately so - `bin/castle unpack`, `install`, `commit`, against a + release started by `bin/`. + + What it takes to make that work is spread across both halves of the pair, and + each piece is asserted here because nothing smaller can be: + + * `release_handler` calls `heart:set_cmd/1` while preparing the reboot, and + that raises `badarg` with no `heart` process, so the install would fail + before anything rebooted. The `env.sh` fragment runs the real heart, + defanged: `HEART_NO_KILL`, no `HEART_COMMAND`, and an inert + `$ROOT/bin/start` to neutralise the temporary command `release_handler` + installs. The fixture supplies its own `-heart` through `rel/vm.args.eex`, + which the launcher passes as `-args_file`, and spells it `-he\art` - + because *two* of them hang the boot, and because the only way to see that + one is to ask erlexec what it made of the file rather than to read the file. + * `release_handler` writes the target to `releases/new_start_erl.data` and + leaves `releases/start_erl.data` naming the version that is still + permanent, so the reboot would come back on the old version. The `env.sh` + fragment selects the provisional one, and only when Castle's own marker + agrees with OTP's. + * nothing in the release restarts it. The test is the supervisor; see + `Forecastle.Deployment.install_supervised!/3`. + + The rollback half is exercised before the commit half, and in that order for a + reason: a provisional release that is killed before `Castle.commit/1` has to + come back as the *previous* version, and the only way to see that is to kill + one. The second install then goes through the same transition again from the + release that rolled back, and is committed. + + mix test --include e2e + + Distribution runs without epmd (see the fixture's `rel/vm.args.eex`), so no + daemon needs to be running on the host for this to work. + """ + + use Forecastle.ReleaseCase + + import Forecastle.Deployment + + alias Forecastle.Fixture + + @moduletag :e2e + + @from "0.1.0" + @to "0.1.1" + + # The greeting changes between the first boot and the provisional one, on + # purpose. `config/runtime.exs` reads it, so the provisional boot answering the + # second value is what says its own providers ran again over the `sys.config` + # Castle materialised - which is a thing nothing asserted before, because until + # now no suite restarted a release after an install. + @first_greeting "hello-from-runtime" + @restart_greeting "hello-after-restart" + + # `CASTLE_INSTALL_TIMEOUT` is generous, because the wait covers a reboot and a + # full cold boot, and bounded, so that a hang fails the suite rather than + # sitting for the five-minute default. + @restart_env [{"CASTLE_INSTALL_TIMEOUT", "120"}, {"SAMPLE_GREETING", @restart_greeting}] + + # The first start is made hostile on purpose. These three are what a deployment + # can already have in its environment - a systemd unit that once ran a release + # with heart as a watchdog, an image that sets them for something else - and + # every one of them contradicts what this release says it does: a command to + # restart with, heart's killing turned back on, and a timeout short enough to + # bite. The whole restart transition then runs under them, so a fragment that + # only *defaulted* these would have this suite exercising a release with a live + # watchdog beside its supervisor. + # + # Measured, and worth knowing before reading the assertions: an inherited + # HEART_COMMAND is not merely a restart authority for an unexpected death. + # heart runs it on an *orderly* halt too - "Erlang has closed. Executed ... -> + # 0. Terminating." - so with one in the environment every `bin/ stop` + # would start the release again. + @hostile_heart [ + {"HEART_COMMAND", "/usr/local/bin/restart-me"}, + {"HEART_NO_KILL", "FALSE"}, + {"HEART_BEAT_TIMEOUT", "11"} + ] + + # The -heart this deployment already has arrives from the fixture's own + # `rel/vm.args.eex`, which the launcher passes to the emulator as `-args_file`, + # and it is asked for at *assembly* time so that only the releases this suite + # builds carry it: -heart makes heart print `heart_beat_kill_pid = ` on + # every VM started with that file, `eval` included, and + # `Forecastle.ConfigurationTest` asserts on the exact output of an `eval`. + # + # **It is spelled `-he\art`, because that is the case no reading of the + # environment can answer.** erlexec removes the backslash and hands the emulator + # `-heart`; a fragment that matched text, split fields or scanned the file + # appends a second one, `init:get_argument(heart)` answers `{ok, [[], []]}`, + # `heart:check_start_heart/0` raises a case_clause at heart.erl:348 and the boot + # hangs having printed nothing. Three versions of that guard shipped, each with a + # unit test that shared its model; the fragment now asks `erl -emu_args_exit` + # what the argument vector came out as. The rest of the note lives in that file, + # which is where a reader looking for it will go. + # + # It has to be exercised on a real boot rather than only in the shell: what a + # unit test can assert about the fragment is what the fragment says, what + # `-emu_args_exit` reports is what erlexec assembled, and what hangs is the + # emulator. A fragment that misses this flag makes the suite time out inside + # `start!/2` rather than fail an assertion, which is exactly why that call + # carries a deadline. Which values the guard recognises, one at a time and + # against a real erlexec, is `Forecastle.EnvScriptTest`'s. + @heart_in_vm_args [{"SAMPLE_HEART", "-he\\art"}] + + # ERL_AFLAGS carries the tab probe and *only* the tab probe: a -heart in both it + # and vm.args would itself be the two-flag hang. `-env` is what makes the tab + # observable - erlexec splits that variable into arguments on tabs as readily as + # on spaces, measured - so a node answering with the value is a node whose + # argument list was really split on them. It is also an ordinary flag variable + # standing beside the measurement, carrying no heart of its own: the probe reads + # it on every start here and the boot still gets exactly the one flag vm.args + # supplies. This used to be phrased as the probe's *gate* not tripping on it; + # there is no gate any more, and what the case says now is that asking about an + # unremarkable variable does not manufacture a second flag. + @tabbed_heart [{"ERL_AFLAGS", "-env\tCASTLE_TAB_PROBE\ttabbed"}] + + @probe_report ~s|IO.puts(inspect(System.get_env("CASTLE_TAB_PROBE")))| + + # What the running node says about the environment it was started with, and + # about the command heart is actually holding. The environment is what heart's + # port program reads, so it is the effective configuration rather than a + # description of one. + @heart_report """ + IO.puts(inspect({ + System.get_env("HEART_COMMAND"), + System.get_env("HEART_NO_KILL"), + System.get_env("HEART_BEAT_TIMEOUT") + })) + """ + + setup_all do + workspace = Fixture.workspace() + relup = Path.join(workspace, "relup") + + deploy = assemble!(into: "restart-deploy", vsn: @from, env: @heart_in_vm_args) + next = assemble!(into: "restart-next", vsn: @to, env: @heart_in_vm_args) + + # `--restart`, explicitly, for the reason the hot suite asks for `--hot`: the + # transition is the subject, so the task rather than an assertion further down + # is what fails if it stops being generatable. `auto` would judge this edge + # hot - the dependency's appup covers its move - so it is not the strategy + # that produces the relup this suite needs. + make_relup!({deploy, @from}, {next, @to}, ["--restart"]) + ^next = assemble!(into: "restart-next", vsn: @to, env: @heart_in_vm_args) + + File.cp!( + Path.join(next, "sample-#{@to}.tar.gz"), + Path.join(deploy, "releases/sample-#{@to}.tar.gz") + ) + + on_exit(fn -> + cmd(Path.join(deploy, "bin/sample"), ["stop"]) + File.rm(relup) + end) + + # Captured rather than discarded: the fragment warns when it overrides an + # inherited heart setting, and that warning is part of what this suite + # asserts. + hostile_start = + start!( + deploy, + [{"SAMPLE_GREETING", @first_greeting}] ++ @hostile_heart ++ @tabbed_heart + ) + + booted = %{ + os_pid: os_pid(deploy), + heart: rpc!(deploy, "IO.puts(inspect(:erlang.whereis(:heart)))"), + heart_cmd: rpc!(deploy, "IO.puts(inspect(:heart.get_cmd()))"), + heart_env: rpc!(deploy, @heart_report), + heart_args: rpc!(deploy, "IO.puts(inspect(:init.get_argument(:heart)))"), + tab_probe: rpc!(deploy, @probe_report), + erl_aflags: rpc!(deploy, ~s|IO.puts(inspect(System.get_env("ERL_AFLAGS")))|), + elixir_erl_options: + rpc!(deploy, ~s|IO.puts(inspect(System.get_env("ELIXIR_ERL_OPTIONS")))|), + start_output: hostile_start, + counter: rpc!(deploy, "IO.puts(inspect(Sample.Counter.info()))"), + releases: castle!(deploy, ["releases"]), + start_erl: File.read!(Path.join(deploy, "releases/start_erl.data")) + } + + castle!(deploy, ["unpack", @to]) + + # The first transition, abandoned. `install` reboots the node, the launcher + # selects the provisional version on the way back up, and nothing has been + # committed - so what a crash from here has to do is come back on @from. + {provisional_output, provisional_status} = install_supervised!(deploy, @to, @restart_env) + + provisional = %{ + output: provisional_output, + status: provisional_status, + os_pid: os_pid(deploy), + counter: rpc!(deploy, "IO.puts(inspect(Sample.Counter.info()))"), + greeting: rpc!(deploy, "IO.puts(Sample.greeting())"), + env_marker: rpc!(deploy, "IO.puts(Sample.env_marker())"), + release_env: rpc!(deploy, "IO.puts(inspect(Sample.release_env()))"), + releases: castle!(deploy, ["releases"]), + version: launcher!(deploy, ["version"]), + start_erl: File.read!(Path.join(deploy, "releases/start_erl.data")), + pending?: File.exists?(Path.join(deploy, "releases/castle-restart-pending")), + marker?: File.exists?(Path.join(deploy, "releases/new_start_erl.data")), + castle_names: Path.wildcard(Path.join(deploy, "releases/castle-*")) + } + + # A crash, not a stop: `bin/sample stop` would be an orderly shutdown, and + # what has to be survivable is the other kind. + {_output, 0} = System.cmd("kill", ["-9", provisional.os_pid]) + await_exit!(provisional.os_pid) + start!(deploy, [{"SAMPLE_GREETING", @first_greeting}]) + + rolled_back = %{ + counter: rpc!(deploy, "IO.puts(inspect(Sample.Counter.info()))"), + releases: castle!(deploy, ["releases"]), + version: launcher!(deploy, ["version"]) + } + + # And again, from the release that came back, this time through to a commit. + {installed_output, installed_status} = install_supervised!(deploy, @to, @restart_env) + + installed = %{output: installed_output, status: installed_status} + + committed = %{output: castle!(deploy, ["commit"])} + + committed = + Map.merge(committed, %{ + releases: castle!(deploy, ["releases"]), + version: launcher!(deploy, ["version"]), + start_erl: File.read!(Path.join(deploy, "releases/start_erl.data")) + }) + + # One more restart, with no marker anywhere: what an ordinary start boots is + # the other half of what `commit` means. + committed_pid = os_pid(deploy) + launcher!(deploy, ["stop"]) + await_exit!(committed_pid) + + # The one start in this suite with nothing hostile in its environment, and + # nothing for the fragment to select either, so it is what says the heart + # warnings above are about the environment rather than about every start. + quiet_start = start!(deploy, [{"SAMPLE_GREETING", @first_greeting}]) + + restarted = %{ + start_output: quiet_start, + heart_env: rpc!(deploy, @heart_report), + counter: rpc!(deploy, "IO.puts(inspect(Sample.Counter.info()))"), + releases: castle!(deploy, ["releases"]) + } + + {:ok, + deploy: deploy, + booted: booted, + provisional: provisional, + rolled_back: rolled_back, + installed: installed, + committed: committed, + restarted: restarted} + end + + describe "heart, on a release that is going to need it" do + test "is running", %{booted: booted} do + # Not because the deployment wants a watchdog - it does not, and heart is + # configured to do nothing - but because heart:set_cmd/1 sends to the + # registered name and raises badarg when nothing is there, which is what + # made a restart transition fail before it could reboot. + assert booted.heart =~ ~r/^#PID version` prints "the release name and version to be booted", + # which it takes from start_erl.data - and it is not a command that starts + # the system, so the fragment does not select anything for it. So it names + # @from while @to is what is running, and that is right rather than a bug in + # either: what it is reporting is where an ordinary restart would land, + # which for an uncommitted provisional release is the rollback target. + assert provisional.version == "sample #{@from}" + end + + test "resolved every path from the provisional version's directory", + %{deploy: deploy, provisional: provisional} do + # This is what says the hook re-execs the launcher rather than assigning + # RELEASE_VSN and returning. By the time env.sh is sourced the launcher has + # already computed REL_VSN_DIR from start_erl.data, and RELEASE_VM_ARGS - + # which runtime.exs reads with fetch_env! - is derived from it *after*. + # Assigning the version in place would have booted @from's vm.args, sys.config + # and boot script under @to's name; only an exec recomputes them. + assert provisional.release_env =~ "#{deploy}/releases/#{@to}/vm.args" + end + + test "runs the project's own env.sh across the re-exec", %{provisional: provisional} do + # The fragment is appended to whatever the project supplied, so the project's + # half runs on both passes. A marker that survived says the second pass was a + # real launcher invocation rather than something reconstructed. + assert provisional.env_marker == "preserved" + end + + test "re-runs the provisional version's own config providers", + %{booted: booted, provisional: provisional} do + # A cold boot of @to, through Elixir's own pipeline, over the sys.config + # Castle's peer materialised before the install was asked for. The peer + # resolved it with SAMPLE_GREETING at its first value; this boot sees the + # second, and answers with it - so materialising did not freeze the + # configuration, which is what the header Mix wrote is preserved for. + assert booted.counter == ~s({"#{@from}", 0}) + assert provisional.greeting == @restart_greeting + end + + test "leaves the new version current and the old one permanent", + %{provisional: provisional} do + # release_handler persisted @to as tmp_current before the reboot; + # transform_release/3 wrote that back as unpacked on disk and set_current/2 + # made it current in the record the handler holds, because init:script_id() + # names it. Which is what Castle.running/1 needs, and what commit needs. + assert provisional.releases =~ ~r/#{@to}\s+current/ + assert provisional.releases =~ ~r/#{@from}\s+permanent/ + end + + test "does not touch what an ordinary restart boots", + %{booted: booted, provisional: provisional} do + # start_erl.data is written only by make_permanent, and this is the whole + # rollback property: until the upgrade is committed, the version a restart + # selects is still the one that was permanent before. + assert provisional.start_erl == booted.start_erl + assert provisional.start_erl =~ @from + end + + test "consumes both markers", %{provisional: provisional} do + # One-shot, and both of them, so that a second start does not select the + # provisional version again - which is what makes the rollback below + # possible at all. + refute provisional.pending?, "Castle's restart marker survived the boot that used it" + refute provisional.marker?, "releases/new_start_erl.data survived the boot that used it" + end + + test "leaves nothing of the protocol behind", %{provisional: provisional} do + # Every name either side of this puts in the releases directory begins + # `castle-`: the marker, the claim the hook renames it to, and the working + # directory Castle stages the marker in before linking it into place. An + # install and a provisional boot have to leave none of them. + assert provisional.castle_names == [] + end + end + + describe "a provisional release that dies before it is committed" do + test "comes back as the previous permanent version", %{rolled_back: rolled_back} do + # No marker, so the stock launcher reads start_erl.data, which + # make_permanent never wrote. Nothing intervened and nothing had to. + assert rolled_back.version == "sample #{@from}" + assert rolled_back.counter == ~s({"#{@from}", 0}) + end + + test "leaves the version it rolled back from unpacked", %{rolled_back: rolled_back} do + assert rolled_back.releases =~ ~r/#{@to}\s+unpacked/ + assert rolled_back.releases =~ ~r/#{@from}\s+permanent/ + end + end + + describe "committing a restart transition" do + test "installs again from the release that rolled back", %{installed: installed} do + assert installed.status == 0, installed.output + end + + test "reports what was committed", %{committed: committed} do + assert committed.output =~ "Committed #{@to}." + end + + test "makes it permanent", %{committed: committed} do + assert committed.releases =~ ~r/#{@to}\s+permanent/ + assert committed.releases =~ ~r/#{@from}\s+old/ + end + + test "points the stock launcher's version selection at it", %{committed: committed} do + # The file Castle never writes itself: make_permanent/1 does, and it is what + # turns the provisional version into the one an ordinary start boots. + assert committed.start_erl =~ @to + assert committed.version == "sample #{@to}" + end + + test "is what an ordinary restart then boots", %{restarted: restarted} do + # With no marker in sight, so nothing but start_erl.data selected it. + assert restarted.counter == ~s({"#{@to}", 0}) + assert restarted.releases =~ ~r/#{@to}\s+permanent/ + end + end +end diff --git a/test/support/deployment.ex b/test/support/deployment.ex index ab0ebe5..80b61cb 100644 --- a/test/support/deployment.ex +++ b/test/support/deployment.ex @@ -20,6 +20,11 @@ defmodule Forecastle.Deployment do alias Forecastle.Fixture + # How long `daemon` itself is given to return. Generous, because on a first + # start it sources `env.sh`, which runs a preboot VM to create + # `releases/RELEASES` and waits for it. + @start_timeout 180_000 + @doc """ Generates a relup between two assembled releases, into the workspace. @@ -53,10 +58,104 @@ defmodule Forecastle.Deployment do relup end - @doc "Starts an assembled release as a daemon and waits for it to answer." + @doc """ + Starts an assembled release as a daemon and waits for it to answer. + + Returns what the launcher printed, which is how a suite gets at what `env.sh` + said on the way past - the two streams merged, the way `Forecastle.Fixture` + merges them. `launcher!/3` raises if the start itself failed, so a caller that + ignores the return value still gets that. + + **The launcher is given a deadline, and that is not belt-and-braces.** A boot + that hangs is a real failure mode of what this suite covers - two `-heart` + flags leave `heart:check_start_heart/0` with no clause for `{ok, [[], []]}` and + the node never finishes starting, having printed nothing - and it hangs + *inside* `daemon` rather than after it, because `env.sh` runs a preboot VM + synchronously on a first start and that VM inherits the same options. + `System.cmd/3` has no deadline of its own and `setup_all` has no ExUnit + timeout, so without this a regression in the `-heart` guard stops the suite for + as long as whatever is running it will wait. Measured, by putting the guard + back the way it was. + """ def start!(deploy, env \\ []) do - launcher!(deploy, ["daemon"], env) + started = Task.async(fn -> launcher!(deploy, ["daemon"], env) end) + + output = + case Task.yield(started, @start_timeout) do + {:ok, output} -> output + _no_answer -> abandoned!(started, deploy) + end + await_boot!(deploy) + output + end + + defp abandoned!(started, deploy) do + Task.shutdown(started, :brutal_kill) + + flunk( + "#{deploy} did not finish starting within #{div(@start_timeout, 1000)}s. " <> + "A launcher that never returns is a boot that hung rather than one that " <> + "failed, and the usual cause is the VM being given two -heart flags: " <> + "init:get_argument(heart) reports {ok, [[], []]}, which heart's own " <> + "startup check has no clause for. Nothing is printed when that happens, " <> + "so there is no output to report here." + ) + end + + @doc """ + The operating system pid of the running release, as the release reports it. + + `bin/ pid` is an rpc, so this is the beam's own `System.pid/0` rather + than anything about the process that asked - which is what makes it usable + both for telling one incarnation of the node from another and for waiting on + the first to go away. + """ + def os_pid(deploy), do: launcher!(deploy, ["pid"]) + + @doc """ + Waits until the operating system process `pid` is gone, or fails the test. + + Asked of the operating system rather than of the node: a node that has stopped + answering rpc is not necessarily a process that has exited, and starting the + replacement while the old beam still holds the distribution port is how a + supervised restart turns into a name clash instead of a boot. + """ + def await_exit!(pid, attempts \\ 300) + + def await_exit!(pid, 0), do: flunk("process #{pid} was still running at the timeout") + + def await_exit!(pid, attempts) do + case System.cmd("ps", ["-o", "pid=", "-p", pid], stderr_to_stdout: true) do + {_output, 0} -> Process.sleep(100) && await_exit!(pid, attempts - 1) + {_output, _} -> :ok + end + end + + @doc """ + Installs `vsn` through `bin/castle` while acting as the release's supervisor. + + A transition that restarts the emulator reboots the node, and nothing inside + the release starts it again - that is the whole design: `bin/start` is inert + and systemd, Docker or runit owns the restart. So the test has to be the + supervisor. `bin/castle install` is run in a task, because it keeps asking the + system what it is running until the version it installed answers; this waits + for the old process to go, starts the release again, and then collects what the + install made of it. + + Returns `{output, status}`, with the two streams merged the way + `Forecastle.Fixture.cmd/4` merges them. + """ + def install_supervised!(deploy, vsn, env \\ []) do + pid = os_pid(deploy) + castle = Path.join(deploy, "bin/castle") + + installing = Task.async(fn -> Fixture.cmd(castle, ["install", vsn], env) end) + + await_exit!(pid) + start!(deploy, env) + + Task.await(installing, 300_000) end @doc "Waits until the release accepts an rpc, or fails the test." diff --git a/test/support/fixture.ex b/test/support/fixture.ex index 86e1c46..6ad160e 100644 --- a/test/support/fixture.ex +++ b/test/support/fixture.ex @@ -21,8 +21,31 @@ defmodule Forecastle.Fixture do # Mix variables that would otherwise leak from the parent test run into the # fixture build and silently redirect its output, and release variables that # would leak into a launcher the tests invoke. + # + # `ERL_AFLAGS`, `ERL_FLAGS` and `ERL_ZFLAGS` are here alongside + # `ELIXIR_ERL_OPTIONS` because all four carry flags to the emulator - erlexec + # prepends the first and appends the other two to the command line it builds - + # so any of them set in a developer's shell or a CI image would put a `-heart` + # into every start these suites make. A suite that wants one there sets it + # itself, which is what `Forecastle.RestartUpgradeTest` does. + # + # `ERL_OTP_FLAGS` is a fifth such variable - erlexec prepends it too - + # and it leaks the same way, but it cannot be written into this list because its + # name carries the emulator's OTP major. It is appended in `env/1` instead. The + # `env.sh` fragment measures that variable rather than modelling it now, which + # makes one leaked into an e2e start a way to hang a boot on two `-heart` flags + # rather than merely a way to add an unexpected one. + # + # `RELEASE_VM_ARGS` and `RELEASE_REMOTE_VM_ARGS` name the args file, which + # carries the same flags, and they belong here for a sharper reason than the + # others: they do not merely *add* a flag, they redirect the launcher to a + # different args file entirely. One set in the environment would have every + # release these suites assemble boot on some other project's vm.args, which would + # present as an assembly bug rather than as a leaked variable. @scrubbed ~w(MIX_BUILD_PATH MIX_BUILD_ROOT MIX_DEPS_PATH MIX_TARGET MIX_QUIET - MIX_DEBUG ERL_LIBS ELIXIR_ERL_OPTIONS RELEASE_ROOT RELEASE_NAME + MIX_DEBUG ERL_LIBS ELIXIR_ERL_OPTIONS ERL_AFLAGS ERL_FLAGS + ERL_ZFLAGS RELEASE_VM_ARGS RELEASE_REMOTE_VM_ARGS + RELEASE_ROOT RELEASE_NAME RELEASE_VSN RELEASE_COOKIE RELEASE_NODE RELEASE_TMP) def start_link(_opts \\ []) do @@ -96,6 +119,8 @@ defmodule Forecastle.Fixture do defp ensure_prepared(workspace), do: {workspace, workspace} defp env(extra) do - Enum.map(@scrubbed, &{&1, nil}) ++ [{"MIX_ENV", "prod"}, {"FORECASTLE_PATH", @root}] ++ extra + scrubbed = ["ERL_OTP#{:erlang.system_info(:otp_release)}_FLAGS" | @scrubbed] + + Enum.map(scrubbed, &{&1, nil}) ++ [{"MIX_ENV", "prod"}, {"FORECASTLE_PATH", @root}] ++ extra end end