Re-monolithize the Ansible orchestration null_resources behind a structured-logging orchestrator script

[gwright99]

Summary

Collapse most of the 10 sequential null_resource blocks in 011_configure_vm.tf back into a single null_resource that delegates to a remote orchestrator script. The orchestrator wraps each Ansible playbook / shell step in structured log markers so per-stage failure attribution survives — which was the original reason for atomizing in the first place — but eliminates the per-apply plan noise that the atomization produces today.

Backlog item — not for immediate implementation. Captured here for the next maintenance cycle that has appetite for this kind of restructuring.

Background

The current chain in 011_configure_vm.tf is 10 null_resource blocks executed sequentially: ssh_connectivity_check → file_transfer → host_configuration → ansible_setup → system_packages → update_configuration_files → pull_containers_run_tower → wait_for_tower → patch_groundswell → run_seqerakit.

Each block invokes local-exec to SSH into the EC2 and run a short shell command (typically ansible-playbook -i inventory.ini NN_some_step.yml). Every block uses triggers = { always_run = "${timestamp()}" }, so every block re-runs on every apply.

The header comment on the file is explicit about why this shape exists:

While useful from an administrative view, [the monolithic original] had coarse runtime granularity
-- you could not easily see what command was executing and stack traces from failed executions.
Accepting repetitive boilerplate in return for finer granularity and more visibility.

That decision was correct given the logging situation at the time. Ansible playbook failures inside a monolithic step were a debugging nightmare — terraform output was an undifferentiated wall of text, and identifying which playbook out of N had failed required scrolling through hundreds of lines of mixed Ansible output.

Atomization solved the visibility problem at the cost of:

• Plan-output noise. Every apply shows 10 null_resource lines being recreated, even when nothing meaningful changed. Operators have to mentally tune out the noise to see real changes.
• Per-step SSH/EICE handshake overhead. Mitigated partially by the SSH ControlMaster work, but still present for the first connection and for the file-transfer step.

Proposed change

1. Remote orchestrator script

Ship a single bash orchestrator script (assets/src/bash/remote/orchestrate.sh.tpl or similar) that runs all the per-stage commands previously spread across the 10 null_resources. Each stage is wrapped in structured log markers that survive arbitrary failure modes. Sketch:

#!/usr/bin/env bash
set -euo pipefail

LOG=/var/log/tower-installer/apply-$(date -u +%Y%m%dT%H%M%SZ).log
mkdir -p "$(dirname "$LOG")"
exec > >(tee -a "$LOG") 2>&1

CURRENT_STAGE=""
trap '[[ -n "$CURRENT_STAGE" ]] && echo "==== STAGE FAILED: $CURRENT_STAGE (exit=$?) ===="' ERR

stage() {
  CURRENT_STAGE=$1
  shift
  local start=$(date +%s)
  echo "==== STAGE START: $CURRENT_STAGE  $(date -u +%FT%TZ) ===="
  "$@"
  local end=$(date +%s)
  echo "==== STAGE OK:    $CURRENT_STAGE  ($((end - start))s) ===="
  CURRENT_STAGE=""
}

cd /home/ec2-user

stage host_configuration  bash target/bash/remote/cleanse_and_configure_host.sh
stage ansible_wait        bash target/ansible/00_wait_for_ansible.sh
stage system_packages     ansible-playbook -i target/ansible/inventory.ini target/ansible/01_load_system_packages.yml
stage update_configs      ansible-playbook -i target/ansible/inventory.ini target/ansible/02_update_file_configurations.yml
stage pull_containers     ansible-playbook -i target/ansible/inventory.ini target/ansible/03_pull_containers_and_run_tower.yml
stage wait_for_tower      ansible-playbook -i target/ansible/inventory.ini target/ansible/04_wait_for_tower.yml
stage patch_groundswell   ansible-playbook -i target/ansible/inventory.ini target/ansible/05_patch_groundswell.yml
[[ "${RUN_SEQERAKIT:-false}" == "true" ]] && stage seqerakit ansible-playbook -i target/ansible/inventory.ini target/ansible/06_run_seqerakit.yml

Key properties:

• Bracketing markers (==== STAGE START: …, ==== STAGE OK: …, ==== STAGE FAILED: …) are consistent and grep-able. A single grep 'STAGE FAILED' /var/log/tower-installer/apply-*.log finds the failing stage in any apply.
• trap … ERR plus set -e guarantees a STAGE FAILED: line whenever any stage's command exits non-zero, regardless of what tool produced the failure (bash, Ansible, Docker, etc.).
• Last STAGE START: without a matching STAGE OK: is the failure point, by construction. No need to interpret tool-specific output.
• Per-stage elapsed time prints with the OK marker, so apply-time analysis (which stage is slow) is trivial.
• Persistent log on the host at /var/log/tower-installer/apply-*.log means post-mortem analysis doesn't require re-running terraform with TF_LOG=DEBUG. SSH in, read the log.
• exec > >(tee -a $LOG) 2>&1 captures both stdout and stderr in the log file and still sends them to the SSH session for terraform to surface.

2. Terraform side: collapse the 10 null_resources

resource "null_resource" "ssh_connectivity_check" {
  # Keep this one — it's a local-side connectivity probe, not part of the remote orchestration.
  count    = var.flag_vm_copy_files_to_instance == true ? 1 : 0
  triggers = { always_run = timestamp() }
  # ...existing local-exec block unchanged...
}

resource "null_resource" "file_transfer" {
  # Keep this one — copies assets/target/ to the host. Has to run before orchestrate.sh exists on the host.
  count      = var.flag_vm_copy_files_to_instance == true ? 1 : 0
  triggers   = { always_run = timestamp() }
  depends_on = [null_resource.ssh_connectivity_check]
  # ...existing local-exec block unchanged...
}

resource "null_resource" "configure_vm" {
  # Replaces: host_configuration, ansible_setup, system_packages, update_configuration_files,
  # pull_containers_run_tower, wait_for_tower, patch_groundswell, run_seqerkit
  count      = var.flag_vm_copy_files_to_instance == true ? 1 : 0
  triggers   = { always_run = timestamp() }
  depends_on = [null_resource.file_transfer]

  provisioner "local-exec" {
    quiet       = false  # Let the orchestrator's stage markers stream to terraform output
    command     = "ssh -T ${var.app_name} 'RUN_SEQERAKIT=${var.flag_run_seqerakit} bash /home/ec2-user/target/bash/remote/orchestrate.sh'"
    interpreter = ["/bin/bash", "-c"]
  }
}

Three null_resources instead of ten. The two we keep (ssh_connectivity_check, file_transfer) are genuinely local-side prerequisites — they can't be folded into a remote orchestrator that depends on files the remote doesn't have yet. Everything that was a remote SSH-invoke step collapses into configure_vm.

3. Drift compensation by default

triggers = { always_run = timestamp() } stays as the trigger for configure_vm. Every apply re-runs the full orchestration → Ansible re-applies its idempotent tasks → drift gets corrected. The "noise" of one resource always re-running is one line in the plan, not ten.

If a deployer ever wants to skip the reconciliation (e.g., quick infra-only iteration without touching the running Tower), that's a separate decision and can be wired as an explicit tfvars flag (flag_skip_vm_reconcile = true) — orthogonal to this restructuring.

Expected outcome

Plan output, today

Plan: 10 to add, 0 to change, 10 to destroy.

  # null_resource.ssh_connectivity_check[0] must be replaced
-/+ resource "null_resource" "ssh_connectivity_check" { ... }
  # null_resource.file_transfer[0] must be replaced
-/+ resource "null_resource" "file_transfer" { ... }
  # null_resource.host_configuration[0] must be replaced
-/+ resource "null_resource" "host_configuration" { ... }
  ... (7 more lines like this)

Plan output, after this change

Plan: 3 to add, 0 to change, 3 to destroy.

  # null_resource.ssh_connectivity_check[0] must be replaced
-/+ resource "null_resource" "ssh_connectivity_check" { ... }
  # null_resource.file_transfer[0] must be replaced
-/+ resource "null_resource" "file_transfer" { ... }
  # null_resource.configure_vm[0] must be replaced
-/+ resource "null_resource" "configure_vm" { ... }

Apply output during normal run

null_resource.configure_vm[0]: Creating...
null_resource.configure_vm[0] (local-exec): ==== STAGE START: host_configuration  2026-06-27T14:23:01Z ====
null_resource.configure_vm[0] (local-exec): [Ansible's normal output …]
null_resource.configure_vm[0] (local-exec): ==== STAGE OK:    host_configuration  (4s) ====
null_resource.configure_vm[0] (local-exec): ==== STAGE START: ansible_wait  2026-06-27T14:23:05Z ====
... (continues stage by stage)
null_resource.configure_vm[0]: Creation complete after 2m26s

Apply output during a failure

null_resource.configure_vm[0] (local-exec): ==== STAGE START: patch_groundswell  2026-06-27T14:25:12Z ====
null_resource.configure_vm[0] (local-exec): [Ansible's error output …]
null_resource.configure_vm[0] (local-exec): ==== STAGE FAILED: patch_groundswell (exit=2) ====
Error: local-exec provisioner error

A single grep against the on-host log file pulls the exact failing stage and its timing.

What we lose by re-monolithizing

Being honest about the trade:

1. Per-stage retry attribution at the Terraform level. Today, if patch_groundswell fails and you fix the root cause, the next apply only re-runs patch_groundswell and downstream. After this change, it re-runs from host_configuration.
   • Mitigation: Ansible playbooks are already idempotent. Re-running stage 1 against a host that's already been configured = changed: false everywhere = a few seconds of evaluation, not a real "redo." On the happy path the cost is small.
   • The pathological case is a long-running early stage that's expensive even when idempotent (rare in practice for this stack — system_packages with yum is the slowest, and even that's seconds when everything's already installed).
2. Ability to terraform taint a specific Ansible step. You can't taint null_resource.patch_groundswell[0] because that resource won't exist. Tainting configure_vm re-runs the orchestrator from the top.
   • Mitigation: for the rare "I need to re-run exactly one playbook in isolation" case, SSH in and run ansible-playbook -i target/ansible/inventory.ini target/ansible/05_patch_groundswell.yml directly. That's faster than a terraform apply anyway.
3. Per-stage timing in terraform's own output. The Creation complete after Ns line goes away for individual stages. The orchestrator's stage markers replace it but they're inside terraform's stdout stream rather than first-class.
   • Mitigation: the on-host log file has timestamps for everything, and stage durations are printed with each STAGE OK: line. Operators who want per-stage timing have a better place to look than terraform's output anyway.
4. Failure mode if the orchestrator script itself has a bug. If stage() or the trap is broken, you lose the structured failure attribution and you're back to "wall of Ansible output."
   • Mitigation: the orchestrator is small (~30 lines of bash) and trivially unit-testable with shellcheck + a small harness. Worth a few tests covering the success path, ERR-trap behaviour, and RUN_SEQERAKIT toggle. See test matrix.
5. One large blast radius for a script bug. If the orchestrator misbehaves in some new way, it affects every apply for every site. With 10 null_resources, a bug in one provisioner block was scoped to that step.
   • Mitigation: ship behind a feature flag for the first release window — flag_use_orchestrator = true defaulted to true but operators can roll back to the old chain by flipping it. Remove the old chain in a follow-up release once confidence has built. See TBDs.

None of these are dealbreakers. The biggest one in practice is (1), and Ansible idempotency takes most of the sting out of it.

What this issue changes about the existing backlog

This is worth being explicit about because it has implications for tickets already filed:

• The SSH ControlMaster work still helps but the magnitude shrinks. After this change, the orchestrator does its work over a single SSH session — ControlMaster only matters for the two remaining external-to-orchestrator SSH calls (ssh_connectivity_check, file_transfer). Still worth keeping the directives in place, but the headline benchmark (~57% wall-clock reduction) was measured with 10 SSH connections; the post-monolith picture will show less.

Net: this issue subsumes part of the previously-planned optimization work and shrinks the rest. Worth landing this before doing the per-step hash work that the docker-compose ticket implicitly assumed.

What to test before merging (when this is picked up)

Scenario	What to verify
Fresh deploy (no prior state on host)	Orchestrator runs all stages cleanly; log file exists at /var/log/tower-installer/apply-*.log; each stage has matched STAGE START: / STAGE OK: markers
Re-apply on healthy host	Orchestrator runs all stages; Ansible reports changed: false for most tasks; total time is short (idempotency working)
Intentional stage failure (e.g., break a playbook deliberately)	Terraform fails; orchestrator's last line is STAGE FAILED: <stage_name> (exit=<n>); log file persists on host with full trace
Mid-stage SSH disconnect (simulate network blip)	Terraform fails cleanly; orchestrator's trap ERR handler still fires; log file shows what was in flight
flag_run_seqerakit = false	Seqerakit stage is skipped (no STAGE START: seqerakit in the log); no spurious failure
flag_run_seqerakit = true	Seqerakit stage runs after patch_groundswell; appears in the log
Apply with flag_skip_vm_reconcile = true (if that escape hatch is implemented)	configure_vm does not run; no other regression
Drift-recovery scenario	Manually break something on the host (stop a container, edit a config file). Next apply runs orchestrator → drift is corrected. Verify in the log that Ansible re-asserted the bad state
Apply, kill terraform mid-stage, re-apply	Re-apply runs orchestrator from the top; idempotent stages no-op; broken state is corrected if any
shellcheck on the orchestrator script	No findings (or annotated findings only)
Orchestrator unit harness	Tests pass: stage() prints expected markers on success; ERR trap fires STAGE FAILED: with correct stage name; RUN_SEQERAKIT env var gates the optional stage correctly
Log file rotation / disk usage	After N apply runs, log directory size is bounded (rotation policy or size cap in place)

TBDs

• Where the orchestrator script lives. Suggestion: assets/src/bash/remote/orchestrate.sh.tpl, rendered via templatefile() so it can interpolate values like the playbook directory or version-pinned paths. Decide before implementation.
• Feature-flag the rollout. Recommended: flag_use_orchestrator default true, with the old 10-step chain kept in a legacy_011_configure_vm.tf.disabled file (or behind the inverse of the flag) for one release window. Then the old chain gets deleted entirely in the following release. Avoids "we broke every site on day one of the new shape."
• Log file rotation policy. /var/log/tower-installer/apply-*.log accumulates one file per apply. A long-lived host that gets re-applied weekly for a year has 52 logs. Decide: rotate at file count (keep last 20), rotate at age (delete > 30 days), or accept unbounded with a documented cleanup command. Each has trade-offs for forensic value vs disk hygiene.
• Whether to keep ssh_connectivity_check and file_transfer as separate resources. Pros of keeping them separate: clear local-side responsibility, fail-fast on connectivity issues before the heavy work begins. Cons of keeping them separate: still two extra entries in the plan output. Reasonable to keep them separate; revisit if the noise still bothers anyone after the main collapse.
• An escape hatch for "I want to skip reconcile on this apply." A flag_skip_vm_reconcile tfvars option would let an operator do an infra-only apply without paying the orchestrator cost. Default false. Useful for fast iteration on tfvars values that only affect Terraform-side resources (e.g., tagging changes, output formatting). Implement now, or defer to a follow-up?

Related

• Subsumes / changes scope of: docker-compose-speedup, per-step hash-trigger work
• Pairs with (reduced benefit): SSH ControlMaster — still helps the remaining external SSH calls but the headline win shrinks once the orchestrator monolithises the bulk of the chain
• Original architectural context: the July 28/25 header comment on 011_configure_vm.tf explains why atomization happened in the first place. This issue is essentially the reversal of that decision, made possible by the structured-logging discipline that wasn't in place at the time.
• Background discussion: thread under #404 where the broader "pipeline pays a fixed cost per step regardless of need" pattern was identified and the trade-offs between per-step optimization and re-monolithization were weighed.