Skip to content

Commit 69ce2c8

Browse files
daithihearnclaude
daithihearn
and
claude
committed
feat: add a hibernate switch for the dev environment
Dev bills the same idle as busy. hibernate-dev.yml parks its ECS service at zero tasks on demand and restores it to the autoscaling floor on the way back, sharing deploy.yml's dev concurrency group so the two cannot interleave. Target tracking is suspended before the count drops, never after: Application Auto Scaling holds a service at its target's min_capacity, so a live target scales it straight back out. Terraform does not manage suspended_state, so that survives every apply until a wake lifts it — which is why the roll job now asserts the service is awake before it deploys. That assertion reads the service rather than a flag, so it also heals a half-applied hibernate, and it only acts below the floor. It lives in roll rather than its own job because a job referencing a protected environment adds another approval gate to prod. Only the task stops; the NAT gateway, both ALBs, the cache and the WAF bill on regardless. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent 656effa commit 69ce2c8

3 files changed

Lines changed: 255 additions & 1 deletion

File tree

‎.github/workflows/README.md‎

Lines changed: 36 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -60,7 +60,7 @@ same image, so parity is structural rather than maintained by hand.
6060
1. `guard` - Resolves the target, the tag and its content SHA, refuses anything not reachable from `main`, and derives the per-environment names below
6161
2. `promote` - Copies the GHCR manifest list into that account's ECR, keeping both arches. Each tag is written only if absent
6262
3. `terraform` - Applies each stack serially in dependency order, planning immediately before each apply
63-
4. `roll` - Moves the service to the new task definition revision unless it already runs it, then checks `/api/health` and that `/api/metrics` is 403, both over the ALB by name
63+
4. `roll` - Wakes the service if it is parked, moves it to the new task definition revision unless it already runs it, then checks `/api/health` and that `/api/metrics` is 403, both over the ALB by name
6464
5. `draft-release` - Calls `prepare-release.yml` (dev only, and not on dispatch)
6565

6666
**Domains:** `https://dev.block-explorer.vechain.org`, `https://block-explorer.vechain.org`
@@ -116,6 +116,38 @@ drafting a release for it would offer prod that rollback as the next cut.
116116

117117
---
118118

119+
### Hibernate Dev (`hibernate-dev.yml`)
120+
121+
**Triggers:** manual dispatch, `mode` of `hibernate` or `wake`
122+
123+
Parks `block-explorer-dev-frontend` at zero tasks when dev is not needed, and restores it to its
124+
autoscaling floor on the way back. It shares `deploy.yml`'s `deploy-dev` concurrency group, so it
125+
cannot interleave with a dev deploy.
126+
127+
Order is load-bearing: target tracking is suspended **before** the count drops, because Application
128+
Auto Scaling holds a service at its target's `min_capacity` and a live target scales it straight back
129+
out. Terraform does not manage `suspended_state`, so a suspension survives every apply — which is
130+
also why hibernating and then deploying without waking would leave a service that cannot scale.
131+
132+
The service and cluster names come from `terraform/frontend`'s outputs rather than being spelled out
133+
in the workflow: a wrong name scales nothing and still reports success.
134+
135+
Waking is idempotent and lives in two places. This workflow's `wake` mode is the operator's button;
136+
`deploy.yml`'s `roll` job asserts the same thing on every deploy, so a deploy can never land on a
137+
parked service. It reads the service rather than a flag, which means it also heals a hibernate that
138+
half-applied and a service someone scaled down by hand, and it only acts *below* the floor — capacity
139+
autoscaling put above it is not the deploy's to reset. It sits in `roll` rather than in its own job
140+
because a job referencing a protected environment adds another approval gate to prod.
141+
142+
**While hibernated:** `https://dev.block-explorer.vechain.org` returns 503. Nothing alarms — dev sets
143+
`alerts_enabled: false`, which builds no alarms and no AMP alert rules at all, so there is no red
144+
state to sit in and nothing to silence before parking the service.
145+
146+
Only the ECS task stops. The NAT gateway, both ALBs, the Valkey cache and the WAF bill on, and they
147+
are most of what dev costs — hibernating buys back roughly the Fargate line, not the environment.
148+
149+
---
150+
119151
### Static Checks (`static-checks.yml`)
120152

121153
Runs the org's reusable `checkov.yaml` and `action-lint.yaml` on every PR and on `main`, behind a
@@ -324,6 +356,9 @@ lock. The cost is that a queued teardown can be evicted by a newer run, which is
324356
- **Cache:** shared with dev, namespaced by image tag
325357
- **Auto-cleanup:** destroyed when the PR closes, and swept every six hours regardless
326358

359+
### Dev
360+
- **Hibernation:** `hibernate-dev.yml` parks the service at zero tasks on demand; any dev deploy wakes it
361+
327362
### Image Lifecycle (ECR)
328363
- Keeps last 30 production images (tagged `v.*`)
329364
- Keeps last 10 preview images (tagged `pr-*`)

‎.github/workflows/deploy.yml‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -405,6 +405,59 @@ jobs:
405405
aws-region: ${{ env.AWS_REGION }}
406406
role-to-assume: ${{ vars.AWS_OIDC_ROLE_ARN }}
407407

408+
# Reverses hibernate-dev.yml, and heals one that only half-applied. Reads
409+
# the service rather than a flag, so a hand-scaled service wakes too, and
410+
# only acts below the floor — autoscaling above it is not ours to reset.
411+
- name: Wake the service if it is parked
412+
env:
413+
CLUSTER: ${{ needs.terraform.outputs.cluster }}
414+
SERVICE: ${{ needs.terraform.outputs.service }}
415+
run: |
416+
set -euo pipefail
417+
418+
resource="service/${CLUSTER}/${SERVICE}"
419+
target=$(aws application-autoscaling describe-scalable-targets \
420+
--service-namespace ecs \
421+
--scalable-dimension ecs:service:DesiredCount \
422+
--resource-ids "$resource" \
423+
--no-cli-pager --query 'ScalableTargets[0]' --output json)
424+
425+
floor=$(jq -r '.MinCapacity // 1' <<<"$target")
426+
suspended=$(jq -r '[.SuspendedState // {} | .[]] | any' <<<"$target")
427+
desired=$(aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \
428+
--no-cli-pager --query 'services[0].desiredCount' --output text)
429+
430+
if [ "$desired" -ge "$floor" ] && [ "$suspended" = "false" ]; then
431+
echo "${SERVICE} is running ${desired} task(s) and scaling freely."
432+
exit 0
433+
fi
434+
435+
restored=()
436+
437+
if [ "$suspended" = "true" ]; then
438+
aws application-autoscaling register-scalable-target \
439+
--service-namespace ecs \
440+
--scalable-dimension ecs:service:DesiredCount \
441+
--resource-id "$resource" \
442+
--suspended-state '{"DynamicScalingInSuspended":false,"DynamicScalingOutSuspended":false,"ScheduledScalingSuspended":false}' \
443+
--no-cli-pager
444+
restored+=("target tracking resumed")
445+
fi
446+
447+
if [ "$desired" -lt "$floor" ]; then
448+
aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" \
449+
--desired-count "$floor" --no-cli-pager --query 'service.desiredCount' --output text
450+
restored+=("desired count ${desired} -> ${floor}")
451+
fi
452+
453+
did=$(printf '%s, ' "${restored[@]}")
454+
{
455+
echo "### Woken from hibernation"
456+
echo
457+
echo "\`${SERVICE}\`: ${did%, }."
458+
echo
459+
} >> "$GITHUB_STEP_SUMMARY"
460+
408461
# The revision rather than the image tag, so a cpu, memory, env or sidecar
409462
# change that leaves the image alone still rolls.
410463
- name: Check whether a roll is needed

‎.github/workflows/hibernate-dev.yml‎

Lines changed: 166 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,166 @@
1+
name: Hibernate Dev
2+
3+
# Dev bills the same idle as busy. This parks its ECS service at zero tasks and
4+
# brings it back on demand. A dev deploy wakes it too, so forgetting to wake
5+
# costs nothing.
6+
on:
7+
workflow_dispatch:
8+
inputs:
9+
mode:
10+
description: "hibernate parks the service at zero tasks; wake restores it"
11+
required: true
12+
type: choice
13+
options:
14+
- hibernate
15+
- wake
16+
default: hibernate
17+
18+
run-name: ${{ inputs.mode == 'wake' && 'Wake dev' || 'Hibernate dev' }}
19+
20+
# deploy.yml's dev group, so this can never interleave with a dev deploy.
21+
concurrency:
22+
group: deploy-dev
23+
cancel-in-progress: false
24+
25+
env:
26+
AWS_REGION: eu-west-1
27+
TERRAFORM_VERSION: 1.16.0
28+
29+
jobs:
30+
capacity:
31+
name: Set Dev Capacity
32+
runs-on: ubuntu-latest
33+
permissions:
34+
id-token: write
35+
contents: read
36+
# Declared for vars.AWS_OIDC_ROLE_ARN, which resolves per Environment. No
37+
# `url`: half of what this workflow does is make that URL stop answering.
38+
environment:
39+
name: dev
40+
steps:
41+
- name: Checkout code
42+
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
43+
44+
- name: Configure AWS credentials
45+
uses: aws-actions/configure-aws-credentials@e6de054238d6b7531b4efff3b6587d9aade6a06c # v6.2.3
46+
with:
47+
aws-region: ${{ env.AWS_REGION }}
48+
role-to-assume: ${{ vars.AWS_OIDC_ROLE_ARN }}
49+
50+
- name: Setup Terraform
51+
uses: hashicorp/setup-terraform@dfe3c3f87815947d99a8997f908cb6525fc44e9e # v4.0.1
52+
with:
53+
terraform_version: ${{ env.TERRAFORM_VERSION }}
54+
terraform_wrapper: false
55+
56+
# From state, the way the deploy reads them: a name spelled out here would
57+
# scale nothing and still report success.
58+
- name: Read service outputs
59+
id: outputs
60+
working-directory: terraform/frontend
61+
run: |
62+
set -euo pipefail
63+
64+
terraform init -input=false -backend-config=../environments/dev/backend.config
65+
terraform workspace select dev
66+
{
67+
echo "cluster=$(terraform output -raw cluster_name)"
68+
echo "service=$(terraform output -raw service_name)"
69+
} >> "$GITHUB_OUTPUT"
70+
71+
# Suspended before the count drops, never after: Application Auto Scaling
72+
# holds the service at the target's min_capacity, so a live target scales
73+
# it straight back out. Terraform does not manage suspended_state, so the
74+
# suspension survives every apply until a wake lifts it.
75+
- name: Park the service
76+
if: inputs.mode == 'hibernate'
77+
env:
78+
CLUSTER: ${{ steps.outputs.outputs.cluster }}
79+
SERVICE: ${{ steps.outputs.outputs.service }}
80+
run: |
81+
set -euo pipefail
82+
83+
resource="service/${CLUSTER}/${SERVICE}"
84+
target=$(aws application-autoscaling describe-scalable-targets \
85+
--service-namespace ecs \
86+
--scalable-dimension ecs:service:DesiredCount \
87+
--resource-ids "$resource" \
88+
--no-cli-pager --query 'ScalableTargets[0].ResourceId' --output text)
89+
90+
if [ "$target" != "$resource" ]; then
91+
echo "No scalable target on ${SERVICE}; nothing to suspend."
92+
else
93+
aws application-autoscaling register-scalable-target \
94+
--service-namespace ecs \
95+
--scalable-dimension ecs:service:DesiredCount \
96+
--resource-id "$resource" \
97+
--suspended-state '{"DynamicScalingInSuspended":true,"DynamicScalingOutSuspended":true,"ScheduledScalingSuspended":true}' \
98+
--no-cli-pager
99+
echo "Target tracking suspended on ${SERVICE}."
100+
fi
101+
102+
aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" \
103+
--desired-count 0 --no-cli-pager --query 'service.desiredCount' --output text
104+
aws ecs wait services-stable --cluster "$CLUSTER" --services "$SERVICE"
105+
echo "${SERVICE} is drained."
106+
107+
# Restores to the target's own floor rather than to 1, so this stays right
108+
# if desired_count in the env YAML ever changes, and only from below it —
109+
# pressing wake on a service that is already up must not scale it in.
110+
- name: Wake the service
111+
if: inputs.mode == 'wake'
112+
env:
113+
CLUSTER: ${{ steps.outputs.outputs.cluster }}
114+
SERVICE: ${{ steps.outputs.outputs.service }}
115+
run: |
116+
set -euo pipefail
117+
118+
resource="service/${CLUSTER}/${SERVICE}"
119+
target=$(aws application-autoscaling describe-scalable-targets \
120+
--service-namespace ecs \
121+
--scalable-dimension ecs:service:DesiredCount \
122+
--resource-ids "$resource" \
123+
--no-cli-pager --query 'ScalableTargets[0]' --output json)
124+
125+
floor=$(jq -r '.MinCapacity // 1' <<<"$target")
126+
desired=$(aws ecs describe-services --cluster "$CLUSTER" --services "$SERVICE" \
127+
--no-cli-pager --query 'services[0].desiredCount' --output text)
128+
129+
if [ "$(jq -r '[.SuspendedState // {} | .[]] | any' <<<"$target")" = "true" ]; then
130+
aws application-autoscaling register-scalable-target \
131+
--service-namespace ecs \
132+
--scalable-dimension ecs:service:DesiredCount \
133+
--resource-id "$resource" \
134+
--suspended-state '{"DynamicScalingInSuspended":false,"DynamicScalingOutSuspended":false,"ScheduledScalingSuspended":false}' \
135+
--no-cli-pager
136+
echo "Target tracking resumed on ${SERVICE}."
137+
fi
138+
139+
if [ "$desired" -lt "$floor" ]; then
140+
aws ecs update-service --cluster "$CLUSTER" --service "$SERVICE" \
141+
--desired-count "$floor" --no-cli-pager --query 'service.desiredCount' --output text
142+
fi
143+
144+
aws ecs wait services-stable --cluster "$CLUSTER" --services "$SERVICE"
145+
echo "${SERVICE} is serving on the revision it was parked with."
146+
147+
- name: Summary
148+
env:
149+
MODE: ${{ inputs.mode }}
150+
SERVICE: ${{ steps.outputs.outputs.service }}
151+
run: |
152+
{
153+
if [ "$MODE" = "hibernate" ]; then
154+
echo "### Dev hibernated"
155+
echo
156+
echo "- \`${SERVICE}\` is at zero tasks with target tracking suspended."
157+
echo "- https://dev.block-explorer.vechain.org now returns 503 — the ALB has no targets."
158+
echo "- Nothing alarms: dev sets \`alerts_enabled: false\`, so no alarms or AMP alert rules exist."
159+
echo "- Wake it by re-running this workflow with \`wake\`, or by deploying dev — the deploy wakes it on its own."
160+
else
161+
echo "### Dev woken"
162+
echo
163+
echo "- \`${SERVICE}\` is serving again and target tracking is live."
164+
echo "- https://dev.block-explorer.vechain.org"
165+
fi
166+
} >> "$GITHUB_STEP_SUMMARY"

0 commit comments

Comments
 (0)