Skip to content

Commit 35bf33f

Browse files
Merge issue #96: accept partial ECS deployment receipts safely
2 parents 75048ec + 7847285 commit 35bf33f

8 files changed

Lines changed: 3042 additions & 476 deletions

File tree

_docs/compatibility/development-terminology-allowlist.json

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -180,15 +180,15 @@
180180
},
181181
{
182182
"path": "core/tests/test_deployment_release.py",
183-
"sha256": "81fa8cab67c79820ec4d09affc73a3f98c0445d1fc6c6b5cc4683d0e038033c6",
183+
"sha256": "ec57d47da62e84830502f1d85aafeca4e30b2c7b9cb3bdd032e70351f56e70ca",
184184
"class": "legacy_contract_test",
185185
"reason": "Release tests exercise exact physical task, secret, repository, and tag values.",
186186
"follow_up": "#94",
187187
"expected_count": 5
188188
},
189189
{
190190
"path": "core/tests/test_deployment_workflow.py",
191-
"sha256": "0d8ddc78f70978ad964456d1df7fb2278e7298baa72bd709944648ccf443cd34",
191+
"sha256": "4988cbb3f08250db969fdcad5d56bb2161940cb389d52f5d5975ba57c89d83c0",
192192
"class": "legacy_contract_test",
193193
"reason": "Workflow tests exercise removed inputs, frozen Gate-B records, and exact physical values.",
194194
"follow_up": "#94",

_docs/runbooks/development-release.md

Lines changed: 63 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -746,10 +746,12 @@ replace them with name searches.
746746
## Automatic compensation
747747

748748
A migration launch failure, timeout, missing exit code, or nonzero exit changes no service. A
749-
failure after the web update attempts restoration of **both** prior exact service task-definition
750-
ARNs and desired counts, even if worker had not moved. It waits for both services, enforces at
751-
most one running/pending worker, validates prior digest/SHA, and verifies prior public health.
752-
The database remains migrated forward.
749+
failure after web mutation restores the prior exact web task definition, count, and receipt-bound
750+
deployment ID. If worker `UpdateService` was actually invoked, its restoration is receipt-bound as
751+
well. If worker was untouched, compensation issues no worker mutation and instead read-only proves
752+
its captured task definition, count, PRIMARY deployment ID, terminal state, and singleton bound as
753+
part of the exact pair. It then validates the prior digest/SHA and public health. The database
754+
remains migrated forward.
753755

754756
The workflow concurrency group is `website-development-release` with cancellation disabled. Never
755757
cancel an in-progress release to start another one.
@@ -959,9 +961,11 @@ promotion or rollback that intentionally starts/replaces web receives the explic
959961
240-second web-stabilization budget; 240 seconds is also its hard maximum. Only a forward promotion
960962
or rollback that starts/replaces the singleton worker receives the separate explicit, code-owned
961963
420-second worker-stabilization budget; 420 seconds is also its hard maximum. Neither value is a
962-
workflow-dispatch input or an arbitrary operator override. The controller makes one final ECS
963-
service observation at the monotonic deadline and never sleeps or polls again afterward. Exact
964-
completion in that observation succeeds; an incomplete or invalid observation fails.
964+
workflow-dispatch input or an arbitrary operator override. A mutating `UpdateService` call starts
965+
only while time remains. Read-only polling makes exactly one final ECS service observation at the
966+
monotonic deadline and never sleeps or polls again afterward. Exact completion in that observation
967+
succeeds; an incomplete or invalid observation fails. Any response returned after the deadline is
968+
discarded.
965969

966970
Web completion still requires its expected task-definition ARN and desired count, exactly one
967971
`PRIMARY` deployment with that definition and count, exact service and primary running/pending
@@ -974,12 +978,28 @@ pending task; queue activity, heartbeat, or a processed job is not completion.
974978

975979
Every service mutation is receipt-bound. Before `UpdateService`, the controller records that exact
976980
workload as attempted and supplies its captured terminal predecessor: the exact task-definition
977-
ARN, desired count, and unique PRIMARY deployment ID. The response must identify the configured
978-
service, the requested target tuple, and one new non-empty PRIMARY deployment ID. That ID must
979-
differ even for a same-ARN/count force-new deployment. All response deployments must be the exact
980-
receipt target or an exact phase predecessor; a third ID, task-definition/count cross-pair,
981-
malformed count, failed target, or old completed PRIMARY returned as the new deployment fails the
982-
mutation.
981+
ARN, desired count, and unique PRIMARY deployment ID. The controller establishes the phase's
982+
absolute monotonic deadline before the API call. The acknowledgement, any immediate reconciliation,
983+
and receipt-bound stabilization share that deadline; none may reset it.
984+
985+
A complete acknowledgement binds one new non-empty deployment ID for the requested task-definition
986+
ARN. The immutable receipt target remains the requested tuple. For a positive requested count, ECS
987+
may initially return that exact new PRIMARY at deployment-level desired/running/pending/failed
988+
`0/0/0/0` and `IN_PROGRESS`, while the service-level tuple already has the requested count. That
989+
AWS initialization shape binds identity but is poll-only. It does not rewrite `B/1` to `B/0`, prove
990+
stabilization, start public health, or permit the worker mutation.
991+
992+
A structurally partial acknowledgement is reconciled with an immediate `DescribeServices` call;
993+
there is no preliminary sleep. Every present member must be correctly typed, and the complete set
994+
of present deployment members must be extendable to at least one allowed target, initialization,
995+
or predecessor shape. Omitting an ID or another member cannot hide an already-provable tuple,
996+
count, or rollout-state contradiction. Genuinely missing information remains unknown, and missing
997+
identity is never synthesized from the request. Reconciliation may bind only one exact new target
998+
deployment ID distinct from every predecessor ID. A third ID, multiple candidates,
999+
task-definition/count cross-pair, malformed present member, failed target, positive failed tasks,
1000+
or completed-inexact target fails immediately. A service-level zero target for a positive request,
1001+
or a zero-count candidate with a positive running, pending, or failed-task count or a state other
1002+
than `IN_PROGRESS`, is also an immediate contradiction.
9831003

9841004
Subsequent service reads may temporarily cross the service-level and PRIMARY target tuples or
9851005
return an exact captured predecessor after the receipt has already appeared. These recognized
@@ -994,22 +1014,48 @@ deployment, and strict target-only validation failed before ECS could publish th
9941014
Compensation then encountered the inverse replica-ordering window. This run remains failed; it is
9951015
not evidence of a successful release.
9961016

1017+
Run `31279458131` attempt 1 on 2026-08-08 supplied the initialization evidence. Its exact new web
1018+
PRIMARY acknowledgement carried the requested task-definition ARN and a new deployment ID, but the
1019+
deployment still reported `0/0/0/0 IN_PROGRESS` while service desired count was 1. The old receipt
1020+
parser rejected that provider-valid acknowledgement, and both restorative acknowledgements hit the
1021+
same false negative. The run remains failed even though the candidate web task appeared publicly
1022+
later: it produced no accepted receipt, worker proof, terminal pair, smoke result, or successful
1023+
release record.
1024+
9971025
If an attempted `UpdateService` response is lost or invalid, recovery uses the same absolute
9981026
180-second per-service deadline for candidate reconciliation and the restorative receipt wait.
9991027
It polls only the captured terminal tuple and the actually attempted target until that target is
1000-
observed as the unique PRIMARY and its deployment ID can be bound. The deadline is finite,
1001-
inclusive, cannot exceed the general-stage maximum, and is never restarted between capture and
1002-
restore. A speculative workload that was never invoked is absent from the recovery allowlist.
1028+
observed as the unique PRIMARY and its deployment ID can be bound while time remains. The deadline
1029+
is finite, cannot exceed the general-stage maximum, and is never restarted between capture and
1030+
restore. A capture that returns exactly at the deadline cannot start the restorative mutation. A
1031+
speculative workload that was never invoked is absent from the recovery allowlist.
10031032
An attempted candidate may be failed while its restorative receipt replaces it, but the recovery
10041033
receipt itself may not fail. Every restorative call must return a new receipt, including an
10051034
`A -> A` force-new recovery; old `A COMPLETED` is always a predecessor, never recovery success.
1035+
If web fails before worker `UpdateService` is invoked, compensation does not force a new worker
1036+
deployment. It read-only verifies the captured worker's exact terminal tuple and singleton state as
1037+
part of the final pair proof. Once worker mutation was actually attempted, its restoration remains
1038+
receipt-bound. Artifact-finalization recovery intentionally restores both workloads because both
1039+
belong to the failed release that had already reached terminal proof.
10061040

10071041
For triage, compare redacted service/PRIMARY tuples and deployment IDs against the recorded receipt
1008-
and phase predecessors. Do not retry the mutation, add an unrecognized identity, pre-sleep before
1042+
and phase predecessors. Forward, rollback, compensation, and artifact-finalization recovery expose
1043+
each bound receipt using only its workload, deployment ID, allowlisted binding reason, and whether
1044+
an exact terminal reconciliation observation was carried. The summary is recorded before waiting,
1045+
so it remains available after a post-binding failure. Evidence uses only allowlisted reason codes
1046+
for complete binding, zero-count initialization, partial acknowledgement reconciliation,
1047+
contradiction, and receipt deadline expiry; it never stores the raw provider payload. Do not retry
1048+
the mutation, add an unrecognized identity, pre-sleep before
10091049
the first observation, or infer adoption from a running task, target health, logs, or an old
10101050
completed deployment. A candidate that never becomes the unique PRIMARY by the shared recovery
10111051
deadline, or any third/cross-paired identity, leaves recovery failed closed for operator review.
10121052

1053+
Restorative failure classification retains `receipt_deadline_expired` only when every observed
1054+
restorative error is that allowlisted deadline reason. If any workload, terminal, or health error is
1055+
`contract_contradiction`, that contradiction takes precedence. A generic exception or an unknown
1056+
reason is also collapsed to `contract_contradiction`; raw exception messages and provider payloads
1057+
are never propagated into evidence or CLI output.
1058+
10131059
The conservative critical-stage recovery envelope is
10141060
`180 + 120 + 240 + 180 + 420 + 180 + 360 + 720 = 2400` seconds: migration observation,
10151061
stopped-migration terminal proof, web stabilization, public readiness/liveness, worker

0 commit comments

Comments
 (0)