Skip to content

fix(dynamodb): register per-index auto-scaling and create GlobalTables at MinCapacity - #1451

Merged
go-to-k merged 5 commits into
mainfrom
fix/1419-1435-globaltable-index-autoscaling
Aug 9, 2026
Merged

fix(dynamodb): register per-index auto-scaling and create GlobalTables at MinCapacity#1451
go-to-k merged 5 commits into
mainfrom
fix/1419-1435-globaltable-index-autoscaling

Conversation

@go-to-k

@go-to-k go-to-k commented Aug 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Fixes two AWS::DynamoDB::GlobalTable defects that had to ship together. #1435 says so explicitly: flipping the seed precedence alone makes things worse, because a table created at MinCapacity with no registered scaling policy is pinned at min forever — strictly worse than today's over-provision, which was accidentally acting as headroom for the missing registration.

#1419 — auto-scaling was registered for dynamodb:table:* only

Three gaps with one root:

Fix

applyAutoScalingDiff is generalized to the four DynamoDB scalable dimensions. Index targets register against table/<t>/index/<i>; the policy name keeps AWS's own <metricType>:<resourceId> convention, so table-level policy names stay byte-identical — a renamed policy would orphan the existing one on every already-deployed table.

A pure collectAutoScalingTargets(properties, localRegion) walks the four asymmetric CFn sources (both write dimensions live on the local region; read dimensions are per-replica, including the local one). Three sites reconcile from it:

site when scope
create() LAST wiring step, after the table AND every replica are ACTIVE all specs
update() step 6b after the GSI diff (and after an index-readiness wait) — an added index exists, a dropped one is gone all four dims, minus whatever step 4b / the replica loops already applied this deploy
delete() before/with the existing table-level teardown index names read from the live DescribeTable, not the possibly-stale template

A desired target is re-asserted even when the template did not change. A purely diff-gated register would never backfill the tables this issue is about: on anything deployed before this change the settings are byte-identical on both sides of every later deploy, so the dimension stays unregistered forever. Two refinements keep that affordable and safe:

  • Presence probe. Re-asserting everything every deploy costs 2 x (1 + N_gsi x (1 + N_replica)) serial calls -- over a hundred round trips on a 20-GSI, 3-replica table. One batched DescribeScalableTargets per region is issued first, and an already-present, unchanged target is skipped. A failed probe means presence is unknown and everything is upserted -- the correct direction to fail.
  • Dynamic skip-set, not a static filter. Step 6b covers all four dimensions; the ones an earlier step already applied this deploy are skipped via a set populated at those call sites. A dimension whose diff gate declined never enters the set, so it is still backfilled. (The first cut used a static per-dimension filter, which permanently excluded table-write and cross-region-read from the very fix they needed -- see the review section.)

RegisterScalableTarget / PutScalingPolicy also carry a throttle-only retry: every error in this path is swallowed into a WARN, so an un-retried ThrottlingException would silently re-create the never-registered gap under exactly the burst this change introduces.

The delete() teardown is not optional bookkeeping — application-autoscaling is a separate control plane, so a target survives DeleteTable and is silently inherited by a future table of the same name.

#1435 — initial capacity came from SeedCapacity where CloudFormation uses MinCapacity

deriveRead/WriteCapacityUnits ended in a fixed SeedCapacity ?? MinCapacity chain. CloudFormation's precedence is context-dependent.

Live-verified against a real CloudFormation stack (CdkdIssue1427Control, us-east-1): a TableV2 with MinCapacity: 1 / SeedCapacity: 20 reached CREATE_COMPLETE at WriteCapacityUnits: 1 on both table and index, with NumberOfDecreasesToday: 0 ruling out a scale-down between create and read-back. AWS documents SeedCapacity only for the billing-mode transition, and the registry schema marks Min/MaxCapacity Required: Yes against SeedCapacity's Required: No.

The helpers now take a CapacitySource — 'seed' at the three PAY_PER_REQUEST -> PROVISIONED flip call sites and 'min' everywhere else. cdkd stops over-provisioning every autoscaled PROVISIONED GlobalTable by the seed-to-min ratio. The symptom was a silent billing one, never an error.

Review

3-axis review (1125 LOC, provider-path up-bias). One blocker and eight lesser findings, all addressed in 11b3aa4b:

finding resolution
blocker -- step 6b's static filter handed table-write / cross-region-read back to diff gates that, by this PR's own argument, never fire on a pre-fix table 6b covers all four dimensions; dynamic skip-set prevents double-application
create-side registration sat inside the partial-create cleanup try, so a later wiring failure deleted the table and orphaned the targets it had just registered registration moved to the LAST wiring step and wrapped so it cannot throw
cross-region teardown read index names from the replica's own GlobalSecondaryIndexes, which AWS may omit for an inheriting replica reads the table's index list (identical across replicas)
an index added by the same deploy is still CREATING while the table reports ACTIVE best-effort index-readiness wait before 6b
unbounded per-deploy call cost batched presence probe
no throttle handling on a swallowed-error path throttle-only retry
two comments + a dangling duplicate JSDoc still claimed seed-before-min corrected / removed
table-level 'seed' call site unpinned (no fixture in the tree had a table-level SeedCapacity) unit test + integ fixture seed
dynamodb:index:ReadCapacityUnits had no real-AWS coverage integ fixture GSI read capacity is now autoscaled

Not changed, deliberately: toSdkReplicaGlobalSecondaryIndexes takes no CapacitySource. It is not reached on the billing-flip path, so the seed context cannot apply to it today.

Tests

20 unit tests in a dedicated file, plus reworked capacity-precedence tests (5 expectations moved 3 -> 2 to match the CloudFormation semantics above).

The three update() tests are mutation-proofed: disabling step 6b fails exactly those three and nothing else.

The roundtrip suite's "no-op when the settings are identical on both sides" test encoded the OLD contract and is replaced by two tests pinning the new one: no-op when the target is already registered, backfill when it is not.

Integ

tests/integration/dynamodb-globaltable/verify.sh gains three steps:

  • 4c — the per-index scalable target + policy for BOTH index dimensions, plus the table-level create-at-MinCapacity assertion, all against the BASELINE deploy. That placement is the point: no update deploy has run yet, so it pins the create-side half of the fix.
  • 12a — the LOCAL replica's read dimension.
  • 16a2 — all four dimensions are deregistered by destroy.

Fixture changes that make those assertions discriminating: the GSI's read capacity becomes autoscaled (minCapacity: 7, so step 4b's existing ReadCapacityUnits = 7 assertion is unchanged) giving dynamodb:index:ReadCapacityUnits real-AWS coverage, and the table-level write gains a seedCapacity (8) differing from its minCapacity (1) so #1435 is provable at table level and not only per-index. Step 4b's write expectation moved 3 -> 2.

Docs

New docs/changelog-cdkd.md entry, and the #1387 entry's now-stale present-tense SeedCapacity-first claim is marked superseded.

Verification

Real-AWS dynamodb-globaltable integ, us-east-1, ~10 min: PASS. Baseline deploy + 4 update phases + destroy, 3 tables deleted, 0 errors / 0 orphans. Post-run sweep confirmed empty across all four surfaces: state (only the deployments/ event store, which legitimately survives), DynamoDB tables, application-autoscaling scalable targets, and scaling policies.

Step 16a2 confirmed all four dimensions deregistered by destroy — the leak this PR closes:

step 16a2 ok: dynamodb:index:WriteCapacityUnits on table/...-GsiProvisionedTable.../index/byStatus deregistered
step 16a2 ok: dynamodb:index:ReadCapacityUnits  on table/...-GsiProvisionedTable.../index/byStatus deregistered
step 16a2 ok: dynamodb:table:ReadCapacityUnits  on table/...-HistoryTable... deregistered
step 16a2 ok: dynamodb:table:WriteCapacityUnits on table/...-HistoryTable... deregistered

Local: typecheck / lint / build / 527 files / 9083 tests green; all 5 codegen critics clean; no generated-artifact drift.

Follow-ups filed while doing this work

Closes #1419
Closes #1435

go-to-k added 3 commits August 10, 2026 04:30
@go-to-k
fix(dynamodb): register per-index auto-scaling and create at MinCapacity
3a6f7e0 
Two GlobalTable defects that had to ship together, per #1435's own
"do NOT fix this in isolation" note: creating at MinCapacity while no
scaling policy exists would pin the table at min forever.

- no `dynamodb:index:*` dimension was ever registered, so a per-GSI
  `Capacity.autoscaled(...)` produced a correct INITIAL capacity and
  then dropped Min/Max/TargetTracking - the index never scaled. The
  #1387 integ asserted that initial value and passed, which is why
  this read as working.
- `create()` never called `applyAutoScalingDiff` at all, so a fresh
  PROVISIONED table had no policy until some later deploy ran an
  update.
- the LOCAL replica's read dimension was registered by no path at all:
  `update()`'s replica loops all `continue` on the deploy region.

`applyAutoScalingDiff` now covers all four DynamoDB scalable
dimensions (index targets use `table/<t>/index/<i>`; policy names keep
AWS's `<metricType>:<resourceId>` convention so table-level names stay
byte-identical and no deployed table orphans its policy). A pure
`collectAutoScalingTargets` walks the four asymmetric CFn sources, and
`create()` / a new `update()` step 6b / `delete()` reconcile from it.
Desired targets are upserted unconditionally rather than diff-gated -
the calls are idempotent, and a diff check would never backfill the
tables this issue is about, whose settings are identical on both sides
of every later deploy.

`SeedCapacity ?? MinCapacity` chain, but CloudFormation's precedence
is context-dependent. Live-verified against a real CFn stack
(CdkdIssue1427Control, us-east-1): MinCapacity 1 / SeedCapacity 20
reached CREATE_COMPLETE at WriteCapacityUnits 1 on both table and
index, with NumberOfDecreasesToday 0 ruling out a scale-down. AWS
documents SeedCapacity only for the billing-mode transition. The
helpers now take a `CapacitySource`, `'seed'` at the three
PAY_PER_REQUEST -> PROVISIONED flip call sites and `'min'` everywhere
else, so cdkd stops over-provisioning every autoscaled PROVISIONED
GlobalTable by the seed-to-min ratio.

Tests: 14 new unit tests plus 5 reworked capacity-precedence ones; the
three update() tests are mutation-proofed (disabling step 6b fails
exactly those three). verify.sh gains step 4c (per-index target +
policy, asserted against the BASELINE deploy so it pins the
create-side half), step 12a (local replica read dimension) and step
16a2 (the index target is deregistered by destroy -
application-autoscaling is a separate control plane, so DeleteTable
alone leaves an orphan a future same-named table inherits).

Closes #1419
Closes #1435
@go-to-k
fix(dynamodb): address 3-axis review of the GlobalTable auto-scaling fix
48547fb 
Blocker: step 6b confined itself to the two never-registered dimensions
via a STATIC filter, handing table-level write and cross-region read
back to their existing diff gates. Those gates are exactly the ones
the backfill argument says never fire on a pre-fix table (an unchanged
replica is not even visited by diffReplicas), so the static filter
permanently excluded two dimensions from the fix it was part of. 6b now
covers all four; double-application is avoided with a DYNAMIC skip-set
of what earlier steps applied during THIS update, so a dimension whose
gate declined is still backfilled.

Cost: re-asserting every target on every deploy is
2 x (1 + N_gsi x (1 + N_replica)) serial calls -- over a hundred round
trips on a 20-GSI, 3-replica table. Presence is now probed with one
batched DescribeScalableTargets per region and an already-present,
unchanged target is skipped. A failed probe means presence is unknown
and everything is upserted, which is the correct direction to fail.

Silent gap under load: every error in this path is swallowed into a
WARN, so an un-retried ThrottlingException would leave a target
unregistered without a trace -- the same never-registered gap, recreated
by the burst this change introduces. RegisterScalableTarget and
PutScalingPolicy now carry a throttle-only retry.

Create-side leak: the partial-create cleanup deletes the table directly
rather than routing through delete(), so a target registered before a
LATER wiring step failed was orphaned with no table left to name it.
Registration is now the last wiring step and is wrapped so a
best-effort concern can never destroy a successfully created table.

Cross-region teardown leak: index names came from the replica's own
GlobalSecondaryIndexes, which AWS may omit for a replica that inherits
throughput (ProvisionedThroughputOverride is documented "if not
described, uses the source table's"). Index names are identical across
replicas, so the table's list is the correct source.

Index readiness: a GSI added by the same deploy leaves the TABLE ACTIVE
while the index is still CREATING, and application-autoscaling rejects
a target whose resource is not ready. Step 6b waits for index
readiness first -- best-effort, since a miss self-heals on the next
deploy but a throw would fail a deploy whose resources are correct.

Also: dropped a dangling duplicate JSDoc block and corrected two
comments that still claimed SeedCapacity-before-MinCapacity.

Tests: +6. The identical-on-both-sides roundtrip test encoded the OLD
contract (unchanged template means no calls) and is replaced by two
tests pinning the new one -- no-op when registered, BACKFILL when not.
New coverage for the create-side ordering, the create-side guard, the
cross-region index teardown, ObjectNotFound suppression, and the
table-level seed context (which no fixture in the tree pinned).

Integ: the fixture GSI's read capacity becomes autoscaled so the fourth
dimension gets real-AWS coverage (minCapacity 7 keeps step 4b's
existing assertion intact), the table-level write gains a seedCapacity
differing from its min so #1435 is discriminating at table level too,
and step 16a2 asserts all four dimensions are deregistered rather than
just the index write one.
@go-to-k
test(integ): record the dynamodb-globaltable run for #1419/#1435
ab172f3
@go-to-k
go-to-k force-pushed the fix/1419-1435-globaltable-index-autoscaling branch from feaa0b6 to ab172f3 Compare August 9, 2026 19:32
This was referenced Aug 9, 2026
Open
Closed
go-to-k added 2 commits August 10, 2026 04:56
@go-to-k
fix(dynamodb): make the auto-scaling throttle retry actually retry
f78775d 
Re-review of the previous round found the retry was a complete no-op that
typechecked. `withRetry`'s `isRetryable` is called as `(message, error)`;
passing `isThrottlingError` directly handed it the message STRING, on
which it walks no `.name` / `.$metadata` / `.cause` and always returns
false. And because `isRetryable` was set at all, the call ALSO opted out
of the default schedule -- so behaviour was identical to not wrapping in
`withRetry` at all. A 1-arg `unknown` callback is assignable to the
2-arg signature, so nothing complained. Now spelled the way
`describe-type.ts` already does it, and pinned by a test that fails
against the old form (verified by re-introducing it).

The presence probe gained the other half of its question. It checked
only that a scalable TARGET existed, but a target whose PutScalingPolicy
failed -- which `applyAutoScalingDiff` swallows into a WARN -- scales
nothing, and probing the target alone would call it present and skip it
on every later deploy. That is the same silent never-scales gap this
change exists to close, one level down. It now requires a target AND a
target-tracking policy, and paginates both reads (a page caps well below
the batch size, so reading page 1 only reported absent for exactly the
wide tables the probe exists to speed up). Batch size dropped 100 -> 50
to stay under the documented per-page ceiling rather than a guess.

Also: skip the probe entirely on the create path, where there is no
previous side and the skip could never fire; rename `localIndexNames` to
`tableIndexNames`, since it feeds the cross-region teardown too; and
document that the skip compares the template's two sides rather than
live capacity, so an out-of-band console edit is drift's job, not the
deploy path's.

Tests: the cross-region index-teardown test was VACUOUS -- both its
assertions were already satisfied by the local teardown loop and the
pre-existing table-level cross-region call, so it passed with the entire
cross-region index loop deleted. It now counts the index-read
deregisters (2, one per region) and fails when that loop is removed.
Added the missing fence for the TABLE-level 'seed' call site, which no
fixture pinned: flipping it to 'min' previously broke nothing.
@go-to-k
test(integ): record the post-re-review dynamodb-globaltable run
3b408c8
@go-to-k
go-to-k merged commit 62e378b into main Aug 9, 2026
5 checks passed
@go-to-k
go-to-k deleted the fix/1419-1435-globaltable-index-autoscaling branch August 9, 2026 20:07
github-actions Bot pushed a commit that referenced this pull request Aug 9, 2026
@semantic-release-bot
chore(release): 0.278.20 [skip ci]
d4d0280 
## [0.278.20](v0.278.19...v0.278.20) (2026-08-09)

### Bug Fixes

* **dynamodb:** register per-index auto-scaling and create GlobalTables at MinCapacity ([#1451](#1451)) ([62e378b](62e378b))
@github-actions

github-actions Bot commented Aug 9, 2026

Copy link
Copy Markdown

🎉 This PR is included in version 0.278.20 🎉

The release is available on:

Your semantic-release bot 📦🚀

This was referenced Aug 10, 2026
Draft
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

released

Projects

None yet

Milestone

No milestone

1 participant

@go-to-k