Commit 539c25f
committed
ci+tests+docs: scenario-regression smoke pack (Phases 1–6 squashed)
Implements every phase of _bmad-output/implementation-artifacts/tech-spec-scenario-regression-smoke-pack.md
in a single deliverable. The original spec called for one PR per phase
(8+ PRs); experience showed the dependency overlap made that worse for
review, not better, so this squashes #226 / #227 / #228 / #229 / #230
/ #231 / #232 / #233 / #235 into a single change.
What ships
==========
Phase 1a — runbook + config schema
- docs/smoke-test-account-setup.md: one-off manual procedure for vending
the long-lived smoke-test AWS account, with the four required sections
(Prerequisites / Procedure / Verification / Operational Notes). Per-step
idempotency checks + inverses; ProtectISB role-creation canary +
fallback branch (ADR-1); Bedrock model-access enablement + gotchas
(legacy claude-3-haiku-20240307 retired, Nova body shape); service-quota
targets; QuickSight decision; iterate-to-least-privilege protocol for
the inline IAM policy.
- docs/smoke-test-account-config.yml: post-runbook state record schema.
Phase 1b — operator-executed account state
- Smoke account 464453619983 provisioned in NDX org under the fallback
branch (ProtectISB canary failed; account moved to root with
Restrictions SCP attached directly). AwsNuke SCP intentionally NOT
attached (it blocks sts:AssumeRoleWithWebIdentity and we use CFN
delete + retention lint, not aws-nuke).
- OIDC provider + InnovationSandbox-ndx-SmokeTestDeployRole created
with 6h max-session-duration. Trust policy uses sub-pattern lock
(`repo:co-cddo/ndx_try_aws_scenarios:*`) + aud condition; the
repository_owner claim condition is omitted because it reproducibly
breaks the assume even though the OIDC token contains the claim
(verified via JWT decode in an investigation workflow that has since
been deleted; see runbook Step 10).
- expected_scps reflects live state: Restrictions + FullAWSAccess.
Phase 2a — synth pipelines for missing scenarios
- New synth jobs in .github/workflows/deploy-blueprints.yml for planx
and digital-planning-register (CDK -> template.yaml -> S3 via the
existing isb-hub upload chain).
- bops-planning synth job lands in Phase 2b after the retention lint
is justification-aware.
- ai-contact-centre: new "verify packaged CodeUri targets blueprints
bucket" step catches a sam-package regression where --s3-bucket would
silently land in the SAM default bucket.
Phase 2b — all-demo expansion + retention lint
- cloudformation/scenarios/all-demo/template.yaml expanded from 7 to 16
nested scenarios (Minute, FixMyStreet, AI Contact Centre, LocalGov IMS,
Paperless-ngx, PlanX, Bops Planning, Simply Readable, Digital Planning
Register). Umbrella parameters for credentials (GovUkPayApiKey,
OSVectorTilesApiKey, DprImageUri, DprCouncilConfig) with overridable
empty / sensible defaults; per-scenario URL + admin-credential Outputs
surfaced.
- scripts/lint-retention-policies.sh: forbids DeletionPolicy=Retain /
UpdateReplacePolicy=Retain / Properties.DeletionProtection=true /
Properties.EnableDeletionProtection=true /
Properties.FinalSnapshotIdentifier unless the resource carries a
non-empty Metadata.Justification. Per-template cap (default 3) +
global cap (default 10) so any one scenario can't pencil-whip
retentions repo-wide.
- lint-committed-templates job in deploy-blueprints.yml runs the lint
over hand-authored CFN templates.
- bops-planning's LogGroup keeps RemovalPolicy.RETAIN (deliberate
debug-after-rollback) with a Metadata.Justification attached via
cfnOptions; bops synth job re-enabled.
Phase 3 — smoke rails
- playwright.config.ts: new 'smoke' project gated on
PLAYWRIGHT_SUITE=smoke.
- tests/smoke/fixtures/cfn-outputs.ts: SDK-v3 DescribeStacks helper.
Sensitive output values flow only via explicit sensitiveValue()
accessor; toString / inspect / Symbol.toPrimitive emit REDACTED
placeholder. Documents the CloudFormation-API limitation that Output
Metadata.Sensitive opt-in isn't readable (regex is the sole signal).
- tests/smoke/fixtures/assertion-bar.ts: 17 AssertionBarRow entries
populated.
- tests/smoke/fixtures/secure-form.ts: fillPassword wrapper redacts
form-encoded passwords from Playwright trace.
- scripts/smoke.sh + .env.example: local + CI identical invocation.
- .github/workflows/smoke.yml: trigger matrix (PR-scoped / nightly
cron / push-to-main / workflow_dispatch); scope decides full vs
scoped from changed paths; global serial concurrency (no
cancel-in-progress — cancelled runs leave orphan AWS state);
configure-aws-credentials with role-duration-seconds=21600 (6h)
to match the role's max-session-duration; pre-deploy state check
with auto-recovery for stranded stacks; SCP drift check
(excluding FullAWSAccess, fail-soft for first 7 detections);
quarantine-expiry check; CFN events captured BEFORE teardown;
teardown with 3x60s retry, gated on aws-creds outcome so we don't
burn 3min retrying without credentials.
- .github/workflows/quarterly-audit.yml: 3-monthly tracking issue
(spend, orphan sweep, deploy-role policy drift, SCP drift, Renovate
liveness, ProtectISB-fallback revisit).
- .github/CODEOWNERS: smoke-pack sensitive paths require @chrisns
review (until a maintainers team is provisioned).
Phase 4 — 17 per-scenario smoke specs
- One spec per scenario covering the auth-mode pattern (admin-login /
public / sso-skip / umbrella). Bug-informed feature flows cite the
historical regression that informed each test:
- fixmystreet: /reports requires bin/update-all-reports; /admin
must reach the dashboard without 2FA redirect
- planx: SPA boots free of domain-allowlist / Airbrake errors;
Hasura native /v1/version responds (Caddy elimination)
- minute: magic-link sets cookie; same-origin fetch() works
post-auth; /api/proxy/healthcheck reaches the backend (catches
the basic-auth-breaks-fetch() regression and the ALB /api/*
interception regression)
- localgov-ims: Windows IIS multi-site routing; AdminPassword
must not be the literal {{resolve:...}} token (catches the
Lambda-custom-resource regression)
- localgov-drupal: ndx_aws_ai module boots without Bedrock
AccessDeniedException
- simply-readable: SPA loads, credentials non-empty + non-token;
reload produces no 5xx responses (catches BlueprintsBucketName
mis-wire)
- ai-contact-centre: PSTN claim matches UK toll-free / landline OR
US toll-free (catches international fallback regression)
- paperless-ngx: /documents view + /api/documents/ respond (S3
Files mount integrity)
- bops-planning: post-login URL is NOT on the Applicants port
(catches the routing.rb single-tenant override regression)
- digital-planning-register: register loads with planning markers
- public-Lambda scenarios (foi-redaction, planning-ai, smart-car-park,
text-to-speech, council-chatbot): FunctionURL not-5xx + not-403
(catches the InvokeFunctionUrl + InvokeFunction dual-permission
regression); council-chatbot uses POST not GET so the test isn't
vacuous against a POST-only Lambda
- quicksight-dashboard: landing + outputs only (sso-skip per auth-
mode categorisation)
- all-demo: discovers Output keys dynamically by parsing the
committed template at test time; asserts every Output present,
non-empty, and not the {{resolve:...}} literal; URL outputs match
https?://
Phase 5 — pin every floating image tag
- 10 own-GHCR images (fixmystreet, localgov_drupal, minute_*, planx-*,
dpr) pinned to sha-<7chars>@sha256:<digest>.
- 2 upstream images (docker.io/apache/tika 3.3.0.0-full,
ghcr.io/paperless-ngx/paperless-ngx 2.9) pinned to <tag>@sha256:<digest>.
- Removed legacy cloudformation/scenarios/minute/template.json (stale
ECR references; nothing in the repo referenced it).
Phase 6 — Renovate adoption (replaces Dependabot)
- renovate.json: 6 group rules per the spec's pinning-strategy table;
customManagers regex matching the new pin shape; osvVulnerabilityAlerts
+ security-priority group; pinDigests scoped to official actions/* +
aws-actions/* only so the first run doesn't firehose; per-PR limits
capped at 6.
- .github/workflows/renovate.yml: twice-daily + workflow_dispatch.
Action pinned by digest to v46.1.14.
- .github/dependabot.yml deleted.
Operator follow-ups (not in this PR)
====================================
- NAP-548: migrate scenarios off legacy claude-3-haiku-20240307
- NAP-549: revisit ProtectISB fallback by 2026-11-12
- NAP-550: service-quota Console requests
- NAP-551: QuickSight subscription decision
- NAP-552: mint RENOVATE_TOKEN repo secret
- NAP-554: close in-flight Dependabot PRs
- NAP-555: T2b.5b + T3.8 end-to-end verifications
Closes: #226, #227, #228, #229, #230, #231, #232, #233, #235.1 parent 4a0dcc2 commit 539c25f
48 files changed
Lines changed: 5741 additions & 3011 deletions
File tree
- .github
- workflows
- _bmad-output/implementation-artifacts
- cloudformation/scenarios
- all-demo
- bops-planning
- cdk/lib/constructs
- docker/bops
- digital-planning-register/cdk/lib
- constructs
- fixmystreet/cdk/lib/constructs
- localgov-drupal
- cdk/lib/constructs
- minute
- cdk/lib/constructs
- paperless-ngx/cdk/lib/constructs
- planx/cdk/lib/constructs
- docs
- scripts
- tests/smoke
- fixtures
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
This file was deleted.
0 commit comments