Upstreaming LZA-grade AWS control-plane depth — a sequenced, per-scope PR plan (feedback wanted before we open anything)

[sam-fakhreddine]

Hi floci maintainers 👋

We've been running floci as the target for a full Landing Zone Accelerator (LZA) deployment, and to get an org-scale CDK/CodePipeline run to go green end-to-end we built out a fair amount of AWS control-plane depth on our fork. We'd like to contribute it back — but on your terms and at your pace, not as a wall of PRs dropped on an already-busy queue.

This post is the "ask first" step. Before we open a single PR, we want to know: is this direction welcome, and how would you like us to sequence it?

What it adds, at a glance

• AWS Organizations — new service: org/root/OU/account lifecycle, all four policy types, handshakes, delegated admins, service access, resource policy.
• IAM SCP enforcement — SCPs as a hard deny-ceiling evaluated before identity policies; account-root bounded by SCPs; aws:PrincipalArn populated for principal-scoped conditions.
• Service Quotas and RAM — two new services.
• S3 — opt-in global bucket namespace + replication config.
• CloudFormation — CDK Provider framework custom resources (async onEvent/isComplete), account-scoped stacks/exports (Fn::ImportValue), real changeset diffs, SSM-typed parameters, and provisioning of CodeBuild/CodePipeline resources.
• CodeBuild — curated-image resolution, faithful buildspec execution, file-mode/symlink fidelity, and concurrency/disk bounds for fan-out.
• CodePipeline — EventBridge events, SNS approvals, V2 stage conditions, real CodeBuild-action integration.
• Step Functions — ASL Retry (used by the CDK provider polling loop).
• Cross-account Lambda invoke, transparent AWS-endpoint interception for spawned containers, Config compliance loop, plus a few test-hardening fixes.

Existing behavior is unchanged by default

Everything governance-related is opt-in and default-off. Two flags gate every SCP/identity check:

• FLOCI_SERVICES_IAM_ENFORCEMENT_ENABLED (default false)
• FLOCI_SERVICES_ORGANIZATIONS_SCP_ENFORCEMENT_ENABLED (default false)

The S3 global namespace is likewise behind FLOCI_SERVICES_S3_GLOBAL_BUCKET_NAMESPACE (default off). Turn nothing on and main behaves exactly as it does today.

We did the merge-overlap homework

Before proposing anything we audited all open PRs on the repo, to land with the community rather than across it:

• No feature is a duplicate. Every capability is either a brand-new service absent from main or an enhancement to an existing service that no open PR is also making.
• Merge overlap is real but mostly mechanical — most shared-file edits are additive registry entries that resolve trivially. The one genuine hotspot is CloudFormationResourceProvisioner.java, which a large number of open PRs touch.

Because of that hotspot, we do not want to submit a monolith. The plan below is a set of per-scope PRs, sequenced safe → hot, so the zero-collision new services land first and the contested provisioner comes last with the smallest possible surface.

Proposed PR sequence (per-scope, dependency-ordered)

Each is feat(<scope>), one feature per PR, reconstructed fresh on top of current main, with its own doc updates and tests.

#	PR	Collision risk	Depends on
1	feat(organizations): AWS Organizations service	none (new files)	—
2	feat(servicequotas,ram): Service Quotas + RAM sharing	none (new files)	—
3	feat(iam): enforce SCPs, populate aws:PrincipalArn	shared IAM files	PR 1
4	feat(s3): global bucket namespace + replication config	shared S3 files	—
5	feat: cross-account routing + container reliability	low	—
6	feat(codebuild): LZA-grade build execution	low	PR 5
7	feat(codepipeline): events, approvals, V2 conditions, CodeBuild actions	low/med	PR 6
8	feat(cloudformation): CDK Provider framework, account-scoped stacks, service provisioning	high (the hotspot)	PR 6 + PR 7

We'd open these in waves (1–2 first, then 3–4, then 5, then 6→7→8), waiting for each wave to be reviewed/mergeable before the next — so you're never looking at all eight at once.

Complementary to in-flight work

The biggest related effort we saw is the community's CDK-enablement series (individual AWS::* resource types so CDK-synthesized templates deploy). Our CloudFormation work — the async custom-resource layer those templates ultimately exercise — is meant to complete that story, not compete with it. We're glad to sequence PR 8 behind that series and coordinate directly with its authors, and likewise to coordinate on the IAM-enforcement PR with anyone currently touching the IAM filter.

What we'd like from you

1. Is this direction something you want upstream at all? A "no" or "only parts of it" now saves everyone review time.
2. How would you like it paced/sequenced? Happy to follow your preferred order, split any PR finer, or hold pieces indefinitely.
3. Who should we coordinate with on the IAM-enforcement and CloudFormation/CDK overlaps?

No rush and no pressure — we'll wait for your read before opening the first PR. Thanks for building floci; it's been genuinely useful to run a real LZA against. 🙏

[sam-fakhreddine]

To make this reviewable async without putting anything in your PR queue, the eight per-scope branches are on our fork now — here are the diffs, one link per proposed PR:

#	PR	Diff
1	feat(organizations)	main → feature/organizations
2	feat(servicequotas,ram)	main → feature/servicequotas-ram
3	feat(iam)	feature/organizations → feature/iam
4	feat(s3)	main → feature/s3
5	feat (cross-account + reliability)	main → feature/cross-account-reliability
6	feat(codebuild)	feature/cross-account-reliability → feature/codebuild
7	feat(codepipeline)	feature/codebuild → feature/codepipeline
8	feat(cloudformation)	feature/codepipeline → feature/cloudformation

On the stacking: PRs 3, 6, 7, and 8 are stacked, so each link above shows only that PR's own delta on top of its parent branch (matching the "Depends on" column). PRs 1, 2, 4, and 5 are independent and diff straight against main. When we actually open these upstream they'll rebase onto main as each parent merges.

These are branches only — no PRs opened, nothing in your queue, nothing running your CI. Browse whenever suits you, and we'll take our cue on order and pacing from your reply above.

[sam-fakhreddine]

cc @hectorvent — flagging this for your eyes since it touches the CloudFormation / CDK provider path most directly. Very happy to sequence the hotter PRs (especially the CloudFormation one) behind whatever you and the CDK-series folks have in flight, and to adjust scope or ordering however works best for the roadmap. No rush at all — just wanted to make sure it was on your radar.

[hectorvent]

Hi @sam-fakhreddine,

This sounds great and will be a solid addition to Floci.

One request: could you hold off on opening the CloudFormation PRs for now? I want to merge what's already in flight and then refactor. Right now the CloudFormation handling is a God class and it's extremely hard to maintain. I'll ping you once it's ready on your side.

[sam-fakhreddine]

In the mean time, I am going to stress test this bigly.

[sam-fakhreddine]

@hectorvent adding ControlTower

[sam-fakhreddine]

Update, plus one question at the end.

Thanks @hectorvent — understood on CloudFormation, and we've held it. No CF PR is coming until you say so.

cc @pgermosen — adding you since CODEOWNERS routes /src/, /docs/ and /compatibility-tests/ to both of you, and nearly everything below lands in /src/. Very happy to take direction from either of you.

Since the last comment the work moved on a lot, so we've rebuilt the fork from scratch. The eight branches linked above still exist, but they've been rebased onto current main and extended, and there are now 46 branches rather than eight. Rather than dump 46 links on you, here's the shape and the homework.

Why this took a while

Worth explaining the gap since the last comment, because it's the reason we trust this code.

None of it was written from specs. Every branch came out of running a real org-scale Landing Zone
Accelerator deployment against Floci, end to end, over and over — and then fixing whatever the run
actually hit. We did that across three LZA releases — 1.14.5, 1.15.5 and 1.16.0 — building a parent
environment for each, and kept going until the same config produced the same result run-to-run
rather than passing once and failing the next time for a different reason. Reaching that consistency,
not writing the features, was most of the elapsed time. Chasing a failure that only reproduces every
third run is slow work, and several of the fixes below only became findable once the runs were stable
enough to tell a real defect from noise.

That process logged 34 distinct defects, and it is where the unglamorous half of this work comes
from — the account-scoping bugs, the CodeBuild fan-out and disk bounds, the Docker connection
handling, the custom-resource timeout semantics. Those were not things we predicted; they were
things a CDK deployment of this size found for us, usually at hour three of a pipeline run.

The practical upshot for review: the branches that LZA exercises hard (Organizations, Control Tower,
CodeBuild, CodePipeline, IAM, S3, EC2, CloudFormation) have a lot of real mileage behind them. The
API-parity branches have much less — they were filled in from Botocore shapes and have only their own
tests behind them. We'd rather say that plainly than present the two as equally proven, and it's part
of why we're asking you to pick the order rather than guessing at it ourselves.

Everything is rebased onto current main

All 46 sit on main as of 39ed7712. Not "roughly current" — every branch literally contains that commit. Our fork's main is synced to yours too, so the compare links diff against the same thing you're looking at.

Four themes

1. New services — organizations, servicequotas-ram, controltower, network-firewall, service-catalog, sso-admin, route53resolver, firehose, logs
2. API-parity fills on existing services — acm, apigateway, autoscaling, ec2-ipam, ec2-ebs-default-encryption, ec2-vpc-endpoint-security-groups, iam-group-and-instance-profile-tagging, ram-resource-share-lifecycle, kms-grant-fidelity, dynamodb-hybrid-persistence, ssm-*, and friends
3. Container/runtime reliability — cross-account-reliability, docker-connection-pools, docker-connection-stability, codebuild*, codepipeline*
4. CloudFormation — held, not proposed

Full list: https://github.com/sam-fakhreddine/floci/branches

Some of these are deliberately stacked (e.g. codebuild → docker-connection-pools → docker-connection-stability), so each one's diff shows only its own delta. We've verified every parent/child link survived the rebase.

What we checked before proposing anything

Rebase-tested all 46 against current main. 24 rebase completely clean. The rest conflict only on the additive hotspots — EmulatorConfig.java service toggles (your #2422 touched exactly that spot), ResolvedServiceCatalog, and the README/docs service counts. Mechanical, not semantic. Every branch whose conflicts we resolved has been compile-checked.

Checked our own work against what recently landed in main, and cut things:

• Dropped a commit of ours entirely — our Lambda ARN-account invoke fix is superseded by your fix(lambda): resolve ARN and function URL invocations in owning account #2414, which is more thorough than ours was. Removed from all four branches that carried it.
• Deleted a duplicate branch of our own that had a byte-identical diff to another.
• Reworked our Route 53 work. We had a branch that overlapped fix: preserve Route 53 private hosted-zone VPC associations #2342. Looking closely, fix: preserve Route 53 private hosted-zone VPC associations #2342 landed the VpcAssociation model and create-time association but not the association lifecycle actions. So rather than propose a competing model, we threw ours away, adopted yours, and re-derived only what's missing — AssociateVPCWithHostedZone, DisassociateVPCFromHostedZone, CreateVPCAssociationAuthorization, ListHostedZonesByVPC — as feature/route53-vpc-association-actions. Your existing Route 53 tests still pass alongside the new ones.

Note that ec2-ipam deliberately excludes its CloudFormation provisioning commit; that's held with the rest of the CF work.

On the CloudFormation hold — findings, and a question

We held, and used the time to re-derive our CloudFormation work onto current main rather than sit on a stale branch. Two things came out of it that you should know before deciding anything.

First: you've already built some of it. Re-deriving commit by commit, two of ours turned out to be redundant against main:

• Skipping resources whose Condition evaluates false — yours is more complete than ours was; it also handles deletion on the update path and tracks delete failures. Dropped.
• Account-scoping stacks and exports — main already has AccountAwareStorageBackend<Stack>, per-account exports, and legacy-key migration. Dropped.

So the CloudFormation contribution is smaller than our original post implied. We'd rather tell you that than have you find it in review.

Second: it no longer grows the God class. Our footprint in CloudFormationResourceProvisioner was +70/−15 lines out of ~7,000. After restructuring it is +6/−15 — the file ends up 9 lines shorter than it is on main today.

What moved out:

• The bulk (~55 lines) was CDK Provider-framework async detection and an idle-timeout budget. That was never resource-type logic, so it now lives in its own collaborator; the provisioner keeps a one-line delegation.
• Inline-zip packaging moved to its own class — a straight removal.
• What remains is that, plus a 3-line Fn::GetAtt fix on AWS::SSM::Parameter.
• CodeBuild and CodePipeline provisioning were already new files under provisioners/, touching nothing in the God class.

Being straight with you: the God class isn't our only overlap. We also change CloudFormationService.java by a few hundred lines (changeset diffs, SSM-typed parameters, the CDK Provider custom-resource path), so this doesn't magically make the work refactor-proof.

The question: what's left that touches that file is exactly the work you're about to do yourself, and we don't want to collide with you. Do you want any of this before your refactor, or should the CloudFormation set stay parked until you ping? We're happy to sit on it indefinitely — just tell us which.

We ran deep reviews on three branches and fixed what they found

Rather than hand you unexamined code, we ran multi-agent reviews on one branch from each family and
fixed everything real they turned up — 8 genuine bugs across 16 findings. A sample of what that
caught, all now fixed with regression tests that we verified fail against the unfixed code:

• ImportApiKeys rejected AWS's own CSV format. We required lowercase name,value,enabled;
  AWS uses Name,Key,Description,Enabled,UsagePlanIds. The CLI, the SDKs and Terraform would all
  have got a 400 on first contact.
• UpdateModel and UpdateUsagePlan ignored the patchOperations envelope, so a real SDK call
  returned HTTP 200 with the resource unchanged — a silent no-op, which is worse than an error.
• A non-numeric authorizerResultTtlInSeconds permanently bricked the authorizer: the store
  write happened before serialisation, so every later GetAuthorizer 500'd until delete-and-recreate.
• TLS SANs missed virtual-hosted S3. A wildcard matches one label, so *.amazonaws.com never
  covered my-bucket.s3.<region>.amazonaws.com — the DNS spoof routed those to us and the handshake
  then failed.
• Rejected writes left partial state in Organizations: TagResource applied tags before checking
  the 50-tag limit, and UpdatePolicy renamed before validating content.

We also found and fixed a class of our own docs errors — duplicated service counts and table rows —
across every affected branch.

Two things we noticed in main while doing this, which are yours rather than ours and which we have
NOT touched in any feature branch, to keep those diffs focused:

• docs/services/index.md links to five anchors that no page defines (cloudwatch.md#metrics,
  dynamodb.md#streams, ses.md#v2, api-gateway.md#v2, appconfig.md#data-plane). The file
  already uses the {#anchor} convention elsewhere, so it is a mechanical fix. Happy to send it as
  its own small docs PR if useful.
• A test-isolation issue: LambdaFunctionUrlAccountRoutingTest, LambdaReactiveSyncIntegrationTest
  and LambdaVersionIntegrationTest fail when run in the same selection as the CloudFormation tests,
  and pass in isolation. Reproducible on clean main (768 tests, 4 failures), so it is not from our
  work — but it is a false-positive generator for anyone reviewing a branch that touches Lambda.

Five of them are open as PRs on our fork, if that's easier to read

Rather than ask you to browse compare links, we've opened preview PRs against our own fork — one
per family, five because that's the cap you'd face upstream. Nothing is in your queue, nothing counts
against that cap, no reviewers are assigned, and you get no notification unless you choose to comment:

Family	PR	Provenance
New service	#2 organizations	LZA-verified + SDK compatibility test
Runtime / container	#3 codebuild	LZA-verified
API parity	#4 apigateway	Botocore-derived, not LZA-exercised
Cross-account	#5 cross-account-reliability	LZA-verified
IAM / SCP enforcement	#6 iam	LZA-verified

Each follows your PR template, including an honest answer to the AWS Compatibility question — which
SDK and CLI versions actually verified the wire protocol, and where we have no such verification.
PR #4 is deliberately the weakest of the five and says so; it's the family your own review rubric
would have the most to say about.

#6 is stacked on #2 rather than on main, so its diff shows only its own 1,824 lines instead of
replaying Organizations. That's how we'd sequence it upstream too, and it's worth seeing because 11
of the 46 branches are stack children — reviewing a parent makes each child small.

What we'd like — and a concrete offer

We're not opening anything upstream yet, and we won't until you say so.

The main question: what's your preferred strategy here? We have far more than the 5-PR cap
allows, it's your roadmap and your review time, and we'd rather be told than guess. Specifically:

1. Sequencing — which ≤5 would you like first? Our instinct is the clean-rebasing new services,
   but we're happy to start anywhere, split any branch finer, or drop pieces entirely.
2. Pace — one wave at a time with each landing before the next, or something slower? We're not
   in a hurry and would rather not crowd your queue.
3. Scope — is there anything here you'd simply rather not carry? A "no thanks" on any branch
   saves us both the review round-trip.
4. CloudFormation — before or after your refactor (see above)?

On those preview PRs above: they are diff-and-comment surfaces only — our fork has CI disabled,
so there are no check runs on them. The branches are verified, we just did it locally: the full
suite (12,194 tests) passes on the integration rollup that merges all 46 together, and each branch
passes its own focused selection. We did not want to imply a green tick you cannot see.

If you would rather not look at any of it until the CloudFormation refactor is done, that is
completely fine — nothing here is time-sensitive.

[pgermosen]

@sam-fakhreddine Lets start with the new services, rebase, and point to main floci

[sam-fakhreddine]

Wave 1 is open — the new services, rebased onto current main as requested:

PR	Service	Provenance
#2578	feat(controltower) — landing-zone + baseline lifecycle, guardrail reconciliation	LZA-verified
#2579	feat(servicequotas,ram) — Service Quotas + RAM resource sharing	LZA-verified
#2580	feat(networkfirewall) — Network Firewall	LZA-verified
#2581	feat(ssoadmin) — Identity Center ListInstances stub	LZA-verified
#2582	feat(route53resolver) — Route 53 Resolver + DNS Firewall lists	LZA-verified
#2583	feat(firehose) — Start/StopDeliveryStreamEncryption	Botocore-derived
#2584	feat(logs) — Associate/DisassociateKmsKey	Botocore-derived
#2585	feat(servicecatalog) — Service Catalog + Control Tower Account Factory	mixed (Account Factory LZA-verified, parity surface Botocore-derived)

Notes from the rebase:

• Organizations is yours now. Your feat(organizations): add AWS Organizations service #2489 landed while our branches waited, and it is a strict superset of our implementation except one operation — so we dropped our Organizations branch entirely rather than PR it. The one gap (ListAccountsWithInvalidEffectivePolicy, as an honest empty-list stub) is a 13-line branch we are holding for a later wave. Our IAM/SCP-enforcement work has been re-ported onto your Organizations implementation and is ready for wave 2.
• Same story on transit gateways (feat(ec2): support transit gateways #2329/feat(ec2): support transit gateway VPC attachments #2347/feat(ec2): support transit gateway route tables, associations and routes #2348 cover everything we had except ExportTransitGatewayRoutes), warm-pool isolation, and FilterLogEvents logStreamName — all our overlapping branches were dropped as superseded.
• Each PR passed make docs-check locally, including registering the new services in the generated action-table tooling.
• Sorry for going three over the 5-PR guideline in one go — 6 of the 8 are small, and we will not open wave 2 until these clear. Happy to close any you would rather not carry.

CloudFormation remains held per your ask.

[pgermosen]

I take a look when I have time (probably tomorrow)