Skip to content

Lambda: legacy code-directory reclaim lock doesn't cover cross-account concurrency (follow-up from #2646) #2661

Description

@pgermosen

Summary

The lock protecting Lambda's legacy-code-directory reclaim logic (introduced in #2646) is keyed by the function's own ARN, which includes the account id. But the directory it protects — the pre-account-scoped code path — can be shared by two different accounts' same-named functions from before the account-scoping migration. Two different accounts' operations lock on two different ARNs, so they never serialize against each other for that shared resource.

Scenario

Account A: publishVersion("shared-fn")     — locks on A's ARN, reads $LATEST.codeLocalPath = legacyPath
Account B: updateFunctionCode("shared-fn") — locks on B's ARN (a different lock), runs concurrently
  B's extractZipCodeBytes migrates B's $LATEST off legacyPath, then checks whether legacyPath is
  still referenced anywhere — if A's new version snapshot hasn't been persisted yet, B sees nothing
  live and deletes legacyPath while A is still mid-publish

If B's delete wins that race, A's freshly published version snapshots codeLocalPath pointing at a directory that's just been removed, and that version's next cold invoke fails with a missing-code error.

Why this is a follow-up rather than blocking #2646

  • Same failure mode as the three cross-account/race bugs already found and fixed in feat(lambda): code-signing endpoints and account settings #2646 (Greptile rounds bede98fc1, f3837be32, aa93885ef) — a single function's next cold invoke fails, recoverable by redeploying that function.
  • Self-expiring: it only exists in the one-time transitional window right after upgrading from a pre-account-scoped Floci version, for functions that happen to share a name across accounts. Once every affected function is touched once post-upgrade, the shared legacy directory disappears for good.
  • Doesn't touch any of feat(lambda): code-signing endpoints and account settings #2646's actual new surface (GetAccountSettings, ListFunctionsByCodeSigningConfig, the Runtime API port-pool fix).
  • Very low practical likelihood: requires two different accounts to hold the exact same function name, both still on the legacy path, touched at the same instant.

Suggested fix

Add a lock keyed on functionName alone (not the per-account ARN) around just the legacy-path reclaim/reference-check critical section, so two accounts' operations on the same legacy directory serialize even though their per-function lockForConcurrencyOp locks don't overlap. Scope it narrowly so it doesn't serialize unrelated per-account work.

Alternatively, if the risk is judged acceptable given how narrow and self-expiring it is, document it as a known limitation of the migration window.

Follow-up from review of #2646.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinglambdaAWS Lambda

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions