Scope events cache entries to the shard context instance - #11660
Draft
prathyushpv wants to merge 2 commits into
Draft
Scope events cache entries to the shard context instance#11660prathyushpv wants to merge 2 commits into
prathyushpv wants to merge 2 commits into
Conversation
Events are cached when they are built, before the transaction is persisted. If that write loses the shard's range CAS, mutable state's NextEventID is unchanged and a later transaction can persist a different event at the same event ID, leaving the cache holding an event that no longer matches history. Nothing invalidates it: DeleteEvent has no production callers. Adding ShardUUID to EventKey makes entries unreachable after the shard context is rebuilt, mirroring the workflow context cache, whose Key already carries a ShardUUID from GetOwner().
- Collapse the nine EventKey literals into MutableStateImpl.eventKey so a future call site cannot omit the shard scope. - Drive the reproduction test through ApplyActivityTaskScheduledEvent: a timer event is never cached in production, so the previous fixture could not occur, and calling writeEventToCache directly skipped the plumbing the test claims to cover. Also assert the caching instance still reads its own entry, so an unstable ShardUUID cannot pass silently. - Assert the shard-scoped key in the NDC replication test; it was the only production site whose removal no test caught. - Stop newTestContext mutating the caller's ShardInfo, which gave two contexts the same owner when a caller reused the pointer. - Fold the duplicate invalid-key tests into one table and name the shard UUID in the validateKey warning.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed?
Added
ShardUUIDtoevents.EventKey, populated fromshardContext.GetOwner(), so cache entries become unreachable once the shard context is rebuilt.Why?
Events are cached when they are built, before the transaction persists. A write that loses the shard's range CAS leaves the entry behind, a later transaction reuses that event ID for a different event, and
transfer_queue_active_task_executor.go:967then nil-derefs the mismatched attributes. Before #11450 a shard reload dropped the cache; now it is process-lifetime, so this mirrors the workflow context cache, whoseKeyalready carries aShardUUID.How did you test it?
TestStaleCachedEventNotServedAfterShardReloadpanics onmainand passes here.