Skip to content

Commit abe57af

Browse files
committed
fix: self-review findings from Phase 10 live-scenario branch (test scope claims, naming)
- ResolveAsync concurrency test renamed/documented to claim only what it actually proves (single-process lock serialization), not cross-process orphan reconciliation. - Cross-contamination and Unicode round-trip tests now restrict matches to the RecentMessage tier specifically -- NAMS entity search is workspace-wide, not conversation-scoped, so checking all items made both tests noisier and their claims broader than what NAMS actually guarantees. - Naming brought in line with the file's existing "Eventually..." convention for eventually- consistent assertions; added a comment on the cancellation test's timing assumption. Re-verified: 7/7 live tests still green against the real NAMS SaaS after these fixes.
1 parent 84c9867 commit abe57af

2 files changed

Lines changed: 72 additions & 15 deletions

File tree

docs/reviews/NAMS_Phase10_LiveScenarioExpansion_PlanningAndImplementationPlan.md

Lines changed: 34 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,36 @@ real NAMS SaaS (`agent-memory-dotnet-dev` workspace):
4545
MAF-layer mapping/gating logic (`NamsMafTypeMapper`, Phase 6), already unit-tested there; they're not
4646
live-NAMS-specific behavior to re-verify against the real service.
4747

48-
## 3. Explicitly deferred
48+
## 3. Self-review findings and fixes
49+
50+
Self-review (2 angles, given the smaller diff: correctness + combined cross-file/conventions) found the
51+
test *logic*, not just the code, needed tightening -- a live test passing once against an eventually
52+
consistent external service doesn't by itself prove the logic is sound:
53+
54+
- **`ResolveAsync_SameIdentityResolvedConcurrently_...`**: traced the actual execution and confirmed this
55+
single-process test can only ever exercise `KeyedAsyncLock` serialization + the already-populated-store
56+
fast path -- it structurally cannot reach the cross-process "lost the race, reconcile an orphaned
57+
conversation" branch (that needs two separate resolver instances racing a shared store, already listed
58+
under "Multi-instance mapping" in §3). Renamed `...ReconcilesToOneConversation` ->
59+
`...SerializesToOneConversation` and added a doc comment stating exactly what this does and doesn't prove.
60+
- **Cross-contamination test**: `NamsRecallOptions.IncludeEntitySearch` defaults `true` and wasn't
61+
overridden, so the original assertion checked ALL recalled items -- including entities, which are
62+
workspace-wide, not conversation-scoped (per the `NamsAgent` sample's own README). That made the test's
63+
actual scope wider than its documented intent and a source of unrelated flakiness. Fixed to filter to
64+
`NamsRecallCategory.RecentMessage` only, matching the documented scope exactly. Renamed to
65+
`...EventuallyRecallOnlyTheirOwnRecentMessage` (also fixes a naming-convention gap the other review angle
66+
found: new eventually-consistent tests should say "Eventually", matching the file's existing two).
67+
- **Unicode round-trip test**: "byte-for-byte" was only actually guaranteed for the `RecentMessage` tier
68+
(`NamsRecallService.MapMessage` passes `Content` through verbatim); reflections/observations are
69+
NAMS-synthesized text with no such guarantee. Fixed the match condition to require
70+
`Category == NamsRecallCategory.RecentMessage`, making the claim proven rather than probabilistic.
71+
Renamed to `...EventuallyRoundTripsByteForByteInRecentMessages`.
72+
- **Cancellation test**: added a one-line comment stating the `CancelAfter(1ms)` timing is an environment
73+
assumption (this session's observed 400-1000ms live NAMS latency), not a language/runtime guarantee.
74+
75+
Re-ran all 7 live tests after the fixes: still 7/7 green, ~7s.
76+
77+
## 4. Explicitly deferred
4978

5079
- **Multi-instance mapping** (separate resolver processes, crash reconciliation) -- needs actual
5180
multi-process orchestration, a bigger investment than fits this increment.
@@ -58,16 +87,17 @@ real NAMS SaaS (`agent-memory-dotnet-dev` workspace):
5887
- **Payload: empty / max size / multi-message / non-text** -- Unicode was judged the highest-value single
5988
addition for this pass; the rest are reasonable follow-ups, not done here.
6089

61-
## 4. Verification
90+
## 5. Verification
6291

6392
- `dotnet build AgentMemory.slnx -c Release` -- 0 warnings, 0 errors.
6493
- `dotnet test tests/AgentMemory.Tests.Integration --filter "FullyQualifiedName~Nams.NamsLiveConnectivityTests"`
6594
-- **7/7 live, ~7s total** (3 original + 4 new).
6695
- `dotnet test tests/AgentMemory.Tests.Unit` -- full suite green, unaffected (no unit-level changes this
6796
phase).
6897

69-
## 5. Definition of done
98+
## 6. Definition of done
7099

71100
- [x] 4 new live scenarios added and passing against the real NAMS SaaS.
72101
- [x] Coverage overlap and deferred scope explicitly documented (no silent gaps).
73-
- [ ] Self-reviewed, PR opened, CI green (live tests report Skipped there, as established), merged to `main`.
102+
- [x] Self-reviewed and fixes applied.
103+
- [ ] PR opened, CI green (live tests report Skipped there, as established), merged to `main`.

tests/AgentMemory.Tests.Integration/Nams/NamsLiveConnectivityTests.cs

Lines changed: 38 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -98,8 +98,16 @@ await persistence.PersistTurnAsync(
9898
found.Should().BeTrue($"an entity named '{entityName}' should eventually be extracted and recallable");
9999
}
100100

101+
/// <summary>
102+
/// Proves the single-process path only: the shared <c>KeyedAsyncLock</c> serializes the second call
103+
/// behind the first, which then takes the already-populated-store fast path -- it never reaches
104+
/// <c>CreateConversationAsync</c> itself. This does NOT exercise the cross-process "lost the race,
105+
/// reconcile an orphaned conversation" branch (that needs two separate resolver instances/processes
106+
/// racing against a genuinely shared store -- out of scope here, see the Phase 10 planning doc's
107+
/// "Multi-instance mapping" deferral).
108+
/// </summary>
101109
[LiveNamsFact]
102-
public async Task ResolveAsync_SameIdentityResolvedConcurrently_ReconcilesToOneConversation()
110+
public async Task ResolveAsync_SameIdentityResolvedConcurrently_SerializesToOneConversation()
103111
{
104112
var resolver = _fixture.Services!.GetRequiredService<INamsConversationResolver>();
105113
var identity = UniqueIdentity(); // one identity, resolved twice concurrently below
@@ -114,8 +122,14 @@ public async Task ResolveAsync_SameIdentityResolvedConcurrently_ReconcilesToOneC
114122
"exactly one of the two concurrent resolutions should have actually created the conversation");
115123
}
116124

125+
/// <summary>
126+
/// Deliberately scoped to the <see cref="NamsRecallCategory.RecentMessage"/> tier only (conversation-
127+
/// scoped) -- NAMS entity search is workspace-wide, not conversation-scoped (documented in
128+
/// <c>AgentMemory.Sample.NamsAgent</c>'s README), so including entity-tier items here would make this
129+
/// test noisy/flaky for a reason unrelated to what it's actually verifying.
130+
/// </summary>
117131
[LiveNamsFact]
118-
public async Task PersistTurnAsync_TwoConcurrentUsers_DoNotCrossContaminateChatHistory()
132+
public async Task PersistTurnAsync_TwoConcurrentUsers_EventuallyRecallOnlyTheirOwnRecentMessage()
119133
{
120134
var services = _fixture.Services!;
121135
var resolver = services.GetRequiredService<INamsConversationResolver>();
@@ -139,16 +153,18 @@ await Task.WhenAll(
139153
{
140154
var recalledA = await recall.RecallAsync(conversationA.NamsConversationId, markerA, CancellationToken.None);
141155
var recalledB = await recall.RecallAsync(conversationB.NamsConversationId, markerB, CancellationToken.None);
142-
var aHasOwnMarker = recalledA.Items.Any(i => i.Content.Contains(markerA, StringComparison.Ordinal));
143-
var bHasOwnMarker = recalledB.Items.Any(i => i.Content.Contains(markerB, StringComparison.Ordinal));
156+
var recentA = recalledA.Items.Where(i => i.Category == NamsRecallCategory.RecentMessage).ToList();
157+
var recentB = recalledB.Items.Where(i => i.Category == NamsRecallCategory.RecentMessage).ToList();
158+
var aHasOwnMarker = recentA.Any(i => i.Content.Contains(markerA, StringComparison.Ordinal));
159+
var bHasOwnMarker = recentB.Any(i => i.Content.Contains(markerB, StringComparison.Ordinal));
144160
var noCrossContamination =
145-
!recalledA.Items.Any(i => i.Content.Contains(markerB, StringComparison.Ordinal))
146-
&& !recalledB.Items.Any(i => i.Content.Contains(markerA, StringComparison.Ordinal));
161+
!recentA.Any(i => i.Content.Contains(markerB, StringComparison.Ordinal))
162+
&& !recentB.Any(i => i.Content.Contains(markerA, StringComparison.Ordinal));
147163
return aHasOwnMarker && bHasOwnMarker && noCrossContamination;
148164
},
149165
timeout: TimeSpan.FromSeconds(30));
150166

151-
found.Should().BeTrue("each conversation's recall should see only its own persisted message, never the other's");
167+
found.Should().BeTrue("each conversation's recent-message tier should show only its own persisted message, never the other's");
152168
}
153169

154170
[LiveNamsFact]
@@ -160,15 +176,25 @@ public async Task RecallAsync_CancelledBeforeTheHttpRoundTripCompletes_Propagate
160176
var conversation = await resolver.ResolveAsync(UniqueIdentity(), CancellationToken.None);
161177

162178
using var cts = new CancellationTokenSource();
163-
cts.CancelAfter(TimeSpan.FromMilliseconds(1)); // a live round trip to NAMS never completes this fast
179+
// Environment assumption, not a language/runtime guarantee: this session's observed live NAMS
180+
// latency is consistently 400-1000ms, so 1ms reliably cancels first. If NAMS round-trips ever get
181+
// fast enough to race this, this test (not the production code) would need a longer delay.
182+
cts.CancelAfter(TimeSpan.FromMilliseconds(1));
164183

165184
var act = () => recall.RecallAsync(conversation.NamsConversationId, "test", cts.Token);
166185

167186
await act.Should().ThrowAsync<OperationCanceledException>();
168187
}
169188

189+
/// <summary>
190+
/// Matches only <see cref="NamsRecallCategory.RecentMessage"/> items -- <see cref="MapMessage"/>-mapped
191+
/// (in <c>NamsRecallService</c>) recent messages pass <c>Content</c> through verbatim, so a match there
192+
/// genuinely proves a byte-for-byte round trip. Reflections/observations are NAMS-synthesized text, not
193+
/// guaranteed to preserve the original bytes -- matching against those tiers too would make "byte for
194+
/// byte" a probabilistic claim rather than a proven one.
195+
/// </summary>
170196
[LiveNamsFact]
171-
public async Task PersistTurnAsync_UnicodeAndEmojiContent_RoundTripsByteForByte()
197+
public async Task PersistTurnAsync_UnicodeAndEmojiContent_EventuallyRoundTripsByteForByteInRecentMessages()
172198
{
173199
var services = _fixture.Services!;
174200
var resolver = services.GetRequiredService<INamsConversationResolver>();
@@ -184,11 +210,12 @@ await persistence.PersistTurnAsync(
184210
async () =>
185211
{
186212
var recalled = await recall.RecallAsync(conversation.NamsConversationId, marker, CancellationToken.None);
187-
return recalled.Items.Any(i => i.Content.Contains(marker, StringComparison.Ordinal));
213+
return recalled.Items.Any(i =>
214+
i.Category == NamsRecallCategory.RecentMessage && i.Content.Contains(marker, StringComparison.Ordinal));
188215
},
189216
timeout: TimeSpan.FromSeconds(30));
190217

191-
found.Should().BeTrue("Unicode/emoji content should round-trip through NAMS byte-for-byte");
218+
found.Should().BeTrue("Unicode/emoji content should round-trip through NAMS byte-for-byte in the recent-messages tier");
192219
}
193220

194221
private static NamsConversationIdentity UniqueIdentity([CallerMemberName] string testName = "") => new()

0 commit comments

Comments
 (0)