Skip to content

Add literal search for archived observations - #14

Open
aalvsz wants to merge 1 commit into
NVlabs:mainfrom
aalvsz:observation-search
Open

Add literal search for archived observations#14
aalvsz wants to merge 1 commit into
NVlabs:mainfrom
aalvsz:observation-search

Conversation

@aalvsz

@aalvsz aalvsz commented Sep 11, 2026

Copy link
Copy Markdown

Summary

  • Add optional literal UTF-8 search to obs_recall for archived large tool results.
  • Return exact byte spans, exclusive ends, LF-based line numbers, bounded UTF-8-safe source context, and continuation metadata.
  • Preserve the existing paged-recall path when query is omitted.
  • Add validation for malformed Unicode, query size, output limits, cancellation, file guards, overlaps, and continuation.
  • Add a reproducible local comparison script.

Validation

  • npm run check — 17 test files, 142 tests, typecheck, and package dry run passed.
  • npm audit --audit-level=high — passed at the requested severity; two pre-existing moderate Vitest advisories remain.
  • node scripts/check-pi-compat.mjs — passed.
  • npx --no-install vitest run tests/all-mechanisms.test.ts — 4 tests passed.
  • Independent full-buffer oracle — 23 cases / 57 search calls passed.

Real-output benchmark

Five real rg commands from the repository produced 30–166 KiB archived outputs. In a 50-case targeted benchmark (eight observed literals with at most 20 occurrences plus two misses per artifact):

metric paged recall literal search change
tool calls 330 53 83.94% fewer
returned bytes 4,900,500 204,085 95.84% fewer

All 50 cases were source-oracle verified, including offsets, line numbers, contexts, and continuation. A separate dense stress run is retained: queries matching nearly every line required more output than paging, which documents the feature's workload boundary.

These are retrieval measurements over real command output. They do not claim improved model-answer accuracy, provider-token usage, or billed cost. Timings are descriptive local wall times.

@gaoanze888 gaoanze888 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the literal-search semantics, UTF-8/chunk boundaries, overlapping matches, continuation, output limits, cancellation, and file guards. I ran npm run check locally: typecheck, 17 test files / 142 tests, and package dry-run all passed.

I also integration-merged this with #13 because both touch Observation Pack; the merge was clean and the combined suite passed 17 files / 144 tests. The PR size is justified by the streaming search boundary work plus substantial tests and a reproducible benchmark, rather than unrelated production scope. I did not find a blocking correctness issue. Nice discipline around reporting returned bytes/calls without overclaiming model accuracy or billed cost.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants