Skip to content

feat(recall): add corpus browser page - #1243

Merged
wesm merged 4 commits into
mainfrom
feat/recall-page
Aug 1, 2026
Merged

feat(recall): add corpus browser page#1243
wesm merged 4 commits into
mainfrom
feat/recall-page

Conversation

@wesm

@wesm wesm commented Jul 23, 2026

Copy link
Copy Markdown
Member

Adds Recall as a top-level corpus browser. Accepted entries appear in a padded, single-row table for quick comparison; each row expands in place to show the full fact, trigger, uncertainty, extraction metadata, provenance state, and transcript evidence. Search and filters cover project, type, active extraction generation, and review state.

Browsing stays bounded with cursor pagination. Ranked searches preserve relevance semantics, reject stale continuations after corpus mutations, and disclose the existing 500-result cap. The page also reports extraction coverage and stays unavailable on read-only backends.

@roborev-ci

roborev-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown

roborev: Combined Review (5f8ae7a)

The Recall browser is generally sound, but two medium-severity usability gaps should be addressed.

Medium

  • frontend/src/lib/components/recall/RecallPage.svelte:91 — The generation selector includes building and retired generations, while entry requests omit status and therefore return only accepted entries. Because machine-generated entries for building and retired generations are archived, selecting those generations misleadingly shows an empty result set. Limit the selector to active generations, or include archived entries and display their lifecycle status.

  • frontend/src/lib/api/recall.ts:15 — Results are permanently capped at 200 with no pagination, load-more action, or truncation indicator, making entries beyond the first page inaccessible. Add cursor-based pagination to the endpoint and UI, along with a total or truncation indicator.


Reviewers: 2 done | Synthesis: codex, 11s | Total: 7m11s

@roborev-ci

roborev-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown

roborev: Combined Review (c3f7ef0)

The PR adds useful Recall browsing features, but it introduces a material search-semantics regression that should be fixed before merging.

Medium

  • internal/server/recall.go:105 — Recall search bypasses the ranked query path. Requests containing q now use the raw recency-ordered list query instead of QueryRecallEntries. This breaks direct/HTTP parity and loses full-text, evidence, and metadata matching, relevance ordering, and score metadata. Queries containing only ignored terms may also return the entire unfiltered corpus.

    Suggested fix: Preserve the ranked query path for the existing API, or add a separate, explicit recency-paginated browser mode with equivalent search semantics. Normalized queries containing no searchable terms should return an empty result.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 7m18s

@roborev-ci

roborev-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown

roborev: Combined Review (ef454cd)

The PR has one medium-severity pagination consistency issue; no material security regressions were found.

Medium

  • internal/server/recall.go:212 — Ranked pagination reruns searches against the current corpus and slices by offset. If background extraction inserts a higher-ranked entry between page requests, results can shift, causing duplicate or skipped entries; duplicate IDs can also conflict with the keyed Svelte list. Bind ranked cursors to a corpus revision and reject or restart stale pagination, or preserve a stable result snapshot across pages.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 6m45s

@roborev-ci

roborev-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown

roborev: Combined Review (60be1b3)

Changes requested: ranked pagination has two medium-severity correctness issues.

Medium

  • internal/server/recall.go:233 — Ranked cursors use RecallCorpusRevision, which does not track all fields affecting ranked membership or ordering, including metadata, evidence, review state, and trusted-only fields such as provenance_ok. Mutations can therefore cause skipped or duplicated entries. Use a dedicated pagination revision covering every relevant field and evidence mutation.

  • internal/server/recall.go:240 — Ranked pagination treats MaxRecallEntryLimit as a cumulative cap, so searches with more than 500 matches silently stop at 500 and reject subsequent cursors. Implement true offset/keyset pagination with a per-page limit, or explicitly expose the truncation.


Reviewers: 2 done | Synthesis: codex, 7s | Total: 6m25s

@roborev-ci

roborev-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown

roborev: Combined Review (440a110)

Medium-severity cursor invalidation bug found; no high or critical issues identified.

Medium

  • internal/db/recall.go:365 — A full resync copies the old query revision unchanged. If every old Recall entry is omitted because its source session was not preserved, no entry/evidence trigger advances the revision, leaving ranked cursors valid even though the corpus is now empty.
    • Fix: Ensure the replacement revision is strictly newer whenever copying an existing corpus, and add coverage for an all-entries-skipped resync.

Reviewers: 2 done | Synthesis: codex, 7s | Total: 9m58s

@roborev-ci

roborev-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown

roborev: Combined Review (d117939)

The Recall browser is sound overall, but needs a read-only-mode availability guard before merging.

Medium

  • Recall is unusable in PostgreSQL/read-only modefrontend/src/lib/components/layout/AppHeader.svelte:84

    The Recall tab and route are exposed unconditionally, while PostgreSQL/read-only stores return ErrReadOnly for Recall queries. In supported pg serve mode, the page always reports a loading error.

    Fix: Hide or disable Recall when sync.readOnly/settings.readOnly is true, and handle direct /recall navigation with an unavailable state or redirect.


Reviewers: 2 done | Synthesis: codex, 8s | Total: 12m58s

@roborev-ci

roborev-ci Bot commented Jul 23, 2026

Copy link
Copy Markdown

roborev: Combined Review (0d6be12)

No Medium, High, or Critical findings; the reviewed changes are clean at the requested severity threshold.


Reviewers: 2 done | Synthesis: codex, 6s | Total: 9m13s

@wesm

wesm commented Jul 23, 2026

Copy link
Copy Markdown
Member Author

I need to do some QA on this before merging

@mjacobs

mjacobs commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

I tested #1243 locally end-to-end this evening.

test setup: checked out the PR, pointed [recall.extract] at gemma-4-31B-it-qat behind llama.cpp (OpenAI-compatible endpoint) on a RTX 5090, and ran the full extract → activate → browse flow against:

synthetic archive

a first test (testfixture-generated, ~14 sessions) to shake out the pipeline before committing a several GPU hours processing my real session data.

Looks good, no functional issues. extract doctor → clean probe on the first try, extraction ran with zero schema failures (Note: llama.cpp enforces the json_schema response format with grammar-constrained decoding, so a local gemma that was never tuned for this still produces valid entries every time; IME gemma/gemini wouldn't always do that naturally).

my real session data

Same setup with real data completed in a few hours (I don't have specific timestamps off hand, but it was a few hours) processed 483 sessions with 5 sessions that couldn't complete the extraction. Note that 488 sessions was a small subset of my overall session data, but those were the sessions that were eligible for extraction given the expected constraints of the extraction (tl;dr protecting potentially sensitive data).

Recall UI

Recall page does what the PR description claims: status card with coverage/backlog, project/type/generation/review-state filters, lexical search flipping between keyset and ranked pagination, and the evidence links jump to the source transcript at the right ordinals.

Screenshot:
snippy_1784867769694
(I blurred several text blocks)

Some minor adjustments to the UI may be worthwhile aesthetic enhancements, but definitely not blocking IMO.

@wesm

wesm commented Jul 24, 2026

Copy link
Copy Markdown
Member Author

Thank you. I'll work on making the UI less ugly! I'm not sure the 3-column layout is useful either

@roborev-ci

roborev-ci Bot commented Jul 24, 2026

Copy link
Copy Markdown

roborev: Combined Review (0d6be12)

No Medium, High, or Critical findings; the changes look clean overall.


Reviewers: 2 done | Synthesis: codex, 3s | Total: 23m37s

Distilled recall needs a corpus-wide inspection surface that stays useful across both small and large archives. The browser keeps management out of scope while providing bounded, stable access to accepted entries and extraction coverage.

- Paginate corpus browsing and limit generation filters to served entries
- Preserve ranked search semantics and reject stale pagination cursors
- Track every query-relevant mutation, including empty resync replacement
- Hide the browser when the active backend cannot serve Recall
- Replace the dense card grid with a padded, expandable fact table
@wesm
wesm force-pushed the feat/recall-page branch from 0d6be12 to a24d5c3 Compare August 1, 2026 12:03
@roborev-ci

roborev-ci Bot commented Aug 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (a24d5c3)

The Recall feature is generally sound, but two medium-severity issues affect pagination correctness and mobile navigation.

Medium

  • Unstable ranked paginationinternal/server/recall.go:252
    Ranked pagination reruns the query with an increasing Limit, while source diversification depends on that limit. This can change the ranked prefix between pages, causing duplicate or skipped entries when results contain multiple entries from the same source.
    Fix: Produce a deterministic, limit-independent ranked order before applying the cursor offset. Add pagination coverage combining duplicate-source and unique-source sessions.

  • Recall missing from mobile navigationfrontend/src/lib/components/layout/ThreeColumnLayout.svelte:333
    Recall is available in the desktop header but absent from the mobile drawer. Opening the drawer from a direct Recall route navigates to Sessions, and the drawer offers no way back to Recall.
    Fix: Add an availability-gated Recall button to the mobile navigation and test it at the mobile breakpoint.


Reviewers: 2 done | Synthesis: codex, 12s | Total: 9m41s

Ranked cursors require every request to slice the same diversified order; otherwise changing the query limit between pages can duplicate one source and omit another. Build the complete unique-source-first ordering before applying the caller limit so its prefixes remain stable.\n\nThe mobile drawer also needs the same backend capability gate as desktop navigation so supported users can return to Recall after the hamburger routes through Sessions.
@roborev-ci

roborev-ci Bot commented Aug 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (76fc8a6)

The PR has one medium-severity issue: the Recall generation filter can hide valid source runs.

Medium

  • Generation filter omits valid retired or reconcile-only source runsfrontend/src/lib/components/recall/RecallPage.svelte:97, cmd/agentsview/main.go:494

    The selector only lists active generations supplied by a configured extraction manager. Human-reviewed entries can belong to retired generations, while reconcile-only mode retains an active corpus without exposing a status provider. Consequently, valid source_run_id values cannot be selected.

    Suggested fix: Populate the selector from distinct source runs in the served corpus, or expose database-backed generation metadata independently of extraction configuration. Do not restrict options to state === "active".


Reviewers: 2 done | Synthesis: codex, 11s | Total: 16m25s

Extraction-manager lifecycle state is not the same as the corpus exposed by Recall. Human review can preserve entries from retired runs, and reconciliation can serve imported entries without a manager status provider.

Expose distinct source runs from accepted entries through the existing status response so the browser offers every selectable run while continuing to exclude archived staging generations.
@roborev-ci

roborev-ci Bot commented Aug 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (77f6399)

The PR has one medium-severity freshness issue; no critical or high-severity findings.

Medium

  • frontend/src/lib/components/recall/RecallPage.svelte:199 — Recall entries and extraction coverage are fetched only on mount or filter changes. If background extraction completes while the page is open, the corpus and status remain stale indefinitely with no refresh affordance. Add periodic or manual refresh, or an extraction-completion notification that reloads the first entry page and status.

Reviewers: 2 done | Synthesis: codex, 9s | Total: 11m1s

Extraction can finish while the Recall browser remains mounted, leaving both visible entries and coverage metrics outdated with no way for the user to reconcile them. Use the shared refresh control so one bounded refresh reloads both views and keeps the page current on the standard interval.
@roborev-ci

roborev-ci Bot commented Aug 1, 2026

Copy link
Copy Markdown

roborev: Combined Review (7b83c49)

The Recall browser changes are clean with no Medium, High, or Critical findings.


Reviewers: 2 done | Synthesis: codex, 9s | Total: 12m18s

@wesm
wesm merged commit ecd0eba into main Aug 1, 2026
23 checks passed
@wesm
wesm deleted the feat/recall-page branch August 1, 2026 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants