Proposal to add Tri-Axis Retrieval scorer and an Evaluation Framework

[Rahul-Malasani]

I've been building a local, privacy-first AI agent for macOS — a floating bubble that reads live machine state (CPU, RAM, running processes, active app) and acts on it. The missing piece was persistent memory: the agent could answer questions about right now, but had no structured recall of what the user was doing an hour ago, or why a process spiked yesterday.

CatchMe's activity tree is exactly the missing piece that I needed. But when I looked at the retrieval layer, I noticed a couple of gaps that affects my use case directly — and probably others.

CatchMe's retrieval is currently driven primarily by the relevancy of the sent query . Recency is implicit, and the actual event significance is not weighted at all.

In real world usage not all events deserve equal surfacing. A 3 hour debugging session is more important than a normal tab switch, even if the tab switch happened more recently. Without a significance axis, a recency-biased ranker buries high-value events from a few hours ago which are exactly the ones a machine-aware agent needs to surface for queries like "why did the app crash?" or "what was I blocked on this morning?"

There is also currently no evaluation metrics to either measure or compare the retrieval quality.

1. Tri-Axis Scorer

Scores every candidate event along three independent axes:

final_score = w_rec · recency + w_rel · relevance + w_sig · significance

Recency — exponential decay from event timestamp. Half-life configurable (default 24 h). Yesterday → ~0.5, one week ago → ~0.07.

Relevancy — keyword overlap between query tokens and event summary/content, with an app-name boost. Offline fallback by default; in production can delegate to CatchMe's existing LLM traversal.

Significance combines:

• Event type baseline ( for example -error logs > deep work > terminal commands > tab switches)
• Duration (30-min sustained work scores higher than a 5-second switch)
• High-signal keywords in summary (crash, fix, deploy, merge, incident )
• Activity tree path depth (/work/projects/app/debug > /work)

Weights are caller-tunable per query intent:

"Why did X break?" — significance + relevance matter most
scorer = TriAxisScorer(weights=(0.15, 0.45, 0.40))

"What was I just doing?" — recency matters most
scorer = TriAxisScorer(weights=(0.60, 0.25, 0.15))

2. Evaluation Framework

A lightweight but complete eval suite:

• Synthetic benchmark — 13-event realistic working session with 5 annotated queries and hand-labelled ground-truth event IDs
• Metrics — Precision@K, Recall@K, MRR, NDCG@K
• Baseline — recency-only ranker (approximates current CatchMe behaviour)
• Comparison report — head-to-head delta table printed on each run

I have working code for both the files. I can also test this against my own agent project after integration ,and give real benchmark not just a synthetic benchmark.

Happy to share the code or discuss the approach further.

[m13v]

the thing that kills this in practice is duration-as-significance not separating deep work from idle. 30 min on one app is either flow or a forgotten youtube tab. we instrumented keystroke cadence plus window-focus events/min to distinguish those before weighting duration, else significance just collapses into longest-running idle. also a 24h half-life murders retrospective queries like 'what was I blocked on monday' - we ended up with per-intent half-lives rather than a global one, retrospective classes need a flatter decay curve than 'what was I just doing'.

[pancacake]

Yea that's a good point! We'll try to improve this asap! Thanks for your comments!