See what your agent retrieved, and what it actually used.
Your agent retrieved twelve documents and answered from three. The other nine cost you tokens, latency, and a wider surface for the model to go wrong on. Nothing in your stack tells you which nine.
Graphsight traces the run and marks each retrieved item overlaps the answer or no lexical trace in answer, scored against the final answer. No LLM re-reads your evidence, no score is invented, and nothing leaves localhost.
A real run. PR #101 scored 0.910, the highest of anything retrieved, and the answer
never used it. PR #412 scored 0.340 and is the one that answered.
Jump to: what this is · architecture · routing · benchmarks · project status · contributing
New to this? Your first trace is a 10-minute walkthrough with a complete runnable example. Otherwise:
pip install graphsight graphsight-langgraphAdd one callback handler to any LangGraph agent:
from graphsight_langgraph import LangGraphTracer, capture
tracer = LangGraphTracer()
result = graph.invoke(inputs, config={"callbacks": [tracer]})
capture(tracer, query="why is checkout failing?", answer=result["answer"])
# -> .graphsight/20260728T184600_why-is-checkout-failing.jsonPassing answer= is what unlocks the retrieved-vs-used split. Without it,
everything renders plain and no usage is claimed.
Then open it:
graphsight .graphsight/A local server on 127.0.0.1:4630 renders the run: the node graph, the retrieval steps, and
the retrieved-vs-used verdict per item. The viewer has zero dependencies, stdlib
http.server plus a bundled UI build. The tracer depends only on langchain-core.
No account, no API key, no telemetry, no cloud.
The trace contract is framework neutral. LangGraph is one producer, not a requirement. If you wrote your retrieval loop yourself, construct the trace from whatever you already have and skip the callback entirely:
from graphsight_langgraph import AgentTrace, Retrieval, RetrievedItem, save_trace
from graphsight_langgraph._text import tokens
atoks = tokens(answer)
def overlap(text): # same heuristic the tracer uses
t = tokens(text)
return round(len(atoks & t) / min(len(t), len(atoks)), 3) if t and atoks else None
trace = AgentTrace(query=query, framework="custom", answer=answer)
trace.retrievals.append(Retrieval(
span_id="retrieve", query=query, arm="vector",
items=[RetrievedItem(id=c.id, label=c.title, content=c.text,
score=c.score, answer_overlap=overlap(c.text))
for c in chunks],
))
print(save_trace(trace)) # -> .graphsight/<timestamp>_<slug>.jsonanswer_overlap is what drives the used/ignored split. Leave it None and every item
renders plain, which is the honest default when there is nothing to compare against.
Label a run and later runs can recall it. Only runs labelled success are ever recalled, so a
bad run can't teach the next one.
capture(tracer, query=q, answer=a, outcome="success") # saved to .graphsight/
recall("who broke checkout?") # past runs that asked something similar
EpisodeRetriever().invoke(question) # the same, traced like any other retrieverMatching is lexical, so a reworded question with different words won't match. Recalled episodes
appear as nodes in the viewer. backend/scripts/episodic_agent.py is a working LangGraph agent
that uses this with the graph engine, and --no-memory gives you the baseline.
graphsight-memory is its own package: no torch, no API key, one local file.
Its README has the full API.
An agent can write facts into the graph, correct them, and delete them. They sit beside the facts read from GitHub without ever overwriting them.
from graphsight_memory import Memory
mem = Memory("team.lbug")
mem.sync_github("org/repo")
mem.remember("checkout-service", "OWNED_BY", "payments team", source="run-42")
mem.remember("checkout-service", "OWNED_BY", "billing team", source="run-57", replace=True)
print(mem.recall("who owns checkout-service?"))- Labelled. Remembered facts reach the model as
checkout-service OWNED_BY billing team (remembered, source: run-57), and they weigh less than any fact read from GitHub. - Replaced, not lost.
replace=Trueends the older fact. It stops being retrieved but stays in the store with the time it ended. - Forget is scoped.
forget()deletes only agent facts, history included, and removes nodes it created that are left with no edges. Facts read from GitHub can't be forgotten this way. - Over MCP as
rememberandforget, which exist only whenTRACERAG_MCP_WRITE_TOKENis set and check that token on every call.
Known limit: after a replace, the old object's name can still surface as a node with no live facts, because its ended fact is kept as history.
Measured: on 22 held-out tasks about facts stated earlier, graph memory answered 16, episodic recall 14, and no memory 5. Graph memory clearly beats no memory, and it is tied with episodic recall. It answered 5 of 6 changed facts against episodic's 3, which is too few to call. It also gives confident wrong answers on questions that sound like a stated fact. Details.
Joins: on 28 tasks written for a repo nothing was tuned on (pydantic), each needing a told fact joined to a GitHub fact, graph memory answered 27 and episodic recall 10, and every episodic miss was a question that starts from GitHub. Putting the whole snapshot in the prompt did as well as graph memory. Details.
Two halves, deliberately separable.
| Package | What it does | Dependencies |
|---|---|---|
graphsight-langgraph |
One BaseCallbackHandler → framework-neutral AgentTrace. Also ships graphsight-github-trace (repo → traced run in one command) |
langchain-core |
graphsight |
Local viewer, bundled Studio UI + stdlib server. graphsight trace.json or graphsight .graphsight/ |
none |
This half works on flat traces. No graph, no database, no ingestion. If your agent does plain vector RAG, you still get the retrieved-vs-used signal, run history, and run diffing.
A local-first hybrid store where vectors and the relationship graph live in one .lbug
file, no graph-DB ↔ vector-DB sync problem, behind a two-tier entity resolver and an
intent-weighted router that emits a full trace_log for every query.
This is what makes retrieval causal rather than merely similar: an agent asking "what broke
the session flow?" should reach yesterday's PR through AUTHORED → RESOLVES, not an
eight-month-old PR that happens to read like the question.
The engine is the graphsight-memory package. It embeds with MiniLM on ONNX, so it installs
without torch and runs without API keys. An LLM is optional: it extracts facts from text in
learn(), and classifies a question the keyword check can't place. The backend adds the
web API, document ingest with spaCy and GLiNER, and the Studio.
One hybrid store, queried by two arms (vector + graph), fused by intent, and every step is recorded.
Multi-hop retrieval for agents fails for three infrastructural reasons, and Graphsight addresses each:
| Failure | Fix |
|---|---|
| Dual-store sync hell, keeping a graph DB and a vector DB consistent | Both live natively in one embedded LadybugDB file |
Entity drift, unconstrained extraction turns PaymentService, payments-v2, pay_svc into three nodes |
Two-tier curation: cheap vector auto-merge, LLM adjudicates only the grey zone |
| Black-box routing, no insight into why a hybrid retriever chose a graph edge over a chunk | Every query returns a structured trace_log the Studio renders |
The anti-slop principle throughout: evidence is never paraphrased by an LLM, and scores are never invented. Overlap is lexical and reproducible. If we can't measure something, we don't claim it.
flowchart TB
subgraph INGEST["1 · Ingestion"]
direction TB
SRC["GitHub payloads · docs"]
SRC --> STRUCT["GitHubGraphBuilder<br/>reads relations from the API:<br/>author · reviews · closes-refs · files"]
SRC --> TEXT["sliding window → NER →<br/>two-tier curation"]
STRUCT --> TYPED["typed, timestamped edges<br/>AUTHORED · RESOLVES · TOUCHES"]
TEXT --> PROX["CO_OCCURS<br/>(proximity, weighted lowest)"]
end
subgraph STORE["2 · LadybugDB, one .lbug"]
direction LR
ENT[("Entity<br/>id · label · type · ts<br/>embedding FLOAT 384")]
DOCN[("Document<br/>id · path · content")]
ENT -->|"RELATES_TO<br/>confidence · relation · ts"| ENT
DOCN -->|MENTIONS| ENT
HNSW["HNSW index"]
ENT --- HNSW
end
TYPED --> ENT
PROX --> ENT
subgraph QUERY["3 · Query, two arms, fused by intent"]
direction TB
Q["query"] --> INTENT{"intent?"}
INTENT -->|relational| WR["alpha=0.15 beta=0.85"]
INTENT -->|semantic| WS["alpha=0.80 beta=0.20"]
Q --> VEC["vector arm<br/>HNSW top-k"]
Q --> GR["graph arm<br/>multiplicative BFS<br/>path = product of edge weights"]
VEC --> FUSE["score = (alpha·sv + beta·sg) × recency"]
GR --> FUSE
WR --> FUSE
WS --> FUSE
end
ENT --> VEC
ENT --> GR
FUSE --> TRACE["trace_log:<br/>seeds · hops + relation ·<br/>recency per node"]
TRACE --> UI["Graphsight Studio"]
| Layer | Technology | Role |
|---|---|---|
| Embeddings | sentence-transformers · all-MiniLM-L6-v2 |
384-dim vectors (cosine) |
| Entity extraction | GLiNER, with spaCy as fallback | Zero-shot NER. spaCy runs when GLiNER can't load |
| Storage | LadybugDB (embedded, Kùzu-lineage) | Single-file hybrid vector + graph store, native HNSW |
| LLM, router intent | Groq · qwen/qwen3.8-27b |
Query intent classification; falls back to OpenRouter when unset |
| LLM, generation | OpenRouter · anthropic/claude-haiku-5.5 |
Grounded answers + node summaries |
| API | FastAPI + Uvicorn | trace · subgraph · answer · graphs · history |
| Concurrency | read connection pool + bounded embed pool | parallel graph/vector reads; embeds off request threads |
| Auth | Clerk + PyJWT / JWKS | Networkless verification; dev-bypass when unconfigured |
| History | Neon Postgres via SQLModel | Users, sessions, persisted trace logs |
| Frontend | Vite + React 18 + ReactFlow · Tailwind | Graphsight Studio |
| Observability | Sentry (opt-in via DSN) | Full-stack error + perf tracing |
| Agent interface | MCP SDK | Typed semantic tools at /mcp |
A single generic Entity table (id, label, type, ts, embedding) keeps the schema dynamic
across all entity labels. Document nodes hold one sliding-window chunk each. Edges:
Document -[MENTIONS]-> Entity and Entity -[RELATES_TO]-> Entity, carrying
confidence, relation, ts.
Typed relations, weighted by how much they actually tell you:
| Relation | Weight | Source |
|---|---|---|
AUTHORED |
0.95 | GitHub API |
RESOLVES |
0.92 | "Fixes #N" in PR body / commit message |
REVIEWED |
0.85 | /pulls/N/reviews |
TOUCHES |
0.80 | PR file list |
PART_OF |
0.80 | commit → repo |
REPORTED |
0.75 | issue author |
CO_OCCURS |
0.35 | same-window text proximity |
Text ingestion can only produce CO_OCCURS. Structure produces the rest, so a two-hop
AUTHORED → RESOLVES chain outranks a one-hop proximity guess. The store's merge only ever strengthens
an edge. The two GitHub passes still write different node ids, so a structural edge doesn't yet
upgrade a text-pass guess: they meet at query time through the shared label.
REVIEWED comes from /pulls/N/reviews, not requested_reviewers. That field is a
pending ask which empties once someone actually reviews, so reading it both misses real
reviewers and credits people who never opened the code. Self-approvals are excluded.
Recency. ts is the underlying event time (0 when unknown) and drives
score *= 0.5 ** (age_days / half_life), floored at RECENCY_FLOOR = 0.35. Half-life is
per type, Ticket 21d, Commit 45d, PR 60d, File 120d, Person 180d, Service/Repo 365d.
Undated entities are left alone rather than penalised, so a graph without dates behaves exactly
as it did before.
Migration is automatic. Opening an older store adds the missing columns via
ALTER TABLE ... ADD ... DEFAULT, keeping every row, and reprices legacy edges, the pre-0.3
writer stored them all at a degenerate 1.0, which flattened traversal and would have
outranked every typed relation forever. Verified lossless on a real 83-node store. Reingest to
populate real timestamps and relations.
backend/
├── api.py # FastAPI app + lifespan (warmup, pools, MCP mount)
├── auth.py # Clerk JWT verify
graphsight-memory/graphsight_memory/ # the engine, its own package
│ ├── api.py # Memory: sync_github, remember, retract, forget, learn, recall
│ ├── embed.py # MiniLM on ONNX (fastembed), sentence-transformers fallback
│ ├── github.py # GitHub REST: merged PRs, issues, commits, files, reviews
│ ├── relations.py # the fixed relation vocabulary
│ ├── config.py # single source of truth: weights, half-lives, thresholds
│ ├── db.py # LadybugDB: pools, HNSW, typed edges, schema migration
│ ├── extract.py # GLiNER (→ spaCy fallback) + sliding window + LRU cache
│ ├── curation.py # two-tier resolution (vector fast-merge + LLM grey zone)
│ ├── github_graph.py # GitHub payloads → typed, timestamped edges
│ ├── recency.py # age decay: 0.5 ** (age_days / half_life), floored
│ ├── router.py # intent classify + dual-stream fusion + trace_log
│ └── integrations/langchain.py # drop-in BaseRetriever
├── scripts/ # ingest · ingest_github · episodic_agent · stress_test
├── models/ · routers/ # chat history
└── tests/
├── test_recency.py · test_router_scoring.py · test_github_graph.py # scoring, no DB
├── test_db_integration.py # real .lbug: DDL, migration, MERGE semantics
└── test_ingest_github_wiring.py # the live path reaches the builder
frontend/ # Vite + React 18, landing + Graphsight Studio
└── src/components/
├── landing/ # marketing page at `/`
├── memory/ # Studio at `/studio`: CommandPanel · GraphCanvas · Inspector
graphsight-langgraph/ # standalone tracer: any LangGraph agent → AgentTrace v0.2
graphsight/ # pip-installable local viewer (bundled UI + stdlib server)
BETA.md # "Beta for Friends", 10-minute test script + feedback questions
docs/PITCH.md # 90-second builder pitch, demo script, hard-question answers
Unconstrained extraction produces near-duplicates. Asking an LLM about every pair is expensive and non-deterministic. So:
- Fast merge (0 LLM). Cosine ≥ 0.92 against an existing node → same entity, merge.
- Grey zone (0.85 to 0.92). Ask Groq a single yes/no question. Fail-safe is NO. An unreachable LLM mints a new node rather than silently collapsing two real entities.
- Below 0.85. New node, slug id.
Canonical labels are never clobbered by a noisier surface form; only ts moves forward, because
recency is the newest fact about a node.
Intent decides the weights. Lexical markers first (who, which PR, caused by →
relational; explain, architecture, how does → semantic), falling back to a one-token LLM
call with a 6-second timeout that fails safe to semantic.
relational alpha=0.15 beta=0.85 semantic alpha=0.80 beta=0.20
score_total = (alpha · score_vector + beta · score_graph) × recency
The graph arm is a multiplicative BFS. A path's score is the product of its edge weights, so
depth costs you, and a hop through AUTHORED (0.95) survives far better than one through
CO_OCCURS (0.35). Seeds come from deterministic entity links first, then fuzzy cosine hits.
Hub throttling is per relation, not per node. A repo TOUCHES everything, so that relation
carries no signal and is dropped, but the repo's other edges still work. Judging the whole node
would hide any PR with a wide diff, which is the opposite of what you want. (This was a real
bug: enabling --files gave real PRs enough TOUCHES edges to trip a per-node filter, making
47% of pallets/click's PRs untraversable and collapsing traversal to one hop.)
Everything is recorded. trace_log carries the intent weights, the vector seeds, every hop
with its relation, and a per-node recency breakdown (age_days, factor). The Studio renders
relation labels on traced edges and an age chip on nodes, so you can see why something ranked.
End to end, real embeddings. pallets/click, 50 PRs + 50 issues + 50 commits with files
and reviews, all-MiniLM-L6-v2 on CPU, warm:
p50 71ms · p95 90ms · mean 71ms · min/max 56/92ms (n=32, 274-node graph)
ingest 78.2s cold start: 46s import + 8.4s model load
That p50 is the whole route() path: query embedding, HNSW search, graph traversal, fusion,
document attach. The cold start is paid once per process and is entirely torch plus the
model, which is the strongest argument for an ONNX or hosted embedding option.
Scale and concurrency. fastapi/fastapi, 100 PRs + 100 issues + 100 commits, with a
stubbed embedder so the numbers isolate the store rather than the model:
graph 1274 nodes, 1633 typed edges
query p50 18ms · p95 21ms · max 36ms
concurrency 115 q/s across 32 threads, 0 errors (read pool = 10)
ingest 98.4s wall (concurrent per-PR detail fetch)
The gap between 18ms and 71ms is the embedder: roughly 50ms per query to encode.
Hostile input. No crash on: null user, null body, malformed dates, emoji logins, a
5000-repeat body, a self-referencing PR, or a 2000-file diff. Empty / whitespace / 50k-char /
null-byte queries all return normally. A Cypher injection attempt
('; MATCH (n) DETACH DELETE n; //) left all 1274 nodes intact. The parameterized queries
hold.
Test suite. 80 tests. Scoring logic runs against a fake store (no driver, no model
download); test_db_integration.py runs the real DDL, migration and MERGE ... ON MATCH
clauses through LadybugDB. CI installs pytest ladybug numpy pandas requests tqdm (no torch),
so the whole job finishes in seconds.
The web surface (/studio) renders a query's traced path: the node graph laid out with dagre,
the execution stepper, the fused scores, and clickable citations that pan the camera to the
cited node. Nodes show their age, the inspector shows the recency multiplier, and traced edges
show their relation.
Other routes: /memory/preview (mock data, no backend), /memory/import (render an external
LangGraph trace with no backend at all).
Run history lands in .graphsight/, browsable and diffable, two runs side by side, so you
can see what changed between them.
Current evaluation: 175 tasks on frozen snapshots of two repos, graded by exact match
against GitHub's records. pytest is held out: its tasks were committed before it was ever run.
Method, caveats and every per-question result.
| Setup | click (dev) | pytest (held-out) | Both |
|---|---|---|---|
| Full context | 80/81 | 91/94 | 171/175 |
| BM25 | 75/81 | 80/94 | 155/175 |
| Graph | 69/81 | 85/94 | 154/175 |
| Vector | 42/81 | 46/94 | 88/175 |
The graph and BM25 are tied: each was alone in being right on 18 and 19 tasks, and they split the two repos. The graph clearly beats plain vector search. Token savings against BM25 ran from 57% on click to 16% on pytest, so they don't generalize. Episodic recall improved 4 follow-up tasks and worsened none, on tasks built so a past run could help, which isn't proof yet.
Full context beat every retriever, 171 of 175, because each snapshot is small enough to fit in one prompt. These runs compare the retrievers with each other. They don't show that retrieval is needed at this size, and nothing here has been measured on a repo too large for one prompt.
Honest accounting of what is finished, what is partial, and what is not measured.
| Layer | Status |
|---|---|
graphsight viewer + graphsight-langgraph tracer |
Real, published at 0.3.1, install-verified from PyPI |
| Retrieval engine (ingest → curate → route → trace) | Real, 80 tests, live-verified against fastapi/fastapi and pallets/click |
| Structured GitHub ingest (typed, dated edges) | Real, exercised against the live API |
| Studio UI (trace canvas, streamed answers, citations, sessions) | Real, wired to the live API with an offline sample fallback |
| Multi-tenant pipeline | On the saas-control-plane branch, not main; see Multi-tenant mode |
MCP server (trace_impact / search_context / find_entity) |
Real, mounted at /mcp by default (TRACERAG_MCP_ENABLED=0 disables) |
| Landing page waitlist form | Real, posts to VITE_WAITLIST_ENDPOINT; refuses to report success when unset |
Single-writer LadybugDB lock (the API must start after ingest; single worker). In-memory rate-limit counters, so a single worker. Retrieval quality is unmeasured. The engine's correctness and performance are verified, but every automated test stubs the embedder, so vector relevance itself has no regression coverage. And the accuracy ceiling in Benchmarks.
The engine needs Python 3.12 or newer. On 3.11, LadybugDB 0.18.3 loads the VECTOR extension fine but
CREATE_VECTOR_INDEXdies withRuntimeError: Caught an unknown exception!, so ingestion fails at the index build. Verified working on 3.12.3 and 3.13.7. (Thegraphsightandgraphsight-langgraphpackages are unaffected, they never touch LadybugDB and support 3.10+.)Windows: install from a short path.
torchships header files deep enough to exceed the 260-character path limit, so a venv under a long nested directory fails withOSError: [Errno 2] No such file or directory: ...predicated_tile_access_iterator_residual_last.h. Either enable Long Path support or put the venv somewhere short likeC:\venv.
pip install -e ./graphsight-memory
cd backend
pip install -r requirements.txt
python -m spacy download en_core_web_sm # NER fallback
# .env: GROQ_API_KEY + OPENROUTER_API_KEY (see .env.example)
# optional: CLERK_ISSUER + APP_DATABASE_URL for auth + history
# ingest your own documents
python scripts/ingest.py --datasets ./datasets --reset
# ...or a GitHub repo. Two passes land in one graph: PR prose through extraction,
# and the same payloads through GitHubGraphBuilder, which reads authorship,
# reviews, "Fixes #N" and touched files as typed, dated edges.
python scripts/ingest_github.py --repo pallets/click --issues 50 --commits 50 --files --reviews
# --files / --reviews each cost one extra request per PR (fetched concurrently);
# without them there are no TOUCHES / REVIEWED edges.
# --no-text writes structured edges only, skipping extraction entirely.
# GITHUB_TOKEN in .env raises the rate limit from 60/hr to 5000/hr.
# serve (AFTER ingest: single-writer DB lock, single worker)
uvicorn api:app --reload --port 8000 # docs at /docs
# frontend, in a second shell
cd ../frontend; npm install; npm run dev
# http://localhost:5173/ -> landing
# http://localhost:5173/studio -> Graphsight StudioNot on main. It lives on the
saas-control-plane branch:
per-org isolated graphs, compiled by a Celery pipeline (GitHub → NLP → Postgres → compile → S3)
and atomically swapped onto serving pods. Parked until there is a first tenant; its heavy pipeline
steps were still stubs when it was moved off main.
| Endpoint | Purpose |
|---|---|
POST /api/trace |
Hybrid retrieval → results + trace_log |
POST /api/subgraph |
Nodes + 1-hop neighbours + edges for the canvas |
POST /api/answer |
Streamed, grounded answer with citations |
GET/POST /api/graphs · /switch |
List and hot-swap the active .lbug |
GET/POST /api/sessions · /traces |
Persisted chat history (ownership-checked) |
POST /api/summarize |
One-sentence node summary (server-cached) |
/mcp |
MCP tools: trace_impact · search_context · find_entity |
Open source, MIT, and genuinely early. The most useful things you could send:
Adapters for other frameworks. LangGraph is the only one wired up today. Every producer
emits the same AgentTrace contract (see schema.py),
so a LlamaIndex, DSPy, or plain-OpenAI adapter is a few hundred lines and needs nothing from
the viewer.
Attack the used/ignored heuristic. It is lexical overlap at a 0.2 threshold. It is fooled by paraphrase, and it is blind to a chunk the model used as a negative constraint, which leaves no words behind. The planned fix is offline leave-one-out ablation as ground truth, with precision and recall published against it. If you have measured this properly, tell me what you found.
Break it on your data. Non-English repos, monorepos, private corpora. It has been run against a handful of public repos, not hundreds.
# package tests, from the repo root
pip install -e ./graphsight -e ./graphsight-langgraph
pip install pytest langgraph
pytest graphsight/tests graphsight-langgraph/tests -q
# engine tests, no torch needed (the embedder is stubbed)
pip install -e ./graphsight-memory pytest tqdm
pytest -q graphsight-memory/tests
cd backend
pytest -q tests/test_recency.py tests/test_router_scoring.py \
tests/test_github_graph.py tests/test_db_integration.py \
tests/test_ingest_github_wiring.pyThese are the same commands CI runs. Engine tests need Python 3.12+ (see Running the engine locally). Issues and PRs both welcome, and a bug report with a trace file attached is worth more than either.
MIT, see LICENSE.
