Skip to content

About

Project memory for coding agents: GitHub history as a typed graph, plus a tracer that shows what your agent retrieved and what it used.

Resources

Stars

5 stars

Watchers

0 watching

Forks

Repository files navigation

Graphsight

See what your agent retrieved, and what it actually used.

PyPI PyPI Python 3.10+ License: MIT

Your agent retrieved twelve documents and answered from three. The other nine cost you tokens, latency, and a wider surface for the model to go wrong on. Nothing in your stack tells you which nine.

Graphsight traces the run and marks each retrieved item overlaps the answer or no lexical trace in answer, scored against the final answer. No LLM re-reads your evidence, no score is invented, and nothing leaves localhost.

Graphsight showing a retrieved-but-unused document

A real run. PR #101 scored 0.910, the highest of anything retrieved, and the answer never used it. PR #412 scored 0.340 and is the one that answered.

Jump to: what this is · architecture · routing · benchmarks · project status · contributing


Quickstart

New to this? Your first trace is a 10-minute walkthrough with a complete runnable example. Otherwise:

pip install graphsight graphsight-langgraph

Add one callback handler to any LangGraph agent:

from graphsight_langgraph import LangGraphTracer, capture

tracer = LangGraphTracer()
result = graph.invoke(inputs, config={"callbacks": [tracer]})

capture(tracer, query="why is checkout failing?", answer=result["answer"])
# -> .graphsight/20260728T184600_why-is-checkout-failing.json

Passing answer= is what unlocks the retrieved-vs-used split. Without it, everything renders plain and no usage is claimed.

Then open it:

graphsight .graphsight/

A local server on 127.0.0.1:4630 renders the run: the node graph, the retrieval steps, and the retrieved-vs-used verdict per item. The viewer has zero dependencies, stdlib http.server plus a bundled UI build. The tracer depends only on langchain-core.

No account, no API key, no telemetry, no cloud.

No framework? Build the trace directly

The trace contract is framework neutral. LangGraph is one producer, not a requirement. If you wrote your retrieval loop yourself, construct the trace from whatever you already have and skip the callback entirely:

from graphsight_langgraph import AgentTrace, Retrieval, RetrievedItem, save_trace
from graphsight_langgraph._text import tokens

atoks = tokens(answer)
def overlap(text):                      # same heuristic the tracer uses
    t = tokens(text)
    return round(len(atoks & t) / min(len(t), len(atoks)), 3) if t and atoks else None

trace = AgentTrace(query=query, framework="custom", answer=answer)
trace.retrievals.append(Retrieval(
    span_id="retrieve", query=query, arm="vector",
    items=[RetrievedItem(id=c.id, label=c.title, content=c.text,
                         score=c.score, answer_overlap=overlap(c.text))
           for c in chunks],
))
print(save_trace(trace))                # -> .graphsight/<timestamp>_<slug>.json

answer_overlap is what drives the used/ignored split. Leave it None and every item renders plain, which is the honest default when there is nothing to compare against.

Episodic memory

Label a run and later runs can recall it. Only runs labelled success are ever recalled, so a bad run can't teach the next one.

capture(tracer, query=q, answer=a, outcome="success")   # saved to .graphsight/

recall("who broke checkout?")        # past runs that asked something similar
EpisodeRetriever().invoke(question)  # the same, traced like any other retriever

Matching is lexical, so a reworded question with different words won't match. Recalled episodes appear as nodes in the viewer. backend/scripts/episodic_agent.py is a working LangGraph agent that uses this with the graph engine, and --no-memory gives you the baseline.

Graph memory

graphsight-memory is its own package: no torch, no API key, one local file. Its README has the full API.

An agent can write facts into the graph, correct them, and delete them. They sit beside the facts read from GitHub without ever overwriting them.

from graphsight_memory import Memory

mem = Memory("team.lbug")
mem.sync_github("org/repo")
mem.remember("checkout-service", "OWNED_BY", "payments team", source="run-42")
mem.remember("checkout-service", "OWNED_BY", "billing team", source="run-57", replace=True)
print(mem.recall("who owns checkout-service?"))
  • Labelled. Remembered facts reach the model as checkout-service OWNED_BY billing team (remembered, source: run-57), and they weigh less than any fact read from GitHub.
  • Replaced, not lost. replace=True ends the older fact. It stops being retrieved but stays in the store with the time it ended.
  • Forget is scoped. forget() deletes only agent facts, history included, and removes nodes it created that are left with no edges. Facts read from GitHub can't be forgotten this way.
  • Over MCP as remember and forget, which exist only when TRACERAG_MCP_WRITE_TOKEN is set and check that token on every call.

Known limit: after a replace, the old object's name can still surface as a node with no live facts, because its ended fact is kept as history.

Measured: on 22 held-out tasks about facts stated earlier, graph memory answered 16, episodic recall 14, and no memory 5. Graph memory clearly beats no memory, and it is tied with episodic recall. It answered 5 of 6 changed facts against episodic's 3, which is too few to call. It also gives confident wrong answers on questions that sound like a stated fact. Details.

Joins: on 28 tasks written for a repo nothing was tuned on (pydantic), each needing a told fact joined to a GitHub fact, graph memory answered 27 and episodic recall 10, and every episodic miss was a question that starts from GitHub. Putting the whole snapshot in the prompt did as well as graph memory. Details.

What this actually is

Two halves, deliberately separable.

1. The tracer + viewer: shipped, use it today

Package What it does Dependencies
graphsight-langgraph One BaseCallbackHandler → framework-neutral AgentTrace. Also ships graphsight-github-trace (repo → traced run in one command) langchain-core
graphsight Local viewer, bundled Studio UI + stdlib server. graphsight trace.json or graphsight .graphsight/ none

This half works on flat traces. No graph, no database, no ingestion. If your agent does plain vector RAG, you still get the retrieved-vs-used signal, run history, and run diffing.

2. The graph memory engine: graphsight-memory

A local-first hybrid store where vectors and the relationship graph live in one .lbug file, no graph-DB ↔ vector-DB sync problem, behind a two-tier entity resolver and an intent-weighted router that emits a full trace_log for every query.

This is what makes retrieval causal rather than merely similar: an agent asking "what broke the session flow?" should reach yesterday's PR through AUTHORED → RESOLVES, not an eight-month-old PR that happens to read like the question.

The engine is the graphsight-memory package. It embeds with MiniLM on ONNX, so it installs without torch and runs without API keys. An LLM is optional: it extracts facts from text in learn(), and classifies a question the keyword check can't place. The backend adds the web API, document ingest with spaCy and GLiNER, and the Studio.


The one idea

One hybrid store, queried by two arms (vector + graph), fused by intent, and every step is recorded.

Multi-hop retrieval for agents fails for three infrastructural reasons, and Graphsight addresses each:

Failure Fix
Dual-store sync hell, keeping a graph DB and a vector DB consistent Both live natively in one embedded LadybugDB file
Entity drift, unconstrained extraction turns PaymentService, payments-v2, pay_svc into three nodes Two-tier curation: cheap vector auto-merge, LLM adjudicates only the grey zone
Black-box routing, no insight into why a hybrid retriever chose a graph edge over a chunk Every query returns a structured trace_log the Studio renders

The anti-slop principle throughout: evidence is never paraphrased by an LLM, and scores are never invented. Overlap is lexical and reproducible. If we can't measure something, we don't claim it.


Architecture

flowchart TB
  subgraph INGEST["1 · Ingestion"]
    direction TB
    SRC["GitHub payloads · docs"]
    SRC --> STRUCT["GitHubGraphBuilder<br/>reads relations from the API:<br/>author · reviews · closes-refs · files"]
    SRC --> TEXT["sliding window → NER →<br/>two-tier curation"]
    STRUCT --> TYPED["typed, timestamped edges<br/>AUTHORED · RESOLVES · TOUCHES"]
    TEXT --> PROX["CO_OCCURS<br/>(proximity, weighted lowest)"]
  end

  subgraph STORE["2 · LadybugDB, one .lbug"]
    direction LR
    ENT[("Entity<br/>id · label · type · ts<br/>embedding FLOAT 384")]
    DOCN[("Document<br/>id · path · content")]
    ENT -->|"RELATES_TO<br/>confidence · relation · ts"| ENT
    DOCN -->|MENTIONS| ENT
    HNSW["HNSW index"]
    ENT --- HNSW
  end
  TYPED --> ENT
  PROX --> ENT

  subgraph QUERY["3 · Query, two arms, fused by intent"]
    direction TB
    Q["query"] --> INTENT{"intent?"}
    INTENT -->|relational| WR["alpha=0.15 beta=0.85"]
    INTENT -->|semantic| WS["alpha=0.80 beta=0.20"]
    Q --> VEC["vector arm<br/>HNSW top-k"]
    Q --> GR["graph arm<br/>multiplicative BFS<br/>path = product of edge weights"]
    VEC --> FUSE["score = (alpha·sv + beta·sg) × recency"]
    GR --> FUSE
    WR --> FUSE
    WS --> FUSE
  end
  ENT --> VEC
  ENT --> GR
  FUSE --> TRACE["trace_log:<br/>seeds · hops + relation ·<br/>recency per node"]
  TRACE --> UI["Graphsight Studio"]
Loading

Stack

Layer Technology Role
Embeddings sentence-transformers · all-MiniLM-L6-v2 384-dim vectors (cosine)
Entity extraction GLiNER, with spaCy as fallback Zero-shot NER. spaCy runs when GLiNER can't load
Storage LadybugDB (embedded, Kùzu-lineage) Single-file hybrid vector + graph store, native HNSW
LLM, router intent Groq · qwen/qwen3.8-27b Query intent classification; falls back to OpenRouter when unset
LLM, generation OpenRouter · anthropic/claude-haiku-5.5 Grounded answers + node summaries
API FastAPI + Uvicorn trace · subgraph · answer · graphs · history
Concurrency read connection pool + bounded embed pool parallel graph/vector reads; embeds off request threads
Auth Clerk + PyJWT / JWKS Networkless verification; dev-bypass when unconfigured
History Neon Postgres via SQLModel Users, sessions, persisted trace logs
Frontend Vite + React 18 + ReactFlow · Tailwind Graphsight Studio
Observability Sentry (opt-in via DSN) Full-stack error + perf tracing
Agent interface MCP SDK Typed semantic tools at /mcp

Data model

A single generic Entity table (id, label, type, ts, embedding) keeps the schema dynamic across all entity labels. Document nodes hold one sliding-window chunk each. Edges: Document -[MENTIONS]-> Entity and Entity -[RELATES_TO]-> Entity, carrying confidence, relation, ts.

Typed relations, weighted by how much they actually tell you:

Relation Weight Source
AUTHORED 0.95 GitHub API
RESOLVES 0.92 "Fixes #N" in PR body / commit message
REVIEWED 0.85 /pulls/N/reviews
TOUCHES 0.80 PR file list
PART_OF 0.80 commit → repo
REPORTED 0.75 issue author
CO_OCCURS 0.35 same-window text proximity

Text ingestion can only produce CO_OCCURS. Structure produces the rest, so a two-hop AUTHORED → RESOLVES chain outranks a one-hop proximity guess. The store's merge only ever strengthens an edge. The two GitHub passes still write different node ids, so a structural edge doesn't yet upgrade a text-pass guess: they meet at query time through the shared label.

REVIEWED comes from /pulls/N/reviews, not requested_reviewers. That field is a pending ask which empties once someone actually reviews, so reading it both misses real reviewers and credits people who never opened the code. Self-approvals are excluded.

Recency. ts is the underlying event time (0 when unknown) and drives score *= 0.5 ** (age_days / half_life), floored at RECENCY_FLOOR = 0.35. Half-life is per type, Ticket 21d, Commit 45d, PR 60d, File 120d, Person 180d, Service/Repo 365d. Undated entities are left alone rather than penalised, so a graph without dates behaves exactly as it did before.

Migration is automatic. Opening an older store adds the missing columns via ALTER TABLE ... ADD ... DEFAULT, keeping every row, and reprices legacy edges, the pre-0.3 writer stored them all at a degenerate 1.0, which flattened traversal and would have outranked every typed relation forever. Verified lossless on a real 83-node store. Reingest to populate real timestamps and relations.

Repository layout

backend/
├── api.py                  # FastAPI app + lifespan (warmup, pools, MCP mount)
├── auth.py                 # Clerk JWT verify
graphsight-memory/graphsight_memory/   # the engine, its own package
│   ├── api.py              # Memory: sync_github, remember, retract, forget, learn, recall
│   ├── embed.py            # MiniLM on ONNX (fastembed), sentence-transformers fallback
│   ├── github.py           # GitHub REST: merged PRs, issues, commits, files, reviews
│   ├── relations.py        # the fixed relation vocabulary
│   ├── config.py           # single source of truth: weights, half-lives, thresholds
│   ├── db.py               # LadybugDB: pools, HNSW, typed edges, schema migration
│   ├── extract.py          # GLiNER (→ spaCy fallback) + sliding window + LRU cache
│   ├── curation.py         # two-tier resolution (vector fast-merge + LLM grey zone)
│   ├── github_graph.py     # GitHub payloads → typed, timestamped edges
│   ├── recency.py          # age decay: 0.5 ** (age_days / half_life), floored
│   ├── router.py           # intent classify + dual-stream fusion + trace_log
│   └── integrations/langchain.py   # drop-in BaseRetriever
├── scripts/                # ingest · ingest_github · episodic_agent · stress_test
├── models/ · routers/      # chat history
└── tests/
    ├── test_recency.py · test_router_scoring.py · test_github_graph.py   # scoring, no DB
    ├── test_db_integration.py           # real .lbug: DDL, migration, MERGE semantics
    └── test_ingest_github_wiring.py     # the live path reaches the builder

frontend/                   # Vite + React 18, landing + Graphsight Studio
└── src/components/
    ├── landing/            # marketing page at `/`
    ├── memory/             # Studio at `/studio`: CommandPanel · GraphCanvas · Inspector

graphsight-langgraph/       # standalone tracer: any LangGraph agent → AgentTrace v0.2
graphsight/                 # pip-installable local viewer (bundled UI + stdlib server)
BETA.md                     # "Beta for Friends", 10-minute test script + feedback questions
docs/PITCH.md               # 90-second builder pitch, demo script, hard-question answers

Entity resolution: the two-tier janitor

Unconstrained extraction produces near-duplicates. Asking an LLM about every pair is expensive and non-deterministic. So:

  1. Fast merge (0 LLM). Cosine ≥ 0.92 against an existing node → same entity, merge.
  2. Grey zone (0.85 to 0.92). Ask Groq a single yes/no question. Fail-safe is NO. An unreachable LLM mints a new node rather than silently collapsing two real entities.
  3. Below 0.85. New node, slug id.

Canonical labels are never clobbered by a noisier surface form; only ts moves forward, because recency is the newest fact about a node.


Hybrid routing

Intent decides the weights. Lexical markers first (who, which PR, caused by → relational; explain, architecture, how does → semantic), falling back to a one-token LLM call with a 6-second timeout that fails safe to semantic.

relational  alpha=0.15  beta=0.85       semantic  alpha=0.80  beta=0.20
score_total = (alpha · score_vector + beta · score_graph) × recency

The graph arm is a multiplicative BFS. A path's score is the product of its edge weights, so depth costs you, and a hop through AUTHORED (0.95) survives far better than one through CO_OCCURS (0.35). Seeds come from deterministic entity links first, then fuzzy cosine hits.

Hub throttling is per relation, not per node. A repo TOUCHES everything, so that relation carries no signal and is dropped, but the repo's other edges still work. Judging the whole node would hide any PR with a wide diff, which is the opposite of what you want. (This was a real bug: enabling --files gave real PRs enough TOUCHES edges to trip a per-node filter, making 47% of pallets/click's PRs untraversable and collapsing traversal to one hop.)

Everything is recorded. trace_log carries the intent weights, the vector seeds, every hop with its relation, and a per-node recency breakdown (age_days, factor). The Studio renders relation labels on traced edges and an age chip on nodes, so you can see why something ranked.


Measured behaviour

End to end, real embeddings. pallets/click, 50 PRs + 50 issues + 50 commits with files and reviews, all-MiniLM-L6-v2 on CPU, warm:

p50 71ms · p95 90ms · mean 71ms · min/max 56/92ms   (n=32, 274-node graph)
ingest 78.2s   cold start: 46s import + 8.4s model load

That p50 is the whole route() path: query embedding, HNSW search, graph traversal, fusion, document attach. The cold start is paid once per process and is entirely torch plus the model, which is the strongest argument for an ONNX or hosted embedding option.

Scale and concurrency. fastapi/fastapi, 100 PRs + 100 issues + 100 commits, with a stubbed embedder so the numbers isolate the store rather than the model:

graph        1274 nodes, 1633 typed edges
query        p50 18ms · p95 21ms · max 36ms
concurrency  115 q/s across 32 threads, 0 errors (read pool = 10)
ingest       98.4s wall (concurrent per-PR detail fetch)

The gap between 18ms and 71ms is the embedder: roughly 50ms per query to encode.

Hostile input. No crash on: null user, null body, malformed dates, emoji logins, a 5000-repeat body, a self-referencing PR, or a 2000-file diff. Empty / whitespace / 50k-char / null-byte queries all return normally. A Cypher injection attempt ('; MATCH (n) DETACH DELETE n; //) left all 1274 nodes intact. The parameterized queries hold.

Test suite. 80 tests. Scoring logic runs against a fake store (no driver, no model download); test_db_integration.py runs the real DDL, migration and MERGE ... ON MATCH clauses through LadybugDB. CI installs pytest ladybug numpy pandas requests tqdm (no torch), so the whole job finishes in seconds.


Graphsight Studio

The web surface (/studio) renders a query's traced path: the node graph laid out with dagre, the execution stepper, the fused scores, and clickable citations that pan the camera to the cited node. Nodes show their age, the inspector shows the recency multiplier, and traced edges show their relation.

Other routes: /memory/preview (mock data, no backend), /memory/import (render an external LangGraph trace with no backend at all).

Run history lands in .graphsight/, browsable and diffable, two runs side by side, so you can see what changed between them.


Benchmarks & known constraints

Current evaluation: 175 tasks on frozen snapshots of two repos, graded by exact match against GitHub's records. pytest is held out: its tasks were committed before it was ever run. Method, caveats and every per-question result.

Setup click (dev) pytest (held-out) Both
Full context 80/81 91/94 171/175
BM25 75/81 80/94 155/175
Graph 69/81 85/94 154/175
Vector 42/81 46/94 88/175

The graph and BM25 are tied: each was alone in being right on 18 and 19 tasks, and they split the two repos. The graph clearly beats plain vector search. Token savings against BM25 ran from 57% on click to 16% on pytest, so they don't generalize. Episodic recall improved 4 follow-up tasks and worsened none, on tasks built so a past run could help, which isn't proof yet.

Full context beat every retriever, 171 of 175, because each snapshot is small enough to fit in one prompt. These runs compare the retrievers with each other. They don't show that retrieval is needed at this size, and nothing here has been measured on a repo too large for one prompt.


Project status

Honest accounting of what is finished, what is partial, and what is not measured.

What's real vs. what's mocked

Layer Status
graphsight viewer + graphsight-langgraph tracer Real, published at 0.3.1, install-verified from PyPI
Retrieval engine (ingest → curate → route → trace) Real, 80 tests, live-verified against fastapi/fastapi and pallets/click
Structured GitHub ingest (typed, dated edges) Real, exercised against the live API
Studio UI (trace canvas, streamed answers, citations, sessions) Real, wired to the live API with an offline sample fallback
Multi-tenant pipeline On the saas-control-plane branch, not main; see Multi-tenant mode
MCP server (trace_impact / search_context / find_entity) Real, mounted at /mcp by default (TRACERAG_MCP_ENABLED=0 disables)
Landing page waitlist form Real, posts to VITE_WAITLIST_ENDPOINT; refuses to report success when unset

Known gaps

Single-writer LadybugDB lock (the API must start after ingest; single worker). In-memory rate-limit counters, so a single worker. Retrieval quality is unmeasured. The engine's correctness and performance are verified, but every automated test stubs the embedder, so vector relevance itself has no regression coverage. And the accuracy ceiling in Benchmarks.


Running the engine locally

The engine (local, single-tenant)

The engine needs Python 3.12 or newer. On 3.11, LadybugDB 0.18.3 loads the VECTOR extension fine but CREATE_VECTOR_INDEX dies with RuntimeError: Caught an unknown exception!, so ingestion fails at the index build. Verified working on 3.12.3 and 3.13.7. (The graphsight and graphsight-langgraph packages are unaffected, they never touch LadybugDB and support 3.10+.)

Windows: install from a short path. torch ships header files deep enough to exceed the 260-character path limit, so a venv under a long nested directory fails with OSError: [Errno 2] No such file or directory: ...predicated_tile_access_iterator_residual_last.h. Either enable Long Path support or put the venv somewhere short like C:\venv.

pip install -e ./graphsight-memory
cd backend
pip install -r requirements.txt
python -m spacy download en_core_web_sm        # NER fallback
#  .env: GROQ_API_KEY + OPENROUTER_API_KEY               (see .env.example)
#  optional: CLERK_ISSUER + APP_DATABASE_URL for auth + history

# ingest your own documents
python scripts/ingest.py --datasets ./datasets --reset

# ...or a GitHub repo. Two passes land in one graph: PR prose through extraction,
# and the same payloads through GitHubGraphBuilder, which reads authorship,
# reviews, "Fixes #N" and touched files as typed, dated edges.
python scripts/ingest_github.py --repo pallets/click --issues 50 --commits 50 --files --reviews
#   --files / --reviews each cost one extra request per PR (fetched concurrently);
#   without them there are no TOUCHES / REVIEWED edges.
#   --no-text writes structured edges only, skipping extraction entirely.
#   GITHUB_TOKEN in .env raises the rate limit from 60/hr to 5000/hr.

# serve (AFTER ingest: single-writer DB lock, single worker)
uvicorn api:app --reload --port 8000           # docs at /docs

# frontend, in a second shell
cd ../frontend; npm install; npm run dev
#   http://localhost:5173/         -> landing
#   http://localhost:5173/studio   -> Graphsight Studio

Multi-tenant mode

Not on main. It lives on the saas-control-plane branch: per-org isolated graphs, compiled by a Celery pipeline (GitHub → NLP → Postgres → compile → S3) and atomically swapped onto serving pods. Parked until there is a first tenant; its heavy pipeline steps were still stubs when it was moved off main.


API surface

Endpoint Purpose
POST /api/trace Hybrid retrieval → results + trace_log
POST /api/subgraph Nodes + 1-hop neighbours + edges for the canvas
POST /api/answer Streamed, grounded answer with citations
GET/POST /api/graphs · /switch List and hot-swap the active .lbug
GET/POST /api/sessions · /traces Persisted chat history (ownership-checked)
POST /api/summarize One-sentence node summary (server-cached)
/mcp MCP tools: trace_impact · search_context · find_entity

Contributing

Open source, MIT, and genuinely early. The most useful things you could send:

Adapters for other frameworks. LangGraph is the only one wired up today. Every producer emits the same AgentTrace contract (see schema.py), so a LlamaIndex, DSPy, or plain-OpenAI adapter is a few hundred lines and needs nothing from the viewer.

Attack the used/ignored heuristic. It is lexical overlap at a 0.2 threshold. It is fooled by paraphrase, and it is blind to a chunk the model used as a negative constraint, which leaves no words behind. The planned fix is offline leave-one-out ablation as ground truth, with precision and recall published against it. If you have measured this properly, tell me what you found.

Break it on your data. Non-English repos, monorepos, private corpora. It has been run against a handful of public repos, not hundreds.

# package tests, from the repo root
pip install -e ./graphsight -e ./graphsight-langgraph
pip install pytest langgraph
pytest graphsight/tests graphsight-langgraph/tests -q

# engine tests, no torch needed (the embedder is stubbed)
pip install -e ./graphsight-memory pytest tqdm
pytest -q graphsight-memory/tests
cd backend
pytest -q tests/test_recency.py tests/test_router_scoring.py \
          tests/test_github_graph.py tests/test_db_integration.py \
          tests/test_ingest_github_wiring.py

These are the same commands CI runs. Engine tests need Python 3.12+ (see Running the engine locally). Issues and PRs both welcome, and a bug report with a trace file attached is worth more than either.


License

MIT, see LICENSE.

About

Project memory for coding agents: GitHub history as a typed graph, plus a tracer that shows what your agent retrieved and what it used.

Resources

Stars

5 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages