Skip to content

README.md

Latest commit

 

History

History
102 lines (71 loc) · 4.3 KB

README.md

File metadata and controls

102 lines (71 loc) · 4.3 KB
Raw

Context Engine

A passive personal-context database that any AI tool can query over MCP.

It learns how you work — your prompts, terminal commands, and editor activity — and exposes that as relevant, weighted context so an AI starts each conversation already knowing who you are and what you're working on, instead of from a blank slate.

Crucially, it ranks context by semantic relevance × weight × recency and applies time decay, so stale or off-topic memories fade instead of polluting every answer (the classic "you mentioned a PDF once and now it's cited everywhere" problem).

Everything runs locally. Embeddings come from a local Ollama model — no API keys, no data leaves your machine.

How it works

Collectors ──► Store (SQLite + vectors) ──► MCP server ──► any AI tool
 claude hook      relevance + decay          get_user_context()
 terminal                                    add_fact()
 vscode                                       list_recent()
  • Collectors capture observations and funnel them through one ingest.add_entry pipeline (redacts secrets, drops noise, dedupes).
  • Store (src/context_engine/store.py) keeps each entry with weight, frequency, explicit, and last_seen_at. A daily decay job lowers weights; reinforcement on near-duplicates raises them.
  • Retrieval (retrieval.py) embeds the query, scores every entry by cosine × weight × recency, and drops anything below a similarity threshold — that filter is what keeps unrelated context out.
  • MCP server (mcp_server.py) exposes the three tools any MCP client can call.

Setup

# 1. Embedding model (needs Ollama running)
bash scripts/pull_model.sh                      # pulls nomic-embed-text

# 2. Python env + install
conda activate con_eng                           # Python 3.11
pip install -e .

# 3. Config
cp config.example.toml config.toml               # tweak thresholds if you like

Register the MCP server with Claude Code

claude mcp add context-engine -- \
  /home/ishan/miniconda3/envs/con_eng/bin/python -m context_engine.mcp_server

Then in any session, Claude can call get_user_context("<topic>") before answering.

Enable the collectors

# Claude Code prompts (append-safe; backs up settings.json, keeps existing hooks)
python scripts/install_claude_hook.py --python /home/ishan/miniconda3/envs/con_eng/bin/python

# Terminal commands (auto-detects bash/zsh)
bash scripts/install_shell_hook.sh && source ~/.bashrc

# VS Code: build & install the extension, then ingest its events periodically
cd vscode-extension && npm install && npm run compile   # then load/package the extension
python -m context_engine.collectors.vscode              # ingests new editor events

Daily decay (cron)

0 3 * * * /home/ishan/miniconda3/envs/con_eng/bin/python \
    /home/ishan/Desktop/Ishan/Coding/Projects/context-engine/scripts/decay.py \
    >> /tmp/context-engine-decay.log 2>&1

Tuning

All knobs live in config.toml:

Setting Effect
retrieval.min_similarity Higher = stricter relevance, less leakage (less recall).
decay.rate / explicit_rate How fast incidental vs. deliberate context fades.
ingest.dedupe_similarity How aggressively repeats reinforce one entry vs. add new rows.

Note on small-model recall: nomic-embed-text is a compact local model. On short, generic queries two loosely-related entries can both clear the threshold. Weighting/frequency separates them over time, and you can raise min_similarity if you see leakage.

Privacy

  • Data lives only in data/context.db (gitignored).
  • Secret-looking strings (API keys, tokens, JWTs, KEY=...) are filtered before storage.
  • list_recent lets you (or the AI, on your behalf) audit exactly what's stored.

Roadmap

  • ChatGPT history via a browser extension (source='chatgpt' is already reserved).
  • Swap brute-force cosine for sqlite-vec if the store ever grows large.

Layout

src/context_engine/   config, embeddings, store, retrieval, ingest, mcp_server, collectors/
scripts/              pull_model, install_claude_hook, install_shell_hook, decay
vscode-extension/     TypeScript activity logger (writes JSONL)
data/                 the SQLite db (gitignored)