Building a Self-Improving Personal Knowledge Base Powered by LLM
-
- Raw storage — everything you ingest goes into raw/. Web articles, PDFs, images, notes. Nothing is processed here; it’s just a staging area. You feed this layer.
-
- The wiki — a compiled, LLM-managed collection of markdown files in Obsidian format. Every concept gets its own article. Every source gets a summary. Everything is linked with Obsidian [[wikilinks]]. The LLM owns this layer entirely.
-
- Outputs — Q&A answers, synthesis reports, lint reports, slides, charts. These get filed back into the wiki so future queries can reference them.
New Video: Obsidian + Karpathy = 95% Cheaper “RAG” in Claude Code
claude-memory-compiler Adapted from Karpathy's LLM Knowledge Base architecture, but instead of clipping web articles, the raw data is your own conversations with Claude Code. When a session ends (or auto-compacts mid-session), Claude Code hooks capture the conversation transcript and spawn a background process that uses the Claude Agent SDK to extract the important stuff - decisions, lessons learned, patterns, gotchas - and appends it to a daily log. You then compile those daily logs into structured, cross-referenced knowledge articles organized by concept. Retrieval uses a simple index file instead of RAG - no vector database, no embeddings, just markdown.
AI Knowledge Layer (and why your agents are useless without it)
+-------------------------------------------------------+ | YOUR AGENTS | | (writer, researcher, strategist, analyst) | +---------------------------+---------------------------+ | reads from | reads from v v +------------------+ +-------------------+ | KNOWLEDGE BASE | | BRAND FOUNDATION | | LAYER (KBL) | | (BF) | | | | | | dynamic | | static | | agent-maintained| | human-edited | | grows over time | | your voice, your | | wiki pages, | | rules, your | | sources, index | | positioning | +--------+---------+ +-------------------+ | compiles from | +--------+---------+ | raw/ inbox | | tweets, articles | | bookmarks, PDFs | | notes, ideas | +-------------------+
memoriki Personal knowledge base with real memory. Combines LLM Wiki (Andrej Karpathy) + MemPalace (MCP server).
Wiki gives structure. MemPalace gives memory.
# 1. Clone
git clone https://github.com/AyanbekDos/memoriki.git my-knowledge-base
cd my-knowledge-base
# 2. Install MemPalace
pip install mempalace
mempalace init .
# 3. Connect MemPalace to Claude Code
claude mcp add mempalace -- python -m mempalace.mcp_server
# 4. Drop your first source
cp ~/some-article.md raw/
# 5. Launch Claude Code and start ingesting
claude
# > Read raw/some-article.md and ingest it into the wiki
When Your AI Wiki Outgrows the Context Window — A Practical Guide to RAG
Schema-First Extraction for LLM Wikis with GLiNER2
llm_wiki A personal knowledge base that builds itself. LLM reads your documents, builds a structured wiki, and keeps it current.
OpenKB — Open LLM Knowledge Base
OpenKB (Open Knowledge Base) is an open-source system (in CLI) that compiles raw documents into a structured, interlinked wiki-style knowledge base using LLMs, powered by PageIndex for vectorless long document retrieval.
raw/ You drop files here
│
├─ Short docs ──→ markitdown ──→ LLM reads full text
│ │
├─ Long PDFs ──→ PageIndex ────→ LLM reads document trees
│ │
│ ▼
│ Wiki Compilation (using LLM)
│ │
▼ ▼
wiki/
├── index.md Knowledge base overview
├── log.md Operations timeline
├── AGENTS.md Wiki schema (LLM instructions)
├── sources/ Full-text conversions
├── summaries/ Per-document summaries
├── concepts/ Cross-document synthesis ← the good stuff
├── explorations/ Saved query results
└── reports/ Lint reports
Turn your raw notes into a self-improving, interlinked wiki — powered by a local LLM.
Drop a markdown file into a folder. The pipeline reads it, extracts concepts, and creates or updates wiki articles with the new knowledge. Reject a draft and explain why — the next compile addresses your feedback. Over time your wiki compounds: every note you add (and every draft you review) makes the whole smarter.
Local-first, provider-flexible. Runs 100% locally with Ollama by default. Also works with any OpenAI-compatible endpoint — Groq, Together AI, LM Studio, vLLM, Azure OpenAI, and more.
Andrej Karpathy proposed LLM-Wiki: an LLM that builds and maintains a persistent, structured wiki from your sources — not a throwaway RAG answer, but compounding knowledge that grows smarter with every paper you feed it.
ΩmegaWiki takes that idea and runs the full distance. It's not just a wiki builder — it's a complete research lifecycle platform: from paper ingestion → knowledge graph → gap detection → idea generation → experiment design → paper writing → peer review response. All driven by 24 Claude Code skills, all centered on one wiki as the single source of truth.
Drop your .tex / .pdf files in a folder. Run one command. Get a fully cross-referenced knowledge base — and then use it to generate novel research ideas, design experiments, write papers, and respond to reviewers.
meg_brain_decoding_neuralset_cnn_Marktechpost
Beyond RAG: Why Knowledge Engineering Becomes the Real Moat in the Agent Era
And three core operations:
-
Ingest: parse one new source, summarize, cross-link, update many pages.
-
Query: answer from wiki pages (not just raw chunks), file high-value answers back as pages.
-
Lint: periodically detect contradictions, stale claims, orphan pages, missing links.
Second Brain is an attempt to make a digital brain that approximates the real thing. It's part knowledge engine, part personal operator, and part programmable automation layer.
It continuously indexes your files, remembers durable context, searches the web when local knowledge is not enough, and runs tools and shell commands. It lives in your terminal and Telegram, so your assistant is available everywhere.
Instead of being "just a chatbot," it turns your machine into a system that can observe, search, reason, and act. Point it at your world, give it tools, and it becomes a private AI layer for research, reminders, recurring work, and everyday operations.