feat(memory): add markdown tiered memory with examples

[vearne]

Introduce MarkdownTieredMemory for daily short-term markdown files and LLM/noop-promoted long-term summary facts, plus LLMFactCompressor adapter and markdown_memory demos.

Summary by CodeRabbit

Release Notes

• New Features
  • Added a tiered memory storage system that organizes messages across short-term (daily) and long-term (summarized) storage
  • Added automatic promotion and compression of aged messages to conserve storage space
  • Added optional AI-powered message summarization for intelligent long-term memory management
  • Added message organization through marking and selective filtering capabilities
  • Added configurable retention policies and automatic archival thresholds
  • Added usage examples demonstrating the new memory system

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: f3d06e78-2ad0-47cb-857d-2ca7ea2928da

📥 Commits

Reviewing files that changed from the base of the PR and between 12ab40a and 5ce9f02.

⛔ Files ignored due to path filters (1)

• go.sum is excluded by !**/*.sum

📒 Files selected for processing (10)

• examples/markdown_memory/main.go
• examples/markdown_memory_llm/main.go
• go.mod
• pkg/memory/llm_fact_compressor.go
• pkg/memory/llm_fact_compressor_test.go
• pkg/memory/markdown_codec.go
• pkg/memory/markdown_meta.go
• pkg/memory/markdown_tiered.go
• pkg/memory/markdown_tiered_test.go
• pkg/memory/noop_compressor.go

---

📝 Walkthrough

Walkthrough

This PR introduces a filesystem-backed tiered memory store for persistent conversation storage. Messages are organized into short-term (daily) and long-term (compressed summary) markdown files, with automatic promotion and compaction controlled by configurable age and size thresholds. The implementation includes markdown serialization with YAML metadata, two compression strategies (no-op and LLM-based fact extraction), and complete CRUD/retrieval APIs with mark-based filtering.

Changes

Markdown Tiered Memory Storage

Layer / File(s)	Summary
Serialization Foundation: Markdown Codec & Metadata Persistence pkg/memory/markdown_codec.go, pkg/memory/markdown_meta.go, pkg/memory/markdown_tiered_test.go (round-trip)	Encodes messages to markdown sections with YAML frontmatter and JSON bodies; decodes sections back into structured messages. A JSON metadata index (meta.json) tracks message location (tier, file, section offset) and compaction statistics by message ID.
Compression Strategies: Noop & LLM Compressors go.mod, pkg/memory/noop_compressor.go, pkg/memory/llm_fact_compressor.go, pkg/memory/llm_fact_compressor_test.go, pkg/memory/markdown_tiered_test.go (compressor tests)	NoopCompressor truncates and reformats selected trailing messages as summaries. LLMFactCompressor calls a chat model to extract independent facts from messages, parses dash-prefixed lines, and falls back to the noop strategy when the model returns no facts. Both emit summary messages with tier/kind metadata. YAML dependency added for metadata serialization.
Tiered Memory Store: Core Implementation & APIs pkg/memory/markdown_tiered.go, pkg/memory/markdown_tiered_test.go (comprehensive tests)	MarkdownTieredMemory manages short-term files (one per day) and long-term summary files. Add appends to today's file; promotion scans for expired days, compresses their messages into long-term summaries via a configured compressor, and removes the expired short-term file. Long-term compaction runs when entry count or byte total exceeds configured thresholds, merges older entries into new chunk files, and rebuilds the index. Retrieval APIs (GetMessages, GetMemory) apply mark filtering across both tiers. Mutation APIs support delete by ID/mark, clear, and update marks, with index/statistics rebuilt after modifications.
Example Usage: Basic & LLM-Integrated Demos examples/markdown_memory/main.go, examples/markdown_memory_llm/main.go	Basic example initializes a temp store, seeds an old short-term file, configures tiering with sync compaction, adds today's messages with marks, prints the file tree, and inspects metadata. LLM example wires an OpenAI chat model to LLMFactCompressor, triggers promotion by adding a message, retrieves long-term summaries, and optionally demonstrates ReAct agent integration.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

🐰 A fluffy file tree grows in markdown dreams,
Old messages fade to summary streams,
LLM facts dance with noop compress delight,
Tiered tiers tier, promotions ignite!
Meta-json indexes all the stored gold,
Compact and clean, the memory unfolds. 🌿

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch cursor/markdown-tiered-memory

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}