Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
notes/
197 changes: 197 additions & 0 deletions Design/local markdown storage.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
# Local Markdown Storage Architecture

### 1. Problem Statement

Most note-taking apps store user data in cloud services or proprietary formats, which reduces user control and offline reliability.

This project needs a simple, local-first storage system for markdown notes.

The system should work without internet, respect user privacy, and scale to thousands of files without slowing down. The goal is to make local files the single source of truth.

### 2. Goals

Offline-first experience :
NO Internet,
NO Cloud Serives,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Typo: "Serives" → "Services".

-    NO Cloud Serives,
+    NO Cloud Services,
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
NO Cloud Serives,
NO Cloud Services,
🤖 Prompt for AI Agents
In `@Design/local` markdown storage.md at line 15, The phrase "NO Cloud Serives"
contains a typo; update the text to read "NO Cloud Services" (replace the string
"NO Cloud Serives" with "NO Cloud Services") so the word "Services" is spelled
correctly.

NO Server

User fully owns their data (plain markdown files)

Scales to large note collections (thousands of notes)

Easy to inspect, backup, and move files

### 3. Non-Goals

Cloud sync
Real-time collaboration
AI processing (handled in separate layers later)

### 4. Constraints

Desktop app built with Electron

Must work consistently on Windows, macOS, and Linux

Privacy-first: no hidden data collection

File system is the source of truth

### 5. High-Level Architecture

The app treats a Editor folder as a “notes workspace.”
Each note is stored as a markdown(.md) file on disk.

The app maintains:

"in-memory index " and "local db(SQLite)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Mismatched quotation marks.

The line has an opening " before local db(SQLite) but no closing ", and there's a stray space inside the first quoted phrase.

-  "in-memory index " and "local db(SQLite) 
+  "in-memory index" and "local db (SQLite)"
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"in-memory index " and "local db(SQLite)
"in-memory index" and "local db (SQLite)"
🤖 Prompt for AI Agents
In `@Design/local` markdown storage.md at line 47, Fix the mismatched and
inconsistent quotation marks in the phrase containing "in-memory index " and
"local db(SQLite)"; remove the stray trailing space inside the first quoted
phrase so it reads "in-memory index", add the missing closing quote after local
db(SQLite) so it reads "local db(SQLite)", and ensure both quoted phrases use
the same quote style and spacing.


A file watcher to detect changes ( like : File added , file changed, file deleted , file renamed )

A metadata layer derived from frontmatter

All edits made inside the app directly modify markdown files, and external file changes are reflected back in the UI.

### 6. Storage Model

Notes are stored as (.md) files

Folder-based organization (nested folders = categories)

File system is the single source of truth

No proprietary database format for note content

* STRUCTURE :

Notes/
|--->Note1.md
|--->Note2.md

### 7. Folder Watching Strategy

WHY needed?
User may write notes outside the app like (vs code, notepad) then app automatically detect changes and update by using FOLDER WATCHER

*Events handled:

File created → add note to index

File updated → refresh note content

File deleted → remove from index

Folder renamed → update paths

Bulk changes → trigger index rebuild

*Tools:
chokidar (Node.js file watcher)

### 8. Metadata Handling (Simple)

Metadata means extra info about a note like:
title
tags
created date
updated date

We store metadata inside the markdown file using frontmatter, so the file is still readable and portable.

* Example :
---
title: My First Note
tags: [gsoc, ideas]
created: 2026-02-12
---
This is my note content.

### 9. Performance Considerations (Simple)

To keep the app fast even with many notes:

-Only load a note when the user opens it (lazy loading)
-Don’t read all file contents at startup
-Build a small index for quick search
-Update index only for changed files
-Run heavy work in background
-Cache recently opened notes
-Avoid reloading everything on every file change
---
Why this helps:

Faster app startup
Smooth UI
Works well even with thousands of notes
---

### 10. Trade-offs & Risks (IMPORTANT PART)

* TRADE-OFFS

Using local .md files as the primary storage keeps data transparent and portable, but it means we rely on the operating system’s file system guarantees instead of having full transactional safety like a database.

The file watcher is lightweight and real-time, but behavior can vary slightly across Windows, macOS, and Linux due to OS-level differences in file system events.

Storing metadata in frontmatter keeps notes portable, but requires parsing files on change, which adds a small performance cost during large batch updates.

Initial indexing can be slower on first launch for users with very large existing note folders, since the app builds its index from disk.



* RISKS

If the same note is edited simultaneously in the app and an external editor, content conflicts may occur.

Some file system features (like symlinks or restricted permissions) can cause watcher errors or skipped files.

Users can accidentally move or delete folders outside the app, causing missing notes until the index is rebuilt.

Malformed frontmatter or corrupted markdown files can break metadata parsing.

Performance may degrade for extremely large workspaces (10k+ files) on low-end machines.



* MITIGATIONS

Detect and surface conflicts instead of silently overwriting changes.

Gracefully handle watcher and file permission errors with user-friendly warnings.

Automatically trigger a safe background re-index when unexpected file changes are detected.

Validate frontmatter before parsing and skip broken files without crashing.

Allow users to exclude heavy folders from watching and indexing.



### 12. Summary
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Section numbering skips from 10 to 12.

Section 11 is missing — ### 12. Summary should likely be ### 11. Summary, and subsequent sections renumbered accordingly.

🤖 Prompt for AI Agents
In `@Design/local` markdown storage.md at line 170, The section numbering skips
from 10 to 12: rename the header "### 12. Summary" to "### 11. Summary" and
adjust any following numbered section headers so numbering is consecutive (e.g.,
decrement subsequent headings by one); search for the literal header "### 12.
Summary" and update it and any later section numbers to maintain proper
sequence.

This architecture treats the local file system as the source of truth for markdown notes.
It ensures privacy, offline support, and long-term user ownership.
The design is simple, transparent, and extensible, making it suitable for building advanced features like search and AI on top without compromising user control.


### 13. RESPONSIBILITY

1️⃣ Where are notes stored?
Notes are stored as plain .md files on the user’s local file system.

2️⃣ How are they organized?
Notes are organized using folders and subfolders, just like normal files on a computer.

3️⃣ How does the app read them?
The app reads markdown files from disk and loads content only when a note is opened.

4️⃣ How does it detect changes?
A file watcher listens for create/update/delete events and updates the app in real time.

5️⃣ How does it scale?
It uses lazy loading, incremental indexing, and background scans to handle large collections.

6️⃣ How can AI sit on top later?
AI can build embeddings and search indexes on top of the local markdown files without owning the data.



Loading