Skip to content

Commit 3324c34

Browse files
ThaddeusJiangThaddeusJiang
committed
docs: add English usage and contribution guides
1 parent 4d62e88 commit 3324c34

2 files changed

Lines changed: 278 additions & 0 deletions

File tree

‎AGENTS.md‎

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# AGENTS.md
2+
3+
This file is the development and contribution contract for AI coding agents working in this repository.
4+
5+
## Project purpose
6+
7+
`tiddlywiki-nmem-importer` imports tiddlers from one Node.js TiddlyWiki into Nowledge Mem. It boots the real Wiki with the npm `tiddlywiki` package, renders WikiText to HTML, converts that HTML to GitHub Flavored Markdown, and upserts each result as a Memory through the installed `nmem` CLI.
8+
9+
`README.md` is the user-facing source of truth for installation and usage. Keep it synchronized whenever CLI behavior changes.
10+
11+
## Non-negotiable behavior
12+
13+
- The current working directory is the only Wiki input.
14+
- The command must be run from a directory containing a readable `tiddlywiki.info`.
15+
- Do not restore `--wiki`, hardcoded Wiki names, parent-directory scans, or sibling-directory scans.
16+
- Default execution is a dry-run. Only `--apply` may write to Nowledge Mem.
17+
- Never modify, move, or delete source tiddlers.
18+
- Use the installed current `nmem`; never hardcode a required nmem version.
19+
- Require the `nmem` CLI and service to report matching versions before writing.
20+
- Reject remote Nowledge Mem services by default; require `--allow-remote` for an explicit override.
21+
- Preserve deterministic Memory IDs based on source Wiki name and tiddler title so reruns remain idempotent.
22+
- Preserve the default sensitive-title filter unless a deliberate behavior change includes tests and README updates.
23+
24+
## Language and runtime
25+
26+
- TypeScript everywhere.
27+
- Do not add `.js` or `.mjs` source files.
28+
- Run TypeScript directly with [Nub](https://github.com/nubjs/nub); do not use Bun.
29+
- Keep `strict` TypeScript enabled.
30+
- Use ESM imports with explicit `.ts` extensions, matching the existing codebase.
31+
- Prefer small functions and declarative data transformations over unnecessary classes.
32+
- Do not add a dependency when the Node.js standard library or an existing dependency is sufficient.
33+
34+
## Repository map
35+
36+
- `src/cli.ts`: CLI orchestration, current-directory validation, reporting, previews, and concurrent imports.
37+
- `src/core.ts`: tiddler classification, metadata, stable IDs, HTML-to-Markdown conversion, and media warnings.
38+
- `src/tiddlywiki-worker.ts`: boots TiddlyWiki and sends records over IPC.
39+
- `src/tiddlywiki.ts`: owns worker lifecycle, IPC validation, and diagnostics.
40+
- `src/nmem.ts`: validates the active nmem service and invokes `nmem memories add`.
41+
- `src/options.ts`: parses supported command-line options.
42+
- `test/`: Node test runner coverage and a minimal TiddlyWiki fixture.
43+
- `reports/`, `previews/`: generated output; both are ignored and must not be committed.
44+
45+
## Architecture constraints
46+
47+
Keep TiddlyWiki execution in the child worker. TiddlyWiki boot diagnostics must not be mixed with structured tiddler records; records travel through IPC and stderr is collected separately.
48+
49+
Conversion behavior is type-dependent:
50+
51+
- `text/vnd.tiddlywiki` and the empty/default type are rendered by TiddlyWiki before Turndown conversion.
52+
- `text/markdown` and `text/plain` use their source text directly.
53+
- Unsupported binary types, system tiddlers, drafts, empty tiddlers, and sensitive-title tiddlers are classified and reported instead of imported.
54+
55+
Memory content includes TiddlyWiki source metadata in front matter. Keep source Wiki, title, tags, created time, and modified time available unless a documented migration replaces them.
56+
57+
The nmem process must receive Memory content through stdin. Do not place note content in command-line arguments or logs.
58+
59+
## Setup
60+
61+
```bash
62+
mise install
63+
npm ci
64+
```
65+
66+
The repository pins its Node, Nub, npm package, and TypeScript toolchain through `mise.toml`, `package.json`, and `package-lock.json`. The nmem version is intentionally not pinned in source code.
67+
68+
## Required validation
69+
70+
Run both commands after every source or test change:
71+
72+
```bash
73+
mise exec -- nub run typecheck
74+
mise exec -- nub run test
75+
```
76+
77+
For CLI behavior involving Wiki discovery or rendering, also run a dry-run from `test/fixtures/wiki` or another disposable Wiki root. Never use `--apply` in tests or validation unless the user explicitly authorizes writes to Nowledge Mem.
78+
79+
Tests must cover behavior, not implementation details. Add or update tests when changing:
80+
81+
- CLI options and defaults
82+
- tiddler classification
83+
- WikiText/HTML/Markdown conversion
84+
- deterministic IDs or metadata
85+
- nmem compatibility and command arguments
86+
- worker IPC and multiline content
87+
88+
## Contribution workflow
89+
90+
1. Read `README.md`, this file, and the source files relevant to the requested behavior.
91+
2. Inspect the repository for all references before changing a public option, report field, ID algorithm, label, or conversion rule.
92+
3. Keep the patch scoped to the request and preserve unrelated user changes.
93+
4. Update tests and user documentation in the same change when behavior changes.
94+
5. Run typecheck and the full test suite.
95+
6. Review the final diff for generated files, note content, credentials, and accidental dependency changes.
96+
7. Do not commit, push, publish, or create a pull request unless the user explicitly asks.
97+
98+
## Documentation contract
99+
100+
Every project must contain both files:
101+
102+
- `README.md`: written for human users, centered on setup and practical usage.
103+
- `AGENTS.md`: written for AI coding agents, centered on architecture, development constraints, validation, and contribution.
104+
105+
When project behavior changes, update the relevant sections rather than allowing either document to become historical or aspirational.

‎README.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
# TiddlyWiki to Nowledge Mem Importer
2+
3+
Batch-convert tiddlers from a Node.js TiddlyWiki into Markdown and import them as Memories in [Nowledge Mem](https://mem.nowledge.co/).
4+
5+
The importer boots the real Wiki through the TiddlyWiki npm package. WikiText is rendered to HTML by TiddlyWiki and then converted to GitHub Flavored Markdown. The default mode is a dry-run: it does not modify the source Wiki or write to Nowledge Mem.
6+
7+
## Prerequisites
8+
9+
- A Node.js TiddlyWiki whose root directory contains `tiddlywiki.info`
10+
- [mise](https://mise.jdx.dev/)
11+
- The latest installed `nmem` CLI and a healthy Nowledge Mem service running the same version
12+
13+
Check Nowledge Mem before importing:
14+
15+
```bash
16+
nmem --version
17+
nmem status
18+
```
19+
20+
The importer only permits a local service (`localhost`, `127.0.0.1`, or `::1`) by default.
21+
22+
## Installation
23+
24+
```bash
25+
git clone https://github.com/ThaddeusJiang/tiddlywiki-nmem-importer.git
26+
cd tiddlywiki-nmem-importer
27+
mise install
28+
npm ci
29+
```
30+
31+
Keep the absolute path to the cloned repository. The examples below use `/path/to/tiddlywiki-nmem-importer`.
32+
33+
## Usage
34+
35+
### 1. Enter the TiddlyWiki root directory
36+
37+
Run the importer from the root of the target TiddlyWiki. The current working directory is the only data source; the importer does not scan parent directories, sibling directories, or a predefined Wiki list.
38+
39+
```bash
40+
cd /path/to/my-wiki
41+
test -r tiddlywiki.info
42+
```
43+
44+
### 2. Start with a dry-run
45+
46+
```bash
47+
MISE_CONFIG_FILE=/path/to/tiddlywiki-nmem-importer/mise.toml \
48+
mise exec -- nub \
49+
/path/to/tiddlywiki-nmem-importer/src/cli.ts
50+
```
51+
52+
A dry-run loads, filters, and converts tiddlers and writes a JSON report, but it never calls `nmem memories add`. By default, reports are written to the importer's `reports/` directory.
53+
54+
Limit the run to the first 20 importable tiddlers and write Markdown previews:
55+
56+
```bash
57+
MISE_CONFIG_FILE=/path/to/tiddlywiki-nmem-importer/mise.toml \
58+
mise exec -- nub \
59+
/path/to/tiddlywiki-nmem-importer/src/cli.ts \
60+
--limit 20 \
61+
--preview-dir /tmp/tiddlywiki-nmem-preview \
62+
--report /tmp/tiddlywiki-nmem-report.json
63+
```
64+
65+
Review `failed`, `warnings`, and `skipped` in the report before applying the import.
66+
67+
### 3. Import Memories
68+
69+
Add `--apply` after confirming the dry-run results:
70+
71+
```bash
72+
MISE_CONFIG_FILE=/path/to/tiddlywiki-nmem-importer/mise.toml \
73+
mise exec -- nub \
74+
/path/to/tiddlywiki-nmem-importer/src/cli.ts \
75+
--apply
76+
```
77+
78+
Choose a Nowledge Mem space and change the write concurrency when needed:
79+
80+
```bash
81+
MISE_CONFIG_FILE=/path/to/tiddlywiki-nmem-importer/mise.toml \
82+
mise exec -- nub \
83+
/path/to/tiddlywiki-nmem-importer/src/cli.ts \
84+
--apply \
85+
--space-id personal \
86+
--jobs 2
87+
```
88+
89+
Each Memory receives a stable ID derived from the Wiki directory name and tiddler title. Repeated runs upsert the same Memory instead of creating duplicate entries for the same title.
90+
91+
## Command-line options
92+
93+
| Option | Description |
94+
| --- | --- |
95+
| `--apply` | Write to Nowledge Mem; omit it for a dry-run |
96+
| `--limit <count>` | Process at most this many importable tiddlers |
97+
| `--jobs <count>` | Set concurrent writes; default: `4` |
98+
| `--space-id <id>` | Select a Nowledge Mem space; default: `default` |
99+
| `--include-sensitive` | Include tiddlers whose titles appear sensitive |
100+
| `--allow-remote` | Permit writes to a non-local Nowledge Mem service |
101+
| `--preview-dir <path>` | Write converted Markdown previews |
102+
| `--report <path>` | Choose the JSON report path |
103+
| `-h`, `--help` | Show command help |
104+
105+
The importer does not support `--wiki`. To import another Wiki, first change to that Wiki's root directory.
106+
107+
## Import rules
108+
109+
The importer skips these tiddlers by default:
110+
111+
- System tiddlers whose titles start with `$:/`
112+
- Drafts
113+
- Empty content
114+
- Unsupported binary types
115+
- Titles containing terms associated with tokens, API keys, secrets, passwords, or credentials
116+
117+
Sensitive-content detection checks titles only; it does not inspect tiddler bodies. Review the destination's access scope before using `--include-sensitive`.
118+
119+
Content conversion follows these rules:
120+
121+
- `text/vnd.tiddlywiki`: rendered by the active TiddlyWiki runtime and converted to GFM Markdown
122+
- `text/markdown` and `text/plain`: source text is used directly
123+
- Tags, creation time, modification time, source Wiki, and original title: stored in Markdown front matter
124+
- Base64-embedded images: replaced with an omission marker instead of storing the binary data
125+
- Local image references: preserved and reported as warnings
126+
127+
Imported Memories receive `tiddlywiki` and `tiddlywiki-<wiki-name>` labels and use `tiddlywiki` as their source.
128+
129+
## Safety
130+
131+
- Dry-run by default
132+
- Never modifies or deletes source tiddlers
133+
- Rejects remote Nowledge Mem services by default
134+
- Confirms that the `nmem` CLI and service versions match without pinning a specific version
135+
- Retries failed writes up to three times and records failures in the report
136+
- Skips titles that appear sensitive by default
137+
138+
## Troubleshooting
139+
140+
### The current directory is not a TiddlyWiki root
141+
142+
If the importer reports:
143+
144+
```text
145+
Current directory is not a TiddlyWiki root
146+
```
147+
148+
Change to the directory containing `tiddlywiki.info` and run the command again.
149+
150+
### The nmem CLI and service versions do not match
151+
152+
Update and restart Nowledge Mem so these commands report the same version:
153+
154+
```bash
155+
nmem --version
156+
nmem status --json
157+
```
158+
159+
### Preview conversion without importing
160+
161+
Omit `--apply` and use `--preview-dir` to write Markdown previews.
162+
163+
## Development
164+
165+
```bash
166+
cd /path/to/tiddlywiki-nmem-importer
167+
mise install
168+
npm ci
169+
mise exec -- nub run typecheck
170+
mise exec -- nub run test
171+
```
172+
173+
See [AGENTS.md](AGENTS.md) for development constraints and contribution guidance.

0 commit comments

Comments
 (0)