Skip to content

Commit 30a1e01

Browse files
authored
feat(export/import): OKF (Open Knowledge Format) bundle interop (#103)
Add lossless OKF v0.1 round-trip to wiki-export and wiki-import. OKF (Google's knowledge-catalog) is a vendor-neutral spec for knowledge as markdown + YAML frontmatter — structurally near-identical to this vault. - wiki-export: new "OKF bundle" output mode (wiki-export/okf/), the canonical frontmatter mapping (category/summary/sources/updated ⇄ type/description/resource/timestamp), [[wikilink]]→file-relative markdown link transform, per-folder index.md generation, okf_version. - wiki-import: source-type detection (graph.json vs OKF dir) + Step 4-OKF that reconstructs full pages (not stubs) via reverse mapping. - AGENTS.md: routing triggers + OKF-conformance architecture note. Native frontmatter keys ride along as OKF extension frontmatter, making the round-trip lossless — unlike the stub-only graph.json round-trip. Forward-references (wikilinks to not-yet-written pages) are preserved as links per OKF §5.3 rather than dropped to plain text. Verified on st3ve (46-page vault): 46/46 exported, 0 OKF §9 conformance issues, 25/25 frontmatter lossless, 46/46 body/link-graph lossless.
1 parent 9e85b62 commit 30a1e01

3 files changed

Lines changed: 141 additions & 22 deletions

File tree

.skills/wiki-export/SKILL.md

Lines changed: 61 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,11 @@ name: wiki-export
33
description: >
44
Export the Obsidian wiki's knowledge graph to structured formats for use in external tools.
55
Use this skill when the user says "export wiki", "export graph", "export to JSON", "export to Gephi",
6-
"export to Neo4j", "graphml", "visualize wiki", "knowledge graph export", or wants to use their
6+
"export to Neo4j", "graphml", "visualize wiki", "knowledge graph export", "export to OKF",
7+
"OKF bundle", "open knowledge format", "export as markdown bundle", or wants to use their
78
wiki data in another tool. Outputs graph.json, graph.graphml, cypher.txt (Neo4j), and graph.html
8-
(interactive browser visualization) into a wiki-export/ directory at the vault root.
9+
(interactive browser visualization) into a wiki-export/ directory at the vault root, plus an
10+
optional OKF (Open Knowledge Format) markdown bundle under wiki-export/okf/.
911
---
1012

1113
# Wiki Export — Knowledge Graph Export
@@ -305,6 +307,55 @@ Replace `/* NODES_JSON */` and `/* EDGES_JSON */` with the actual JSON arrays yo
305307

306308
---
307309

310+
## Step 3.5: OKF Bundle Export (optional)
311+
312+
Run this step **only** when the user asks for OKF / a markdown bundle (phrases like "export to OKF", "OKF bundle", "open knowledge format", "export as markdown bundle"). It is additive — the four graph files above are always produced; this writes an extra full-fidelity markdown bundle.
313+
314+
The four graph files are a *lossy* projection (graph skeleton only). An **OKF bundle is the actual page bodies**, so an export→`wiki-import` round-trip through OKF preserves full content, and the bundle drops straight into MkDocs, Notion, Hugo, GitHub's renderer, or any [Open Knowledge Format](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) consumer.
315+
316+
### Canonical frontmatter mapping (obsidian-wiki ⇄ OKF)
317+
318+
This table is the single source of truth for the mapping; `wiki-import` references it for the reverse direction.
319+
320+
| OKF key | ← export from | → import to | Notes |
321+
|---------------|-------------------------|--------------------------|-------|
322+
| `type` (required) | `category`, title-cased | `category` (lower-cased) | `concepts``Concept`, `entities``Entity`, `skills``Skill`, `references``Reference`, `synthesis``Synthesis`, `projects``Project`, `journal``Journal`. OKF requires `type`; consumers tolerate any string. |
323+
| `title` | `title` | `title` | Verbatim. |
324+
| `description` | `summary` | `summary` | Our one-line `summary:` is exactly OKF's `description` (used in `index.md` entries). |
325+
| `tags` | `tags` | `tags` | Verbatim list. `visibility/*` system tags pass through unchanged. |
326+
| `timestamp` | `updated` | `updated` | ISO 8601 both sides. |
327+
| `resource` | first `sources:` entry **iff** it is an `http(s)://` URL || Optional; omit when no source URL. Most pages describe abstract knowledge and have none. |
328+
| *(extensions)* | `category`, `sources`, `created`, `relationships`, `lifecycle`, `tier`, `base_confidence`, … | preserved verbatim | OKF §4.1 permits arbitrary keys and requires consumers to preserve them. **Writing our native keys as OKF extension frontmatter is what makes the round-trip lossless** — on import, preserved `category`/`created`/`sources` are preferred over re-deriving from `type`. |
329+
330+
### Steps
331+
332+
Reuse the node list from Step 1 (with any active project/visibility filters already applied). Write a directory tree under `wiki-export/okf/`:
333+
334+
1. **One file per in-scope page.** For each page, parse its frontmatter, apply the mapping table above to build the OKF frontmatter (required `type` first, then `title`, `description`, `tags`, `timestamp`, optional `resource`, then the preserved extension keys), transform the body links (below), and write to `wiki-export/okf/<category>/<slug>.md` — same relative path the page has in the vault.
335+
336+
2. **Body link transform** (`[[wikilinks]]` → standard markdown links):
337+
- `[[concepts/transformers]]``[<target title>](<file-relative path>.md)`, e.g. from `entities/foo.md` a link to `concepts/transformers` becomes `[Transformer Architecture](../concepts/transformers.md)`. Link text = the target page's `title` (fall back to the target id if unknown).
338+
- `[[target|display]]``[display](<rel path>.md)`.
339+
- Use **file-relative** paths (`../concepts/x.md`), **never** `/`-absolute — `/`-rooted links break GitHub rendering. (This matches knowledge-catalog's own production agent.)
340+
- Resolve link targets with the same normalization used in Step 1 (lowercase, spaces→hyphens, strip `.md`). Handle unresolved targets by form, so forward-references survive the round-trip:
341+
- **Resolves to an in-scope page** → relative markdown link to it.
342+
- **Path-form target** (contains a `/`, e.g. `[[concepts/attention-mechanism]]`) with **no page yet**, and not excluded by an active filter → still emit the relative markdown link. OKF §5.3 treats a missing target as not-yet-written knowledge, and keeping the link makes the user's forward-references lossless on re-import. (Verified on st3ve: dropping these silently deleted real `[[wikilinks]]`.)
343+
- **Excluded by an active project/visibility filter** → plain text. Do not emit a path pointing into filtered-out content.
344+
- **Bare-title target** with no match (e.g. `[[tractorex]]` when no such page exists in scope) → plain text; there is no reliable path to write.
345+
- Leave existing external `http(s)://` links and `# Citations` sections untouched.
346+
347+
3. **Generate `index.md` files** (OKF §6 progressive disclosure; these contain no per-entry frontmatter):
348+
- Bundle root `wiki-export/okf/index.md` — a `# Subdirectories` section listing each category folder: `* [<category>](<category>/index.md) - <one-line description of the category>`. This is the **only** index permitted frontmatter: add a single key `okf_version: "0.1"` (OKF §11).
349+
- One `index.md` per category folder listing its pages: `* [<title>](<slug>.md) - <description from the page's summary>`.
350+
351+
4. **Copy `log.md`** from the vault root to `wiki-export/okf/log.md` as-is (OKF §7 treats the leading bold action word as convention, so the existing line-based log is conformant).
352+
353+
5. **Filters.** Honor the same project/visibility filters as the graph export — filtered pages are omitted from the bundle and their inbound links degrade to plain text per step 2.
354+
355+
Excluded from the bundle (same as the graph export): `_archives/`, `_raw/`, `.obsidian/`, `_insights.md`, `_meta/*.base`, and the vault root `index.md` (regenerated above).
356+
357+
---
358+
308359
## Step 4: Print Summary
309360

310361
```
@@ -315,6 +366,11 @@ Wiki export complete → wiki-export/
315366
graph.html — interactive browser visualization (open in any browser)
316367
```
317368

369+
Append this line only when the OKF bundle was produced (Step 3.5):
370+
```
371+
okf/ — OKF v0.1 markdown bundle (N pages, lossless; import via wiki-import)
372+
```
373+
318374
Append filter notes when active:
319375
```
320376
(filtered: project:prismor — 19 of 67 pages)
@@ -324,7 +380,8 @@ Only include lines for filters that were actually applied.
324380

325381
## Notes
326382

327-
- **Re-running is safe** — all output files are overwritten on each run
328-
- **Broken wikilinks are skipped** — only edges to pages that exist in the vault are exported
383+
- **Re-running is safe** — all output files (and the `okf/` bundle) are overwritten on each run
384+
- **Broken wikilinks are skipped** — only edges to pages that exist in the vault are exported; in the OKF bundle, a wikilink to a missing/filtered page degrades to plain text
385+
- **OKF is the lossless format**`graph.json` reconstructs only stubs on import, while the `okf/` bundle preserves full page bodies. Use OKF for vault-to-vault transfer and external markdown tools (MkDocs, Notion, GitHub); use the graph files for analysis tools (Gephi, Neo4j)
329386
- **The `wiki-export/` directory should be gitignored** if the vault is version-controlled — these are derived artifacts
330387
- **`graph.json` is the primary format** — the others are derived from it. If a future tool supports graph queries natively, point it at `graph.json`

.skills/wiki-import/SKILL.md

Lines changed: 76 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,69 @@
11
---
22
name: wiki-import
33
description: >
4-
Import a wiki knowledge graph from a graph.json export file into the current vault.
4+
Import a wiki knowledge graph into the current vault — either from a graph.json export
5+
file (stubs) or from an OKF (Open Knowledge Format) markdown bundle (full page bodies).
56
Use this skill when the user says "import wiki", "import from export", "load graph.json",
6-
"import vault", "/wiki-import", or wants to transfer pages from one vault to another
7-
using the output of wiki-export.
7+
"import vault", "import OKF bundle", "import OKF", "load OKF", "import markdown bundle",
8+
"/wiki-import", or wants to transfer pages from one vault to another using the output of wiki-export.
89
---
910

10-
# Wiki Import — Reconstruct Pages from graph.json
11+
# Wiki Import — Reconstruct Pages from an Export
1112

12-
You are importing a vault's knowledge graph from a `graph.json` export (produced by `wiki-export`) into the current vault. This reconstructs page stubs with correct frontmatter, wikilinks, and typed relationships, then updates all vault metadata.
13+
You are importing a vault's knowledge into the current vault from one of two sources produced by `wiki-export`:
14+
15+
- **`graph.json`** — the graph skeleton. Reconstructs page **stubs** (frontmatter, typed relationships, a `## Related` link list — no body). Lossy.
16+
- **OKF bundle** (a `wiki-export/okf/` directory) — the actual markdown files. Reconstructs **full pages** with their real bodies. Lossless. Use this for true vault-to-vault transfer.
17+
18+
Either way, the import writes pages with correct frontmatter and wikilinks, then updates all vault metadata. **Step 2, Step 3 (graph only), and Step 5 are shared; Step 4 forks by source type.**
1319

1420
## Before You Start
1521

1622
1. **Resolve config** — follow the Config Resolution Protocol in `llm-wiki/SKILL.md` (walk up CWD for `.env``~/.obsidian-wiki/config` → prompt setup). This gives `OBSIDIAN_VAULT_PATH`.
1723
2. Read `$OBSIDIAN_VAULT_PATH/AGENTS.md` if it exists — apply any owner-specific conventions.
1824

19-
## Step 1: Locate and Validate Source
25+
## Step 1: Locate and Detect Source Type
2026

2127
**Find the import source:**
2228
- If the user provided a path argument, use it directly.
23-
- Otherwise auto-detect `./wiki-export/graph.json` in the current directory.
29+
- Otherwise auto-detect, in order: `./wiki-export/okf/` (a directory) → `./wiki-export/graph.json` (a file).
2430
- If neither exists, ask the user for the path.
2531

26-
**Validate the file:**
32+
**Detect the source type:**
33+
- The path is a **file ending in `.json`****graph.json import** (validate below, then Step 3 + Step 4-Graph).
34+
- The path is a **directory** containing `.md` files with OKF frontmatter (a `type:` key), and/or a root `index.md` with `okf_version`**OKF bundle import** (skip Step 3; go to Step 4-OKF).
35+
- Anything else → report what's wrong and stop.
36+
37+
**Validate a graph.json source:**
2738
- Must be valid JSON
2839
- Must have top-level keys: `nodes` (array), `links` (array), `graph` (object)
2940
- Must have at least 1 node
3041

3142
If validation fails, report what's wrong and stop.
3243

44+
**Validate an OKF bundle source:**
45+
- Must contain at least 1 non-reserved `.md` file (i.e. not `index.md`/`log.md`) with parseable YAML frontmatter containing a non-empty `type`.
46+
- A `.md` with no frontmatter or no `type` is skipped (with a count), not fatal — OKF consumers are permissive (OKF §9).
47+
3348
**Show a preview before importing:**
49+
50+
graph.json:
3451
```
35-
Import preview
52+
Import preview (graph.json — stubs)
3653
Source: <path> (exported at <graph.exported_at>)
3754
Nodes: N total (concepts: A, entities: B, skills: C, references: D, ...)
3855
Links: M edges (X typed, Y untyped)
3956
Target: $OBSIDIAN_VAULT_PATH
4057
```
4158

59+
OKF bundle:
60+
```
61+
Import preview (OKF bundle — full pages)
62+
Source: <dir> (okf_version <ver if present>)
63+
Pages: N total (concepts: A, entities: B, skills: C, references: D, ...)
64+
Target: $OBSIDIAN_VAULT_PATH
65+
```
66+
4267
## Step 2: Determine Conflict Resolution Mode
4368

4469
Read the user's phrasing to determine mode. Default is `merge`.
@@ -49,7 +74,7 @@ Read the user's phrasing to determine mode. Default is `merge`.
4974
| `skip` | "skip existing", "don't overwrite", "only new pages" | Leave existing pages completely untouched; only create pages that don't exist yet. |
5075
| `overwrite` | "overwrite", "replace existing", "force import" | Replace all matched pages with freshly reconstructed stubs regardless of existing content. |
5176

52-
## Step 3: Build Internal Maps
77+
## Step 3: Build Internal Maps *(graph.json only — skip for OKF bundles)*
5378

5479
Before writing anything, build two maps from the `links` array:
5580

@@ -66,7 +91,7 @@ typed_edges["concepts/transformers"] = [
6691
]
6792
```
6893

69-
## Step 4: Reconstruct Pages
94+
## Step 4-Graph: Reconstruct Pages from graph.json *(graph.json source only)*
7095

7196
Record counts: `created = 0`, `skipped = 0`, `merged = 0`.
7297

@@ -136,21 +161,56 @@ If `adjacency[node.id]` is empty, omit the `## Related` section entirely.
136161
6. **Body**: scan for a `## Related` section. If it exists, append any missing wikilinks from `adjacency[node.id]` that aren't already linked anywhere in the body. If no `## Related` section exists, append one with the missing links.
137162
7. Leave the rest of the body untouched.
138163

164+
## Step 4-OKF: Reconstruct Pages from an OKF bundle *(OKF source only)*
165+
166+
Record counts: `created = 0`, `skipped = 0`, `merged = 0`, `unparseable = 0`.
167+
168+
Walk the bundle directory tree. For each `.md` file that is **not** a reserved file (`index.md`, `log.md`):
169+
170+
1. Parse the YAML frontmatter. If it has no frontmatter or no non-empty `type`, increment `unparseable` and skip the file.
171+
2. Compute the **concept id** = the file's path relative to the bundle root, with `.md` stripped (e.g. `concepts/transformers.md``concepts/transformers`). The target page is `$VAULT/<concept-id>.md`.
172+
3. **Reverse-map frontmatter** (the inverse of the canonical mapping table in `wiki-export` Step 3.5):
173+
- `title``title`.
174+
- `category` ← the preserved `category` extension key if present; **else** lower-case the directory prefix of the concept id (`concepts/…``concepts`); **else** derive from `type` (`Concept``concepts`, `Entity``entities`, `Skill``skills`, `Reference``references`, `Synthesis``synthesis`, `Project``projects`, `Journal``journal`).
175+
- `tags``tags`.
176+
- `summary``description`.
177+
- `updated``timestamp` (or now if absent).
178+
- `created` ← the preserved `created` extension key if present, else now.
179+
- `sources` ← the preserved `sources` extension key if present; else `["imported from OKF bundle <bundle path>"]`. If a `resource` URL is present and not already in `sources`, add it.
180+
- Carry through any other preserved extension keys verbatim (`relationships`, `lifecycle`, `tier`, `base_confidence`, …). These make the round-trip lossless.
181+
4. **Reverse-transform body links** — markdown links that point at `.md` paths become wikilinks (this restores both real cross-links and forward-references the exporter preserved per `wiki-export` Step 3.5):
182+
- `[text](../concepts/transformers.md)` or `[text](/concepts/transformers.md)` → resolve the path (relative to this file's dir, or bundle-root for `/`-absolute) to a concept id → `[[concepts/transformers]]`, or `[[concepts/transformers|text]]` when `text` differs from the target's title. The target's title comes from the bundle page when it exists; otherwise compare against the last path segment.
183+
- This applies **even when the target page is not in the bundle** — a path-form link to a not-yet-written page round-trips back to a dangling `[[wikilink]]` (Obsidian supports these; OKF §5.3 expects them). Do not leave it as a markdown link.
184+
- When `OBSIDIAN_LINK_FORMAT=markdown` is set in config, **keep** markdown links (just rewrite the path to be vault-relative); do not convert to wikilinks.
185+
- Leave external `http(s)://` links and `# Citations` sections untouched.
186+
5. **Write the page** using the conflict mode from Step 2:
187+
- **merge + exists** → reverse-map frontmatter and merge it into the existing page (union `tags`; fill `summary`/`sources` only if missing; union `relationships`; refresh `updated`). For the body, OKF carries a real body: replace the existing body with the bundle body **only if the existing page is a stub** (body is just a heading + `## Related`); otherwise keep the existing body and append any bundle `# Citations` / new `##` sections not already present. Increment `merged`.
188+
- **merge + doesn't exist** → write the full page (frontmatter + full bundle body). Increment `created`.
189+
- **skip + exists** → increment `skipped`, continue.
190+
- **skip + doesn't exist** → write the full page. Increment `created`.
191+
- **overwrite + exists** → write the full page, replacing the existing one. Increment `merged` (label as `Replaced` in the summary).
192+
- **overwrite + doesn't exist** → write the full page. Increment `created`.
193+
6. Ensure the parent directory (`$VAULT/concepts/`, etc.) exists before writing.
194+
195+
Unlike the graph.json path, **do not** generate a `## Related` stub section — the bundle body already contains the real cross-links.
196+
139197
## Step 5: Update Vault Metadata
140198

141199
### `.manifest.json`
142200

143201
Add a new entry keyed by the canonical path of the graph.json file:
144202

145203
```json
146-
"<absolute path to graph.json>": {
204+
"<absolute path to source>": {
147205
"ingested_at": "<ISO timestamp>",
148206
"source_type": "wiki-export",
149207
"pages_created": ["list/of/created/pages.md"],
150208
"pages_updated": ["list/of/merged/pages.md"]
151209
}
152210
```
153211

212+
Set `source_type` to `"wiki-export"` for a graph.json import or `"okf-bundle"` for an OKF bundle import. Key the entry by the absolute path of the source (the `graph.json` file or the bundle directory).
213+
154214
Also increment:
155215
- `stats.total_sources_ingested` by 1
156216
- `stats.total_pages` by the count of pages actually created (not skipped/merged)
@@ -194,18 +254,18 @@ Update the `updated:` frontmatter timestamp. Leave other hot.md sections (Active
194254

195255
```
196256
Wiki import complete → $OBSIDIAN_VAULT_PATH
197-
Source: <graph.json path>
198-
exported at <graph.exported_at>, <N> nodes, <M> links
257+
Source: <source path> (<graph.json | OKF bundle>)
258+
<graph: exported at <graph.exported_at>, N nodes, M links | okf: N pages, okf_version <ver>>
199259
Created: <X> pages
200260
Merged: <Z> pages (existing pages updated)
201261
Skipped: <Y> pages (only when --skip mode was used)
202262
```
203263

204-
Only show the `Skipped` line if `skip` mode was explicitly requested. If `overwrite` mode was used, label merged pages as `Replaced` instead.
264+
Only show the `Skipped` line if `skip` mode was explicitly requested. If `overwrite` mode was used, label merged pages as `Replaced` instead. For an OKF import, also report `Unparseable: <U> files (no frontmatter/type, skipped)` when `U > 0`.
205265

206266
## Notes
207267

208-
- **Stub quality**: Imported pages are stubs — they have structure and wikilinks but no full body content. They're starting points for future ingestion, not finished pages.
268+
- **Stub vs full pages**: A **graph.json** import produces stubs — structure and wikilinks but no body, a starting point for future ingestion. An **OKF bundle** import produces full pages with real bodies — it is the lossless, content-bearing path and the right choice for vault-to-vault transfer.
209269
- **Re-running is safe**: The default `merge` mode is idempotent — re-running on an unchanged export will update timestamps but won't destroy content. Use `overwrite` only when you want to fully reset pages to stubs.
210270
- **Directory creation**: Always create missing category directories before writing pages.
211271
- **Broken wikilinks**: Since pages are being created together from the same export, most links will resolve. Any node referenced in `links` but absent from `nodes` (broken in the original export) will still appear as a wikilink — it just won't have a corresponding page file, which is valid.

0 commit comments

Comments
 (0)