Spike: offline OneNote .one-file importer (PoC) - #10649
Conversation
Adds a from-scratch TypeScript parser for the OneNote desktop .one / .onetoc2 binary format (MS-ONESTORE revision store + MS-ONE object model), ported from the msiemens/onenote.rs reference. It decodes the file node lists, object-space/revision/object-group graph and property sets to extract the page hierarchy, page titles, body text (reading order) and embedded images/files — all in pure TS (DataView/Uint8Array), so it runs in the Node server and the standalone/WASM build alike, with no external runtime. Technology evaluation (see the report): Rust onenote_parser -> WASM was ruled out (no toolchain here, can't build unattended); Python pyOneNote crashed on the samples and is the wrong runtime for Trilium; a pure-TS parser is the architectural fit and proved out against the onenote.rs desktop test corpus (text, unicode, math, embedded files, a 112 MB file), correctly rejecting the OneDrive/FSSHTTPB variant. Wiring: - services/import/onenote-file/one_parser.ts — the binary decoder. - services/import/onenote-file/importer.ts — parse -> section root note, a note per page (subpage levels nested), text as HTML, images inline and other files as attachments. - dispatch.ts routes the .one/.onetoc2 extension to it (read from the upload buffer, no format tag). - a new "OneNote file" import-dialog provider (offline, no account), distinct from the Graph-based "OneNote" provider. Scope (PoC): formatting, ink, tables, note tags, math and cross-page links are not handled yet; the Graph importer remains the higher-fidelity path for cloud notebooks, while this reaches the offline .one-file case Graph cannot. Tests cover the parser and the end-to-end import against a small desktop .one fixture (from onenote.rs, MPL-2.0). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
🖥️ App preview is ready! 🔗 Preview URL: https://pr-10649.trilium-app.pages.dev ✅ All checks passed This preview will be updated automatically with new commits. |
Bundle ReportChanges will increase total bundle size by 19.46kB (0.02%) ⬆️. This is within the configured threshold ✅ Detailed changes
Affected Assets, Files, and Routes:view changes for bundle: client-esmAssets Changed:
Files in
view changes for bundle: standalone-esmAssets Changed:
|
Greptile SummaryThis PR introduces a pure-TypeScript, offline parser for the OneNote desktop
Confidence Score: 4/5Safe to merge as a draft/PoC; one logic error in the image-vs-file dispatch should be fixed before promotion to production. The importer dispatches content blocks to inline-image or file-attachment rendering using only the file extension, ignoring the importer.ts (image/file dispatch logic) and one_parser.ts (open parser issues from prior review threads, acknowledged as PoC scope). Important Files Changed
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const container = containerId ? space.objects.get(containerId) : undefined; | ||
| if (container?.fileData) { | ||
| const name = stringProp(obj, PROP.ImageFilename) ?? "image"; | ||
| out.push({ kind: "file", name, ext: mediaExtension(name, container.fileExt), bytes: container.fileData, image: true }); | ||
| } | ||
| break; | ||
| } | ||
| case JCID.EmbeddedFileNode: { | ||
| const containerId = objectRefs(obj, PROP.EmbeddedFileContainer)[0]; | ||
| const container = containerId ? space.objects.get(containerId) : undefined; | ||
| if (container?.fileData) { |
There was a problem hiding this comment.
String.fromCharCode(code) treats each 16-bit code unit as a standalone character. UTF-16 encodes any code point above U+FFFF (emoji, many math symbols, CJK Extension B) as two consecutive code units — a high surrogate (0xD800–0xDBFF) followed by a low surrogate (0xDC00–0xDFFF). Calling fromCharCode on each independently produces two garbled replacement characters instead of the intended glyph.
Concretely: a page containing a single emoji like 😀 (U+1F600, encoded as 0xD83D 0xDE00 in UTF-16LE) will appear as two broken characters when imported.
| if (!rootSpace) { | ||
| diagnostics.push("root object space not found"); | ||
| return { pages: [], diagnostics }; | ||
| } | ||
|
|
||
| const sectionId = rootSpace.roots.get(ROLE_DEFAULT_CONTENT); | ||
| const section = sectionId ? rootSpace.objects.get(sectionId) : undefined; | ||
| if (!section) { | ||
| diagnostics.push("section node not found"); | ||
| return { pages: [], diagnostics }; | ||
| } | ||
| if (section.jcid === JCID.TocContainer) { | ||
| diagnostics.push("this is a .onetoc2 table-of-contents, not a section"); | ||
| } | ||
|
|
||
| const pages: OnePage[] = []; | ||
| // SectionNode -> ElementChildNodes -> PageSeriesNodes | ||
| for (const seriesId of objectRefs(section, PROP.ElementChildNodes)) { | ||
| const series = rootSpace.objects.get(seriesId); | ||
| if (!series || series.jcid !== JCID.PageSeriesNode) { | ||
| continue; | ||
| } | ||
| const pageSpaceIds = objectSpaceRefs(series, PROP.ChildGraphSpaceElementNodes); | ||
| // Page metadata objects (for level), XORed with the seed. | ||
| const metaIds = objectRefs(series, PROP.MetaDataObjectsAboveGraphSpace).map((id) => xorExGuid(id, XOR_SEED_GUID)); | ||
|
|
||
| pageSpaceIds.forEach((spaceId, index) => { | ||
| const pageSpace = spaces.get(spaceId); | ||
| if (!pageSpace) { | ||
| return; | ||
| } | ||
| let level = 0; | ||
| const metaId = metaIds[index]; | ||
| const meta = metaId ? rootSpace.objects.get(metaId) : undefined; | ||
| if (meta) { | ||
| level = u32Prop(meta, PROP.PageLevel) ?? 0; |
There was a problem hiding this comment.
GlobalIdTableEntry2FNDX / Entry3FNDX nodes silently dropped
Both parseRevisionList and parseObjectGroup switch on GlobalIdTableEntryFNDX (0x024) only. The two compact-form variants — GlobalIdTableEntry2FNDX (0x025) and GlobalIdTableEntry3FNDX (0x026) — are defined in FN but fall through to default: break, so they never populate the GUID table. Any file that uses these variants will have incomplete ID tables, causing later resolveCompact calls to return <unresolved:N>:n strings. Objects keyed under those IDs are then silently absent from the extracted pages, resulting in missing text or images with no error reported.
| u64(): number { | ||
| const lo = this.u32(); | ||
| const hi = this.u32(); | ||
| return hi * 0x1_0000_0000 + lo; |
There was a problem hiding this comment.
u64() loses precision for offsets above 2⁵³
hi * 0x1_0000_0000 + lo is plain IEEE-754 double arithmetic. JavaScript integers are exact only up to Number.MAX_SAFE_INTEGER (2⁵³ − 1). The cbLength field in file-data objects is also read as u64 and could theoretically hold a large value that miscomputes the byte count, causing bytesN to return incorrect data.
| bytesN(n: number): Uint8Array { | ||
| const out = this.bytes.subarray(this.pos, this.pos + n); | ||
| this.pos += n; | ||
| return out; |
There was a problem hiding this comment.
bytesN silently under-reads on truncated/malformed input
bytes.subarray(this.pos, this.pos + n) clamps at the actual buffer length without signalling an error, while this.pos += n always advances by the full requested amount. If a crafted or truncated file declares a length that exceeds the remaining buffer, subarray returns a shorter slice silently, but subsequent reads start from the advanced (now out-of-range) position. A bounds check before the read would prevent silent mis-parses.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
The Graph importer can attach each page's original HTML for diagnosing a conversion; the offline .one importer had no equivalent, because there is no source markup — the closest thing is the object graph the page was decoded from, so debug mode now dumps that as JSON per page note. The dump is id-driven rather than name-driven: every object and every property is emitted keyed by its raw jcid / property id, with the parser's tables used only as annotation. The parser already retains whole property sets, so formatting, tables and note tags — everything the importer walks past today — survive into the dump without it having to understand them first. Blobs are base64'd in full for the same reason, and rendered in both text encodings the format uses: a latin-1 run decodes as convincing CJK when read as UTF-16, so picking one would hide half the text behind plausible gibberish. The serializer lives beside the parser rather than inside it, to keep the parser free of the debug format and avoid an import cycle; the parser only gains an option to retain the object spaces it would otherwise discard. Also carries the in-progress FSSHTTPB (OneDrive-downloaded section) detection from the working tree, which touches the same parser file. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 participant
What this is
A from-scratch TypeScript parser for the OneNote desktop
.one/.onetoc2binary format (MS-ONESTORE revision store + MS-ONE object model), plus an importer that turns a.onefile into a Trilium note tree — fully offline, no Microsoft Graph, no account. This reaches the.one-file case the Graph API fundamentally cannot (local/unsynced notebooks, and the content Graph drops).The parser is pure
DataView/Uint8Arraywith no external dependencies or runtime, so it lives intrilium-coreand runs in the Node server and the standalone/WASM build alike.Technology evaluation
I tried three approaches before landing on pure-TS:
onenote_parser→ WASM/nativepyOneNotesidecarmsiemens/onenote.rsreferenceHow it's wired
services/import/onenote-file/one_parser.ts— decodes the file-node lists, object-space/revision/object-group graph and property sets; extracts page hierarchy, titles, body text (reading order) and embedded images/files.services/import/onenote-file/importer.ts— parse → section root note → one note per page (subpage levels nested), text as HTML, images inline, other files as attachments.dispatch.tsroutes the.one/.onetoc2extension to it (read from the upload buffer; noformattag needed).To try it:
pnpm server:start→ Import → OneNote file → drop a.oneexported from OneNote desktop.Proven against real files
Validated against the
onenote.rsdesktop test corpus:.docx, PNGs; parsed a 112 MB file with embedded PDFs)Tests
one_parser.spec.ts— parser: pages/titles/text, embedded-image bytes, format rejectionimporter.spec.ts— end-to-end:.one→ 3-page note tree with HTML body and an inline image attachment5 tests pass;
pnpm typecheckis clean. The fixture is a small desktop.onefromonenote.rs(MPL-2.0), attributed in the spec.Scope / not done yet (why it's a draft)
.onedesktop export /.onetoc2); OneDrive-downloaded FSSHTTPB files are rejected (a separate code path in the reference parser).Open questions for reviewers
.onepath worth productionizing, or should effort stay on the Graph importer?onenote_parser→ WASM for full fidelity once a build toolchain is in place?.onetest fixture (MPL-2.0), or should we generate/host our own?🤖 Generated with Claude Code