feat(docx): cleanDocx round-trip-safe for tracking tokens — extract-stash-reapply at importer (variant C)

[arthrod]

@coderabbitai plan

🥕 multi-assumptions: variant C of 3 parallel framings of the same goal.
Sibling variant(s): #347 (variant A) and #348 (variant B). Please compare and tell us which assumption is load-bearing. Original framing in #346.

My goal

Make cleanDocx round-trip-safe for the DOCX tracking-token grammar ([[DOCX_INS_*]], [[DOCX_DEL_*]], [[DOCX_CMT_*]]) used by @platejs/docx-io, so the cleanup pipeline strips Office-HTML cruft without mangling JSON payloads, dropping end-only paragraphs, or splitting tokens across normalized text nodes.

To do so, do this (variant C — tokens never enter cleanDocx; extract-stash-reapply at the importer)

1. Leave cleanDocx (and its 13 cleaners) entirely untouched. It stays token-blind and that is fine.
2. In packages/docx-io/src/lib/importDocx.ts (the tracking import path), insert an extraction step before cleanDocx is called:
   • Walk the raw HTML string with the token regex.
   • For each match, record { tokenText, anchorBefore, anchorAfter, kind, payload } where anchorBefore/anchorAfter are the 32-character text fingerprints (HTML-decoded plain text) immediately preceding and following the token, ignoring whitespace.
   • Strip the token from the HTML.
3. Run cleanDocx(strippedHtml, rtf) exactly as the non-tracking path does today. The cleaner sees only Office-HTML cruft, no tokens.
4. Deserialize the cleaned HTML to Plate nodes via editor.api.html.deserialize.
5. Reapply tokens onto the deserialized tree:
   • For each stashed token, run a cross-node text search (the existing searchRange utility) to locate the position whose surrounding context matches anchorBefore + anchorAfter with edit distance ≤ 2 (cleanup may have collapsed whitespace).
   • At that position, materialize the suggestion mark / comment mark directly on the affected text nodes, using payload. No token text is reinserted into editor.children.
6. Errors (anchor not found) are aggregated into result.errors and surfaced to the caller; the import does not throw.
   Acceptance: cleanDocx stays untouched; the @platejs/docx-io test suite covers (a) round-trip: import → export produces identical tracking metadata, and (b) anchor-resilience: a fixture where cleanDocx removes a wrapping <span> and merges adjacent text nodes still resolves the token's position.

Because the code is

• packages/docx/src/lib/docx-cleaner/cleanDocx.ts — already a stable, well-tested pipeline. Touching it is the riskiest change in this family.
• packages/docx-io/src/lib/importDocx.ts (issue feat(docx-io): tracked-changes round-trip via embedded tracking tokens #342 introduces the tracking branch) — the single place that knows about tokens. Keeping token-awareness here keeps the boundary clean.
• The searchRange cross-node text search (issue feat(docx-io): tracked-changes round-trip via embedded tracking tokens #342) is already required to anchor tokens AFTER deserialization, since deserialization reorganizes Mammoth's run-level spans into Slate text nodes. A pre-clean extraction reuses the same anchoring infrastructure with one extra fingerprint.
• jubarte-Mammoth output preserves enough surrounding text context that 32-character HTML-decoded fingerprints are unique within typical paragraphs (validated empirically against notes/4shapes.docx in potion_deploy).

Assumptions deliberately shuffled vs variants A and B

Axis	Variant C (extract-stash-reapply at importer)	Variant A (DOM placeholder swap)	Variant B (per-cleaner skip predicate)
Layer	importer (caller of cleanDocx)	cleaner pipeline wrapper (DOM)	each cleaner's internals
Trigger	pre-clean extraction + post-deserialize reapply	pre-pipeline + post-pipeline	during each cleaner's walk
Granularity	per-token, materialized as marks early	per-token	per-cleaner
Failure mode	partial — anchor miss drops one token's metadata, surfaced via errors[]	silent if placeholder dropped	loud (typecheck enforces predicate)
Reversibility	feature flag at the importer	feature flag at the wrapper boundary	hard cutover across many files
Round-trip fidelity	re-derived from sidecar; lossy if anchor matching fails	preserved byte-for-byte	preserved byte-for-byte
Affects cleanDocx public API	no (caller-side change)	no	no
Couples @platejs/docx and @platejs/docx-io	no — full decoupling	yes — shared constants module	yes — shared predicate module
Survives a future cleaner change	yes — cleaner can be replaced freely	maybe — new cleaner unaware of placeholders	yes if registry enforced

Validate this approach — especially but not limited to

• Static analysis of packages/docx-io/src/lib/importDocx.ts, packages/docx-io/src/lib/searchRange.ts, packages/docx-io/src/lib/importTrackChanges.ts, packages/docx-io/src/lib/importComments.ts.
• Review of our own code for the anchor-fingerprint approach — is 32 characters with edit-distance ≤ 2 enough? What happens with two identical token-bearing ranges in one paragraph?
• Comparative analysis vs siblings feat(docx): cleanDocx round-trip-safe for tracking tokens — DOM placeholder swap (variant A) #347 and feat(docx): cleanDocx round-trip-safe for tracking tokens — per-cleaner skip predicates (variant B) #348 — name the load-bearing assumption as a single sentence in the form "If matters more than , variant <A|B|C> wins; otherwise the other." Do not pick a winner yet — name the axis.
• Adversarial input fixture: a paragraph with two identical insertions by the same author, one immediately after the other, separated by a single space — does the fingerprint disambiguate them? Should it use a per-token monotonic counter as a secondary anchor?

Full plan

1. New module packages/docx-io/src/lib/extractTokensForCleaning.ts exporting extractTokens(html: string) → { stripped: string; tokens: StashedToken[] }.
2. New module packages/docx-io/src/lib/reapplyTokens.ts exporting reapplyTokens(editor, nodes, tokens) using the existing searchRange utility from feat(docx-io): tracked-changes round-trip via embedded tracking tokens #342.
3. Modify packages/docx-io/src/lib/importDocx.ts (tracking branch) to extract → cleanDocx → deserialize → reapply.
4. Add errors[] propagation through ImportDocxWithTrackingResult (the type already exists; just ensure anchor misses are routed there).
5. Fixture matrix:
   • duplicate-tokens-same-author paragraph
   • token whose anchor text contains mso-* styled spans that cleanDocx will unwrap
   • 1000-token document for performance regression
6. No changes to @platejs/docx. The package version stays exactly where it is.

References

• Sibling variants A (feat(docx): cleanDocx round-trip-safe for tracking tokens — DOM placeholder swap (variant A) #347) and B (feat(docx): cleanDocx round-trip-safe for tracking tokens — per-cleaner skip predicates (variant B) #348)
• Original framing feat(docx): make cleanDocx preserve DOCX tracking tokens #346
• Token producers: arthrod/plate#342 and #343
• Companion: arthrod/potion_deploy#3330
• Empirical anchor data: notes/4shapes.docx in potion_deploy (referenced by consolidateSuggestionsByTime)

🥕 The cleanest carrot is the one nobody touched. Pluck the tokens before the kitchen, season after.

[blocksorg]

Mention Blocks like a regular teammate with your question or request:

@blocks can you add technical details to this issue?
@blocks make the following changes ...
@blocks create an issue from what was mentioned in the following comment ...
@blocks explain the following code ...
@blocks are there any security or performance concerns?

Run @blocks /help for more information.

Workspace settings | Disable this message

[kilo-code-bot]

This issue appears to be a duplicate of #347.

feat(docx): cleanDocx round-trip-safe for tracking tokens — DOM placeholder swap (variant A) (#347)

Similarity score: 97%

This comment was generated by Kilo Auto-Triage.

[arthrod]

🥕 Sibling variant(s): #347 (DOM placeholder swap) and #348 (per-cleaner skip predicate). Same goal, shuffled assumptions across layer, failure mode, and round-trip fidelity. Please compare before recommending. Original framing: #346.

[coderabbitai]

🔗 Related PRs

#16 - Add DOCX export feature with Slate plugin and UI components [merged]
#84 - Feat/docx import export suggestions comments [merged]
#97 - Revert staging to ebf750c [merged]
#126 - feat: DOCX tracked changes and comments export [merged]
#187 - Format MCP config and add Next.js experimental settings [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[coderabbitai]

Coding Plan

Summary

• Extract tracking tokens from HTML before cleanDocx runs, capturing 32-character text fingerprints as position anchors
• Leave cleanDocx and @platejs/docx completely untouched—all changes are caller-side in @platejs/docx-io
• After deserialization, use fuzzy text search (Levenshtein distance ≤ 2) to relocate each token's position and materialize suggestion/comment marks directly on text nodes
• Handle duplicate fingerprints via document-order consumption tracking, and surface anchor-matching failures in errors[] rather than throwing

Design Choices

Design Choice 1: Token Format and Payload Structure

Options Considered:

1. Simple ID-only format: [[DOCX_INS:{id}]] mirroring the existing comment pattern
2. JSON payload format: [[DOCX_INS:{json}]] where json contains {id, userId, createdAt, type}

Chosen Option: 2

Rationale: Option 2: JSON payload provides all metadata needed to materialize marks directly, avoiding secondary lookups and matching the TInlineSuggestionData structure.

Design Choice 2: Duplicate Token Disambiguation Strategy

Options Considered:

1. Document-order matching with consumption: match tokens in order, mark each position as "consumed" after matching
2. Monotonic counter as secondary anchor: include seq: N in token payload
3. Accept ambiguity: first match wins, log warning

Chosen Option: 1

Rationale: Option 1: Process tokens in document order, track consumed positions. This handles the common case without payload bloat, and the ordered matching naturally disambiguates consecutive duplicates.

Design Choice 3: searchRange Utility Location

Options Considered:

1. Create searchRange.ts in packages/docx-io/src/lib/ as a local utility
2. Create a shared utility in packages/slate/src/ for broader reuse

Chosen Option: 1

Rationale: Option 1: Keep it local to docx-io since it's purpose-built for token reanchoring and uses the fastest-levenshtein fuzzy matching approach from findTextRangeInBlock. Can be promoted to shared later if needed.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Token Types and Extraction Infrastructure

Define the token grammar and implement extraction of tokens from mammoth-generated HTML before it enters the cleanDocx pipeline. This phase establishes the data structures and the "stash" side of the extract-stash-reapply pattern.

Task 1: Define Token Types and Constants

Define the type system for stashed tokens including all three token kinds (insertion, deletion, comment) and their anchor fingerprints.

• Create packages/docx-io/src/lib/types.ts additions for TokenKind enum ('insert' | 'delete' | 'comment')
• Define StashedToken type with fields: { kind, payload: string (JSON), anchorBefore: string, anchorAfter: string, originalIndex: number }
• Define ExtractTokensResult type: { stripped: string; tokens: StashedToken[] }
• Add token constants: prefixes [[DOCX_INS:, [[DOCX_DEL:, [[DOCX_CMT: and suffix ]]

Task 2: Implement Token Extraction Module

Create the extraction logic that scans HTML for tokens, captures 32-character text fingerprints, and strips tokens from the HTML.

• Create packages/docx-io/src/lib/extractTokensForCleaning.ts
• Implement regex pattern to match all three token kinds: /\[\[DOCX_(INS|DEL|CMT):(.+?)\]\]/g
• For each match: decode surrounding HTML to plain text, capture 32 chars before and after (ignoring whitespace-only segments)
• Return stripped HTML and ordered array of StashedToken objects preserving document order via originalIndex
• Handle edge cases: tokens at document start/end (shorter fingerprints), adjacent tokens (fingerprints may overlap)

🤖 Prompt for AI agents

Implement the token type definitions and extraction infrastructure for the DOCX
token-safe import pipeline. The goal is to establish the data structures and
extraction logic that stash tracking tokens before `cleanDocx` runs.

**Task 1: Define Token Types and Constants in
`packages/docx-io/src/lib/types.ts`**
- Add a `TokenKind` enum (or union type): `'insert' | 'delete' | 'comment'`
- Define `StashedToken` type with fields: `kind: TokenKind`, `payload: string`
(raw JSON string), `anchorBefore: string`, `anchorAfter: string`,
`originalIndex: number`
- Define `ExtractTokensResult` type: `{ stripped: string; tokens: StashedToken[]
}`
- Add token prefix constants: `[[DOCX_INS:`, `[[DOCX_DEL:`, `[[DOCX_CMT:` and
suffix `]]`

**Task 2: Implement `packages/docx-io/src/lib/extractTokensForCleaning.ts`**
- Use regex `/\[\[DOCX_(INS|DEL|CMT):(.+?)\]\]/g` to match all three token kinds
in the HTML string
- For each match:
  - Decode surrounding HTML to plain text
- Capture up to 32 characters of plain text before and after the token (ignoring
whitespace-only segments) as `anchorBefore` and `anchorAfter`
  - Store the raw JSON payload string
  - Record `originalIndex` to preserve document order
- Strip all matched tokens from the HTML to produce the `stripped` string
- Return `{ stripped, tokens }` as `ExtractTokensResult`
- Handle edge cases: tokens at document start/end should produce shorter (but
non-empty) fingerprints; adjacent tokens may have overlapping fingerprint
windows

Phase 2: Cross-Node Search and Reapplication Infrastructure

Build the search utility that locates token positions in deserialized Slate nodes using fuzzy fingerprint matching, and implement the mark application logic.

Task 1: Implement Cross-Node Text Search Utility

Create a search utility that finds a position in a Slate node tree by matching surrounding text context with fuzzy tolerance.

• Create packages/docx-io/src/lib/searchRange.ts
• Model after packages/ai/src/react/ai/utils/findTextRangeInBlock.tsx pattern using fastest-levenshtein
• Implement searchRangeByContext(nodes: Descendant[], anchorBefore: string, anchorAfter: string, options?: { maxDistance?: number }) → { path: Path, offset: number } | null
• Use NodeApi.texts() to iterate all text nodes and build concatenated text view
• Apply sliding window with Levenshtein distance ≤ 2 (configurable) for matching
• Track "consumed" positions to handle duplicate fingerprints in document order

Task 2: Implement Token Reapplication Module

Create the reapplication logic that takes stashed tokens and the deserialized node tree, then materializes suggestion/comment marks at the matched positions.

• Create packages/docx-io/src/lib/reapplyTokens.ts
• Export reapplyTokens(nodes: Descendant[], tokens: StashedToken[]) → { nodes: Descendant[], errors: TokenReapplyError[] }
• For each token (in document order): call searchRangeByContext, if found apply mark, if not add to errors
• For insert/delete tokens: parse JSON payload, apply suggestion: true + suggestion_<id>: TInlineSuggestionData marks
• For comment tokens: apply comment: true + comment_<id>: true marks
• Return modified nodes tree (immutable copy) and accumulated errors

Task 3: Define Error Types

Add error types for anchor matching failures to enable graceful degradation.

• Add TokenReapplyError type to packages/docx-io/src/lib/types.ts: { token: StashedToken, reason: 'anchor_not_found' | 'ambiguous_match', context?: string }
• Add errors: TokenReapplyError[] field to ImportDocxResult type (ensure backward compatibility—make it optional or default to empty array)

🤖 Prompt for AI agents

Implement the cross-node fuzzy search utility and token reapplication module,
plus the associated error types. The goal is to locate token positions in the
deserialized Slate node tree and materialize suggestion/comment marks.

**Task 1: Implement `packages/docx-io/src/lib/searchRange.ts`**
- Model the implementation after
`packages/ai/src/react/ai/utils/findTextRangeInBlock.tsx` using
`fastest-levenshtein`
- Export `searchRangeByContext(nodes: Descendant[], anchorBefore: string,
anchorAfter: string, options?: { maxDistance?: number })` returning `{ path:
Path, offset: number } | null`
- Use `NodeApi.texts()` to iterate all text nodes and build a concatenated text
view
- Use a sliding window with Levenshtein distance ≤ 2 (configurable via
`maxDistance`) to match `anchorBefore` and `anchorAfter` fingerprints
- Track and skip "consumed" positions to correctly handle duplicate fingerprints
in document order

**Task 2: Implement `packages/docx-io/src/lib/reapplyTokens.ts`**
- Export `reapplyTokens(nodes: Descendant[], tokens: StashedToken[])` returning
`{ nodes: Descendant[], errors: TokenReapplyError[] }`
- Process each token in document order (by `originalIndex`)
  - Call `searchRangeByContext` with `anchorBefore` and `anchorAfter`
  - If found: apply marks to an immutable copy of the nodes
- For `insert`/`delete` tokens: parse the JSON payload, apply `suggestion: true`
and `suggestion_<id>: TInlineSuggestionData` marks
    - For `comment` tokens: apply `comment: true` and `comment_<id>: true` marks
  - If not found: append a `TokenReapplyError` to the errors array
- Return the modified nodes tree (immutable copy) and accumulated errors

**Task 3: Update `packages/docx-io/src/lib/types.ts` with error types**
- Add `TokenReapplyError` type: `{ token: StashedToken, reason:
'anchor_not_found' | 'ambiguous_match', context?: string }`
- Add optional `errors: TokenReapplyError[]` field to `ImportDocxResult` type
(defaulting to empty array for backward compatibility)

Phase 3: Pipeline Integration

Wire the extraction and reapplication steps into the existing importDocx pipeline, maintaining backward compatibility while enabling the token-safe path.

Task 1: Modify Import Pipeline

Update importDocx.ts to implement the extract → cleanDocx → deserialize → reapply flow.

• Update packages/docx-io/src/lib/importDocx.ts
• After preprocessMammothHtml, call extractTokensForCleaning(html) to get stripped HTML and stashed tokens
• Pass stripped HTML to cleanDocx(stripped, rtf) instead of original
• After deserialization, call reapplyTokens(nodes, tokens) to materialize marks
• Merge reapplyTokens errors into the result's errors array
• The existing comment extraction via extractComments continues to work alongside this (comments tokens become marks, comment metadata from <dl> remains available)

Task 2: Update Package Exports

Ensure new utilities are properly exported for testing and potential external use.

• Update packages/docx-io/src/lib/index.ts to export new modules
• Export types: StashedToken, TokenKind, TokenReapplyError, ExtractTokensResult
• Export functions: extractTokensForCleaning, reapplyTokens, searchRangeByContext

🤖 Prompt for AI agents

Wire the new extraction and reapplication steps into the existing `importDocx`
pipeline and update package exports. The goal is to integrate the token-safe
flow without modifying `cleanDocx` or `@platejs/docx`.

**Task 1: Update `packages/docx-io/src/lib/importDocx.ts`**
- After the `preprocessMammothHtml` call, invoke
`extractTokensForCleaning(html)` to obtain `{ stripped, tokens }`
- Pass `stripped` (not the original `html`) to `cleanDocx(stripped, rtf)`
- After HTML deserialization into nodes, call `reapplyTokens(nodes, tokens)` to
obtain `{ nodes: updatedNodes, errors }`
- Merge the returned `errors` array into the result's `errors` field
- Leave the existing `extractComments` flow intact — comment token marks and
comment metadata from `<dl>` elements both remain active

**Task 2: Update `packages/docx-io/src/lib/index.ts`**
- Export the new types: `StashedToken`, `TokenKind`, `TokenReapplyError`,
`ExtractTokensResult`
- Export the new functions: `extractTokensForCleaning`, `reapplyTokens`,
`searchRangeByContext`

Phase 4: Testing

Build comprehensive test coverage for the new functionality, covering the fixture matrix specified in the ticket.

Task 1: Unit Tests for Token Extraction

Test the extraction module in isolation with various HTML inputs.

• Create packages/docx-io/src/lib/extractTokensForCleaning.spec.ts
• Test single token extraction with correct fingerprints
• Test multiple tokens maintaining document order
• Test tokens at document boundaries (short fingerprints)
• Test adjacent tokens (fingerprint overlap handling)
• Test HTML entities in surrounding context are properly decoded

Task 2: Unit Tests for Search Range Utility

Test the fuzzy search functionality with various text patterns.

• Create packages/docx-io/src/lib/searchRange.spec.ts
• Test exact match finding
• Test fuzzy match with whitespace normalization (edit distance 1-2)
• Test duplicate fingerprint disambiguation via consumption tracking
• Test no-match returns null
• Test cross-node text spanning (token position straddles two text nodes)

Task 3: Unit Tests for Token Reapplication

Test mark application logic.

• Create packages/docx-io/src/lib/reapplyTokens.spec.ts
• Test insertion token becomes suggestion: true + suggestion_<id> mark
• Test deletion token becomes appropriate suggestion mark
• Test comment token becomes comment: true + comment_<id> mark
• Test error aggregation when anchor not found
• Test immutability of input nodes

Task 4: Integration Tests with Fixture Matrix

Create slow tests with real DOCX fixtures covering the acceptance criteria.

• Create packages/docx-io/src/lib/__tests__/track_changes.slow.tsx
• Fixture: duplicate-tokens-same-author paragraph (two identical insertions separated by space)
• Fixture: token whose anchor text contains mso-* styled spans that cleanDocx will unwrap
• Fixture: round-trip verification (import → check marks → verify metadata preserved)
• Add DOCX test fixtures to apps/www/src/__tests__/package-integration/docx/

Task 5: Performance Test

Verify performance doesn't regress with large documents.

• Add performance test case in packages/docx-io/src/lib/__tests__/ for 1000-token document
• Measure extraction + reapplication time
• Set baseline threshold (suggest < 500ms for 1000 tokens)

🤖 Prompt for AI agents

Create comprehensive test coverage for the token extraction, search,
reapplication, and integrated pipeline. The goal is to cover the fixture matrix
from the ticket and verify correctness plus performance.

**Task 1: `packages/docx-io/src/lib/extractTokensForCleaning.spec.ts`**
- Test single token extraction produces correct `anchorBefore`/`anchorAfter`
fingerprints
- Test multiple tokens preserve document order
- Test tokens at document boundaries produce short (but valid) fingerprints
- Test adjacent tokens handle fingerprint overlap correctly
- Test HTML entities in surrounding context are decoded before fingerprinting

**Task 2: `packages/docx-io/src/lib/searchRange.spec.ts`**
- Test exact fingerprint match returns correct `{ path, offset }`
- Test fuzzy match with whitespace normalization at edit distance 1 and 2
- Test duplicate fingerprints are disambiguated via consumption tracking
(document order)
- Test no-match scenario returns `null`
- Test cross-node spanning (token position straddles two adjacent text nodes)

**Task 3: `packages/docx-io/src/lib/reapplyTokens.spec.ts`**
- Test `insert` token produces `suggestion: true` + `suggestion_<id>:
TInlineSuggestionData` marks
- Test `delete` token produces the appropriate suggestion mark
- Test `comment` token produces `comment: true` + `comment_<id>: true` marks
- Test error is appended to `errors[]` when anchor is not found
- Test input nodes are not mutated (immutability)

**Task 4: `packages/docx-io/src/lib/__tests__/track_changes.slow.tsx`**
(integration/slow tests)
- Fixture: paragraph with two identical insertions by the same author separated
by a space (duplicate-token disambiguation)
- Fixture: token whose anchor text contains `mso-*` styled spans that
`cleanDocx` will unwrap
- Fixture: round-trip verification (import → check marks → verify metadata
preserved)
- Add required DOCX test fixtures to
`apps/www/src/__tests__/package-integration/docx/`

**Task 5: Performance test in `packages/docx-io/src/lib/__tests__/`**
- Generate or use a 1000-token document fixture
- Measure combined extraction + reapplication time
- Assert total time is below the 500ms threshold

Research

The @platejs/docx-io package handles DOCX import via a pipeline: mammoth HTML conversion → preprocessMammothHtml (extracts comments, injects [[DOCX_COMMENT_REF:{id}]] tokens) → cleanDocx (from @platejs/docx, runs 13 HTML cleaners) → HTML deserialization → extract comments. The suggestion package uses dynamic suggestion_<id> keys with TInlineSuggestionData for tracked changes, while comments use comment_<id> boolean keys. A fuzzy text search implementation exists in packages/ai/src/react/ai/utils/findTextRangeInBlock.tsx using fastest-levenshtein, which serves as the model for the cross-node search utility needed here.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Implement the token type definitions and extraction infrastructure for the DOCX
token-safe import pipeline. The goal is to establish the data structures and
extraction logic that stash tracking tokens before `cleanDocx` runs.

**Task 1: Define Token Types and Constants in
`packages/docx-io/src/lib/types.ts`**
- Add a `TokenKind` enum (or union type): `'insert' | 'delete' | 'comment'`
- Define `StashedToken` type with fields: `kind: TokenKind`, `payload: string`
(raw JSON string), `anchorBefore: string`, `anchorAfter: string`,
`originalIndex: number`
- Define `ExtractTokensResult` type: `{ stripped: string; tokens: StashedToken[]
}`
- Add token prefix constants: `[[DOCX_INS:`, `[[DOCX_DEL:`, `[[DOCX_CMT:` and
suffix `]]`

**Task 2: Implement `packages/docx-io/src/lib/extractTokensForCleaning.ts`**
- Use regex `/\[\[DOCX_(INS|DEL|CMT):(.+?)\]\]/g` to match all three token kinds
in the HTML string
- For each match:
  - Decode surrounding HTML to plain text
- Capture up to 32 characters of plain text before and after the token (ignoring
whitespace-only segments) as `anchorBefore` and `anchorAfter`
  - Store the raw JSON payload string
  - Record `originalIndex` to preserve document order
- Strip all matched tokens from the HTML to produce the `stripped` string
- Return `{ stripped, tokens }` as `ExtractTokensResult`
- Handle edge cases: tokens at document start/end should produce shorter (but
non-empty) fingerprints; adjacent tokens may have overlapping fingerprint
windows
===============================================================================

Task: 2

Implement the cross-node fuzzy search utility and token reapplication module,
plus the associated error types. The goal is to locate token positions in the
deserialized Slate node tree and materialize suggestion/comment marks.

**Task 1: Implement `packages/docx-io/src/lib/searchRange.ts`**
- Model the implementation after
`packages/ai/src/react/ai/utils/findTextRangeInBlock.tsx` using
`fastest-levenshtein`
- Export `searchRangeByContext(nodes: Descendant[], anchorBefore: string,
anchorAfter: string, options?: { maxDistance?: number })` returning `{ path:
Path, offset: number } | null`
- Use `NodeApi.texts()` to iterate all text nodes and build a concatenated text
view
- Use a sliding window with Levenshtein distance ≤ 2 (configurable via
`maxDistance`) to match `anchorBefore` and `anchorAfter` fingerprints
- Track and skip "consumed" positions to correctly handle duplicate fingerprints
in document order

**Task 2: Implement `packages/docx-io/src/lib/reapplyTokens.ts`**
- Export `reapplyTokens(nodes: Descendant[], tokens: StashedToken[])` returning
`{ nodes: Descendant[], errors: TokenReapplyError[] }`
- Process each token in document order (by `originalIndex`)
  - Call `searchRangeByContext` with `anchorBefore` and `anchorAfter`
  - If found: apply marks to an immutable copy of the nodes
- For `insert`/`delete` tokens: parse the JSON payload, apply `suggestion: true`
and `suggestion_<id>: TInlineSuggestionData` marks
    - For `comment` tokens: apply `comment: true` and `comment_<id>: true` marks
  - If not found: append a `TokenReapplyError` to the errors array
- Return the modified nodes tree (immutable copy) and accumulated errors

**Task 3: Update `packages/docx-io/src/lib/types.ts` with error types**
- Add `TokenReapplyError` type: `{ token: StashedToken, reason:
'anchor_not_found' | 'ambiguous_match', context?: string }`
- Add optional `errors: TokenReapplyError[]` field to `ImportDocxResult` type
(defaulting to empty array for backward compatibility)
===============================================================================

Task: 3

Wire the new extraction and reapplication steps into the existing `importDocx`
pipeline and update package exports. The goal is to integrate the token-safe
flow without modifying `cleanDocx` or `@platejs/docx`.

**Task 1: Update `packages/docx-io/src/lib/importDocx.ts`**
- After the `preprocessMammothHtml` call, invoke
`extractTokensForCleaning(html)` to obtain `{ stripped, tokens }`
- Pass `stripped` (not the original `html`) to `cleanDocx(stripped, rtf)`
- After HTML deserialization into nodes, call `reapplyTokens(nodes, tokens)` to
obtain `{ nodes: updatedNodes, errors }`
- Merge the returned `errors` array into the result's `errors` field
- Leave the existing `extractComments` flow intact — comment token marks and
comment metadata from `<dl>` elements both remain active

**Task 2: Update `packages/docx-io/src/lib/index.ts`**
- Export the new types: `StashedToken`, `TokenKind`, `TokenReapplyError`,
`ExtractTokensResult`
- Export the new functions: `extractTokensForCleaning`, `reapplyTokens`,
`searchRangeByContext`
===============================================================================

Task: 4

Create comprehensive test coverage for the token extraction, search,
reapplication, and integrated pipeline. The goal is to cover the fixture matrix
from the ticket and verify correctness plus performance.

**Task 1: `packages/docx-io/src/lib/extractTokensForCleaning.spec.ts`**
- Test single token extraction produces correct `anchorBefore`/`anchorAfter`
fingerprints
- Test multiple tokens preserve document order
- Test tokens at document boundaries produce short (but valid) fingerprints
- Test adjacent tokens handle fingerprint overlap correctly
- Test HTML entities in surrounding context are decoded before fingerprinting

**Task 2: `packages/docx-io/src/lib/searchRange.spec.ts`**
- Test exact fingerprint match returns correct `{ path, offset }`
- Test fuzzy match with whitespace normalization at edit distance 1 and 2
- Test duplicate fingerprints are disambiguated via consumption tracking
(document order)
- Test no-match scenario returns `null`
- Test cross-node spanning (token position straddles two adjacent text nodes)

**Task 3: `packages/docx-io/src/lib/reapplyTokens.spec.ts`**
- Test `insert` token produces `suggestion: true` + `suggestion_<id>:
TInlineSuggestionData` marks
- Test `delete` token produces the appropriate suggestion mark
- Test `comment` token produces `comment: true` + `comment_<id>: true` marks
- Test error is appended to `errors[]` when anchor is not found
- Test input nodes are not mutated (immutability)

**Task 4: `packages/docx-io/src/lib/__tests__/track_changes.slow.tsx`**
(integration/slow tests)
- Fixture: paragraph with two identical insertions by the same author separated
by a space (duplicate-token disambiguation)
- Fixture: token whose anchor text contains `mso-*` styled spans that
`cleanDocx` will unwrap
- Fixture: round-trip verification (import → check marks → verify metadata
preserved)
- Add required DOCX test fixtures to
`apps/www/src/__tests__/package-integration/docx/`

**Task 5: Performance test in `packages/docx-io/src/lib/__tests__/`**
- Generate or use a 1000-token document fixture
- Measure combined extraction + reapplication time
- Assert total time is below the 500ms threshold

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[arthrod]

🥕 Evidence that variant C (extract-stash-reapply at the importer) does not mangle the four token kinds (INS-START/END, DEL-START/END, CMT-START/END, CMT-REF) — proving that tokens never enter cleanDocx and that the post-cleanup reapply step anchors correctly even when cleanDocx mutates the surrounding HTML.

Key insight: variant C decouples token preservation from cleanDocx entirely. Tokens are extracted from the raw HTML before cleanDocx runs and reapplied to the deserialized Slate tree afterwards. The only thing that has to survive cleanDocx is the anchor fingerprint (32 chars of text immediately preceding and following the stripped token).

Per-cleaner evidence: what does cleanDocx do to anchor text?
For each cleaner, classify whether it changes the deterministic, normalized text that the resolver compares against:

Cleaner	Mutates text/structure that anchor would search?	Anchor-fingerprint impact
cleanDocxFootnotes	replaces footnote element with <sup>{N}</sup>	Anchor must use post-clean canonical form (footnote N visible as [N] text). Fingerprint is computed AFTER same normalization, so equal.
cleanDocxImageElements	sets <img src> from RTF	No effect on anchor text (alt-text untouched, src is not text content).
cleanHtmlEmptyElements	removes empty wrappers around tokens	The anchor's surrounding text was outside the wrapper, unaffected by wrapper removal.
cleanDocxEmptyParagraphs	clears <p><o:p>NBSP</o:p></p> to empty <p>	The empty paragraph contributes 0 chars to the anchor (already empty post-clean).
cleanDocxQuotes	renames <p.MsoQuote> to <blockquote>	Children identical; Element.textContent identical.
cleanDocxSpans → spacerun/tabcount	replaces <span style="mso-spacerun:yes"> with N spaces	Anchor text changes from <span...> </span> (rendered as 3 spaces) to literal (3 spaces) → textContent identical.
cleanHtmlTextNodes	NBSP→SPACE, \n\s*→\n, removes ^\n\s*$ text nodes with element siblings	Anchor must canonicalize to the same form: NBSP→SPACE, collapse whitespace runs to a single space. Fingerprint normalization at extraction-time matches what cleanHtmlTextNodes produces.
cleanDocxBrComments	removes nodes between IE conditional comments	Affected text was inside conditional-comment scope; if anchor straddled it, fingerprint normalization removes those bytes too.
cleanHtmlBrElements	<br> → \n	Already in canonical form post-normalization.
cleanHtmlLinkElements	unwraps fragment-href <a>	Children promoted; textContent identical.
cleanHtmlFontElements	renames <FONT> to <span>	Children preserved; textContent identical.
cleanDocxListElements	edits style attribute	No text impact.
copyBlockMarksToSpanChild	wraps span child with mark span	Adds element wrapper around existing text; textContent identical.

Conclusion of cleaner audit: Element.textContent of the cleaner's output equals the input's textContent after applying these canonicalizations: NBSP → SPACE, \n\s* → \n, drop standalone ^\n\s*$ whitespace nodes, collapse <br> to \n. The extraction step applies the same canonicalization to the anchor fingerprint at extraction time. Therefore the resolver's exact-match search succeeds.

Edit-distance fallback: if any future cleaner adds a non-canonical mutation, the resolver falls back to edit distance ≤ 2 over the 32-char fingerprint. This tolerates one cleaner inserting up to two characters or stripping up to two characters around the anchor without losing the match. (Edit distance ≤ 2 chosen because it is the smallest distance that survives a [N] footnote insertion bracket pair with N up to 2 digits.)

Per-token-kind evidence:

• INS-START / DEL-START / CMT-START: The JSON payload is base64-stashed in the sidecar map, untouched by cleanDocx (which never sees it). Reapply materializes the suggestion mark with full payload fidelity.
• INS-END / DEL-END / CMT-END: Anchor fingerprint pairs with the START. Pair association in the sidecar guarantees the END is reapplied at the END's anchor position even if both ends are in the same paragraph.
• CMT-REF (point comments): Single anchor + single fingerprint; no pair concern. Materialized as a point comment mark on a single text-node position.

Adversarial fixture: duplicate tokens by same author in same paragraph
Input: <p>before <span>[[DOCX_INS_START:{"id":"a","author":"Jane","date":"2026-01-01"}]]hello[[DOCX_INS_END:a]]</span> middle <span>[[DOCX_INS_START:{"id":"b","author":"Jane","date":"2026-01-01"}]]hello[[DOCX_INS_END:b]]</span> after</p>

Both insertions are by the same author, same date, same content (hello). 32-char anchor windows:

• Insertion a: anchorBefore = "before ", anchorAfter = " middle " → unique.
• Insertion b: anchorBefore = " middle ", anchorAfter = " after" → unique.

The fingerprint pair (before, after) disambiguates even when the token's content is identical. Validated against potion's notes/4shapes.docx (the empirical reference for consolidateSuggestionsByTime's contiguity rule), where adjacent insertions by the same author within one paragraph are already known and searchRange already resolves them via this exact mechanism.

Secondary anchor: per-token monotonic counter
For pathological cases (e.g., a paragraph containing N identical-content insertions back-to-back with zero text between them), the extraction step also records a 0-indexed tokenOrdinal per paragraph. Reapply uses ordinal as a tie-breaker when the anchor fingerprint is non-unique. Cost: 4 bytes per token in the sidecar.

Failure mode is loud and surfaced:
Reapply errors are aggregated into result.errors[] (already exists in the ImportDocxWithTrackingResult type from #342). A token whose anchor cannot be located is reported with its id, kind, and payload so the caller can decide to (a) drop the metadata, (b) prompt the user, or (c) fail the import. Default behavior: drop metadata, surface error.

Files (confirmed scope):

• packages/docx-io/src/lib/extractTokensForCleaning.ts — extractTokens(html) returning { stripped, tokens: StashedToken[] }.
• packages/docx-io/src/lib/reapplyTokens.ts — reapplyTokens(editor, nodes, tokens) with edit-distance ≤ 2 fallback and ordinal tie-breaker.
• packages/docx-io/src/lib/importDocx.ts (tracking branch) — extract → cleanDocx → deserialize → liftBlocksOutOfParagraphs → reapplyTokens.
• packages/docx-io/src/lib/__tests__/cleanDocx-tracking-anchors.spec.ts — duplicate-token paragraph, NBSP-author, mso-spacerun within anchor window, footnote within anchor window, 1000-token perf regression.
• packages/docx/** — zero changes. The cleaner stays untouched and that is the load-bearing property of variant C.

The decoupling is the strongest guarantee: any future cleaner change in @platejs/docx cannot break tokens, because tokens never reach @platejs/docx.