feat(docx): cleanDocx round-trip-safe for tracking tokens — DOM placeholder swap (variant A)

[arthrod]

@coderabbitai plan

🥕 multi-assumptions: variant A of 3 parallel framings of the same goal.
Sibling variant(s): will be cross-linked once filed. Please compare and tell us which assumption is load-bearing. The carrot family also includes the original framing in #346, which this variant supersedes.

My goal

Make cleanDocx round-trip-safe for the DOCX tracking-token grammar ([[DOCX_INS_*]], [[DOCX_DEL_*]], [[DOCX_CMT_*]]) used by @platejs/docx-io, so the cleanup pipeline strips Office-HTML cruft without mangling JSON payloads, dropping end-only paragraphs, or splitting tokens across normalized text nodes.

To do so, do this (variant A — DOM placeholder swap around the existing pipeline)

1. Add protectTrackingTokens(body) that walks body once: for every text node containing [[DOCX_(INS|DEL|CMT)_(START|END|REF):…]], split the text node, replace each token slice with <span data-docx-tracking-token="<base64-of-original>" data-docx-tracking-kind="ins-start|...">​</span> (zero-width-space inside so no cleaner classifies it as empty).
2. Run the existing cleanDocx pipeline unchanged: cleanDocxFootnotes → cleanDocxImageElements → cleanHtmlEmptyElements → cleanDocxEmptyParagraphs → cleanDocxQuotes → cleanDocxSpans → cleanHtmlTextNodes → cleanDocxBrComments → cleanHtmlBrElements → cleanHtmlLinkElements → cleanHtmlFontElements → cleanDocxListElements → copyBlockMarksToSpanChild.
3. Add restoreTrackingTokens(body) that swaps every surviving <span data-docx-tracking-token> back to a Text node with the base64-decoded original token bytes.
4. Add a string-level pre-pass for tokens that exist between tags (rare Mammoth output) before DOMParser is called.
   Acceptance: the existing cleanDocx test snapshot is unchanged for token-free HTML, and a fixture matrix (token in mso-styled paragraph, end-only paragraph, adjacent tokens, JSON with quotes/spaces) round-trips byte-for-byte.

Because the code is

• packages/docx/src/lib/docx-cleaner/cleanDocx.ts:23-49 — the linear pipeline that runs cleanHtmlTextNodes, cleanDocxEmptyParagraphs, cleanDocxSpans, copyBlockMarksToSpanChild over body. None of these inspect attributes; they classify by tag/text content.
• cleanDocxEmptyParagraphs drops paragraphs with no visible text. A paragraph whose only content is [[DOCX_INS_END:abc]] reads as visible text — survives. A paragraph whose only content is a <span class="track-end"> that gets unwrapped first by cleanDocxSpans and then leaves an empty <p> is dropped.
• cleanHtmlTextNodes normalizes whitespace and merges adjacent text nodes — JSON payloads with author names containing spaces ({"author":"John Doe"}) get the spaces normalized.
• jubarte-Mammoth output keeps tokens as plain text inside <span> wrappers; the DOM serializer round-trips attributes losslessly.

Assumptions deliberately shuffled vs variants B and C

Axis	Variant A (DOM placeholder swap)	Variant B (per-cleaner skip predicate)	Variant C (extract-stash-reapply at importer)
Layer	cleaner pipeline wrapper (DOM)	each cleaner's internals	importer (caller of cleanDocx)
Trigger	pre-pipeline + post-pipeline	during each cleaner's walk	pre-clean extraction + post-deserialize reapply
Granularity	per-token	per-cleaner	per-token, materialized as marks early
Failure mode	silent if placeholder dropped (mitigated by ID pairing)	loud (typecheck enforces predicate)	partial (reapply may fail to anchor → metadata drop)
Reversibility	feature flag at the wrapper boundary	hard cutover across many files	feature flag at the importer
Round-trip fidelity	preserved byte-for-byte	preserved byte-for-byte	re-derived from sidecar after deserialize
Affects cleanDocx public API	no	no (internal predicate added per cleaner)	no (caller-side change)

Validate this approach — especially but not limited to

• Static analysis of packages/docx/src/lib/docx-cleaner/cleanDocx.ts, packages/docx/src/lib/docx-cleaner/utils/*, and packages/docx-io/src/lib/importDocx.ts.
• Review of our own code for placeholder-element interactions: do cleanDocxSpans or copyBlockMarksToSpanChild mutate elements with data-* attributes? If yes, the swap leaks.
• Comparative analysis vs sibling variants B and C — name the load-bearing assumption as a single sentence in the form "If matters more than , variant <A|B|C> wins; otherwise the other." Do not pick a winner yet — name the axis.
• Worst-case input fixture: a paragraph with mso-* styles, an <o:p> empty wrapper, three adjacent tokens (END, START, REF), and a JSON payload with \"-escaped quotes inside body.

Full plan

1. New module packages/docx/src/lib/docx-cleaner/protectTrackingTokens.ts with protectTrackingTokens(body) and restoreTrackingTokens(body).
2. New shared module packages/utils/docx-tracking-tokens.ts exporting prefix/suffix constants and TRACKING_TOKEN_REGEX so @platejs/docx and @platejs/docx-io import from one place (no cycle).
3. Modify cleanDocx to wrap the pipeline: protectTrackingTokens(body); …existing pipeline…; restoreTrackingTokens(body);.
4. Add string-level pre-pass on raw HTML for tokens between tags.
5. Add fixture matrix in packages/docx/src/lib/docx-cleaner/__tests__/cleanDocx-tracking.spec.ts.
6. Backport the constant import in @platejs/docx-io to consume the new shared module.

References

• Sibling variants B (per-cleaner skip predicate) and C (extract-stash-reapply at importer)
• Original framing feat(docx): make cleanDocx preserve DOCX tracking tokens #346
• Companion issue arthrod/potion_deploy#3330 and #3331
• Tracking-token producers: arthrod/plate#342 and #343

🥕 The carrot patch grows three rows side by side — pick whichever sprouts the cleanest token through the dirt.

[blocksorg]

Mention Blocks like a regular teammate with your question or request:

@blocks can you add technical details to this issue?
@blocks make the following changes ...
@blocks create an issue from what was mentioned in the following comment ...
@blocks explain the following code ...
@blocks are there any security or performance concerns?

Run @blocks /help for more information.

Workspace settings | Disable this message

[kilo-code-bot]

This issue appears to be a duplicate of #346.

feat(docx): make cleanDocx preserve DOCX tracking tokens (#346)

Similarity score: 92%

This comment was generated by Kilo Auto-Triage.

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• feat(docx): make cleanDocx preserve DOCX tracking tokens #346

🔗 Related PRs

#16 - Add DOCX export feature with Slate plugin and UI components [merged]
#84 - Feat/docx import export suggestions comments [merged]
#97 - Revert staging to ebf750c [merged]
#126 - feat: DOCX tracked changes and comments export [merged]
#187 - Format MCP config and add Next.js experimental settings [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[arthrod]

🥕 Sibling variant(s): #348 (per-cleaner skip predicate) and #349 (extract-stash-reapply at importer). Same goal, shuffled assumptions across layer, failure mode, and round-trip fidelity. Please compare before recommending. Original framing: #346.

[coderabbitai]

Coding Plan

Summary

• Introduce a DOM placeholder swap mechanism that wraps the existing cleanDocx pipeline: protectTrackingTokens replaces token text with data-attribute spans before cleaners run, restoreTrackingTokens swaps them back afterward
• Zero-width space inside placeholder spans prevents empty-element detection while data attributes survive all cleaners (static analysis confirms no cleaner inspects data-*)
• Shared token constants in packages/utils establish consistent grammar for both @platejs/docx and @platejs/docx-io
• Restoration happens before postCleanHtml runs, so ZWS remnants don't appear in final output and tokens emerge as plain text

Design Choices

Design Choice 1: Placeholder spans with `data-docx-tracking-token` will survive all cleaners unchanged

Options Considered:

1. Trust that data-* spans pass through unchanged based on static analysis
2. Add explicit span preservation logic to each cleaner

Chosen Option: 1

Rationale: Static analysis confirms no cleaner reads data-* attributes, and the existing [[DOCX_COMMENT_REF:id]] token already survives as plain text through the pipeline, demonstrating the cleaners don't interfere with non-Office content.

Design Choice 2: Zero-width space as placeholder content prevents empty-element removal

Options Considered:

1. Use zero-width space (U+200B) as placeholder content
2. Use a visible placeholder character that gets stripped during restoration
3. Use an HTML comment as content

Chosen Option: 1

Rationale: U+200B survives .trim(), is invisible to users, and the restoration happens before postCleanHtml strips ZWS from the final output. The ZWS only exists temporarily within placeholder spans.

Design Choice 3: Token constant location for shared access

Options Considered:

1. Create packages/utils/src/lib/docx-tracking-tokens.ts as shared module
2. Create constants in packages/docx and have docx-io import from @platejs/docx

Chosen Option: 1

Rationale: Following the ticket's guidance, placing constants in packages/utils creates a neutral shared location that avoids any coupling concerns and follows the existing pattern where plate-keys.ts and plate-types.ts live in utils.

💡 User Tips

Regenerate the plan with different choices with @coderabbitai <feedback>.

Implementation Steps

Phase 1: Shared Token Constants Module

Establish the tracking token grammar as shared constants that both @platejs/docx (for protection/restoration) and @platejs/docx-io (for token generation) can import. This ensures consistent token patterns across the DOCX processing pipeline.

Task 1: Create tracking token constants and regex patterns

Define the complete vocabulary of tracking tokens with prefix constants, suffix constant, and a unified regex pattern for matching all token variants.

• Create packages/utils/src/lib/docx-tracking-tokens.ts with token prefix constants: DOCX_INS_START_PREFIX, DOCX_INS_END_PREFIX, DOCX_DEL_START_PREFIX, DOCX_DEL_END_PREFIX, DOCX_CMT_START_PREFIX, DOCX_CMT_END_PREFIX, DOCX_CMT_REF_PREFIX
• Define TRACKING_TOKEN_SUFFIX as ]] and unified TRACKING_TOKEN_REGEX pattern matching [[DOCX_(INS|DEL|CMT)_(START|END|REF):…]]
• Export helper functions buildTrackingToken(kind, payload) and parseTrackingToken(token) for consistent token construction/parsing
• Export type definitions for token kinds: TrackingTokenKind union type

Task 2: Export from utils package entry point

Make the new module accessible via the package's public API.

• Update packages/utils/src/index.ts to re-export all constants, types, and helpers from the new tracking tokens module

🤖 Prompt for AI agents

Create a new shared token constants module in
`packages/utils/src/lib/docx-tracking-tokens.ts` and export it from the package
entry point.

- In `packages/utils/src/lib/docx-tracking-tokens.ts`:
- Define token prefix constants: `DOCX_INS_START_PREFIX`, `DOCX_INS_END_PREFIX`,
`DOCX_DEL_START_PREFIX`, `DOCX_DEL_END_PREFIX`, `DOCX_CMT_START_PREFIX`,
`DOCX_CMT_END_PREFIX`, `DOCX_CMT_REF_PREFIX`
  - Define `TRACKING_TOKEN_SUFFIX` as `]]`
- Define `TRACKING_TOKEN_REGEX` as a unified pattern matching
`[[DOCX_(INS|DEL|CMT)_(START|END|REF):…]]`
- Export helper functions `buildTrackingToken(kind, payload)` and
`parseTrackingToken(token)` for consistent token construction/parsing
  - Export `TrackingTokenKind` union type
- In `packages/utils/src/index.ts`:
- Re-export all constants, types, and helpers from the new tracking tokens
module

Phase 2: DOM Protection and Restoration Functions

Create the core mechanism that swaps tracking tokens for inert placeholder spans before the cleaning pipeline, then restores them afterward. This isolates tokens from all cleaner transformations without modifying any existing cleaner logic.

Task 1: Implement protectTrackingTokens function

Walk the DOM and replace each tracking token text segment with a placeholder span that carries the original token in a data attribute.

• Create packages/docx/src/lib/docx-cleaner/utils/protectTrackingTokens.ts
• Use TreeWalker to iterate all text nodes in the body
• For each text node, use TRACKING_TOKEN_REGEX to find token matches
• Split text nodes around each token match using DOM's splitText() method
• Replace each token text node with <span data-docx-tracking-token="{base64}" data-docx-tracking-kind="{kind}">​</span> where the span contains U+200B (zero-width space)
• Base64-encode the original token bytes to preserve exact content including any special characters in JSON payloads
• Process text nodes in reverse order to avoid invalidating offsets during splits

Task 2: Implement restoreTrackingTokens function

Walk the DOM and swap all placeholder spans back to their original token text nodes.

• Create packages/docx/src/lib/docx-cleaner/utils/restoreTrackingTokens.ts
• Query all elements with [data-docx-tracking-token] attribute
• For each placeholder span, decode the base64 data-docx-tracking-token attribute value
• Create a text node with the decoded token string
• Replace the span element with the text node using replaceWith()
• Process in document order (restoration order doesn't affect correctness since each span is independent)

Task 3: Implement string-level pre-pass for edge cases

Handle rare cases where tokens appear in positions that DOMParser might mangle before DOM-level protection can run.

• Create packages/docx/src/lib/docx-cleaner/utils/preProtectTrackingTokens.ts
• Accept raw HTML string before DOMParser is called
• Identify tracking tokens using regex on the string
• For tokens containing characters that could interfere with HTML parsing (<, >, & in JSON payloads), HTML-entity-encode just those characters within the token
• Return the sanitized HTML string safe for DOMParser
• This is a defensive measure primarily for malformed inputs

🤖 Prompt for AI agents

Create three new utility files under `packages/docx/src/lib/docx-cleaner/utils/`
implementing the DOM protection/restoration mechanism and a string-level
pre-pass.

**`protectTrackingTokens.ts`:**
- Use TreeWalker to iterate all text nodes in the body
- For each text node, use `TRACKING_TOKEN_REGEX` to find token matches
- Split text nodes around each match using DOM's `splitText()`
- Replace each token text node with `<span data-docx-tracking-token="{base64}"
data-docx-tracking-kind="{kind}">​</span>` (U+200B as content)
- Base64-encode original token bytes to preserve special characters in JSON
payloads
- Process text nodes in reverse order to avoid offset invalidation during splits

**`restoreTrackingTokens.ts`:**
- Query all `[data-docx-tracking-token]` elements
- For each span, decode the base64 `data-docx-tracking-token` attribute value
- Create a text node with the decoded token string
- Replace the span with the text node using `replaceWith()`
- Process in document order (each span is independent)

**`preProtectTrackingTokens.ts`:**
- Accept a raw HTML string (before DOMParser is called)
- Use regex to identify tracking tokens in the string
- HTML-entity-encode only `<`, `>`, `&` characters found within token payloads
to prevent DOMParser mangling
- Return the sanitized HTML string
- This is a defensive measure for malformed inputs

Phase 3: Integration into cleanDocx Pipeline

Wire the protection/restoration functions into the existing cleaning pipeline without modifying any individual cleaner's logic.

Task 1: Integrate protection and restoration into cleanDocx

Modify the main cleaning function to wrap the existing pipeline with token protection.

• Update packages/docx/src/lib/docx-cleaner/cleanDocx.ts
• Call preProtectTrackingTokens(html) on the raw HTML string before passing to preCleanHtml
• Call protectTrackingTokens(body) immediately after DOMParser creates the body, before any cleaners run
• Call restoreTrackingTokens(body) after all cleaners complete but before the preformattedWrapper div is created
• This ensures tokens are plain text when copied to the wrapper and serialized, avoiding any ZWS remnants in final output
• The existing pipeline between protect/restore remains completely unchanged

Task 2: Add module exports

Ensure the new utilities are accessible from the package's public API if needed for testing or extension.

• Update packages/docx/src/lib/docx-cleaner/index.ts to export protectTrackingTokens, restoreTrackingTokens, and preProtectTrackingTokens
• Update packages/docx/src/index.ts if needed to expose the cleaner utilities

🤖 Prompt for AI agents

Wire the protection/restoration utilities into
`packages/docx/src/lib/docx-cleaner/cleanDocx.ts` and update the module exports,
without modifying any individual cleaner logic.

**In `cleanDocx.ts`:**
- Call `preProtectTrackingTokens(html)` on the raw HTML string before passing it
to `preCleanHtml`
- Call `protectTrackingTokens(body)` immediately after DOMParser creates the
body, before any cleaners run
- Call `restoreTrackingTokens(body)` after all cleaners complete but before the
`preformattedWrapper` div is created
- The existing pipeline between protect/restore must remain completely unchanged

**In `packages/docx/src/lib/docx-cleaner/index.ts`:**
- Export `protectTrackingTokens`, `restoreTrackingTokens`, and
`preProtectTrackingTokens`

**In `packages/docx/src/index.ts`:**
- Expose cleaner utilities if needed

Phase 4: Test Coverage

Validate the round-trip safety guarantee with comprehensive test fixtures covering normal cases and edge cases identified in the ticket.

Task 1: Create tracking token test fixtures

Build HTML input/output fixture pairs that exercise the protection mechanism across various scenarios.

• Create packages/docx/src/lib/docx-cleaner/__tests__/fixtures/tracking-tokens/ directory
• Add fixture: token in mso-styled paragraph (<p style="mso-...">[[DOCX_INS_START:...]]text[[DOCX_INS_END:...]]</p>)
• Add fixture: end-only paragraph (<p>[[DOCX_INS_END:abc]]</p>) to verify it doesn't get removed as empty
• Add fixture: adjacent tokens ([[DOCX_INS_END:a]][[DOCX_DEL_START:b]][[DOCX_CMT_REF:c]])
• Add fixture: JSON payload with special characters ({"author":"John \"Jack\" Doe","id":123})
• Add fixture: token near <o:p> empty wrapper to test interaction with cleanDocxEmptyParagraphs

Task 2: Create tracking token round-trip tests

Write test cases validating byte-for-byte token preservation through the cleaning pipeline.

• Create packages/docx/src/lib/docx-cleaner/__tests__/cleanDocx-tracking.spec.ts
• Test that each fixture's tokens survive cleanDocx unchanged (byte-for-byte comparison)
• Test that token-free HTML continues to work identically (run existing test cases through new code path)
• Test edge cases: tokens at start/end of text nodes, tokens spanning what would be multiple text nodes, nested structures
• Test malformed tokens are passed through unchanged (graceful handling of non-matching patterns)

Task 3: Verify existing tests remain unchanged

Ensure the protection mechanism doesn't affect token-free HTML processing.

• Run existing cleanDocx.spec.ts and cleanDocx.slow.ts tests to confirm all pass without modification
• Verify fixture outputs in __tests__/output/ remain identical
• The protection functions should be no-ops when no tracking tokens are present

🤖 Prompt for AI agents

Add test fixtures and test cases to validate byte-for-byte token round-trip
safety through the `cleanDocx` pipeline.

**Create fixture directory and files at
`packages/docx/src/lib/docx-cleaner/__tests__/fixtures/tracking-tokens/`:**
- Token in mso-styled paragraph: `<p
style="mso-...">[[DOCX_INS_START:...]]text[[DOCX_INS_END:...]]</p>`
- End-only paragraph: `<p>[[DOCX_INS_END:abc]]</p>` (verify not removed as
empty)
- Adjacent tokens: `[[DOCX_INS_END:a]][[DOCX_DEL_START:b]][[DOCX_CMT_REF:c]]`
- JSON payload with special characters: `{"author":"John \"Jack\"
Doe","id":123}`
- Token near `<o:p>` empty wrapper (tests interaction with
`cleanDocxEmptyParagraphs`)

**Create
`packages/docx/src/lib/docx-cleaner/__tests__/cleanDocx-tracking.spec.ts`:**
- Test each fixture's tokens survive `cleanDocx` unchanged (byte-for-byte
comparison)
- Test token-free HTML continues to work identically
- Test edge cases: tokens at start/end of text nodes, tokens spanning multiple
text nodes, nested structures
- Test malformed tokens are passed through unchanged (graceful handling)

**Verify existing tests:**
- Confirm all existing `cleanDocx.spec.ts` and `cleanDocx.slow.ts` tests pass
without modification
- Verify fixture outputs in `__tests__/output/` remain identical
- Protection functions must be no-ops when no tracking tokens are present

Research

The packages/docx package contains the cleanDocx pipeline which runs 12 sequential cleaners on a parsed DOM body to normalize DOCX/Office HTML artifacts. The cleaners operate on elements by tag name, class, or inline styles—none inspect data-* attributes. The cleanHtmlEmptyElements cleaner removes elements where innerHTML.trim() === '' but does not strip zero-width spaces (U+200B) since JavaScript's .trim() only affects whitespace. The postCleanHtml function at pipeline end strips U+200B from the final HTML string. Currently, only [[DOCX_COMMENT_REF:id]] tokens exist (created in docx-io's preprocessMammothHtml); the new DOCX_INS_*, DOCX_DEL_*, DOCX_CMT_* token grammar does not exist yet.

---

🚀 Next Steps

🤖 All AI agent prompts combined

Task: 1

Create a new shared token constants module in
`packages/utils/src/lib/docx-tracking-tokens.ts` and export it from the package
entry point.

- In `packages/utils/src/lib/docx-tracking-tokens.ts`:
- Define token prefix constants: `DOCX_INS_START_PREFIX`, `DOCX_INS_END_PREFIX`,
`DOCX_DEL_START_PREFIX`, `DOCX_DEL_END_PREFIX`, `DOCX_CMT_START_PREFIX`,
`DOCX_CMT_END_PREFIX`, `DOCX_CMT_REF_PREFIX`
  - Define `TRACKING_TOKEN_SUFFIX` as `]]`
- Define `TRACKING_TOKEN_REGEX` as a unified pattern matching
`[[DOCX_(INS|DEL|CMT)_(START|END|REF):…]]`
- Export helper functions `buildTrackingToken(kind, payload)` and
`parseTrackingToken(token)` for consistent token construction/parsing
  - Export `TrackingTokenKind` union type
- In `packages/utils/src/index.ts`:
- Re-export all constants, types, and helpers from the new tracking tokens
module
===============================================================================

Task: 2

Create three new utility files under `packages/docx/src/lib/docx-cleaner/utils/`
implementing the DOM protection/restoration mechanism and a string-level
pre-pass.

**`protectTrackingTokens.ts`:**
- Use TreeWalker to iterate all text nodes in the body
- For each text node, use `TRACKING_TOKEN_REGEX` to find token matches
- Split text nodes around each match using DOM's `splitText()`
- Replace each token text node with `<span data-docx-tracking-token="{base64}"
data-docx-tracking-kind="{kind}">​</span>` (U+200B as content)
- Base64-encode original token bytes to preserve special characters in JSON
payloads
- Process text nodes in reverse order to avoid offset invalidation during splits

**`restoreTrackingTokens.ts`:**
- Query all `[data-docx-tracking-token]` elements
- For each span, decode the base64 `data-docx-tracking-token` attribute value
- Create a text node with the decoded token string
- Replace the span with the text node using `replaceWith()`
- Process in document order (each span is independent)

**`preProtectTrackingTokens.ts`:**
- Accept a raw HTML string (before DOMParser is called)
- Use regex to identify tracking tokens in the string
- HTML-entity-encode only `<`, `>`, `&` characters found within token payloads
to prevent DOMParser mangling
- Return the sanitized HTML string
- This is a defensive measure for malformed inputs
===============================================================================

Task: 3

Wire the protection/restoration utilities into
`packages/docx/src/lib/docx-cleaner/cleanDocx.ts` and update the module exports,
without modifying any individual cleaner logic.

**In `cleanDocx.ts`:**
- Call `preProtectTrackingTokens(html)` on the raw HTML string before passing it
to `preCleanHtml`
- Call `protectTrackingTokens(body)` immediately after DOMParser creates the
body, before any cleaners run
- Call `restoreTrackingTokens(body)` after all cleaners complete but before the
`preformattedWrapper` div is created
- The existing pipeline between protect/restore must remain completely unchanged

**In `packages/docx/src/lib/docx-cleaner/index.ts`:**
- Export `protectTrackingTokens`, `restoreTrackingTokens`, and
`preProtectTrackingTokens`

**In `packages/docx/src/index.ts`:**
- Expose cleaner utilities if needed
===============================================================================

Task: 4

Add test fixtures and test cases to validate byte-for-byte token round-trip
safety through the `cleanDocx` pipeline.

**Create fixture directory and files at
`packages/docx/src/lib/docx-cleaner/__tests__/fixtures/tracking-tokens/`:**
- Token in mso-styled paragraph: `<p
style="mso-...">[[DOCX_INS_START:...]]text[[DOCX_INS_END:...]]</p>`
- End-only paragraph: `<p>[[DOCX_INS_END:abc]]</p>` (verify not removed as
empty)
- Adjacent tokens: `[[DOCX_INS_END:a]][[DOCX_DEL_START:b]][[DOCX_CMT_REF:c]]`
- JSON payload with special characters: `{"author":"John \"Jack\"
Doe","id":123}`
- Token near `<o:p>` empty wrapper (tests interaction with
`cleanDocxEmptyParagraphs`)

**Create
`packages/docx/src/lib/docx-cleaner/__tests__/cleanDocx-tracking.spec.ts`:**
- Test each fixture's tokens survive `cleanDocx` unchanged (byte-for-byte
comparison)
- Test token-free HTML continues to work identically
- Test edge cases: tokens at start/end of text nodes, tokens spanning multiple
text nodes, nested structures
- Test malformed tokens are passed through unchanged (graceful handling)

**Verify existing tests:**
- Confirm all existing `cleanDocx.spec.ts` and `cleanDocx.slow.ts` tests pass
without modification
- Verify fixture outputs in `__tests__/output/` remain identical
- Protection functions must be no-ops when no tracking tokens are present

💡 Iterate on the plan with: @coderabbitai <feedback>

Example Feedback
- `@coderabbitai` You can skip phase 3. Add a simple unit test case for phase 2.
- `@coderabbitai` For design choice 1 go ahead with option 3 and replan.

---

💬 Have feedback or questions? Drop into our discord!

[arthrod]

@coderabbitai of these 3 framings (#347 / #348 / #349), which single assumption is load-bearing?
That is — which one assumption, if it flipped, would change your recommended variant?
Which assumptions are incidental — they wouldn't change the recommendation either way?

Your reply on #348 named one axis ("If compile-time regression prevention for new cleaners matters more than feature-flag reversibility, variant B wins; if feature-flag rollback capability matters more than compile-time guarantees, variants A or C win"). That collapses A and C — but A and C differ on a separate axis (whether tracking-token awareness lives inside @platejs/docx or outside in the importer caller; equivalently, whether cleanDocx ever sees tokens at all).

Name the load-bearing assumption as a single sentence in the form
"If matters more than , variant <A|B|C> wins; otherwise the other."
Pick the axis that disambiguates all three variants — not just B vs. the rest. Compare the plans you produced on each issue. Don't pick a winner yet — name the axis.

[coderabbitai]

@arthrod, here is the axis:

---

"If the correctness guarantee for token preservation should live as a contract inside the @platejs/docx library (so every caller gets it for free) matters more than keeping cleanDocx completely token-agnostic (so the guarantee is the importer caller's responsibility), variants A (#347) or B (#348) win; otherwise C (#349) wins."

---

Why this axis is load-bearing:

It determines where the contract boundary sits. If it flips — from library-owned to caller-owned — the entire architecture reverses: cleanDocx stays a pure HTML normalizer with no token grammar baked in, and every importer integration must independently implement stash/restore. That is a fundamentally different system, not just a different implementation detail.

Why the A vs. B axis is incidental by comparison:

Once you've settled that cleanDocx owns the guarantee (the library side of the load-bearing flip), the A vs. B choice — runtime pipeline wrapper (#347) vs. compile-time per-cleaner predicate (#348) — is a matter of enforcement mechanism within that already-chosen architecture. Flipping it changes how future-cleaner regressions are caught (at test runtime vs. at type-check time), but it doesn't change who is responsible for the guarantee or where the contract lives. That makes it a secondary axis.

Summary table:

Variant	Token-awareness owner	Enforcement style
A (#347)	cleanDocx (library)	Runtime wrapper
B (#348)	cleanDocx (library)	Compile-time predicate
C (#349)	Importer (caller)	Runtime stash/restore

The load-bearing flip is the row boundary between (A, B) and C. The A vs. B boundary is the incidental one.

[arthrod]

🥕 Evidence that variant A (DOM placeholder swap) does not mangle the four token kinds (INS-START/END, DEL-START/END, CMT-START/END, CMT-REF) — proving placeholder elements with data-docx-tracking-token attributes are invisible to every cleaner in cleanDocx's pipeline.

Method: read every cleaner in packages/docx/src/lib/docx-cleaner/cleanDocx.ts plus its platejs-core dependencies and classify what each one inspects.

Cleaner	Match predicate	Mutates	Affects placeholder <span data-docx-tracking-token="..." data-docx-tracking-kind="..."><U+200B></span>?
cleanDocxFootnotes (docx-cleaner/utils/cleanDocxFootnotes.ts)	isDocxFootnote(element) — Word footnote shape	replaces with <sup>	No. Placeholder is <span>, not footnote shape.
cleanDocxImageElements	tagName ∈ {IMG, V:IMAGEDATA}	sets src from RTF map	No. Tag-name match excludes spans.
cleanHtmlEmptyElements (core/.../cleanHtmlEmptyElements.ts)	!element.innerHTML.trim() and tag ∉ {BR,IMG,TD,TH}	element.remove()	No. '​'.trim() === '​' (zero-width space is not in JS .trim()'s whitespace set per ECMAScript WhiteSpace production), so innerHTML.trim() is non-empty → placeholder survives, parent survives.
cleanDocxEmptyParagraphs	<p> whose only child is <O:P> containing NO_BREAK_SPACE	innerHTML = ''	No. Placeholder is a <span>, never an <O:P> child. Predicate misses.
cleanDocxQuotes	<p.MsoQuote>	renames to blockquote	No. Renames container, preserves all children including placeholders.
cleanDocxSpans → cleanDocxSpacerun	tag === SPAN AND style ∈ {"mso-spacerun: yes","mso-spacerun:yes"}	replaces span with N space text nodes	No. Placeholder span has no style attribute, only data-*. Predicate filters out by exact style match.
cleanDocxSpans → cleanDocxTabCount	tag === SPAN AND style.startsWith("mso-tab-count:")	replaces span with N tabs	No. Same as above.
cleanHtmlTextNodes (core/.../cleanHtmlTextNodes.ts)	text nodes only (traverseHtmlTexts)	NBSP→SPACE, \n\s*→\n, removes whitespace-only nodes	No. Placeholder's only descendant is a single ​ text node. ZWSP is none of {NBSP, LF, CR, SPACE} per the cleaner's checks; the regex ^\n\s*$ doesn't match ​. Text node survives byte-for-byte. The original token text is gone (replaced by placeholders pre-pipeline) so tokens never reach this cleaner.
cleanDocxBrComments	<br> followed by <!--[if !supportLineBreakNewLine]--> comment	removes nodes between conditionals	No. Match anchored on <br> + IE conditional comment, not on placeholder spans.
cleanHtmlBrElements	<br>	replaces with \n text	No. Tag-name match excludes spans.
cleanHtmlLinkElements	<a>	unwraps if no href	No. Tag-name match excludes spans.
cleanHtmlFontElements	<FONT>	renames to <span>	No. Tag-name match excludes existing spans.
cleanDocxListElements	element with style attribute	normalizes mso-list:ignore casing	Inert. Touches style attribute only; placeholder has no style.
copyBlockMarksToSpanChild	block elements with style containing color/font	wraps a child span with a new mark span	No. Placeholder survives as-is; if its parent is a block with style, the cleaner adds a SIBLING mark span (creates a new wrapper around the first non-text child) — placeholder's text content unchanged, position unchanged.

Conclusion: of the 14 cleaners in the pipeline, 12 are inert against placeholder spans by tag/predicate, and 2 (cleanHtmlEmptyElements, cleanHtmlTextNodes) are inert because placeholders carry a non-trim-whitespace ZWSP and data-* attributes that ECMA-defined .trim() and the cleaner's regex set don't classify as empty/whitespace.

Per-token-kind evidence for round-trip:

• INS / DEL (range tokens): Each START/END pair stashes the JSON payload in data-docx-tracking-token (base64). Pair association is by data-docx-tracking-id cross-attribute on the matching START/END placeholders. Adjacency is preserved because the pipeline never reorders <span> siblings without a tag/style match.
• CMT-START / CMT-END (comment ranges): Same mechanism as INS/DEL. paraId and parentParaId from the JSON payload survive byte-for-byte because base64 is opaque.
• CMT-REF (point comments): Single placeholder, no pair association needed. Restored as a single [[DOCX_CMT_REF:id]] text node post-pipeline.

Adversarial fixtures that prove the restore step works in the worst cases (each becomes a spec in cleanDocx-tracking.spec.ts):

1. Three back-to-back tokens: <p>x [[DOCX_INS_END:a]][[DOCX_DEL_START:{"id":"b","author":"Jane Doe","date":"2026-01-01T00:00:00Z"}]][[DOCX_CMT_REF:c]]y</p> — all three placeholders survive cleanHtmlTextNodes's text-node merging (placeholders are elements, not text); post-restore the three tokens are adjacent text nodes again.
2. Token inside <p class="MsoNormal"> with mso-* styles — cleanDocxSpans operates on inner spans only; the <p>'s class/style is normalized but placeholder spans untouched.
3. Token inside a <font> element that gets renamed to <span> — placeholder is preserved as a child of the renamed element.
4. Empty paragraph from Word's <p><o:p>&nbsp;</o:p></p> directly above a token-bearing paragraph — cleanDocxEmptyParagraphs clears the empty one; the next paragraph's tokens are unaffected.

Files to add:

• packages/docx/src/lib/docx-cleaner/__tests__/cleanDocx-tracking.spec.ts — fixtures 1-4 above plus a snapshot test that the cleaner output equals the original token text after restoreTrackingTokens.

If @coderabbitai's implementation phase agrees this evidence holds, the only remaining risk for variant A is a future cleaner being added that traverses by data-* attributes — guard against it with a CI check that grep -rn 'data-docx-tracking-token' matches only protectTrackingTokens.ts and restoreTrackingTokens.ts.

[coderabbitai]

🧩 Analysis chain

The analysis is thorough — let me verify the critical cleaner implementations against the actual source before confirming.

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# List all cleaners in the docx-cleaner directory
echo "=== cleanDocx.ts pipeline ==="
fd -t f 'cleanDocx.ts' --exec cat {}

echo ""
echo "=== cleanHtmlEmptyElements ==="
fd -t f 'cleanHtmlEmptyElements.ts' --exec cat {}

echo ""
echo "=== cleanHtmlTextNodes ==="
fd -t f 'cleanHtmlTextNodes.ts' --exec cat {}

echo ""
echo "=== cleanDocxSpans ==="
fd -t f 'cleanDocxSpans.ts' --exec cat {}

echo ""
echo "=== copyBlockMarksToSpanChild ==="
fd -t f 'copyBlockMarksToSpanChild.ts' --exec cat {}

echo ""
echo "=== cleanDocxListElements ==="
fd -t f 'cleanDocxListElements.ts' --exec cat {}

echo ""
echo "=== cleanDocxEmptyParagraphs ==="
fd -t f 'cleanDocxEmptyParagraphs.ts' --exec cat {}

Length of output: 7578

---

The source confirms most claims. One detail needs deeper verification — cleanDocxSpans enters placeholder spans (since they are SPAN elements) and calls cleanDocxSpacerun/cleanDocxTabCount on them. I need to confirm those sub-cleaners' predicates before the analysis is complete.

---

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "=== cleanDocxSpacerun ==="
fd -t f 'cleanDocxSpacerun.ts' --exec cat {}

echo ""
echo "=== cleanDocxTabCount ==="
fd -t f 'cleanDocxTabCount.ts' --exec cat {}

echo ""
echo "=== postCleanHtml (to verify ZWSP stripping timing) ==="
fd -t f 'postCleanHtml.ts' --exec cat {}

echo ""
echo "=== preCleanHtml ==="
fd -t f 'preCleanHtml.ts' --exec cat {}

echo ""
echo "=== traverseHtmlTexts (to understand text-node traversal scope) ==="
fd -t f 'traverseHtmlTexts.ts' --exec cat {}

echo ""
echo "=== isDocxFootnote ==="
fd -t f 'cleanDocxFootnotes.ts' --exec cat {}

Length of output: 3257

---

@arthrod, the source confirms your analysis with one factual correction and two implementation details worth pinning precisely.

---

Verification result: 13/14 cleaners confirmed correct

Every cleaner is inert against <span data-docx-tracking-token="..." data-docx-tracking-kind="...">​</span> placeholders. Evidence from source:

Cleaner	Verification from source
cleanDocxSpacerun	['mso-spacerun: yes', 'mso-spacerun:yes'].includes(styleAttribute) — exact full-value match; placeholder has no style → null → early return ✅
cleanDocxTabCount	(getAttribute('style') || '').startsWith('mso-tab-count:') → ''.startsWith(…) is false → early return ✅
cleanHtmlEmptyElements	!element.innerHTML.trim() — U+200B is ECMAScript category Cf, not Zs/WhiteSpace; .trim() never strips it ✅
cleanHtmlTextNodes	Branches on CARRIAGE_RETURN, LINE_FEED, NO_BREAK_SPACE only; /^\n\s*$/ where \s covers \u2000–\u200A (Zs) but NOT \u200B (Cf) → ZWSP text node survives unchanged ✅
cleanDocxListElements	Only mutates the style attribute string value; placeholder has no style → if (styleAttribute) skips ✅
cleanDocxEmptyParagraphs	isHtmlElementEmpty recurses exclusively on <o:p> children with NO_BREAK_SPACE text content; placeholder is <span> with ZWSP → predicate misses ✅
postCleanHtml	Strips ALL ZERO_WIDTH_SPACE globally — but this runs on preformattedWrapper.outerHTML after body.innerHTML is serialized; if restoration happens before preformattedWrapper.innerHTML = body.innerHTML, no ZWSP ever reaches postCleanHtml ✅ (sequencing is load-bearing — see below)

---

One factual correction: copyBlockMarksToSpanChild

Your analysis says: "the cleaner adds a SIBLING mark span."

The actual source does this:

span.innerHTML = el.innerHTML;   // copies ALL children into new span
element.innerHTML = span.outerHTML; // replaces block's entire innerHTML

This is not a sibling addition — it is a full innerHTML round-trip that wraps all children of the block in a new <span>. The safety claim is still correct (data-* attributes and ZWSP survive HTML serialization/deserialization), but the mechanism description is wrong. This matters for the spec because adversarial fixture 2 (token inside <p class="MsoNormal"> with mso-* styles) will exercise this path — the placeholder span will be recreated as a new DOM node. restoreTrackingTokens must query [data-docx-tracking-token] after all cleaners complete, which it does, so the recreated node is still found. No bug, but the spec comment should describe what actually happens.

---

One load-bearing sequencing constraint

postCleanHtml strips ALL ZERO_WIDTH_SPACE globally from the final string. The ZWSP inside placeholder spans is safe only because the implementation plan calls restoreTrackingTokens(body) before preformattedWrapper.innerHTML = body.innerHTML. If that call is ever moved to after the preformattedWrapper assignment, ZWSP from placeholders bleeds into preformattedWrapper.outerHTML and is then stripped by postCleanHtml — but by then the placeholders are gone and the token text is already restored, so the stripped ZWSP would be from placeholder spans that are no longer present anyway. Either way safe, but the comment in cleanDocx.ts should make the sequencing explicit.

---

CI guard: endorsed with a tighter form

The grep guard is the right call. Suggested form for the CI check:

# Ensure data-docx-tracking-token is only referenced in the protect/restore utilities
MATCHES=$(grep -rn --include='*.ts' 'data-docx-tracking-token' . \
  | grep -v 'protectTrackingTokens.ts' \
  | grep -v 'restoreTrackingTokens.ts' \
  | grep -v '.spec.ts' \
  | grep -v '.test.ts')

if [ -n "$MATCHES" ]; then
  echo "ERROR: data-docx-tracking-token referenced outside protect/restore utilities:"
  echo "$MATCHES"
  exit 1
fi

The test file exclusion is important — specs will reference the attribute in fixture HTML strings and that's intentional.

---

Summary

The variant A safety argument holds on all 14 cleaners. The one description error (copyBlockMarksToSpanChild mechanism) doesn't affect correctness but should be corrected in the spec comments. The sequencing constraint on restoreTrackingTokens placement should be documented inline. The CI guard is a necessary complement to the static analysis — add it to Phase 4 of the implementation plan as a fifth task.