helix-translate-pipeline

A glossary-protected, resumable localization pipeline for Markdown content. It translates *.md files from English into any target language over the HelixTranslate engine, protects non-translatable terms so they survive translation verbatim, keeps YAML frontmatter byte-identical, and then runs an independent context-and-spirit review with a different model than the one that produced the translation.

It is a faithful decoupling of the pipeline defined by the HelixConstitution clauses §11.4.140 (all translation through the engine, fail loud, no silent fallback), §11.4.141 (independent review by a different model), and §11.4.236 / §11.4.235 (context + spirit review across seven dimensions).

---

What it does

For every input Markdown file, translate-content.sh:

1. Splits YAML frontmatter from the body. The frontmatter is preserved byte-identical in the output (never translated). The body split keys off the first two --- fences only, so --- horizontal rules inside the body are safe.
2. Protects every glossary term in the body by replacing it with a unique Z9TERM<NNN>Z9 sentinel that survives LLM translation intact (glossary_protect.py protect). Matching is whole-token and case-sensitive, longest-term-first, so Model Context Protocol is protected before MCP and the acronym AI never matches inside email.
3. Translates the protected body en -> <lang> with HelixTranslate, in paragraph-sized chunks (blank-line boundaries under MAXCHARS) to stay under the engine's single-request output cliff. Sentinels always sit within a line and are never split.
4. Restores every sentinel back to its original term (glossary_protect.py restore).
5. Reassembles byte-identical frontmatter + translated body, with a length guard that fails loud if the body came back grossly shorter than the source (catches residual truncation).
6. Writes the result to $OUT_ROOT/<same relative path>.
7. Reviews the output with an independent model (a different provider/model than the translator) and emits a strict PASS/FAIL JSON verdict (review_translation.py). See REVIEW.md.

run-batch.sh wraps the driver with resumable, capped-concurrency orchestration across many languages and documents. See USAGE.md.

---

The fail-loud, no-silent-fallback contract

This pipeline is deliberately loud. It never silently degrades:

• All translation goes through the HelixTranslate engine. There is no hand-translation and no fallback engine. If the engine cannot produce a chunk after ENGINE_RETRIES transient retries (rate-limit / 5xx / timeout, with exponential backoff), the run aborts with a fatal error — it does not emit a partial or improvised translation. (§11.4.140)
• Retries are not a fallback. Retrying the same engine/provider on a transient failure keeps §11.4.140 satisfied because only HelixTranslate ever produces the text. Only a hard, non-transient exhaustion fails loud.
• Truncation is caught, not accepted. Per-chunk and whole-body length guards abort if output is suspiciously short.
• Sentinel integrity is enforced. restore exits non-zero (fail loud) if any Z9TERM-shaped token is left unmapped, or if a mapped sentinel disappeared — both indicate the engine mangled a protected term.
• The review makes a real API call. A missing key or failed call yields verdict=ERROR — never a faked PASS. The verdict is re-computed in Python from the numeric scores, so the gate cannot be talked past by the model.
• Exit code is honest. The driver exits 0 iff every file translated and every review verdict is PASS.

---

The HelixTranslate engine dependency

The translation engine — HelixTranslate — is an external dependency and is NOT vendored in this repository. This pipeline shells out to a prebuilt helixtranslate binary via $HELIX_BIN.

Build it from the helix_translate repository:

# in a checkout of the helix_translate repo
CGO_ENABLED=1 go build -o helixtranslate ./cmd/unified-translator

CGO_ENABLED=1 is required (the engine links native code). Then point the pipeline at the binary:

export HELIX_BIN=/absolute/path/to/helixtranslate
# or drop it as ./helixtranslate next to where you run the pipeline (the default)

The pipeline invokes it per chunk as:

helixtranslate -i <in.md> -o <out.md> \
  -provider <provider> -model <model> -api-key <key> \
  -source-lang en -target-lang <lang> -script <script> -verify=false -timeout 120s

If $HELIX_BIN is missing or not executable, the driver fails loud immediately.

---

API keys — BYOK (bring your own key)

Keys are read from the environment; nothing is bundled or committed.

• Translator (default provider mistral): MISTRAL_API_KEY — required.
• Reviewer (default provider groq): GROQ_API_KEY — required for the review step. Other supported reviewer providers and their key env vars: openrouter → OPENROUTER_API_KEY, mistral → MISTRAL_API_KEY, zhipu → ZHIPU_API_KEY, cohere → COHERE_API_KEY.

As a convenience, if $HOME/api_keys.sh exists it is sourced automatically; otherwise export the keys yourself:

export MISTRAL_API_KEY=...
export GROQ_API_KEY=...

The reviewer must be a different model than the translator (§11.4.141). The defaults already satisfy this (translator = Mistral, reviewer = Groq/Llama).

---

Non-translatable glossary protection

Proper nouns, product names, technologies, and standards must appear verbatim in every language — PostgreSQL must not become ПостгреСКЛ, MCP must not be translated to a phrase. glossary.json lists these terms; the protect/restore sentinel mechanism guarantees they pass through translation untouched, and the reviewer independently verifies preservation (terms_preserved). See GLOSSARY.md for the format and how to extend it.

---

Repository layout

helix-translate-pipeline/
├── README.md                  # this file
├── GLOSSARY.md                # glossary.json format + how to extend
├── REVIEW.md                  # the independent review dimensions
├── USAGE.md                   # CLI usage, batch orchestration, rate-limit lesson
├── LICENSE
├── glossary.json              # non-translatable terms
└── bin/
    ├── translate-content.sh   # the driver: protect -> translate -> restore -> review
    ├── run-batch.sh           # resumable, capped-concurrency batch orchestrator
    ├── glossary_protect.py    # sentinel protect/restore
    └── review_translation.py  # independent context+spirit reviewer

Quick start

# 1. Build the engine (see above) and point HELIX_BIN at it.
export HELIX_BIN=/path/to/helixtranslate
# 2. Provide keys.
export MISTRAL_API_KEY=...   GROQ_API_KEY=...
# 3. Put your English *.md under ./content, then translate to German:
bin/translate-content.sh de
# -> writes ./content_de/**, review JSON under ./evidence/de/

See USAGE.md for everything else.

---

Provenance

Decoupled from the vasic monorepo's _tools/translate/ pipeline. Constitution clauses referenced: §11.4.140 (engine-only, fail loud), §11.4.141 (independent review by a different model), §11.4.236 / §11.4.235 (context + spirit review).