Skip to content

mixed_locale_metadata.md

Latest commit

 

History

History
174 lines (129 loc) · 9.55 KB

mixed_locale_metadata.md

File metadata and controls

174 lines (129 loc) · 9.55 KB

Mixed Locale Metadata — Guardrails and Fix Pattern

🧭 Quick Return to Map

You are in a sub-page of LanguageLocale.
To reorient, go back here:

Think of this page as a desk within a ward.
If you need the full triage and all prescriptions, return to the Emergency Room lobby.

A focused fix when language, script, or region metadata is inconsistent across files, snippets, headers, analyzers, or indexes. Use this page to normalize BCP-47 tags, align analyzers and encoders, and lock the schema so retrieval stays stable.

Open these first

When to use this page

  • Corpus claims lang=en in headers while HTML meta, PDF tags, or file paths imply non-English.
  • Index analyzer set to standard but crawler stamped lang=zh-Hant or lang=ja.
  • Mixed lang values inside one document after conversions or merges.
  • Query comes from a UI locale while the text is in another, leading to wrong analyzer or stopword set.
  • Re-ingest does not update stale lang/script/region fields and recall becomes unstable.

Acceptance targets

  • ΔS(question, retrieved) ≤ 0.45 on three paraphrases.
  • Coverage ≥ 0.70 to the intended section.
  • λ stays convergent across two seeds after metadata normalization.
  • E_resonance flat on long windows after analyzer alignment.

The core problem

Most stacks carry only lang and drop script and region, or they conflate UI locale with content locale. Encoders and analyzers then guess. That guess diverges from retrieval, chunking, and ranking, which raises ΔS and flips λ.

60-second fix

  1. Detect

    • Run lightweight LID on both raw source and post-parser text. Keep the probability.
    • If p_top − p_second < 0.20, mark the snippet mixed and route to re-chunk.

  2. Normalize

    • Stamp BCP-47 tags as three fields: lang, script, region where known.
    • Normalize Unicode to NFC for CJK and most Latin, NFKC only when you must collapse width.

  3. Align analyzers and encoders

    • Choose analyzer by (lang, script) rather than UI locale.
    • Record analyzer_id, encoder_id, normal_form in the snippet.

  4. Verify

    • Re-rank with a deterministic reranker.
    • Recompute ΔS and λ across 3 paraphrases, two seeds. Ship if targets pass.

Minimal contract you must carry

Add these keys to your snippet schema. See the full table in Data Contracts.

{
  "snippet_id": "doc:123#p4",
  "lang": "zh",
  "script": "Hant",
  "region": "TW",
  "normal_form": "NFC",
  "analyzer_id": "icu_zh_Hant",
  "encoder_id": "e5-multilingual",
  "lid": { "label": "zh-Hant", "p": 0.93, "method": "cld3|fasttext" },
  "source_meta": { "declared_lang": "en-US", "html_meta_lang": "zh-Hant", "pdf_lang": "zh" }
}

Rules

  • Prefer content-derived LID over HTTP headers or file paths.
  • If declared_lang disagrees with LID, trust LID and record the disagreement.
  • Never mix snippets with different (lang, script) into the same chunk group.

Quick diagnosis → fix

  • Answers oscillate per run although facts exist → The analyzer follows UI locale. Lock analyzer by snippet fields. See Retrieval Traceability.

  • High similarity yet wrong meaning → Encoded with a multilingual model but index analyzer used wrong token rules. Rebuild with aligned analyzer. See embedding-vs-semantic.md.

  • Mixed half width, punctuation, or digits break matching → Normalize to NFC or NFKC per policy and re-index. See digits_width_punctuation.md.

  • Script switches mid-paragraph → Split at the switch and stamp each part. See script_mixing.md.

  • CJK recall dips after parser change → Check wordbreak rules and ICU configs. See cjk_segmentation_wordbreak.md.

Copy-paste prompt for your LLM step

I uploaded TXT OS and WFGY Problem Map.

My issue looks like mixed locale metadata:
- declared vs detected lang differ,
- analyzer_id does not match (lang, script),
- ΔS(question, retrieved) = {value}, λ = {state}

Tell me:
1) which layer is failing and why,
2) the exact WFGY page to open,
3) the minimal steps to align BCP-47, analyzer, and encoder,
4) a reproducible test that verifies ΔS ≤ 0.45 and λ convergent.
Use BBMC/BBCR/BBAM when relevant.

🔗 Quick-Start Downloads (60 sec)

Tool Link 3-Step Setup
WFGY 1.0 PDF Engine Paper 1️⃣ Download · 2️⃣ Upload to your LLM · 3️⃣ Ask “Answer using WFGY + <your question>”
TXT OS (plain-text OS) TXTOS.txt 1️⃣ Download · 2️⃣ Paste into any LLM chat · 3️⃣ Type “hello world” — OS boots instantly

Explore More

Layer Page What it’s for
⭐ Proof WFGY Recognition Map External citations, integrations, and ecosystem proof
⚙️ Engine WFGY 1.0 Original PDF tension engine and early logic sketch (legacy reference)
⚙️ Engine WFGY 2.0 Production tension kernel for RAG and agent systems
⚙️ Engine WFGY 3.0 TXT based Singularity tension engine (131 S class set)
🗺️ Map Problem Map 1.0 Flagship 16 problem RAG failure taxonomy and fix map
🗺️ Map Problem Map 2.0 Global Debug Card for RAG and agent pipeline diagnosis
🗺️ Map Problem Map 3.0 Global AI troubleshooting atlas and failure pattern map
🧰 App TXT OS .txt semantic OS with fast bootstrap
🧰 App Blah Blah Blah Abstract and paradox Q&A built on TXT OS
🧰 App Blur Blur Blur Text to image generation with semantic control
🏡 Onboarding Starter Village Guided entry point for new users

If this repository helped, starring it improves discovery so more builders can find the docs and tools.
GitHub Repo stars