eroll translit pipe

Author: soodoku

.gitignore

@@ -0,0 +1,29 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.venv/
+.mypy_cache/
+.ruff_cache/
+.pytest_cache/
+htmlcov/
+.coverage
+coverage.xml
+
+# Build/runtime artifacts (any location)
+*.7z
+*.parquet
+*_tokens.jsonl
+*.batchstate.json
+
+# data/: track README + the per-language word-pair corpora (small .csv.gz, <100MB).
+# Large raw rolls (>100MB) and derived files stay out of git -> host on Dataverse
+# (GitHub hard-rejects files >100MB).
+/data/*
+!/data/README.md
+!/data/*.csv.gz
+/data/*_all_clean*.csv.gz
+/data/*_train_v2.csv.gz

Makefile

@@ -0,0 +1,25 @@
+.PHONY: help install dev-install test lint format type-check ci
+
+help: ## Show this help message
+	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*?## "}; {printf "  %-14s %s\n", $$1, $$2}'
+
+install: ## Install package
+	uv sync
+
+dev-install: ## Install with dev dependencies
+	uv sync --group dev
+
+test: ## Run tests
+	uv run python -m pytest
+
+lint: ## Run linter
+	uv run ruff check eroll/ tests/
+
+format: ## Format code
+	uv run black eroll/ tests/
+	uv run isort eroll/ tests/
+
+type-check: ## Run type checker
+	uv run mypy eroll/
+
+ci: lint type-check test ## Run all CI checks

README.md

@@ -0,0 +1,61 @@
+# eroll_transliteration
+
+Reusable pipeline for transliterating Indian **electoral rolls** from native script to
+English, building high-quality `(native, english)` training corpora and improved
+romanized rolls. The LLM step runs in **batch mode** through the
+[`indicate`](https://github.com/in-rolls/indicate) core library; this repo holds the
+roll-specific glue (per-state config, streaming IO, join-back, corpus extraction).
+
+**Forward direction only** (Indic → English). Rolls that are already romanized English
+(e.g. Andhra) are out of scope — there is no native source side to learn from.
+
+## Install
+
+```bash
+uv sync                # uses ../indicate as an editable path dependency (see pyproject)
+```
+
+Set the chosen provider's API key before any LLM step (batch via litellm — OpenAI by
+default; Anthropic is not a batch provider in litellm, use Bedrock for Claude):
+
+```bash
+export OPENAI_API_KEY=...        # or configure another litellm batch provider
+export EROLL_DATA_DIR=/path/to/rolls   # defaults to ./data
+```
+
+## Usage
+
+```bash
+# 1. Profile: unique-token counts + batch-cost estimate (no API calls)
+python -m eroll.transliterate_rolls --state assam profile
+
+# 2. Sample-review 300 random tokens before committing to a full run
+python -m eroll.transliterate_rolls --state assam sample --sample-n 300
+
+# 3. Full run: batch-transliterate all uniques -> parallel parquet (resumable)
+python -m eroll.transliterate_rolls --state assam run
+
+# 4. Extract the (native, english) corpus -> data/assamese.csv.gz
+python -m eroll.transliterate_rolls --state assam extract
+
+# Or gated end-to-end:
+python -m eroll.transliterate_rolls --state assam all
+```
+
+The produced corpus (e.g. `data/assamese.csv.gz`) is handed to `indicate` for model
+training; only the trained model ships in the `indicate` wheel.
+
+## Adding a state
+
+Add one `StateConfig` entry to `eroll/states.py` (name, language, native unicode range,
+input filename, corpus header). Everything else is state-agnostic.
+
+## Layout
+
+| Module | Role |
+|---|---|
+| `eroll/states.py` | Per-state config + `STATES` registry |
+| `eroll/pipeline.py` | Unique-token extraction + word-map join-back |
+| `eroll/extract_pairs.py` | Aligned `(native, english)` pair extraction (modal dedup) |
+| `eroll/transliterate_rolls.py` | Orchestrator CLI (profile/sample/run/extract/all) |
+| `eroll/pricing.py` | Batch-API cost table for the profile step |

data/README.md

@@ -0,0 +1,22 @@
+# Roll data
+
+Roll inputs and pipeline artifacts live here (path overridable via `EROLL_DATA_DIR`).
+Large files are **not** committed — host them on Dataverse or git-LFS, mirroring the
+`indicate` repo's data convention.
+
+## Inputs (place here or point `EROLL_DATA_DIR` at them)
+
+| File | State | Notes |
+|---|---|---|
+| `assam_all_clean+t13n.csv.gz` | Assam | Native Assamese columns + weak libindic `_t13n` columns. The live forward target. Relocated out of the `indicate` repo. |
+| `andhra.7z` | Andhra | **Excluded / not required here** — already romanized English (no native script), so out of scope for a forward Indic→English corpus. Keep wherever you store raw Andhra rolls; the pipeline never reads it. |
+| `punjab_transliteration_subset.parquet` | Punjab | GPT-4o parallel parquet (`<field>` + `<field>_transliterated`). Used only by the `extract_pairs` regression test. Currently lives in `indicate/data/`. |
+
+## Outputs (generated; gitignored)
+
+| File | Produced by |
+|---|---|
+| `<state>_tokens.jsonl` (+ `.batchstate.json`) | batch checkpoint/state |
+| `<state>_parallel.parquet` | `run` (join-back) |
+| `<language>.csv.gz` (e.g. `assamese.csv.gz`) | `extract` — the corpus handed to `indicate` for training |
+| `eval/<state>_sample.tsv` | `sample` — human-review gate |

data/bengali.csv.gz

@@ -0,0 +1,15 @@
+"""eroll -- reusable electoral-roll transliteration pipeline.
+
+Turns Indian electoral rolls (native script) into high-quality Indic->English
+training corpora and improved romanized rolls, using the ``indicate`` core library's
+batch-mode LLM transliteration. Forward direction only (Indic -> English); rolls that
+are already romanized English are out of scope.
+"""
+
+from __future__ import annotations
+
+__version__ = "0.1.0"
+
+from .states import STATES, StateConfig
+
+__all__ = ["STATES", "StateConfig"]

data/hindi.csv.gz

@@ -0,0 +1,100 @@
+"""Extract clean ``(native, english)`` word pairs from a parallel-script parquet.
+
+Generalized from ``indicate``'s ``training/extract_punjabi.py``. For every row/field
+it pulls the native-script runs from ``<field>`` and the Latin runs from the
+positionally-aligned ``<field><translit_suffix>``; when their counts match they zip
+into word pairs. Ambiguous native words (more than one observed English) collapse to
+the **modal** English. Output is a deduped, sorted ``(native, english)`` csv.gz.
+
+Field order and ``batch_size`` are significant for exact reproduction of the original
+Punjabi corpus: they fix the ``Counter`` insertion order (hence modal tie-breaking).
+"""
+
+from __future__ import annotations
+
+import csv
+import gzip
+import re
+from collections import Counter, defaultdict
+from pathlib import Path
+
+import pyarrow.parquet as pq
+from tqdm import tqdm
+
+LATIN_RUN = re.compile(r"[A-Za-z]+")
+
+
+def extract_pairs(
+    *,
+    source: str | Path,
+    out: str | Path,
+    native_run: re.Pattern[str],
+    fields: list[str],
+    header: tuple[str, str],
+    translit_suffix: str = "_transliterated",
+    latin_run: re.Pattern[str] = LATIN_RUN,
+    max_len: int = 40,
+    batch_size: int = 100_000,
+    progress: bool = True,
+) -> dict[str, int]:
+    """Write ``(native, english)`` pairs from ``source`` parquet to ``out`` csv(.gz).
+
+    Returns a stats dict: ``rows_seen, aligned, mismatched, pairs, ambiguous``.
+    """
+    source = Path(source)
+    out = Path(out)
+    parquet = pq.ParquetFile(source)
+    columns = [c for f in fields for c in (f, f"{f}{translit_suffix}")]
+
+    counts: dict[str, Counter[str]] = defaultdict(Counter)
+    rows_seen = aligned = mismatched = 0
+    total = parquet.metadata.num_rows
+
+    bar = tqdm(total=total, desc="rows", unit="row", disable=not progress)
+    for batch in parquet.iter_batches(batch_size=batch_size, columns=columns):
+        cols = batch.to_pydict()
+        batch_len = len(cols[fields[0]])
+        rows_seen += batch_len
+        for field_name in fields:
+            src_col = cols[field_name]
+            tgt_col = cols[f"{field_name}{translit_suffix}"]
+            for src, tgt in zip(src_col, tgt_col, strict=False):
+                if not src or not tgt:
+                    continue
+                native = native_run.findall(src)
+                if not native:
+                    continue
+                latin = latin_run.findall(tgt)
+                if len(native) != len(latin):
+                    mismatched += 1
+                    continue
+                aligned += 1
+                for nat, lat in zip(native, latin, strict=False):
+                    eng = lat.lower()
+                    if 0 < len(nat) <= max_len and 0 < len(eng) <= max_len:
+                        counts[nat][eng] += 1
+        bar.update(batch_len)
+    bar.close()
+
+    ambiguous = 0
+    pairs: list[tuple[str, str]] = []
+    for native_word, eng_counts in counts.items():
+        if len(eng_counts) > 1:
+            ambiguous += 1
+        pairs.append((native_word, eng_counts.most_common(1)[0][0]))
+    pairs.sort()
+
+    out.parent.mkdir(parents=True, exist_ok=True)
+    opener = gzip.open if str(out).endswith(".gz") else open
+    with opener(out, "wt", encoding="utf-8", newline="") as handle:
+        writer = csv.writer(handle)
+        writer.writerow(list(header))
+        writer.writerows(pairs)
+
+    return {
+        "rows_seen": rows_seen,
+        "aligned": aligned,
+        "mismatched": mismatched,
+        "pairs": len(pairs),
+        "ambiguous": ambiguous,
+    }

data/punjabi.csv.gz

@@ -0,0 +1,59 @@
+"""Roll pipeline primitives: unique-token extraction and word-map join-back.
+
+The LLM step itself is delegated to ``indicate.batch.transliterate_tokens_batched``
+(batch-mode, checkpointed). This module only does the cheap, deterministic data work:
+collecting unique native-script tokens from roll columns, and substituting a resolved
+``token -> english`` map back into each cell.
+"""
+
+from __future__ import annotations
+
+import re
+from collections.abc import Iterable
+
+import pandas as pd
+
+
+def extract_unique_native_tokens(
+    chunks: Iterable[pd.DataFrame],
+    columns: Iterable[str],
+    native_run: re.Pattern[str],
+) -> set[str]:
+    """Collect every unique native-script run across ``columns`` of all chunks."""
+    columns = list(columns)
+    tokens: set[str] = set()
+    for df in chunks:
+        for col in columns:
+            if col not in df.columns:
+                continue
+            for value in df[col].dropna().unique():
+                tokens.update(native_run.findall(str(value)))
+    return tokens
+
+
+def join_back(
+    df: pd.DataFrame,
+    columns: Iterable[str],
+    word_map: dict[str, str],
+    native_run: re.Pattern[str],
+    suffix: str = "_t13n_llm",
+) -> pd.DataFrame:
+    """Add ``<col><suffix>`` columns with native runs replaced via ``word_map``.
+
+    Mirrors the notebook's fast path: transliterate each column's *unique* values
+    once, then map back to every row. Unknown runs (not in ``word_map``) are left
+    unchanged, so the output degrades gracefully. Returns ``df`` (mutated in place).
+    """
+    columns = list(columns)
+
+    def _sub(text: object) -> object:
+        if pd.isna(text):
+            return text
+        return native_run.sub(lambda m: word_map.get(m.group(0), m.group(0)), str(text))
+
+    for col in columns:
+        if col not in df.columns:
+            continue
+        value_map = {value: _sub(value) for value in df[col].dropna().unique()}
+        df[f"{col}{suffix}"] = df[col].map(value_map)
+    return df