A 50 pair Tamil instruction tuning seed dataset and validation pipeline for TamilLM, a from scratch Tamil first language model.
# Install dependencies
pip install -r validator/requirements.txt
# Zero-argument run (uses default paths)
python main.py
# Explicit CLI
python validate_sft.py \
--input data/tamil_sft_seed.jsonl \
--clean outputs/clean_sft.jsonl \
--report outputs/validation_report.json
# Run test suite
python -m pytest tests/ -vTamilLM/
├── data/
│ └── tamil_sft_seed.jsonl # Single source of truth for the dataset
├── fixtures/
│ └── bad_examples.jsonl # Test fixtures for the validator
├── outputs/
│ ├── clean.jsonl # Output file generated by the pipeline
│ └── validation_report.json # Final report generated by the pipeline
├── tests/
│ ├── test_checks.py # Test cases for individual rules
│ ├── test_p0_regression.py # P0 regression cases
│ ├── test_utils.py # Test cases for helpers
│ └── test_validator.py # Test cases for orchestrator
├── validator/
│ ├── __init__.py
│ ├── checks.py # Validation rules & checks logic
│ ├── requirements.txt
│ ├── utils.py # Configuration, constants & scoring logic
│ └── validator.py # Orchestrator logic
├── .gitignore # Ignoring outputs, pycache, venv, etc.
├── LICENSE
├── README.md
├── main.py # Default wrapper runner script
└── validate_sft.py # CLI entry point for programmatic usage
Think in Tamil, not translate into Tamil.
Every prompt was written as something a real person would type, not an exam question rephrased in Tamil. Spoken prompts are fragmented, emotionally toned, and code-mixed at varying levels (fully Tamil to heavily Tanglish), because that is how Tamil speakers actually write in digital contexts. This variation is intentional and documented in each record's notes field.
Three registers, deliberately separated.
Spoken Tamil is colloquial and code mixed. Formal Tamil is procedural and neutral, no metaphors, no rhetoric. Literary Tamil is elevated and evocative, used for history, literature, and editorial content. The consistency check in checks.py enforces these boundaries automatically using Tamil verb-ending pattern matching.
Near-duplicates are flagged, not auto-rejected.
Two pairs covering the same topic in different dialects (Coimbatore vs Tirunelveli) share structural similarity but are linguistically distinct. Auto-rejection would remove valid dialectal variation. The validator flags near-duplicates with the matched ID and Jaccard score and routes them to the report for human review.
Character trigrams over word n-grams.
Tamil is agglutinative. Suffixes attach directly to stems - "போகிறான்" and "போகிறாள்" differ by one character but share most of their structure. Character trigrams capture this shared substructure and correctly measure similarity across inflected forms.
Duplicate detection previously ran only on records that passed schema validation — schema-errored records were short-circuited before reaching the DuplicateDetector, so their text was never registered. A later record with the same prompt as an earlier schema-invalid record would not be flagged.
The fix uses two passes. Pass 1 registers every record's prompt and response text (when the field exists, is a string, and is non-empty) regardless of schema validity, using the record's id or a line-number locator (<line_N>) when id is missing. Pass 2 runs the full check suite as before — schema errors still short-circuit content checks, but duplicate findings from Pass 1 are added alongside any schema errors rather than replacing them.
schema.duplicate_id findings are now included in the report's duplicate_count, so duplicate IDs are reflected in the summary total, not only in per-record details.
Exit 0 on data failures, exit 1 on infrastructure failures.
A record failing validation is a data finding, not a tool failure. The validator exits nonzero only when it cannot do its job, file not found, malformed JSONL it cannot parse, missing dependency.
1. Compound characters inflate Tamil script ratio incorrectly.
Tamil vowel signs (ா, ி, ீ) are combining Unicode codepoints that attach to consonants. "கா" is two codepoints but one visible character. A naive character count would inflate the Tamil ratio for text that mixes Tamil consonants with Latin approximations, because each vowel sign is counted as a separate Tamil codepoint. The fix: count every codepoint in U+0B80–U+0BFF individually and measure ratio against total alphabetic codepoints, not visible grapheme clusters. This is correct for ratio measurement even though it overcounts visible characters.
2. Tamil-scripted English loanwords evade the Latin ratio check but expose a deeper labeling problem.
Spoken pairs contain loanwords like "ஹைட்ரேட்டடா" and "பாராசிட்டமால்", phonetically English but written entirely in Tamil script. The english_script_ratio() check correctly passes these because they occupy the Tamil Unicode block. The problem surfaces when a contributor writes the same loanword sometimes in Tamil script and sometimes in Latin across different records - the two versions look like near duplicates with near zero Jaccard similarity because their character trigrams share nothing. The three value flag field resolves this: allowed_code_switch marks intentional Tamil-script mixing, unexpected_latin_text flags unmotivated Latin in an otherwise Tamil response, and the validator enforces this as a closed enum with ERROR severity.
3. Dialectal variants sit just below the near-duplicate threshold and the threshold had to be calibrated to keep them there.
A Tirunelveli dialect pair and a Coimbatore dialect pair covering bus travel share similar sentence structure and topic. Their character trigram Jaccard similarity is approximately 0.65 below the 0.70 threshold, so they correctly pass. This was not obvious in advance. The threshold was calibrated by computing similarity between known good dialectal pairs (which scored 0.55 - 0.68) and known-bad templated pairs with minor word substitutions (which scored 0.78 - 0.91). The 0.70 threshold sits cleanly between these two clusters. This is why near-duplicates are flagged for review rather than auto rejected the threshold is a calibrated heuristic, not a rule.
- Records: 50 total, 50 valid, 0 invalid
- Quality: 99.9/100 (aggregate)
- Issues: 0 errors, 1 warnings, 0 info
- Dupes: 0 duplicate findings
The remaining warning is a false positive from the English ratio check on an intentionally code-mixed spoken pair. It was reviewed and confirmed acceptable.