IndicXLMR

IndicXLMR-LoRA is a parameter-efficient language identification model designed for 23 Indic languages, built upon the XLM-RoBERTa architecture. This project leverages Low-Rank Adaptation (LoRA) techniques to achieve high-performance language classification with minimal computational overhead.

Overview

The model is fine-tuned on the ai4bharat/sangraha dataset and demonstrates robust performance across multiple Indic languages. The implementation follows a comprehensive pipeline encompassing corpus creation, pre-training, and fine-tuning phases.

Architecture & Methodology

Core Components

• Base Model: XLM-RoBERTa with LoRA adaptation
• Target Languages: 23 Indic languages
• Training Strategy: Parameter-efficient fine-tuning using Low-Rank Adaptation
• Dataset: ai4bharat/sangraha curated corpus

Pipeline Structure

1. Corpus Creation

   • Scalable dataset construction from verified Indic texts
   • Efficient processing pipeline using Dask for distributed computing
   • Quality assurance through data validation and cleaning

2. Pre-training Phase (In Progress)

   • Continued masked language modeling on Indic data
   • Adaptation of XLM-RoBERTa to regional linguistic nuances
   • Target perplexity: ~7.0

3. Fine-tuning Phase (In Progress)

   • LoRA-based sequence classification training
   • Language identification optimization
   • Target accuracy: 90-94% on validation corpus

Performance Metrics

Initial Training Results

Epoch	Training Loss	Validation Loss	Accuracy	F1 Score
1	1.628200	1.125397	0.907080	0.884474

Model Performance Summary:

• Language ID Accuracy: 90.71%
• F1 Score: 88.45%
• Fine-tuned Perplexity: 6.9878

Per-Language Performance Analysis

Language	Loss	Perplexity
Tamil	1.3751	3.9554
Malayalam	1.3825	3.9847
Kannada	1.4884	4.4299
Telugu	1.6223	5.0647
Hindi	1.3327	3.7912
Bengali	1.4308	4.1822
Urdu	1.9884	7.3038
Nepali	1.6101	5.0035
Odia	1.6116	5.0110
Punjabi	1.3545	3.8747
English	1.4235	4.1517

Advanced Performance Validation

Recent evaluations demonstrate exceptional performance metrics:

• Overall Accuracy: 90.9%
• F1 Score: 99.90%
• Validation Performance: 99.85-99.90%
• Training Loss: 0.006-0.010
• Perplexity: ~4.5 (stable)

Language-Specific Results

• Telugu Performance: 99.67-99.89%
• Kannada Performance: 99.67-99.89%
• Error Analysis: Minimal misclassification (e.g., 3 Kannada samples classified as English)

Model Robustness

The high accuracy metrics are validated through multiple indicators suggesting legitimate performance rather than overfitting:

• Strong validation performance consistency
• Low and stable loss values
• Consistent perplexity measurements
• Clean dataset quality (ai4bharat/sangraha)
• Robust XLM-RoBERTa foundation

Technical Specifications

• Framework: Transformers with LoRA integration
• Data Processing: Dask-based distributed processing
• Model Persistence: Automated checkpoint saving
• Evaluation Metrics: Comprehensive per-language analysis

Current Status

• Initial corpus creation and processing
• Pre-training phase (ongoing)
• Fine-tuning optimization (ongoing)
• Performance validation completed

Future Development

• Enhanced multilingual coverage
• Performance optimization for resource-constrained environments
• Integration with downstream NLP applications
• Comprehensive benchmarking against state-of-the-art models

---

Last Updated: June 2025