This repository contains the code and sample data for a multilingual Transformer model trained on four historical stages of the Egyptian language: Hieroglyphic, Demotic, Bohairic, and Sahidic. The project investigates semantic alignment and cross-linguistic representation learning under low-resource conditions.
The main training path adopted in this project is BERT-all(bert_all), which corresponds to training from scratch using BERT as the encoder, jointly on all Egyptian language stages and English with a shared byte-level BPE tokenizer.
├── data/
│ ├── *.jsonl # Sample corpora from Thesaurus Linguae Aegyptiae and Coptic Scriptorium (10-sample demo)
│ ├── processed_jsonl_UPOS/ # Cleaned and tokenized corpora used for training and evaluation
│ └── clean_corpora.py # Preprocessing script for corpus normalization and format conversion
│
├── models/
│ └── multi_task_bert_encoder_decoder.py # Model architecture defining a multi-task BERT encoder-decoder structure
│
├── project_tokenizers/
│ └── bert_all/
│ ├── build_byte_level_bpe_with_special_tokens.py # Script for training the BPE tokenizer
│ └── tokenizer.json # Trained tokenizer (Byte-Level BPE)
│
├── resource_eval/
│ └── egyptian_cognate_pairs.jsonl # Expert-curated cognate pairs derived from Thesaurus Linguae Aegyptiae (10 groups for demo)
│
├── results/
│ ├── bert_all_evaluation.csv # Quantitative evaluation results (AUC, accuracy, etc.)
│ ├── tsne/ # Visualization outputs (word embedding distributions)
│ └── *.png / *.pdf # Evaluation figures generated by scripts/eval_*.py
│
├── scripts/
│ ├── run_configs.jsonl # JSONL file listing experimental setups and hyperparameter configurations
│ ├── run_from_jsonl.py # Runner script to execute experiments from the JSONL configuration file
│ ├── eval_semantic_pairs.py # Evaluation of cross-linguistic cognate pairs (computes triple accuracy and AUC)
│ └── eval_tsne.py # t-SNE visualization of learned word embeddings across languages
│
├── training/
│ ├── losses/ # Directory for modular loss functions
│ │ ├── init.py
│ │ └── mlm_loss.py
│ ├── custom_collator.py # Custom data collator for dynamic batching and token masking
│ ├── normalization.py # Two normalization strategies for ancient Egyptian language forms
│ ├── train_main.py # Main training script integrating multi-task loss and evaluation
│ └── utils.py # Dataset loading, logging, and utility functions
├── README.md
├── requirements.txt
To reproduce any experiment:
python -m scripts.run_from_jsonlEach line in run_configs.jsonl defines a full command to train a model, for example:
{"name": "mlm_translation",
"cmd": "python -m training.train_main --model_size bert_all --lang hieroglyphic,demotic,bohairic,sahidic --multi --tokenizer_type shared_bpe --output_dir checkpoints/bert_all_exp1_balanced --epochs 10"}The evaluation scripts generate:
- AUC and accuracy results:
scripts/eval_semantic_pairs.py - t-SNE visualizations:
scripts/eval_tsne.py
Results are stored in /results/.
Due to strict copyright restrictions from the Thesaurus Linguae Aegyptiae and Coptic Scriptorium, we cannot publicly re-distribute the full datasets. Therefore:
- Training Data: A de minimis set of 10 illustrative snippets is provided solely for code integrity verification in
data/. - Evaluation Data: Only 10 groups of expert-curated cognate pairs are included in
resource_eval/egyptian_cognate_pairs.jsonl. - Preprocessing: The
clean_corpora.pyscript is provided to illustrate our exact data cleaning methodology, but it is tailored for the complete, raw dataset and cannot be executed directly on the provided demo files.
Please note that running the provided commands on the demo data is solely to verify the code pipeline and will not reproduce the quantitative results (AUC, accuracy) or visualizations reported in the paper. To fully reproduce our findings, researchers must obtain the original datasets from the respective sources and process them using the provided scripts.