This is a Transformer-based encoder Language Model pre-trained on the masked language modelling objective. It is a RoBERTa architecture. The model was pre-trained from scratch on Early Modern Dutch from the Dutch East India Company Archives, digitized with a Handwritten Text Recognition Module. There was no other data involved in pre-training. The model can be used to further develop NLP technology to be used in software that supports research into the archive. In other words, the model can be fine-tuned for any NLP task, but it is built to be fine-tuned on tasks in the domain of Early Modern VOC Dutch. Download our model here.
The data used for pre-training is publicly available on Dataverse. It comprises a little over 6GB in text data after pre-processing.
Four models were pre-trained. There were two sets of parameter settings, varying in num_training_steps and num_warmup_steps which influence the optimizer settings (AdamW). I pre-trained two seperate models per set of paramater settings in order to investgate stochasticity between initializations. For all four models, these were some of the most important parameter settings (static for all models)
| param | setting |
|---|---|
| learning rate | 0.0003 |
| betas | [ 0.9, 0.98] |
| weight_decay | 0.01 |
| num_train_epochs | 2 |
| per_device_train_batch_size | 40 |
| gradient_accumulation_steps | 10 |
| fp16 | true |
All parameter settings can be found per model on our Huggingface (in files).
The models were pre-trained on the Snellius HPC cluster, funded by Surf and developed within the GLOBALISE project, funded by NWO under project number 175.2019.003
This code was repurposed from pre-training MedRoBERTa.nl
To evaluate the model on binary event detection, see my other repo
- Models on Huggingface
- Presentation comparing different models