Skip to content

DATA_PROVENANCE.md

Latest commit

 

History

History
187 lines (141 loc) · 7.29 KB

DATA_PROVENANCE.md

File metadata and controls

187 lines (141 loc) · 7.29 KB
Raw

Data Provenance — albert. Training Corpus

Model: albert. v3.0
Maintainer: RFI-IRFOS, contact@ternlang.com
Last updated: 2026-05-14

This document satisfies the data governance and provenance requirements of EU AI Act Regulation (EU) 2024/1689, Article 53(1)(c) for the albert. General-Purpose AI model.

Governance Principles

  1. No personal data. The corpus contains no PII, private communications, or data subject to GDPR data subject rights. All sources are either public-domain, openly licensed, or aggregated public content with no individual identification.

  2. Opt-out contact. Any source operator or rights holder who wishes to dispute inclusion of their content should contact contact@ternlang.com. Disputed content will be reviewed and removed from future training runs within 30 days.

  3. Reproducibility. All ingestion scripts are version-controlled in albert-moe-13/scripts/. The exact corpus state at any training run can be reconstructed from the corpus cache (data/corpus_cache.bin).

  4. Chaos invariant. Approximately 10% of training tokens at all times are synthetic noise, adversarial patterns, and mixed-language text generated locally. This corpus tier contains no third-party content.

Corpus Sources

Tier 1 — Public Domain Books (Project Gutenberg)

Property Value
Source Project Gutenberg (gutenberg.org)
Access method Direct HTTP download via download_corpus.py
Content Public-domain literary and reference texts, multilingual
Approximate share ~30% of training tokens
Legal basis Public domain (pre-1928 works; Gutenberg license)
Copyright status No copyright — all works are confirmed public domain
User-Agent disclosed AlbertCorpusBot/1.0 (contact@ternlang.com)

No copyright exceptions are needed. All Gutenberg texts used are explicitly released to the public domain by Project Gutenberg.

Tier 2 — Wikipedia (Multilingual)

Property Value
Source Wikimedia Foundation — English, German, French Wikipedia
Access method Wikimedia REST API (ingest_wiki.py, download_corpus.py)
Content Encyclopedia articles; factual prose
Approximate share ~25% of training tokens
Legal basis Creative Commons Attribution-ShareAlike 4.0 (CC BY-SA 4.0)
Attribution Wikimedia Foundation / Wikipedia contributors
Terms compliance Non-commercial research use; no redistribution of raw dumps
User-Agent disclosed AlbertCorpusBot/1.0 (contact@ternlang.com)

CC BY-SA 4.0 permits use in ML training datasets provided attribution is maintained. Attribution is recorded in this document and in the corpus metadata files.

Tier 3 — OpenWebText (Filtered Common Crawl)

Property Value
Source OpenWebText dataset (Gokaslan & Cohen, 2019)
Access method Pre-processed corpus download
Content Web text filtered by Reddit upvote signals
Approximate share ~15% of training tokens
Legal basis Research use; Common Crawl content is publicly accessible
Copyright status Mixed; Common Crawl content is subject to individual page rights
Filtering applied Quality filter: length > 100 words, no adult content signals
Known limitation Individual page copyright cannot be verified at corpus scale

Use of Common Crawl-derived corpora for non-commercial AI research is established practice and is consistent with the EU AI Act Article 53(1)(c) exception for text and data mining for research purposes under DSM Directive Article 3 (Directive 2019/790/EU).

Tier 4 — Technical Content (GitHub, HN, Developer Blogs)

Property Value
Source GitHub Issues (public repos), Hacker News discussions, developer blogs
Access method ingest_github.py, build_stage10.py
Content Technical debugging discussions, bug reports, Q&A
Approximate share ~10% of training tokens
Legal basis Public content; GitHub public content terms permit crawling
Copyright status Mixed open-source licenses; original authors retain rights
Filtering applied Public repositories only; no private repo content

Tier 5 — Synthetic Chaos Layer

Property Value
Source Generated locally by RFI-IRFOS (build_chaos_corpus.py)
Content Synthetic noise, adversarial patterns, mixed-language text, random token sequences
Approximate share ~10% of training tokens (enforced invariant)
Third-party rights None — entirely generated by the training team
Purpose Robustness training; prevents over-fitting to clean text distributions

This tier contains no third-party content and raises no copyright concerns.

Tier 6 — Structured and Multilingual Content

Property Value
Sources EU AI Act text (GitHub mirror), Linux kernel docs (kernel.org RST), TLDR pages, multilingual EU language samples
Access method download_corpus.py, build_multilingual_corpus.py
Content Technical documentation, structured text, EU legislation
Approximate share ~10% combined
Legal basis EU legislation is public domain; kernel docs under GPL-2.0 (TDM exception applies); TLDR pages MIT license

Tier 7 — Reddit Data (Conditional)

Property Value
Source Reddit public API via reddit_scraper.py
Access method PRAW library; authenticated API access with registered app
Content Public discussion threads from technical subreddits
Status Conditional — dependent on API terms compliance
Legal basis Reddit API terms of service; public content
Credentials Client ID/secret via .env (not committed to repo)

Reddit's Data API ToS (updated 2023) restricts bulk data collection. RFI-IRFOS uses the authenticated API within stated rate limits. If Reddit enforces restrictions that exclude this use case, this data tier will be removed from future training runs.

Text and Data Mining (TDM) Exception

Under EU Directive 2019/790/EU Article 3 (DSM Directive), reproduced in national law across EU member states, text and data mining for scientific research purposes is permitted regardless of the rightsholder's opt-out provisions, subject to lawful access.

albert. is developed as non-commercial AI research by RFI-IRFOS. Corpus collection respects robots.txt where feasible and discloses a contact User-Agent for opt-out requests.

Data Preprocessing

All corpus text undergoes the following pipeline before tokenization:

  1. Language detection — Non-target language content filtered or down-sampled to maintain corpus balance.
  2. Deduplication — Near-duplicate paragraphs removed using shingling hashes.
  3. Quality filtering — Minimum length thresholds; removal of machine-generated boilerplate.
  4. Tokenization — Custom BPE tokenizer (tokenizer_v3/) with 32k vocabulary trained on the combined corpus.
  5. Corpus caching — Final token IDs written to data/corpus_cache.bin for reproducible training.

Contact for Data Governance Inquiries

Organisation: RFI-IRFOS
Email: s.kepp@ternlang.com
Response time: Within 30 days of receipt
Repository: https://github.com/eriirfos-eng/ternary-intelligence-stack