Measuring Meaning Preservation in German-English European Parliament Speeches

This completed research project asks whether official English translations preserve the meaning of original German European Parliament speeches, and whether LaBSE cosine similarity can serve as a credible proxy for that preservation. It studies 2,473 German-English energy-policy speech pairs from 2000 to 2024, compares each true translation pair against a demanding within-topic, within-period shuffled baseline, and finds high overall semantic preservation: true pairs average 0.8959 similarity versus 0.5683 for shuffled non-pairs, with a mean gap of 0.3276.

Portfolio case study: coming soon | Live demo: not applicable yet | Report

Key results

• Official English translations of German European Parliament energy-policy speeches preserve general semantic content well overall.
• True German-English pairs have a mean LaBSE cosine similarity of 0.8959, compared with 0.5683 for shuffled same-topic, same-period non-pairs.
• The mean true-minus-shuffled gap is 0.3276, and every true pair scores above its shuffled comparison in the realised baseline.
• Temporal variation is limited and does not support a simple story of steady improvement from 2000 to 2024.
• Speech length is associated with a smaller translation-linked gap, but the pattern is mixed across model specifications.
• A small bilingual audit supports construct validity: lower embedding similarity corresponds to weaker human-observable preservation of meaning.

Method

The analysis starts from EUPDCorp and reconstructs a German-source subset with available English translations. It then narrows the scope to energy-policy speeches using a dictionary-based topic restriction, removes very short speeches, duplicate text pairs, and cases with substantial non-German material in the German source text.

Meaning preservation is measured as cosine similarity between Language-Agnostic BERT Sentence Embedding (LaBSE) representations of each German original and matched English translation. Longer speeches are chunked before embedding and aggregated into speech-level vectors. The primary benchmark is a shuffled baseline that pairs each German speech with a different English speech from the same topic corpus and broad time period, so the main quantity is translation-linked similarity above topical relatedness. The project also estimates descriptive regression models and validates the embedding measure with a small blinded bilingual audit.

Repository structure

• notebooks/ contains the analysis workflow:
  • 01_corpus_preprocessing.ipynb builds the analysis corpus.
  • 02_semantic_drift_analysis.ipynb computes embeddings, similarity scores, baselines, models, and figures.
  • 03_validity_checks.ipynb supports validation and audit checks.
• reports/report.pdf is the final paper.
• reports/report.tex and reports/bibs/ contain the report source and bibliography.
• reports/outputs/ contains generated tables and figures locally, but these outputs are ignored by Git except for the final report.
• data/ follows raw, interim, and processed stages. Data files are not tracked.
• refs/ stores reference material such as the EUPDCorp codebook.

Installation

Create the Conda environment from the project specification:

conda env create -f environment.yml
conda activate ep-crosslingual-semantic-drift

The environment uses Python 3.11 with Jupyter, pandas, NumPy, matplotlib, scikit-learn, SciPy, transformers, sentence-transformers, statsmodels, and related analysis dependencies.

Reproduction

Run the notebooks in order:

1. notebooks/01_corpus_preprocessing.ipynb
2. notebooks/02_semantic_drift_analysis.ipynb
3. notebooks/03_validity_checks.ipynb

Full reproduction requires the local source data described below. The final paper is available directly at reports/report.pdf.

Data

The main source dataset is the European Parliament Debate Corpus (EUPDCorp), restricted to German-source speeches with English translations. The topic subset is built with a trimmed Lexicoder Policy Agendas dictionary approach, selecting energy-policy speeches for a controlled comparison design.

Raw and intermediate data files are intentionally not tracked in Git because they are large local research artifacts and may have separate redistribution constraints. The repository keeps the expected directory structure with .gitkeep files, while the full analysis expects local data under data/raw/ and generated intermediate files under data/interim/.

Limitations

• The study covers one language pair: German originals and English translations.
• The substantive scope is one dictionary-defined energy-policy subset, not the full European Parliament corpus.
• The institutional context is unusually translation-rich, so results should not be generalised automatically to lower-resource or less professionalised settings.
• LaBSE cosine similarity is a proxy for general semantic preservation, not a complete translation-quality metric.
• The bilingual audit strengthens construct validity but is small and author-coded.
• General semantic preservation does not guarantee that finer-grained tasks such as stance, framing, or tone are equally unaffected by translation.