Skip to content

Commit eb5643a

Browse files
committed
merged GermEval & Update requirements
1 parent 3a1b6f7 commit eb5643a

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

71 files changed

+764213
-310
lines changed

README.md

Lines changed: 39 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -9,27 +9,42 @@ For the current leaderboard and more information check out the [SuperGLEBer Webs
99

1010
This is the updated branch that contains the new and improved version of the SuperGLEBer benchmark.
1111

12+
## Updates
13+
- We added 8 new tasks of the GermEval 2025 shared task.
14+
- Additionally, we added support for LLM2Vec models, with the integration of bidirectional masks (Thanks @vasqu)
15+
16+
1217
## Running Experiments
1318

14-
create all relevant files necessary to schedule runs on a k8s/slurm cluster:
19+
Create all relevant files necessary to schedule runs on a k8s/slurm cluster:
1520

1621
```bash
1722
python src/template_k8s.py
1823
```
1924

20-
running a model on a task:
25+
Running a model on a task:
2126

2227
```bash
2328
python src/train.py +model=gbert_base +train_args=a100 +task=news_class
2429
```
2530

26-
override config keys via CLI:
31+
Override config keys via CLI:
2732

2833
```bash
2934
python src/train.py +model=gbert_base +train_args=a100 +task=news_class train_args.batch_size=1
3035
```
3136

32-
you can find valid parameters in the provided yaml configs: <https://github.com/LSX-UniWue/SuperGLEBer/tree/paper/src/conf>
37+
You can find valid parameters in the provided yaml configs: <https://github.com/LSX-UniWue/SuperGLEBer/tree/paper/src/conf>
38+
39+
40+
## Contact
41+
42+
Feel free to reach out 💡:
43+
[supergleber@informatik.uni-wuerzburg.de](mailto:supergleber@informatik.uni-wuerzburg.de)
44+
45+
46+
47+
3348
## Citation
3449
```bib
3550
@inproceedings{pfister-hotho-2024-supergleber,
@@ -50,3 +65,23 @@ you can find valid parameters in the provided yaml configs: <https://github.com/
5065
abstract = "We assemble a broad Natural Language Understanding benchmark suite for the German language and consequently evaluate a wide array of existing German-capable models in order to create a better understanding of the current state of German LLMs. Our benchmark consists of 29 different tasks ranging over different types such as document classification, sequence tagging, sentence similarity, and question answering, on which we evaluate 10 different German-pretrained models, thereby charting the landscape of German LLMs. In our comprehensive evaluation we find that encoder models are a good choice for most tasks, but also that the largest encoder model does not necessarily perform best for all tasks. We make our benchmark suite and a leaderboard publically available at https://supergleber.professor-x.de and encourage the community to contribute new tasks and evaluate more models on it (https://github.com/LSX-UniWue/SuperGLEBer)."
5166
}
5267
```
68+
69+
For our GermEval 2025 participation cite:
70+
```
71+
@inproceedings{wunderle-etal-2025-die,
72+
title = "Die {S}uper{GLEB}er at {G}erm{E}val 2025 Shared Tasks: Growing Pains - When More Isn{'}t Always Better",
73+
author = "Wunderle, Julia and
74+
Pfister, Jan and
75+
Hotho, Andreas",
76+
editor = "Wartena, Christian and
77+
Heid, Ulrich",
78+
booktitle = "Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops",
79+
month = sep,
80+
year = "2025",
81+
address = "Hannover, Germany",
82+
publisher = "HsH Applied Academics",
83+
url = "https://aclanthology.org/2025.konvens-2.45/",
84+
pages = "479--493"
85+
}
86+
```
87+
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
# Flauscherkennung
2+
3+
Paper: [url](https://aclanthology.org/2025.konvens-2.32/)
4+
5+
6+
### Tasks
7+
Task 1: Binary Classification (True, False)
8+
9+
Task 2: Span detection with category assignment
10+
11+
### Data
12+
13+
| Train | Dev | Test |
14+
|-------|------|------|
15+
| 29645 | 7412 | 9229 |
16+
17+
18+
### Example

data/Germeval/2025/FlauschErkennung/task1/dev.csv

Lines changed: 7413 additions & 0 deletions
Large diffs are not rendered by default.

data/Germeval/2025/FlauschErkennung/task1/test.csv

Lines changed: 9231 additions & 0 deletions
Large diffs are not rendered by default.

data/Germeval/2025/FlauschErkennung/task1/train.csv

Lines changed: 29646 additions & 0 deletions
Large diffs are not rendered by default.

0 commit comments

Comments
 (0)