@@ -11,13 +11,18 @@ geekdocBreadcrumb: false
11
11
# Contribute to LEPISZCZE
12
12
13
13
We invite the community to contribute to ` LEPISZCZE ` by submitting model results. You can either manually fill in your submissions or use the ` embeddings ` library for automatic generation.
14
+ <br />
15
+ <br />
14
16
15
17
## Table of Contents
16
- * [ 1A. Manually Filled Submissions] ( #1a-manually-filled-submissions )
17
- * [ 1B. Automatically Generated Submissions] ( #1b-generation-submission-using-embeddings-library )
18
+ * [ 1.A Manually Filled Submissions] ( #1a-manually-filled-submissions )
19
+ * [ 1.B Example Submissions] ( #1b-example-submissions )
20
+ * [ 1.C Automatically Generated Submissions] ( #1c-generation-submission-using-embeddings-library )
18
21
* [ 2. Submitting submission as PR] ( #2-submit-via-pull-request )
19
22
20
- ## 1A. Manually Filled Submissions
23
+ <br />
24
+
25
+ ## 1.A Manually Filled Submissions
21
26
22
27
Submissions ** must include** the following information:
23
28
@@ -40,11 +45,14 @@ There are also optional submission keys, but we strongly recommend including all
40
45
| ** hparams** | Mapping of hyperparameters with their values. |
41
46
| ** packages** | Mapping of packages used for model training and evaluation, along with their versions. |
42
47
43
- Submissions should be in ` .json ` format.
44
-
45
-
46
- ### Examples
48
+ Submissions should be in ` .json ` format.
49
+ <br />
47
50
51
+ ``` python
52
+ print (" Hello, Hugo!" )
53
+ ```
54
+ ## 1.B Example Submissions
55
+ \
48
56
{{< collapse title="Information Retrieval sample submission file without optional fields." >}}
49
57
{
50
58
"submission_name": "msmarco_bm_25",
@@ -71,8 +79,7 @@ Submissions should be in `.json` format.
71
79
"averaged_over": 1
72
80
}
73
81
{{< /collapse >}}
74
-
75
-
82
+ \
76
83
{{< collapse title="Question Answering sample submission file with packages provided." >}}
77
84
78
85
{
@@ -129,61 +136,59 @@ Submissions should be in `.json` format.
129
136
},
130
137
"averaged_over": 1
131
138
}
132
-
133
139
{{< /collapse >}}
134
-
135
-
136
-
137
- ## 1B. Generation Submission using Embeddings library
140
+ \
141
+ ## 1.C Generation Submission using Embeddings library
138
142
139
143
140
144
- Install ` embeddings ` package
141
145
142
- ``` bash
143
- pip install clarinpl-embeddings
144
- ```
146
+ ``` bash
147
+ pip install clarinpl-embeddings
148
+ ```
145
149
146
150
- Put your data in accordance with comments
147
151
148
- ` ` ` python
149
- import datasets
150
- import numpy as np
151
-
152
- from embeddings.evaluator.evaluation_results import Predictions
153
- from embeddings.evaluator.leaderboard import get_dataset_task
154
- from embeddings.evaluator.submission import AveragedSubmission
155
- from embeddings.utils.utils import get_installed_packages
156
-
157
- DATASET_NAME = " clarin-pl/polemo2-official"
158
- TARGET_COLUMN_NAME = " target"
159
-
160
- hparams = {" hparam_name_1" : 0.2, " hparam_name_2" : 0.1} # put your hyperparameters here!
161
-
162
- dataset = datasets.load_dataset(DATASET_NAME)
163
- y_true = np.array(dataset["test"][TARGET_COLUMN_NAME])
164
- # put your predictions from multiple runs below!
165
- predictions = [
166
- Predictions(
167
- y_true=y_true, y_pred=np.random.randint(low=0, high=4, size=len(y_true))
168
- )
169
- for _ in range(5)
170
- ]
171
-
172
- # make sure you are running on a training env or put exported packages below!
173
- packages = get_installed_packages()
174
- submission = AveragedSubmission.from_predictions(
175
- submission_name="your_submission_name", # put your submission here!
176
- dataset_name=DATASET_NAME,
177
- dataset_version=dataset["train"].info.version.version_str,
178
- embedding_name="your_embedding_model", # put your embedding name here!
179
- predictions=predictions,
180
- hparams=hparams,
181
- packages=packages,
182
- task=get_dataset_task(DATASET_NAME),
152
+ ``` python
153
+ import datasets
154
+ import numpy as np
155
+
156
+ from embeddings.evaluator.evaluation_results import Predictions
157
+ from embeddings.evaluator.leaderboard import get_dataset_task
158
+ from embeddings.evaluator.submission import AveragedSubmission
159
+ from embeddings.utils.utils import get_installed_packages
160
+
161
+ DATASET_NAME = " clarin-pl/polemo2-official"
162
+ TARGET_COLUMN_NAME = " target"
163
+
164
+ hparams = {" hparam_name_1" : 0.2 , " hparam_name_2" : 0.1 } # put your hyperparameters here!
165
+
166
+ dataset = datasets.load_dataset(DATASET_NAME )
167
+ y_true = np.array(dataset[" test" ][TARGET_COLUMN_NAME ])
168
+ # put your predictions from multiple runs below!
169
+ predictions = [
170
+ Predictions(
171
+ y_true = y_true, y_pred = np.random.randint(low = 0 , high = 4 , size = len (y_true))
183
172
)
184
-
185
- submission.save_json()
186
- ` ` `
173
+ for _ in range (5 )
174
+ ]
175
+
176
+ # make sure you are running on a training env or put exported packages below!
177
+ packages = get_installed_packages()
178
+ submission = AveragedSubmission.from_predictions(
179
+ submission_name = " your_submission_name" , # put your submission here!
180
+ dataset_name = DATASET_NAME ,
181
+ dataset_version = dataset[" train" ].info.version.version_str,
182
+ embedding_name = " your_embedding_model" , # put your embedding name here!
183
+ predictions = predictions,
184
+ hparams = hparams,
185
+ packages = packages,
186
+ task = get_dataset_task(DATASET_NAME ),
187
+ )
188
+
189
+ submission.save_json()
190
+ ```
191
+ <br />
187
192
188
193
## 2. Submit via pull request
189
194
0 commit comments