Skip to content

Commit f146d2e

Browse files
Henry010303Henry010303
committed
Add Text Watermarking Task information
1 parent 83608f8 commit f146d2e

File tree

1 file changed

+97
-8
lines changed

1 file changed

+97
-8
lines changed

‎clef26/pan26-web/text-watermarking.html‎

Lines changed: 97 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -28,31 +28,79 @@ <h1 class="uk-margin-remove-top">Text Watermarking 2026</h1>
2828

2929
<h2 id="synopsis">Synopsis</h2>
3030
<ul class="uk-list uk-list-bullet">
31-
<li>Task: TBA.</li>
31+
<li>Task: Insert a watermark into a given text. Then, after we have attacked the text, detect the inserted watermark. </li>
3232
<li>Registration: [<a href="https://clef-labs-registration.dipintra.it/">CLEF labs</a>]</li>
3333
<li>Important dates:
3434
<ul>
3535
<li><strong>May 07, 2026:</strong> software submission</li>
3636
<li><strong>May 28, 2026:</strong> participant notebook submission
3737
[<a href="../../pan-notebook-paper-template/pan-notebook-paper-template.zip">template</a>]
38-
[<a href="https://easychair.org/conferences/?conf=clef2026">submission</a>&nbsp; – <em>select "Stylometry and Digital Text Forensics (PAN)"</em> ]</li>
38+
[<a href="https://easychair.org/conferences/?conf=clef2026">submission</a>&nbsp; – <em>select "Text Watermarking (PAN)"</em> ]</li>
3939
</ul>
4040
</li>
41-
<!-- <li>Data: TBA [<a href="">download</a>]</li>-->
42-
<!-- <li>Evaluation Measures: TBD</li>-->
41+
<li>Data: TBA <!-- [<a href="">download</a>] --> </li>
42+
<li>Evaluation Measures: Accuracy, F1, BLEU, BERTScore </li>
4343
<!-- <li>Baselines: TBA [<a href="">code</a>]</li>-->
4444
</ul>
4545

4646

4747
<h2 id="task-overview">Task Overview</h2>
48-
<p>TBA</p>
48+
<p>In the Text Watermarking task, participants are given a text and must insert a watermark into it. After submitting the watermarked texts and the watermark algorithm, including a watermark detection system, through <a href="www.tira.io">TIRA</a>, the texts are subjected to various attacks. The objective is to detect the watermark after the text has been attacked, thereby demonstrating its robustness against an attacker.</p>
49+
50+
<p>The task is structured as follows:</p>
4951

52+
<ol class="uk-list uk-list-bullet">
53+
<li>Insert a watermark into each text in the provided data set.</li>
54+
<li>Submit the watermarked texts together with the watermarking system, including the watermark detection, through the <a href="https://www.tira.io/task-overview/generative-ai-authorship-verification-panclef-2026/">Tira</a> platform.</li>
55+
<li>We will carry out attacks with varying severity on the watermarked texts.</li>
56+
<li>We will run your watermark detection system on the attacked texts to evaluate its performance in detecting watermarks.</li>
57+
</ol>
5058

5159
<h2 id="submission">Submission</h2>
52-
<p>TBA</p>
60+
<p>Participants will submit their systems as Docker images through the <a href="https://www.tira.io/task-overview/generative-ai-authorship-verification-panclef-2026/">Tira</a> platform. It is not expected that submitted systems are actually <em>trained</em> on Tira, but they must be standalone and runnable on the platform without requiring contact to the outside world (evaluation runs will be sandboxed).</p>
61+
62+
<p>The submitted software must be executable inside the container via a command line call. The script must take two arguments: an input file (an absolute path to the input JSONL file) and an output directory (an absolute path to where the results will be written):</p>
63+
64+
<p>Within Tira, the input file will be called <code>dataset.jsonl</code>, so with the pre-defined Tira placeholders, your software should be invoked like this:</p>
65+
<pre class="prettyprint"><code class="lang-console">$ mySoftware $inputDataset/dataset.jsonl $outputDir</code></pre>
66+
67+
<p>Within <code>$outputDir</code>, a single (!) file with the file extension <code>*.jsonl</code> must be created with the following format:</p>
68+
69+
<pre class="prettyprint"><code class="lang-json">{"id": "bea8cccd-0c99-4977-9c1b-8423a9e1ed96", "label": 1.0}
70+
{"id": "a963d7a0-d7e9-47c0-be84-a40ccc2005c7", "label": 0.0}
71+
...</code></pre>
72+
5373

5474
<h2 id="evaluation">Evaluation</h2>
55-
<p>TBA</p>
75+
<p>Systems will be evaluated with the following metrics:</p>
76+
77+
<ul class="uk-list uk-list-bullet">
78+
<li>Accuracy: The percentage of how often the correct outcome is predicted.</li>
79+
<li>F<sub>1</sub>: The harmonic mean of precision and recall.</li>
80+
<li>BLEU: Similarity of the watermarked text to the original text.</li>
81+
<li>BERTScore: Similarity of the watermarked text and the original text based on the cosine similarity.</li>
82+
<li>The arithmetic mean of all the metrics above.</li>
83+
<li>A confusion matrix for calculating true/false positive/negative rates.</li>
84+
</ul>
85+
86+
<p>The evaluator for the task will output the above measures as JSON like so:</p>
87+
<pre class="prettyprint"><code class="lang-json">{
88+
"accuracy": 0.984,
89+
"f1": 0.98,
90+
"BLEU": 0.901,
91+
"BERTScore": 0.85,
92+
"mean": 0.92875,
93+
"confusion": [
94+
[
95+
1211,
96+
66
97+
],
98+
[
99+
27,
100+
2285
101+
]
102+
]
103+
}</code></pre>
56104

57105
<h2 id="baselines">Baselines</h2>
58106
<p>TBA</p>
@@ -66,7 +114,48 @@ <h2 id="leaderboard">Leaderboard</h2>
66114
<div class="uk-container uk-margin-medium">
67115
<h2 id="related-work">Related Work</h2>
68116
<ul class="uk-list uk-list-bullet">
69-
<li>TBA</li>
117+
<li>
118+
John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and
119+
Tom Goldstein. A watermark for large language models. In Andreas Krause,
120+
Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and
121+
Jonathan Scarlett, editors, International Conference on Machine Learning,
122+
ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Pro-
123+
ceedings of Machine Learning Research, pages 1706117084. PMLR, 2023.
124+
URL https://proceedings.mlr.press/v202/kirchenbauer23a.html
125+
</li>
126+
<li>
127+
Xuandong Zhao, Prabhanjan Vijendra Ananth, Lei Li, and Yu-Xiang Wang.
128+
Provable robust watermarking for ai-generated text. In The Twelfth Interna-
129+
tional Conference on Learning Representations, ICLR 2024, Vienna, Aus-
130+
tria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreview.
131+
net/forum?id=SsmT8aO45L.
132+
</li>
133+
<li>
134+
Yijian Lu, Aiwei Liu, Dianzhi Yu, Jingjing Li, and Irwin King. An entropy-
135+
based text watermarking detection method. In Lun-Wei Ku, Andre Martins,
136+
and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of
137+
the Association for Computational Linguistics (Volume 1: Long Papers),
138+
ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 1172411735.
139+
Association for Computational Linguistics, 2024. doi: 10.18653/V1/2024.
140+
ACL-LONG.630. URL https://doi.org/10.18653/v1/2024.acl-long.
141+
630.
142+
</li>
143+
<li>
144+
Taehyun Lee, Seokhee Hong, Jaewoo Ahn, Ilgee Hong, Hwaran Lee, Sangdoo
145+
Yun, Jamin Shin, and Gunhee Kim. Who wrote this code? watermarking
146+
for code generation. In Lun-Wei Ku, Andre Martins, and Vivek Sriku-
147+
mar, editors, Proceedings of the 62nd Annual Meeting of the Association for
148+
Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok,
149+
Thailand, August 11-16, 2024, pages 48904911. Association for Compu-
150+
tational Linguistics, 2024. doi: 10.18653/V1/2024.ACL-LONG.268. URL
151+
https://doi.org/10.18653/v1/2024.acl-long.268.
152+
</li>
153+
<li>
154+
Aiwei Liu, Leyi Pan, Yijian Lu, Jingjing Li, Xuming Hu, Lijie Wen, Irwin King,
155+
and Philip S. Yu. A survey of text watermarking in the era of large language
156+
models. CoRR, abs/2312.07913, 2023. doi: 10.48550/ARXIV.2312.07913.
157+
URL https://doi.org/10.48550/arXiv.2312.07913.
158+
</li>
70159
</ul>
71160

72161
<h2 id="task-committee">Task Committee</h2>

0 commit comments

Comments
 (0)