Adds parser task using deep biaffine parser (#120)

* Adds metrics for parsing

* Beginning integration

* Adds metrics test.

One major issue is that this requires us to use negative indices for
specials, which breaks assumptions in the indexes. Will have to come
back and fix this.

* Draft of parser and its integration

* More work.

Known issues:

1. I don't think the metrics test is going to work; I will need to shift
   all the head indices by special.OFFSET.
2. I am not passing a parser mask. Do I need to? I think maybe yes.

* Applies shift to metrics test to avoid collisions.

* Moves reverse_edits to data, where it belongs.

It has no effect in the model, so let's get rid of it.

* Days' debugging work

* More work; still debugging

* Optimizes mmap instructions (#116)

* Updates Black version

* Adds logging for vocabularies (#117)

* Adds logging for vocabularies

Sample output:

INFO: 22-Feb-26 17:56:27 - UPOS vocabulary (21): '[PAD]', '[UNK]', '_', 'ADJ', 'ADP', 'ADV', 'AUX', 'CCONJ', 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X', '_'
INFO: 22-Feb-26 17:56:27 - XPOS vocabulary (53): '[PAD]', '[UNK]', '_', '$', "''", ',', '-LRB-', '-RRB-', '.', ':', 'ADD', 'AFX', 'CC', 'CD', 'DT', 'EX', 'FW', 'GW', 'HYPH', 'IN', 'JJ', 'JJR', 'JJS', 'LS', 'MD', 'NFP', 'NN', 'NNP', 'NNPS', 'NNS', 'PDT', 'POS', 'PRP', 'PRP$', 'RB', 'RBR', 'RBS', 'RP', 'SYM', 'TO', 'UH', 'VB', 'VBD', 'VBG', 'VBN', 'VBP', 'VBZ', 'WDT', 'WP', 'WP$', 'WRB', '_', '``'
INFO: 22-Feb-26 17:56:27 - Lemma vocabulary (533): [omitted]
INFO: 22-Feb-26 17:56:27 - Features vocabulary (235): [omitted]

Closes #115.

* black update

* f-string fix

* driveby: silence more warnings

* Avoids "Crashed" status in sweeps. (#118)

See Yoyodyne [#369](CUNY-CL/yoyodyne#369) for
context.

Closes #79.

* Pooling layer efficiency (#119)

* Fix pooling layer regression in UDTubeEncoder.forward

Special cases pooling_layers=1 to use last_hidden_state directly, avoiding
unnecessary allocation of all hidden states. This seems to save a lot of
GPU memory.

A few drive-bys:

1. suppress progress bar during test data generation
2. add "not human-readable" to "[omitted]" when logging lemmas
3. actually log features; why not?
4. pass information about which heads to build to the data module too,
   so it logs properly
5. removes _ from "special", since it doesn't require any special
   treatment in actuality; it's just another tag as far as we're
   concerned.
6. Standardizes trailing """: it's on its own line if the comment is
   more than one line.

* regeneration last-minute fix

* Update special.py

* fix typo

* Optimizes mmap instructions (#116)

* Adds logging for vocabularies (#117)

* Adds logging for vocabularies

Sample output:

INFO: 22-Feb-26 17:56:27 - UPOS vocabulary (21): '[PAD]', '[UNK]', '_', 'ADJ', 'ADP', 'ADV', 'AUX', 'CCONJ', 'DET', 'INTJ', 'NOUN', 'NUM', 'PART', 'PRON', 'PROPN', 'PUNCT', 'SCONJ', 'SYM', 'VERB', 'X', '_'
INFO: 22-Feb-26 17:56:27 - XPOS vocabulary (53): '[PAD]', '[UNK]', '_', '$', "''", ',', '-LRB-', '-RRB-', '.', ':', 'ADD', 'AFX', 'CC', 'CD', 'DT', 'EX', 'FW', 'GW', 'HYPH', 'IN', 'JJ', 'JJR', 'JJS', 'LS', 'MD', 'NFP', 'NN', 'NNP', 'NNPS', 'NNS', 'PDT', 'POS', 'PRP', 'PRP$', 'RB', 'RBR', 'RBS', 'RP', 'SYM', 'TO', 'UH', 'VB', 'VBD', 'VBG', 'VBN', 'VBP', 'VBZ', 'WDT', 'WP', 'WP$', 'WRB', '_', '``'
INFO: 22-Feb-26 17:56:27 - Lemma vocabulary (533): [omitted]
INFO: 22-Feb-26 17:56:27 - Features vocabulary (235): [omitted]

Closes #115.

* black update

* f-string fix

* driveby: silence more warnings

* Avoids "Crashed" status in sweeps. (#118)

See Yoyodyne [#369](CUNY-CL/yoyodyne#369) for
context.

Closes #79.

* Pooling layer efficiency (#119)

* Fix pooling layer regression in UDTubeEncoder.forward

Special cases pooling_layers=1 to use last_hidden_state directly, avoiding
unnecessary allocation of all hidden states. This seems to save a lot of
GPU memory.

A few drive-bys:

1. suppress progress bar during test data generation
2. add "not human-readable" to "[omitted]" when logging lemmas
3. actually log features; why not?
4. pass information about which heads to build to the data module too,
   so it logs properly
5. removes _ from "special", since it doesn't require any special
   treatment in actuality; it's just another tag as far as we're
   concerned.
6. Standardizes trailing """: it's on its own line if the comment is
   more than one line.

* regeneration last-minute fix

* Beginning integration

* Adds metrics test.

One major issue is that this requires us to use negative indices for
specials, which breaks assumptions in the indexes. Will have to come
back and fix this.

* Draft of parser and its integration

* More work.

Known issues:

1. I don't think the metrics test is going to work; I will need to shift
   all the head indices by special.OFFSET.
2. I am not passing a parser mask. Do I need to? I think maybe yes.

* Moves reverse_edits to data, where it belongs.

It has no effect in the model, so let's get rid of it.

* Days' debugging work

* More work; still debugging

* Optimizes mmap instructions (#116)

* Pooling layer efficiency (#119)

* Fix pooling layer regression in UDTubeEncoder.forward

Special cases pooling_layers=1 to use last_hidden_state directly, avoiding
unnecessary allocation of all hidden states. This seems to save a lot of
GPU memory.

A few drive-bys:

1. suppress progress bar during test data generation
2. add "not human-readable" to "[omitted]" when logging lemmas
3. actually log features; why not?
4. pass information about which heads to build to the data module too,
   so it logs properly
5. removes _ from "special", since it doesn't require any special
   treatment in actuality; it's just another tag as far as we're
   concerned.
6. Standardizes trailing """: it's on its own line if the comment is
   more than one line.

* regeneration last-minute fix

* Manual merge

* README and bibliography

* stashing incomplete work

* updates parser

* Parser testing

* Expands grid for biaffine parsing hparams

* Adds the parser itself

* Updates tests

Eliminates a test bug where the file comparisons were against the
hypothesis file!

* reflows README

* updates encoder special-casing logic slightly

* Update mappers.py

* Daniel's suggestion

Author: kylebgorman

README.md

@@ -60,31 +60,31 @@ Dependencies project](https://universaldependencies.org/).
 
 UDTube can perform up to four morphological tasks simultaneously:
 
--   Lemmatization is performed using the `LEMMA` field and [edit
-    scripts](https://aclanthology.org/P14-2111/).
-
+-   Lemmatization is performed using the `LEMMA` field and edit scripts.
 -   [Universal part-of-speech
     tagging](https://universaldependencies.org/u/pos/index.html) is performed
-    using the `UPOS` field: enable with `data: use_upos: true`.
-
+    using the `UPOS` field.
 -   Language-specific part-of-speech tagging is performed using the `XPOS`
-    field: enable with `data: use_xpos: true`.
-
--   Morphological feature tagging is performed using the `FEATS` field: enable
-    with `data: use_feats: true`.
+    field.
+-   Morphological feature tagging is performed using the `FEATS` field.
+-   Dependency parsing is performed using the `HEAD` and `DEPREL` fields, a deep
+    biaffine parser, and minimum spanning tree decoding.
 
 The following caveats apply:
 
+-   By default, lemmatization uses reverse-edit scripts. This is appropriate for
+    predominantly suffixal languages, which are thought to represent the
+    majority of the world's languages. If working with a predominantly prefixal
+    language, disable this with `data: reverse_edits: false`.
 -   Note that many newer Universal Dependencies datasets do not have
-    language-specific part-of-speech-tags.
+    language-specific part-of-speech-tags so this task should be disabled
+    (`data: use_xpos: false`).
 -   The `FEATS` field is treated as a single unit and is not segmented in any
     way.
 -   One can convert from [Universal Dependencies morphological
     features](https://universaldependencies.org/u/feat/index.html) to [UniMorph
     features](https://unimorph.github.io/schema/) using
     [`scripts/convert_to_um.py`](scripts/convert_to_um.py).
--   UDTube does not perform dependency parsing at present, so the `HEAD`,
-    `DEPREL`, and `DEPS` fields are ignored and should be specified as `_`.
 
 ## Usage
 
@@ -132,8 +132,9 @@ supported as they lack an `AutoTokenizer`.
 
 #### Classifier
 
-The classifier layer contains up to four sequential linear heads for the four
-tasks described above. By default all four are enabled.
+The classifier layer contains up to four sequential linear heads for the tagging
+tasks, and a biaffine parser head for the parsing task. By default all heads are
+enabled.
 
 #### Optimization
 
@@ -189,7 +190,7 @@ information](https://github.com/CUNY-CL/yoyodyne/blob/master/README.md#logging).
 
 #### Other options
 
-By default, UDTube attempts to model all four tasks; one can disable the
+By default, UDTube attempts to model all five tasks; one can disable the
 language-specific tagging task using `model: use_xpos: false`, and so on.
 
 Dropout probability is specified using `model: dropout: ...`.
@@ -198,25 +199,19 @@ The encoder has multiple layers. The input to the classifier consists of just
 the last few layers mean-pooled together. The number of layers used for
 mean-pooling is specified using `model: pooling_layers: ...`.
 
-By default, lemmatization uses reverse-edit scripts. This is appropriate for
-predominantly suffixal languages, which are thought to represent the majority of
-the world's languages. If working with a predominantly prefixal language,
-disable this with `model: reverse_edits: false`.
-
 The following YAML snippet shows the default architectural arguments.
 
     ...
     model:
       dropout: 0.5
       encoder: google-bert/bert-base-multilingual-cased
       pooling_layers: 1
-      reverse_edits: true
       use_upos: true
       use_xpos: true
       use_lemma: true
       use_feats: true
+      use_parse: true
       ...
-      
 
 Batch size is specified using `data: batch_size: ...` and defaults to 32.
 
@@ -322,3 +317,6 @@ following document, which describes the model:
 Yakubov, D. 2024. [How do we learn what we cannot
 say?](https://academicworks.cuny.edu/gc_etds/5622/) Master's thesis, CUNY
 Graduate Center.
+
+(See also [`udtube.bib`](udtube.bib) for more work used during the development
+of this library.)

configs/ewt_bert.yaml

@@ -22,12 +22,6 @@ trainer:
 model:
   dropout: 0.4
   encoder: google-bert/bert-base-cased
-  pooling_layers: 4
-  reverse_edits: true
-  use_upos: true
-  use_xpos: true
-  use_lemma: true
-  use_feats: true
   encoder_optimizer:
     class_path: torch.optim.Adam
     init_args:

configs/ewt_distilbert.yaml

@@ -22,12 +22,6 @@ trainer:
 model:
   dropout: 0.4
   encoder: distilbert/distilbert-base-cased
-  pooling_layers: 4
-  reverse_edits: true
-  use_upos: true
-  use_xpos: true
-  use_lemma: true
-  use_feats: true
   encoder_optimizer:
     class_path: torch.optim.Adam
     init_args:
@@ -52,6 +46,7 @@ data:
   test: /Users/Shinji/UD_English-EWT/en_ewt-ud-test.conllu
   predict: /Users/Shinji/UD_English-EWT/en_ewt-ud-test.conllu
   batch_size: 32
+  reverse_edits: true
 checkpoint:
   filename: "model-{epoch:03d}-{val_loss:.4f}"
   monitor: val_loss

configs/ewt_roberta.yaml

@@ -22,12 +22,6 @@ trainer:
 model:
   dropout: 0.4
   encoder: FacebookAI/roberta-base
-  pooling_layers: 4
-  reverse_edits: true
-  use_upos: true
-  use_xpos: true
-  use_lemma: true
-  use_feats: true
   encoder_optimizer:
     class_path: torch.optim.Adam
     init_args:

configs/syntagrus_mbert.yaml

@@ -22,12 +22,7 @@ trainer:
 model:
   dropout: 0.4
   encoder: google-bert/bert-base-multilingual-cased
-  pooling_layers: 4
-  reverse_edits: true
-  use_upos: true
   use_xpos: false
-  use_lemma: true
-  use_feats: true
   encoder_optimizer:
     class_path: torch.optim.Adam
     init_args:

configs/syntagrus_rubert.yaml

@@ -22,12 +22,7 @@ trainer:
 model:
   dropout: 0.4
   encoder: DeepPavlov/rubert
-  pooling_layers: 4
-  reverse_edits: true
-  use_upos: true
   use_xpos: false
-  use_lemma: true
-  use_feats: true
   encoder_optimizer:
     class_path: torch.optim.Adam
     init_args:

configs/syntagrus_xlm-roberta.yaml

@@ -22,12 +22,7 @@ trainer:
 model:
   dropout: 0.4
   encoder: FacebookAI/xlm-roberta-base
-  pooling_layers: 4
-  reverse_edits: true
-  use_upos: true
   use_xpos: false
-  use_lemma: true
-  use_feats: true
   encoder_optimizer:
     class_path: torch.optim.Adam
     init_args:

examples/wandb_sweeps/configs/ewt_grid.yaml

@@ -1,15 +1,14 @@
-method: random
+method: bayes
 metric:
   name: val_loss
   goal: minimize
 parameters:
-  model.use_xpos:
-    value: true
   model.dropout:
     distribution: uniform
     min: 0
     max: 0.5
   model.encoder:
+    distribution: categorical
     values:
       - FacebookAI/roberta-base
       - distilbert/distilbert-base-cased
@@ -18,7 +17,7 @@ parameters:
     distribution: q_uniform
     q: 1
     min: 1
-    max: 8
+    max: 4
   model.encoder_optimizer.class_path:
     value: torch.optim.Adam
   model.encoder_optimizer.init_args.lr:
@@ -31,7 +30,17 @@ parameters:
     distribution: q_uniform
     q: 1
     min: 1
-    max: 20
+    max: 40
+  model.arc_mlp_size:
+    distribution: q_uniform
+    q: 64
+    min: 64
+    max: 512
+  model.deprel_mlp_size:
+    distribution: q_uniform
+    q: 64
+    min: 64
+    max: 256
   model.classifier_optimizer.class_path:
     value: torch.optim.Adam
   model.classifier_optimizer.init_args.lr:
@@ -49,6 +58,7 @@ parameters:
   model.classifier_scheduler.init_args.patience:
     value: 5
   data.batch_size:
+    distribution: categorical
     values:
       - 8
       - 16

examples/wandb_sweeps/configs/gdt_grid.yaml

@@ -1,4 +1,4 @@
-method: random
+method: bayes
 metric:
   name: val_loss
   goal: minimize
@@ -10,14 +10,15 @@ parameters:
     min: 0
     max: 0.5
   model.encoder:
+    distribution: categorical
     values:
       - google-bert/bert-base-multilingual-cased
       - FacebookAI/xlm-roberta-base
   model.pooling_layers:
     distribution: q_uniform
     q: 1
     min: 1
-    max: 8
+    max: 4
   model.encoder_optimizer.class_path:
     value: torch.optim.Adam
   model.encoder_optimizer.init_args.lr:
@@ -30,7 +31,17 @@ parameters:
     distribution: q_uniform
     q: 1
     min: 1
-    max: 20
+    max: 40
+  model.arc_mlp_size:
+    distribution: q_uniform
+    q: 64
+    min: 64
+    max: 512
+  model.deprel_mlp_size:
+    distribution: q_uniform
+    q: 64
+    min: 64
+    max: 256
   model.classifier_optimizer.class_path:
     value: torch.optim.Adam
   model.classifier_optimizer.init_args.lr:
@@ -48,6 +59,7 @@ parameters:
   model.classifier_scheduler.init_args.patience:
     value: 5
   data.batch_size:
+    distribution: categorical
     values:
       - 8
       - 16

examples/wandb_sweeps/configs/syntagrus_grid.yaml

@@ -1,4 +1,4 @@
-method: random
+method: bayes
 metric:
   name: val_loss
   goal: minimize
@@ -10,6 +10,7 @@ parameters:
     min: 0
     max: 0.5
   model.encoder:
+    distribution: categorical
     values:
       - google-bert/bert-base-multilingual-cased
       - FacebookAI/xlm-roberta-base
@@ -18,7 +19,7 @@ parameters:
     distribution: q_uniform
     q: 1
     min: 1
-    max: 8
+    max: 4
   model.encoder_optimizer.class_path:
     value: torch.optim.Adam
   model.encoder_optimizer.init_args.lr:
@@ -31,7 +32,17 @@ parameters:
     distribution: q_uniform
     q: 1
     min: 1
-    max: 20
+    max: 40
+  model.arc_mlp_size:
+    distribution: q_uniform
+    q: 64
+    min: 64
+    max: 512
+  model.deprel_mlp_size:
+    distribution: q_uniform
+    q: 64
+    min: 64
+    max: 256
   model.classifier_optimizer.class_path:
     value: torch.optim.Adam
   model.classifier_optimizer.init_args.lr:
@@ -49,6 +60,7 @@ parameters:
   model.classifier_scheduler.init_args.patience:
     value: 5
   data.batch_size:
+    distribution: categorical
     values:
       - 8
       - 16