Skip to content

README.md

Latest commit

 

History

History
227 lines (208 loc) · 20.2 KB

README.md

File metadata and controls

227 lines (208 loc) · 20.2 KB

Targeted Syntactic Evaluation of Language Models (LM-SynEval)

Paper

Title: Targeted Syntactic Evaluation of Language Models

Authors:: Rebecca Marvin and Tal Linzen

Link: https://doi.org/10.18653/v1/D18-1151

Abstract:

We present a data set for evaluating the grammaticality of the predictions of a language model. We automatically construct a large number of minimally different pairs of English sentences, each consisting of a grammatical and an ungrammatical sentence. The sentence pairs represent different variations of structure-sensitive phenomena: subject-verb agreement, reflexive anaphora and negative polarity items. We expect a language model to assign a higher probability to the grammatical sentence than the ungrammatical one. In an experiment using this data set, an LSTM language model performed poorly on many of the constructions. Multi-task training with a syntactic objective (CCG supertagging) improved the LSTM's accuracy, but a large gap remained between its performance and the accuracy of human participants recruited online. This suggests that there is considerable room for improvement over LSTMs in capturing syntax in a language model.

Homepage: https://github.com/BeckyMarvin/LM_syneval

Language(s): English

License: MIT License

Citation

@inproceedings{marvin-linzen-2018-targeted,
    title = "Targeted Syntactic Evaluation of Language Models",
    author = "Marvin, Rebecca  and
      Linzen, Tal",
    editor = "Riloff, Ellen  and
      Chiang, David  and
      Hockenmaier, Julia  and
      Tsujii, Jun{'}ichi",
    booktitle = "Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing",
    year = "2018",
    address = "Brussels, Belgium",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/D18-1151/",
    doi = "10.18653/v1/D18-1151",
    pages = "1192--1202"
}

Groups, Tags, and Tasks

The tasks are structured hierarchically as listed below. For more detailed explanations, see original paper and repository (linked above). In this implementation, group means are unweighted.

  • lm_syneval: Targeted Syntactic Evaluation of Language Models
    • lm_syneval__agreement: Agreement
      • lm_syneval__agreement__simple_agrmt: Simple agreement
        • lm_syneval__agreement__simple_agrmt__sing_MS_MV:
          • Example: 'The author laughs.' (correct) vs. 'The author laugh.' (incorrect)
        • lm_syneval__agreement__simple_agrmt__plur_MS_MV:
          • Example: 'The authors laugh.' (correct) vs. 'The authors laughs.' (incorrect)
      • lm_syneval__agreement__prep_anim: Agreement across a prepositional phrase with animate subject
        • lm_syneval__agreement__prep_anim__sing_MS_MV_sing_ES:
          • Example: 'The author next to the guard laughs.' (correct) vs. 'The author next to the guard laugh.' (incorrect)
        • lm_syneval__agreement__prep_anim__sing_MS_MV_plur_ES:
          • Example: 'The author next to the guards laughs.' (correct) vs. 'The author next to the guards laugh.' (incorrect)
        • lm_syneval__agreement__prep_anim__plur_MS_MV_sing_ES:
          • Example: 'The authors next to the guard laugh.' (correct) vs. 'The authors next to the guard laughs.' (incorrect)
        • lm_syneval__agreement__prep_anim__plur_MS_MV_plur_ES:
          • Example: 'The authors next to the guards laugh.' (correct) vs. 'The authors next to the guards laughs.' (incorrect)
      • lm_syneval__agreement__prep_inanim: Agreement across a prepositional phrase with inanimate subject
        • lm_syneval__agreement__prep_inanim__sing_IS_IV_sing_ES:
          • Example: 'The movie from the guard is good.' (correct) vs. 'The movie from the guard are good.' (incorrect)
        • lm_syneval__agreement__prep_inanim__sing_IS_IV_plur_ES:
          • Example: 'The movie from the guards is good.' (correct) vs. 'The movie from the guards are good.' (incorrect)
        • lm_syneval__agreement__prep_inanim__plur_IS_IV_sing_ES:
          • Example: 'The movies from the guard are good.' (correct) vs. 'The movies from the guard is good.' (incorrect)
        • lm_syneval__agreement__prep_inanim__plur_IS_IV_plur_ES:
          • Example: 'The movies from the guards are good.' (correct) vs. 'The movies from the guards is good.' (incorrect)
      • lm_syneval__agreement__sent_comp: Agreement in a sentential complement
        • lm_syneval__agreement__sent_comp__sing_MS_MV_sing_BS:
          • Example: 'The mechanic said the author laughs.' (correct) vs. 'The mechanic said the author laugh.' (incorrect)
        • lm_syneval__agreement__sent_comp__sing_MS_MV_plur_BS:
          • Example: 'The mechanics said the author laughs.' (correct) vs. 'The mechanics said the author laugh.' (incorrect)
        • lm_syneval__agreement__sent_comp__plur_MS_MV_sing_BS:
          • Example: 'The mechanic said the authors laugh.' (correct) vs. 'The mechanic said the authors laughs.' (incorrect)
        • lm_syneval__agreement__sent_comp__plur_MS_MV_plur_BS:
          • Example: 'The mechanics said the authors laugh.' (correct) vs. 'The mechanics said the authors laughs.' (incorrect)
      • lm_syneval__agreement__subj_rel: Agreement across a subject relative clause
        • lm_syneval__agreement__subj_rel__sing_MS_EV_MV_sing_ES:
          • Example: 'The author that likes the guard laughs.' (correct) vs. 'The author that likes the guard laugh.' (incorrect)
        • lm_syneval__agreement__subj_rel__sing_MS_EV_MV_plur_ES:
          • Example: 'The author that likes the guards laughs.' (correct) vs. 'The author that likes the guards laugh.' (incorrect)
        • lm_syneval__agreement__subj_rel__plur_MS_EV_MV_sing_ES:
          • Example: 'The authors that like the guard laugh.' (correct) vs. 'The authors that like the guard laughs.' (incorrect)
        • lm_syneval__agreement__subj_rel__plur_MS_EV_MV_plur_ES:
          • Example: 'The authors that like the guards laugh.' (correct) vs. 'The authors that like the guards laughs.' (incorrect)
      • lm_syneval__agreement__vp_coord: Short verb phrase coordination
        • lm_syneval__agreement__vp_coord__sing_MS_MV_MV:
          • Example: 'The author laughs and swims.' (correct) vs. 'The author laughs and swim.' (incorrect)
        • lm_syneval__agreement__vp_coord__plur_MS_MV_MV:
          • Example: 'The authors laugh and swim.' (correct) vs. 'The authors laugh and swims.' (incorrect)
      • lm_syneval__agreement__long_vp_coord: Long verb phrase coordination
        • lm_syneval__agreement__long_vp_coord__sing_MS_LMV_LMV:
          • Example: 'The author knows many different foreign languages and likes to watch television shows.' (correct) vs. 'The author knows many different foreign languages and like to watch television shows.' (incorrect)
        • lm_syneval__agreement__long_vp_coord__plur_MS_LMV_LMV:
          • Example: 'The authors know many different foreign languages and like to watch television shows.' (correct) vs. 'The authors know many different foreign languages and likes to watch television shows.' (incorrect)
      • lm_syneval__agreement__obj_rel_within_anim: Agreement in an object relative clause with animate external subject
        • lm_syneval__agreement__obj_rel_within_anim__sing_ES_EV_sing_MS_MV:
          • Example: 'The author that the guard likes laughs.' (correct) vs. 'The author that the guard like laughs.' (incorrect)
        • lm_syneval__agreement__obj_rel_within_anim__sing_ES_EV_plur_MS_MV:
          • Example: 'The authors that the guard likes laugh.' (correct) vs. 'The authors that the guard like laugh.' (incorrect)
        • lm_syneval__agreement__obj_rel_within_anim__plur_ES_EV_sing_MS_MV:
          • Example: 'The author that the guards like laughs.' (correct) vs. 'The author that the guards likes laughs.' (incorrect)
        • lm_syneval__agreement__obj_rel_within_anim__plur_ES_EV_plur_MS_MV:
          • Example: 'The authors that the guards like laugh.' (correct) vs. 'The authors that the guards likes laugh.' (incorrect)
      • lm_syneval__agreement__obj_rel_within_inanim: Agreement in an object relative clause with inanimate external subject
        • lm_syneval__agreement__obj_rel_within_inanim__sing_ES_EV_sing_IS_IV:
          • Example: 'The movie that the guard likes is good.' (correct) vs. 'The movie that the guard like is good.' (incorrect)
        • lm_syneval__agreement__obj_rel_within_inanim__sing_ES_EV_plur_IS_IV:
          • Example: 'The movies that the guard likes are good.' (correct) vs. 'The movies that the guard like are good.' (incorrect)
        • lm_syneval__agreement__obj_rel_within_inanim__plur_ES_EV_sing_IS_IV:
          • Example: 'The movie that the guards like is good.' (correct) vs. 'The movie that the guards likes is good.' (incorrect)
        • lm_syneval__agreement__obj_rel_within_inanim__plur_ES_EV_plur_IS_IV:
          • Example: 'The movies that the guards like are good.' (correct) vs. 'The movies that the guards likes are good.' (incorrect)
      • lm_syneval__agreement__obj_rel_across_anim: Agreement across an object relative clause with animate external subject
        • lm_syneval__agreement__obj_rel_across_anim__sing_MS_MV_sing_ES_EV:
          • Example: 'The author that the guard likes laughs.' (correct) vs. 'The author that the guard likes laugh.' (incorrect)
        • lm_syneval__agreement__obj_rel_across_anim__sing_MS_MV_plur_ES_EV:
          • Example: 'The author that the guards like laughs.' (correct) vs. 'The author that the guards like laugh.' (incorrect)
        • lm_syneval__agreement__obj_rel_across_anim__plur_MS_MV_sing_ES_EV:
          • Example: 'The authors that the guard likes laugh.' (correct) vs. 'The authors that the guard likes laughs.' (incorrect)
        • lm_syneval__agreement__obj_rel_across_anim__plur_MS_MV_plur_ES_EV:
          • Example: 'The authors that the guards like laugh.' (correct) vs. 'The authors that the guards like laughs.' (incorrect)
      • lm_syneval__agreement__obj_rel_across_inanim: Agreement across an object relative clause with inanimate external subject
        • lm_syneval__agreement__obj_rel_across_inanim__sing_IS_IV_sing_ES_EV:
          • Example: 'The movie that the guard likes is good.' (correct) vs. 'The movie that the guard likes are good.' (incorrect)
        • lm_syneval__agreement__obj_rel_across_inanim__sing_IS_IV_plur_ES_EV:
          • Example: 'The movie that the guards like is good.' (correct) vs. 'The movie that the guards like are good.' (incorrect)
        • lm_syneval__agreement__obj_rel_across_inanim__plur_IS_IV_sing_ES_EV:
          • Example: 'The movies that the guard likes are good.' (correct) vs. 'The movies that the guard likes is good.' (incorrect)
        • lm_syneval__agreement__obj_rel_across_inanim__plur_IS_IV_plur_ES_EV:
          • Example: 'The movies that the guards like are good.' (correct) vs. 'The movies that the guards like is good.' (incorrect)
      • lm_syneval__agreement__obj_rel_no_comp_within_anim: Agreement in an object relative clause (no that) with animate external subject
        • lm_syneval__agreement__obj_rel_no_comp_within_anim__sing_ES_EV_sing_MS_MV:
          • Example: 'The author the guard likes laughs.' (correct) vs. 'The author the guard like laughs.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_within_anim__sing_ES_EV_plur_MS_MV:
          • Example: 'The authors the guard likes laugh.' (correct) vs. 'The authors the guard like laugh.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_within_anim__plur_ES_EV_sing_MS_MV:
          • Example: 'The author the guards like laughs.' (correct) vs. 'The author the guards likes laughs.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_within_anim__plur_ES_EV_plur_MS_MV:
          • Example: 'The authors the guards like laugh.' (correct) vs. 'The authors the guards likes laugh.' (incorrect)
      • lm_syneval__agreement__obj_rel_no_comp_within_inanim: Agreement in an object relative clause (no that) with inanimate external subject
        • lm_syneval__agreement__obj_rel_no_comp_within_inanim__sing_ES_EV_sing_IS_IV:
          • Example: 'The movie the guard likes is good.' (correct) vs. 'The movie the guard like is good.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_within_inanim__sing_ES_EV_plur_IS_IV:
          • Example: 'The movies the guard likes are good.' (correct) vs. 'The movies the guard like are good.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_within_inanim__plur_ES_EV_sing_IS_IV:
          • Example: 'The movie the guards like is good.' (correct) vs. 'The movie the guards likes is good.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_within_inanim__plur_ES_EV_plur_IS_IV:
          • Example: 'The movies the guards like are good.' (correct) vs. 'The movies the guards likes are good.' (incorrect)
      • lm_syneval__agreement__obj_rel_no_comp_across_anim: Agreement across an object relative clause (no that) with animate external subject
        • lm_syneval__agreement__obj_rel_no_comp_across_anim__sing_MS_MV_sing_ES_EV:
          • Example: 'The author the guard likes laughs.' (correct) vs. 'The author the guard like laughs.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_across_anim__sing_MS_MV_plur_ES_EV:
          • Example: 'The authors the guard likes laugh.' (correct) vs. 'The authors the guard like laugh.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_across_anim__plur_MS_MV_sing_ES_EV:
          • Example: 'The author the guards like laughs.' (correct) vs. 'The author the guards likes laughs.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_across_anim__plur_MS_MV_plur_ES_EV:
          • Example: 'The authors the guards like laugh.' (correct) vs. 'The authors the guards likes laugh.' (incorrect)
      • lm_syneval__agreement__obj_rel_no_comp_across_inanim: Agreement across an object relative clause (no that) with inanimate external subject
        • lm_syneval__agreement__obj_rel_no_comp_across_inanim__sing_IS_IV_sing_ES_EV:
          • Example: 'The movie the guard likes is good.' (correct) vs. 'The movie the guard likes are good.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_across_inanim__sing_IS_IV_plur_ES_EV:
          • Example: 'The movie the guards like is good.' (correct) vs. 'The movie the guards like are good.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_across_inanim__plur_IS_IV_sing_ES_EV:
          • Example: 'The movies the guard likes are good.' (correct) vs. 'The movies the guard likes is good.' (incorrect)
        • lm_syneval__agreement__obj_rel_no_comp_across_inanim__plur_IS_IV_plur_ES_EV:
          • Example: 'The movies the guards like are good.' (correct) vs. 'The movies the guards like is good.' (incorrect)
    • lm_syneval__reflexives: Reflexive anaphora
      • lm_syneval__reflexives__simple_reflexives: Simple Reflexives
        • lm_syneval__reflexives__simple_reflexives__sing_MS_ANPHR:
          • Example: 'The author hurt himself.' (correct) vs 'The author hurt themselves.' (incorrect)
        • lm_syneval__reflexives__simple_reflexives__plur_MS_ANPHR:
          • Example: 'The authors hurt themselves.' (correct) vs. 'The authors hurt himself.' (incorrect)
      • lm_syneval__reflexives__reflexive_sent_comp: Reflexives in a sentential complement
        • lm_syneval__reflexives__reflexive_sent_comp__sing_MS_ANPHR_sing_BS:
          • Example: 'The mechanic said the author hurt himself.' (correct) vs. 'The mechanic said the author hurt themselves.' (incorrect)
        • lm_syneval__reflexives__reflexive_sent_comp__sing_MS_ANPHR_plur_BS:
          • Example: 'The mechanics said the author hurt himself.' (correct) vs. 'The mechanics said the author hurt themselves.' (incorrect)
        • lm_syneval__reflexives__reflexive_sent_comp__plur_MS_ANPHR_sing_BS:
          • Example: 'The mechanic said the authors hurt themselves.' (correct) vs. 'The mechanic said the authors hurt himself.' (incorrect)
        • lm_syneval__reflexives__reflexive_sent_comp__plur_MS_ANPHR_plur_BS:
          • Example: 'The mechanics said the authors hurt themselves.' (correct) vs. 'The mechanics said the authors hurt himself.' (incorrect)
      • lm_syneval__reflexives__reflexives_across: Reflexive across an object relative clause
        • lm_syneval__reflexives__reflexives_across__sing_MS_ANPHR_sing_ES_EV:
          • Example: 'The author that the guard likes hurt himself.' (correct) vs. 'The author that the guard likes hurt themselves.' (incorrect)
        • lm_syneval__reflexives__reflexives_across__sing_MS_ANPHR_plur_ES_EV:
          • Example: 'The author that the guards like hurt himself.' (correct) vs. 'The author that the guards like hurt themselves.' (incorrect)
        • lm_syneval__reflexives__reflexives_across__plur_MS_ANPHR_sing_ES_EV:
          • Example: 'The authors that the guard likes hurt themselves.' (correct) vs. 'The authors that the guard likes hurt himself.' (incorrect)
        • lm_syneval__reflexives__reflexives_across__plur_MS_ANPHR_plur_ES_EV:
          • Example: 'The authors that the guards like hurt themselves.' (correct) vs. 'The authors that the guards like hurt himself.' (incorrect)
    • lm_syneval__npi: Negative polarity items
      • lm_syneval__npi__simple_npi_anim: Simple NPI with animate subject
        • lm_syneval__npi__simple_npi_anim__past:
          • Example: 'No authors have ever been popular.' (correct) vs. 'The authors have ever been popular.' (incorrect)
        • lm_syneval__npi__simple_npi_anim__future:
          • Example: 'No authors will ever be popular.' (correct) vs. 'The authors will ever be popular.' (incorrect)
      • lm_syneval__npi__simple_npi_inanim: Simple NPI with imanimate subject
        • lm_syneval__npi__simple_npi_inanim__past:
          • Example: 'No movies have ever been seen.' (correct) vs. 'The movies have ever been seen.' (incorrect)
        • lm_syneval__npi__simple_npi_inanim__future:
          • Example: 'No movies will ever be seen.' (correct) vs. 'The movies will ever be seen.' (incorrect)
      • lm_syneval__npi__npi_across_anim: NPI across a relative clause with animate subject
        • lm_syneval__npi__npi_across_anim__past:
          • Example: 'No authors that the guards like have ever been popular.' (correct) vs. 'The authors that no guards like have ever been popular.' (incorrect)
        • lm_syneval__npi__npi_across_anim__future:
          • Example: 'No authors that the guards like will ever be popular.' (correct) vs. 'The authors that no guards like will ever be popular.' (incorrect)
      • lm_syneval__npi__npi_across_inanim: NPI across a relative clause with imanimate subject
        • lm_syneval__npi__npi_across_inanim__past:
          • Example: 'No movies that the guards like have ever been seen.' (correct) vs. 'The movies that no guards like have ever been seen.' (incorrect)
        • lm_syneval__npi__npi_across_inanim__future:
          • Example: 'No movies that the guards like will ever be seen.' (correct) vs. 'The movies that no guards like will ever be seen.' (incorrect)

Checklist

For adding novel benchmarks/datasets to the library:

  • Is the task an existing benchmark in the literature?
    • Have you referenced the original paper that introduced the task?
    • If yes, does the original paper provide a reference implementation? If so, have you checked against the reference implementation and documented how to run such a test?
      • The original paper evaluates traditional RNN models, which require a very different pipeline to analyze.

Changelog