GENTLE: A Genre-Diverse Multilayer Challenge Set for English NLP and Linguistic Evaluation

What it is about

GENTLE is a small English challenge corpus of about 17k tokens in eight unusual genres: dictionary entries, esports commentary, legal documents, medical notes, poetry, mathematical proofs, syllabuses and threat letters. Each genre is annotated for UD parsing, entities, coreference and discourse. Current systems are run on it, and performance drops for most genres compared with GUM. Dependency parsing from plain text falls to LAS 72.38, about 15 points below the gold-token result, and the worst genres are proofs and syllabuses. Tokenisation fails on bulleted lists and abbreviations, and XPOS tagging fails on mathematical symbols. Human agreement on snippets (LAS 84.66) shows that some of the gap is annotation difficulty.

Ideas

• I1. en ud-eval on GENTLE with gold tokens, per genre: LAS and UAS for en next to EWT test, to get a degradation profile by genre (test only).
• I2. en/src/tok.rs: a rule set for numbered and bulleted list items and legal abbreviations, measured by tokenisation F1 on GENTLE syllabus and legal in end-to-end mode (test only).
• I3. en/src/ptag.rs: a symbol class for mathematical signs (for example ≦, ∈) in the lexicon, measured by XPOS accuracy on the GENTLE proof genre (test only).
• I4. en/src/explain.rs on the paper's human disagreement example: report where en disagrees with the two-annotator gold, to separate model errors from guideline gaps (test only).

For Mova

GENTLE is an out-of-domain test for en, which only the tests may use, because the dataset is listed as non-commercial in Mova's rules. The tokeniser and the tagger are the first points of failure. Relevant chapters are languages/en/07-punctuation-capitals-numbers, 19-lexicon-mwe-register and 37-register-and-variation. Measure first the end-to-end LAS per genre of en on GENTLE, next to its gold-token LAS.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.law-1.17 · arXiv 2306.01966

@inproceedings{aoyama2023gentle,
  title = {{GENTLE: A Genre-Diverse Multilayer Challenge Set for English NLP and Linguistic Evaluation}},
  author = {Aoyama, Tatsuya and Behzad, Shabnam and Gessler, Luke and Levine, Lauren and Lin, Jessica and Liu, Yang Janet and Peng, Siyao and Zhu, Yilun and Zeldes, Amir},
  booktitle = {Proceedings of the 17th Linguistic Annotation Workshop (LAW-XVII)},
  year = {2023},
  doi = {10.18653/v1/2023.law-1.17},
  eprint = {2306.01966},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.law-1.17}
}

References

33 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mitchell Abrams. 2019. Uncovering the genre of threatening texts: A multilayered corpus study. Master’s thesis, Georgetown University. find
2. Katherine Atwell, Junyi Jessy Li, and Malihe Alikhani. 2021. Where are we in discourse relation recognition? In Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 314–325, Singapore and Online. Association for Computational Linguistics. find
3. Isabelle Augenstein, Leon Derczynski, and Kalina Bontcheva. 2017. Generalisation in named entity recognition: A quantitative analysis. Computer Speech & Language, 44:61–83. find
4. Shabnam Behzad and Amir Zeldes. 2020. A cross-genre ensemble approach to robust Reddit part of speech tagging. In Proceedings of the 12th Web as Corpus Workshop, pages 50–56, Marseille, France. European Language Resources Association. find
5. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308. find our card
6. Kim Gerdes. 2013. Collaborative dependency annotation. In Proceedings of the Second International Conference on Dependency Linguistics (DepLing 2013), pages 88–97, Prague, Czech Republic. Charles University in Prague, Matfyzpress, Prague, Czech Republic. find
7. Luke Gessler, Shabnam Behzad, Yang Janet Liu, Siyao Peng, Yilun Zhu, and Amir Zeldes. 2021. Dis-CoDisCo at the DISRPT2021 shared task: A system for discourse segmentation, classification, and connective detection. In Proceedings of the 2nd Shared Task on Discourse Relation Parsing and Treebanking (DISRPT 2021), pages 51–62, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
8. Luke Gessler, Lauren Levine, and Amir Zeldes. 2022. Midas loop: A prioritized human-in-the-loop annotation for large scale multilayer data. In Proceedings of the 16th Linguistic Annotation Workshop (LAW-XVI) within LREC2022, pages 103–110, Marseille, France. European Language Resources Association. find
9. Grigorii Guz and Giuseppe Carenini. 2020. Coreference for discourse parsing: A neural approach. In Proceedings of the First Workshop on Computational Approaches to Discourse, pages 160–167, Online. Association for Computational Linguistics. find
10. Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. NeurIPS. find
11. Vidur Joshi, Matthew Peters, and Mark Hopkins. 2018. Extending a parser to distant domains using a few dozen partially annotated examples. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1190–1199, Melbourne, Australia. Association for Computational Linguistics. find our card
12. Jenna Kanerva and Filip Ginter. 2022. Out-of-domain evaluation of Finnish dependency parsing. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1114–1124, Marseille, France. European Language Resources Association. find
13. Sujian Li, Liang Wang, Ziqiang Cao, and Wenjie Li. 2014. Text-level discourse dependency parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25–35, Baltimore, Maryland. Association for Computational Linguistics. find
14. Jessica Lin and Amir Zeldes. 2021. WikiGUM: Exhaustive entity linking for wikification in 12 genres. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 170–175, Punta Cana, Dominican Republic. Association for Computational Linguistics. ↗
15. Yang Janet Liu and Amir Zeldes. 2023. Why can’t discourse parsing generalize? A thorough investigation of the impact of data diversity. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3104–3122, Dubrovnik, Croatia. Association for Computational Linguistics. find
16. William C. Mann and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization. Text, 8(3):243–281. ↗ our card
17. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Special Issue on Using Large Corpora, Computational Linguistics, 19(2):313–330. find our card
18. Nafise Sadat Moosavi and Michael Strube. 2017. Lexical features in coreference resolution: To be used with caution. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 14–19, Vancouver, Canada. Association for Computational Linguistics. find our card
19. Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017. How much progress have we made on RST discourse parsing? a replication study of recent results on the RST-DT. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1319–1324, Copenhagen, Denmark. Association for Computational Linguistics. find our card
20. Anmol Nayak, Hariprasad Timmapathini, Karthikeyan Ponnalagu, and Vijendran Gopalan Venkoparao. 2020. Domain adaptation challenges of BERT in tokenization and sub-word representations of out-of-vocabulary words. In Proceedings of the First Workshop on Insights from Negative Results in NLP, pages 1–5, Online. Association for Computational Linguistics. find
21. Minh Van Nguyen, Viet Dac Lai, Amir Pouran Ben Veyseh, and Thien Huu Nguyen. 2021. Trankit: A lightweight transformer-based toolkit for multilingual natural language processing. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 80–90, Online. Association for Computational Linguistics. find
22. Siyao Peng and Amir Zeldes. 2018. All roads lead to UD: Converting Stanford and Penn parses to English Universal Dependencies with multilayer annotations. In Proceedings of the Joint Workshop on Linguistic Annotation, Multiword Expressions and Constructions (LAW-MWE-CxG-2018), pages 167–177, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card
23. Barbara Plank. 2016. What to do about non-standard (or non-canonical) language in NLP. KONVENS 2016 Invited Talk. find
24. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101–108, Online. Association for Computational Linguistics. find our card
25. Manuela Sanguinetti, Lauren Cassidy, Cristina Bosco, Özlem Çetinoğlu, Alessandra Teresa Cignarella, Teresa Lynn, Ines Rehbein, Josef Ruppenhofer, Djamé Seddah, and Amir Zeldes. 2022. Treebanking user-generated content: A UD based overview of guidelines, corpora and unified recommendations. Language Resources and Evaluation, 57:493–544. find
26. Amir Zeldes and Dan Simonson. 2016. Different flavors of GUM: Evaluating genre and sentence type effects on multilayer corpus annotation quality. In Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016), pages 68–78, Berlin, Germany. Association for Computational Linguistics. find our card
27. Shuo Zhang and Amir Zeldes. 2017. GitDOX: A linked version controlled online XML editor for manuscript transcription. In Proceedings of FLAIRS 2017, pages 619–623, Marco Island, FL. find
28. Yilun Zhu, Sameer Pradhan, and Amir Zeldes. 2021. OntoGUM: Evaluating contextualized SOTA coreference resolution on 12 more genres. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 461–467, Online. Association for Computational Linguistics. ↗
29. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 2897– 2904, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗
30. Zeqi Tan, Yongliang Shen, Shuai Zhang, Weiming Lu, and Yueting Zhuang. 2021. A sequence-to-set network for nested named entity recognition. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 3936– 3942. International Joint Conferences on Artificial Intelligence Organization. Main Track. find
31. Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Nianwen Xue, Martha Palmer, Jena D. Hwang, Claire Bonial, Jinho Choi, Aous Mansouri, Maha Foster, Abdel aati Hawwary, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, Robert Belvin, and Ann Houston. 2012. OntoNotes release 5.0. Technical report, Linguistic Data Consortium, Philadelphia. find
32. Amir Zeldes. 2016. rstWeb - a browser-based annotation interface for Rhetorical Structure Theory and discourse relations. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 1–5, San Diego, California. Association for Computational Linguistics. find
33. Amir Zeldes. 2017. The GUM Corpus: Creating Multilayer Resources in the Classroom. Language Resources and Evaluation, 51(3):581–612. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/treebanks-2023-aoyama-gentle-genre-diverse-multilayer-challenge/