Are UD Treebanks Getting More Consistent? A Report Card for English UD

Summary

The paper checks whether the two largest English UD treebanks, EWT and GUM, which have different origins and conversion histories, are converging, and whether it makes sense to train joint models on them. First, each pair of consecutive releases from v2.6 to v2.11 is compared with the official CoNLL 2018 scorer, treating the newer release as gold, to locate the changes in tokenization, tags, lemmas and dependencies that caused the largest shifts. Then a parser is trained on every version of each corpus and on both combined, and tested across corpora. Consistency grows with each release both within and between the treebanks, and v2.11 is best on all metrics. Joint training still lags slightly behind in-domain training, but the gap has become very small, so a combined model is reasonable for new genres. The method applies to any language with several treebanks.

How Mova uses it

Background for the English grammar expert system (en/src/expert.rs), whose rules live in en/seeds/errors/:

• english-treebank-conventions.md: divergences between EWT and GUM are treated as conventions, not rule errors; a much higher violation count on GUM than on EWT signals a convention difference.
• adjectives-in-names.md (rule en.errors.adj-not-compound): since v2.8, adjectives in proper names are ADJ with amod, not compound.
• dep-last-resort.md (rule en.errors.dep-last-resort): dep should be rare, following the v2.11 clean-up of dep uses.
• advcl-on-noun.md, flat-compound-direction.md: supporting examples from the paper's appendix.
• consistency-detection-methods.md: one of several sources on detecting annotation errors automatically (release-to-release comparison).

Effectiveness in Mova

Not measured separately. Background reading behind several annotation-checking rules. Rule statistics on the reference treebanks: ADJ as compound of a noun — EWT 2.18: 14, GUM: 57; dep — EWT 2.18: 5, GUM 2.18: 184; advcl under a noun — EWT 2.18: 201 hits, 15 violations.

👨‍🔬💥

More ideas (a second reading)

• I1. Release report for Mova: keep a table of en parse-eval LAS on EWT test for each treebank release and each conversion rule set in treebanks/en/ewt/. Report three seeds with the mean and the standard deviation, as the paper does, so that a change in the data is not confused with seed noise.
• I2. Cross-corpus check: train en on EWT train and test on GUM test, and the reverse. GUM is NC, so the GUM test set is for measurement only, and nothing from GUM goes into the built model. The gap between the two directions is a consistency signal.
• I3. Frozen document subset: the paper fixes the GUM test to documents present since v2.6, so that scores are comparable across releases. Apply the same rule in en folds: split by document id, not by sentence, and keep the document list fixed when the treebank grows.

Topics: Universal Dependencies · Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2302.00636

@inproceedings{zeldes2023are,
  title = {{Are UD Treebanks Getting More Consistent? A Report Card for English UD}},
  author = {Zeldes, Amir and Schneider, Nathan},
  booktitle = {Proceedings of the Sixth Workshop on Universal Dependencies (UDW, GURT/SyntaxFest 2023)},
  year = {2023},
  eprint = {2302.00636},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.00636}
}

References

23 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akshay Aggarwal and Daniel Zeman. 2020. Estimating POS annotation consistency of different treebanks in a language. In Proceedings of the 19th International Workshop on Treebanks and Linguistic Theories, pages 93–110, Düsseldorf, Germany. Association for Computational Linguistics. find our card
2. Héctor Martínez Alonso and Daniel Zeman. 2016. Universal Dependencies for the AnCora treebanks. Procesamiento del Lenguaje Natural, 57:91–98. find
3. Giuseppe Attardi, Daniele Sartiano, and Maria Simi. 2021. Biaffine dependency and semantic graph parsing for Enhanced Universal dependencies. In Proceedings of the 17th International Conference on Parsing Technologies and the IWPT 2021 Shared Task on Parsing into Enhanced Universal Dependencies (IWPT 2021), pages 184–188, Online. Association for Computational Linguistics. find
4. Shabnam Behzad and Amir Zeldes. 2020. A cross-genre ensemble approach to robust Reddit part of speech tagging. In Proceedings of the 12th Web as Corpus Workshop, pages 50–56, Marseille, France. European Language Resources Association. find
5. Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English Web Treebank. Technical Report LDC2012T13, Linguistic Data Consortium, Philadelphia, PA. find
6. Gosse Bouma, Jan Hajic, Dag Haug, Joakim Nivre, Per Erik Solberg, and Lilja Øvrelid. 2018. Expletives in Universal Dependency treebanks. In Proceedings of the Second Workshop on Universal Dependencies (UDW 2018), pages 18–26, Brussels, Belgium. Association for Computational Linguistics. find
7. Marie-Catherine de Marneffe, Matias Grioni, Jenna Kanerva, and Filip Ginter. 2017. Assessing the annotation consistency of the Universal Dependencies corpora. In Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017), pages 108–115, Pisa,Italy. Linköping University Electronic Press. find our card
8. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of LREC 2006, pages 449–454, Genoa, Italy. find our card
9. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308. find our card
10. Tillmann Dönicke, Xiang Yu, and Jonas Kuhn. 2020. Identifying and handling cross-treebank inconsistencies in UD: A pilot study. In Proceedings of the Fourth Workshop on Universal Dependencies (UDW 2020), pages 67–75, Barcelona, Spain (Online). Association for Computational Linguistics. find our card
11. Kira Droganova, Olga Lyashevskaya, and Daniel Zeman. 2018. Data conversion and consistency of monolingual corpora: Russian UD treebanks. In Proceedings of the 17th International Workshopon Treebanks and Linguistic Theories (TLT 2018), pages 53–66„ Linköping, Sweden. Linköping University Electronic Press. find
12. Djamé Seddah, and Amir Zeldes. 2022. Treebanking user-generated content: a UD based overview of guidelines, corpora and unified recommendations. Language Resources and Evaluation. find
13. Nathan Schneider and Amir Zeldes. 2021. Mischievous nominal constructions in Universal Dependencies. In Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021), pages 160– 172, Sofia, Bulgaria. Association for Computational Linguistics. find our card
14. Dmytro Kalpakchi and Johan Boye. 2021. Minor changes make a difference: a case study on the consistency of UD-based dependency parsers. In Proceedings of the Fifth Workshop on Universal Dependencies (UDW, SyntaxFest 2021), pages 96–108, Sofia, Bulgaria. Association for Computational Linguistics. find our card
15. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Christopher D. Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). find
16. Fernando Sánchez León. 2020. Combining different parsers and datasets for CAPITEL UD parsing. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2020) co-located with 36th Conference of the Spanish Society for Natural Language Processing (SEPLN 2020), Málaga, Spain, September 23th, 2020, volume 2664 of CEUR Workshop Proceedings, pages 39–44. CEUR-WS.org. find
17. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗
18. Amir Zeldes, Nick Howell, Noam Ordan, and Yifat Ben Moshe. 2022. A second wave of UD Hebrew treebanking and cross-domain parsing. In Proceedings of EMNLP 2022, Abu Dhabi, UAE. find our card
19. Yang Janet Liu and Amir Zeldes. 2023. Why can’t discourse parsing generalize? A thorough investigation of the impact of data diversity. In Proceedings of EACL 2023, Dubrovnik, Croatia. find
20. Amir Zeldes and Dan Simonson. 2016. Different flavors of GUM: Evaluating genre and sentence type effects on multilayer corpus annotation quality. In Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016), pages 68–78, Berlin, Germany. Association for Computational Linguistics. find our card
21. Max Müller-Eberstein, Rob van der Goot, and Barbara Plank. 2021. Genre as weak supervision for cross-lingual dependency parsing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4786–4802, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
22. Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 shared task: Multilingual parsing from raw text to Universal Dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–21, Brussels, Belgium. Association for Computational Linguistics. find our card
23. Siyao Peng, Yang Janet Liu, and Amir Zeldes. 2022. GCDT: A Chinese RST treebank for multigenre and multilingual discourse parsing. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 382–391, Online only. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/zeldes-2023-are-ud-treebanks-getting-more/