Estimating POS Annotation Consistency of Different Treebanks in a Language

Summary

The paper asks whether several treebanks of the same language, all nominally following the Universal Dependencies guidelines, actually tag parts of speech in the same way. It builds on an asymmetric KL-divergence measure over UPOS trigram distributions (KLcpos3, from Rosa and Žabokrtský 2015), computes it in both directions and adds the two to get a symmetric score, θpos. To make the score interpretable, the authors derive thresholds empirically, separating the effect of data size (cross-validation on Czech-PDT and Estonian-EDT news text) from the effect of genre mix. Roughly, θpos at or below 0.5 indicates consistent annotation and θpos at or above 4.0 inconsistent annotation, with intermediate thresholds depending on whether genres differ. They apply the test to treebank pairs in UD 2.5 and show it can localise a problem, for example to the grammar-examples section of a Finnish treebank. The method does not depend on the quality of any tagger or parser, unlike cross-training approaches, but it only covers POS tags, not syntax.

How Mova uses it

• Summarised as one of five methods for detecting annotation errors automatically in the seed en/seeds/errors/consistency-detection-methods.md, which guides how the grammar expert system (en/src/expert.rs) and its rules are interpreted.
• The idea adopted is the principle that treebank inconsistency leaves statistical traces that can be detected without a trained model; the θpos measure itself is not implemented in code.

Effectiveness in Mova

Background reading only; not implemented and not measured separately.

👨‍🔬💥

Topics: Treebanks · Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.tlt-1.9

@inproceedings{aggarwal2020estimating,
  title = {{Estimating POS Annotation Consistency of Different Treebanks in a Language}},
  author = {Aggarwal, Akshay and Zeman, Daniel},
  booktitle = {Proceedings of the 19th International Workshop on Treebanks and Linguistic Theories},
  year = {2020},
  doi = {10.18653/v1/2020.tlt-1.9},
  url = {https://doi.org/10.18653/v1/2020.tlt-1.9}
}

References

24 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Chiara Alzetta, Felice Dell’Orletta, Simonetta Montemagni, and Giulia Venturi. 2018. Universal Dependencies and Quantitative Typological Trends. A Case Study on Word Order. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May.… find
2. Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018. Towards Better UD Parsing: Deep Contextualized Word Embeddings, Ensemble, and Treebank Concatenation. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 55–64, Brussels,… find
3. Jayeol Chun, Na-Rae Han, Jena D. Hwang, and Jinho D. Choi. 2018. Building Universal Dependency Treebanks in Korean. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), pages 2194–2202, Miyazaki, Japan, May. European Language Resources Association… find
4. Markus Dickinson and W. Detmar Meurers. 2003a. Detecting Errors in Part-of-speech Annotation. In Proceedings of the Tenth Conference on European Chapter of the Association for Computational Linguistics - Volume 1, EACL ’03, pages 107–114, Stroudsburg, PA, USA. Association for Computational Linguisti… find
5. Markus Dickinson and W. Detmar Meurers. 2003b. Detecting Inconsistencies in Treebanks. IEEE Transactions on Learning Technologies - TLT, 01. find
6. Markus Dickinson and W. Detmar Meurers. 2005. Detecting Errors in Discontinuous Structural Annotation. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 322–329, Stroudsburg, PA, USA. Association for Computational Linguistics. find
7. Kira Droganova, Olga Lyashevskaya, and Daniel Zeman. 2018. Data Conversion and Consistency of Monolingual Corpora: Russian UD Treebanks. In Proceedings of the 17th International Workshop on Treebanks and Linguistic Theories (TLT 2018), number 155, pages 53–66, Linköping, Sweden. Linköping University… find
8. Tuomo Kakkonen. 2006. Dependency treebanks: methods, annotation schemes and tools. In Proceedings of the 15th Nordic Conference of Computational Linguistics (NODALIDA 2005), pages 94–104, Joensuu, Finland, May. University of Joensuu, Finland. find
9. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: Annotating Predicate Argument Structure. In Proceedings of the Workshop on Human Language Technology, HLT ’94, page 114–119, USA. Association fo… find our card
10. Héctor Martínez Alonso and Daniel Zeman. 2016. Universal Dependencies for the AnCora treebanks. Procesamiento del Lenguaje Natural, (57):91–98. find
11. Héctor Martínez Alonso, Željko Agić, Barbara Plank, and Anders Søgaard. 2017. Parsing Universal Dependencies without training. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 230–240, Valencia, Spain, April.… find
12. Joakim Nivre, Željko Agić, Lars Ahrenberg, Maria Jesus Aranzabe, Masayuki Asahara, Aitziber Atutxa, Miguel Ballesteros, John Bauer, Kepa Bengoetxea, Yevgeni Berzak, Riyaz Ahmad Bhat, Cristina Bosco, Gosse Bouma, Sam Bowman, Gülşen Cebiroğlu Eryiğit, Giuseppe G. A. Celano, Çağrı Çöltekin, Miriam Conn… find
13. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016b. Universal Dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth I… find our card
14. Joakim Nivre, Mitchell Abrams, Željko Agić, Lars Ahrenberg, Lene Antonsen, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, John Bauer… find
15. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher D. Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the Twelfth International Conference on… find
16. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A Universal Part-of-Speech Tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 2089–2096, Istanbul, Turkey, May. European Languages Resources Association (ELRA). find
17. Martin Popel, Zdeněk Žabokrtský, and Martin Vojtek. 2017. Udapi: Universal API for Universal Dependencies. In Proceedings of the NoDaLiDa 2017 Workshop on Universal Dependencies (UDW 2017), pages 96–101, Gothenburg, Sweden, May. Association for Computational Linguistics. find our card
18. Rudolf Rosa and Zdeněk Žabokrtský. 2015. KLcpos3 - a Language Similarity Measure for Delexicalized Parser Transfer. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Pa… find
19. Daniel Zeman, Ondřej Dušek, David Mareček, Martin Popel, Loganathan Ramasamy, Jan Štěpánek, Zdeněk Žabokrtský, and Jan Hajič. 2014. HamleDT: Harmonized Multi-Language Dependency Treebank. Language Resources and Evaluation, 48(4):601–637. find
20. Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. find
21. CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–21, Brussels, Belgium, October. Association for Computational Linguistics. find our card
22. Daniel Zeman, Joakim Nivre, Mitchell Abrams, Noëmi Aepli, Željko Agić, Lars Ahrenberg, Gabrielė Aleksandravičiūtė, Lene Antonsen, Katya Aplonova, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Miguel Ballester… find
23. A Appendix A: θ pos Scores for UDv2.5 Treebanks, Annotated to Mark Consistent and Inconsistent Treebanks This appendix lists the θ pos scores in the UDv2.5 data (Zeman et al., 2019) with the annotations used as per Table 11. In the listing of scores, small treebanks where the total number of sentenc… find
24. Color Red Green Gray Significance Inconsistent in POS Annotation Consistent in POS Annotation Could Not Be Estimated Superscript Asterisk (∗) Dagger (†) Significance Cannot split into constituent genres Adulterant Genre(s) Present (b) Superscripts against Treebank Names (a) Color Codes Used for Scor… find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aggarwal-2020-estimating-pos-annotation-consistency-different/