Correcting Dependency Annotation Errors

Summary

The paper moves from detecting errors in dependency treebank annotation to correcting them automatically. It explains why this is hard: erroneous annotations often recur identically in similar contexts, so a model trained on the corpus easily learns the noise. The data is the written part of the Swedish Talbanken05 treebank, with correction candidates taken from variation n-grams found by earlier detection work. For each word pair the model predicts a dependency label together with which word is the head, or a special label when there is no relation; classification uses memory-based learning. Simple combinations of local features perform poorly, so the author takes ambiguity classes from strict lexical models and passes them to more flexible POS-based models, as output constraints, as extra features, or both. The best configuration labels about three quarters of the problematic positions correctly, clearly above the no-change baseline. The idea of one model refining another is presented as applicable to other annotation layers and to dependency parsing.

How Mova uses it

• en/seeds/errors/consistency-detection-methods.md: a seed card surveying automatic ways to find annotation errors; this paper is cited for method 1 ("variation nuclei" — the same word pair in the same context with different relations signals an error) and for the result that such positions can be automatically corrected (76.7% vs 70.1% unchanged, as reported in the paper).
• The card feeds the expert-system principle that a rule which fires often on a gold treebank is either wrong or is catching errors in the gold standard, and the noise-robust training in Mova's taggers that turns repeatedly-wrong labels into suspicions rather than facts.

Effectiveness in Mova

Background reading; not measured separately. No Dickinson-style correction classifier is implemented in Mova.

👨‍🔬💥

Topics: Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1609067.1609088

@inproceedings{dickinson2009correcting,
  title = {{Correcting Dependency Annotation Errors}},
  author = {Dickinson, Markus},
  booktitle = {Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009)},
  year = {2009},
  doi = {10.3115/1609067.1609088},
  url = {https://doi.org/10.3115/1609067.1609088}
}

References

26 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Boyd, Adriane, Markus Dickinson and Detmar Meurers (2008). On Detecting Errors in Dependency Treebanks. Research on Language and Computation 6(2), 113–137. find
2. Canisius, Sander, Toine Bogers, Antal van den Bosch, Jeroen Geertzen and Erik Tjong Kim Sang (2006). Dependency parsing by inference over high-recall dependency predictions. In Proceedings of CoNLL-X. New York. find
3. Chen, Wenliang, Youzheng Wu and Hitoshi Isahara (2008). Learning Reliable Information for Dependency Parsing Adaptation. In Proceedings of Coling 2008. Manchester. find
4. Daelemans, Walter (1996). Abstraction Considered Harmful: Lazy Learning of Language Processing. In Proceedings of the 6th Belgian-Dutch Conference on Machine Learning. Maastricht, The Netherlands. find
5. Daelemans, Walter, Jakub Zavrel, Ko Van der Sloot and Antal Van den Bosch (2007). TiMBL: Tilburg Memory Based Learner, version 6.1, Reference Guide. Tech. rep., ILK Research Group. ILK Research Group Technical Report Series no. 07-07. find
6. Dickinson, Markus (2006). From Detecting Errors to Automatically Correcting Them. In Proceedings of EACL-06. Trento, Italy. find
7. Dickinson, Markus (2007). Determining Ambiguity Classes for Part-of-Speech Tagging. In Proceedings of RANLP-07. Borovets, Bulgaria. find
8. Dickinson, Markus and W. Detmar Meurers (2003). Detecting Inconsistencies in Treebanks. In Proceedings of TLT-03. Växjö, Sweden. find
9. Dickinson, Markus and W. Detmar Meurers (2005). Detecting Errors in Discontinuous Structural Annotation. In Proceedings of ACL-05. find
10. Einarsson, Jan (1976). Talbankens skrift-sprøakskonkordans. Tech. rep., Lund University, Dept. of Scandinavian Languages. find
11. Habash, Nizar, Ryan Gabbard, Owen Rambow, Seth Kulick and Mitch Marcus (2007). Determining Case in Arabic: Learning Complex Linguistic Behavior Requires Complex Linguistic Features. In Proceedings of EMNLP-07. find
12. Hogan, Deirdre (2007). Coordinate Noun Phrase Disambiguation in a Generative Parsing Model. In Proceedings of ACL-07. Prague. find
13. Klein, Dan and Christopher D. Manning (2002). A Generative Constituent-Context Model for Improved Grammar Induction. In Proceedings of ACL-02. Philadelphia, PA. find
14. McDonald, Ryan, Kevin Lerman and Fernando Pereira (2006). Multilingual Dependency Analysis with a Two-Stage Discriminative Parser. In Proceedings of CoNLL-X. New York City. find
15. McDonald, Ryan and Joakim Nivre (2007). Characterizing the Errors of Data-Driven Dependency Parsing Models. In Proceedings of EMNLP-CoNLL-07. Prague, pp. 122–131. find our card
16. McDonald, Ryan and Fernando Pereira (2006). Online learning of approximate dependency parsing algorithms. In Proceedings of EACL-06. Trento. find our card
17. Mintz, Toben H. (2006). Finding the verbs: distributional cues to categories available to young learners. In K. Hirsh-Pasek and R. M. Golinkoff (eds.), Action Meets Word: How Children Learn Verbs, New York: Oxford University Press, pp. 31–63. find
18. Nivre, Joakim (2006). Inductive Dependency Parsing. Berlin: Springer. find
19. Nivre, Joakim, Johan Hall, Jens Nilsson, Atanas Chanev, Gulsen Eryigit, Sandra Kubler, Svetoslav Marinov and Erwin Marsi (2007). Malt-Parser: A language-independent system for data-driven dependency parsing. Natural Language Engineering 13(2), 95–135. find
20. Nivre, Joakim and Ryan McDonald (2008). Integrating Graph-Based and Transition-Based Dependency Parsers. In Proceedings of ACL-08: HLT. Columbus, OH. find
21. Nivre, Joakim, Jens Nilsson and Johan Hall (2006). Talbanken05: A Swedish Treebank with Phrase Structure and Dependency Annotation. In Proceedings of LREC-06. Genoa, Italy. find
22. Osborne, Miles (2002). Shallow Parsing using Noisy and Non-Stationary Training Material. In JMLR Special Issue on Machine Learning Approaches to Shallow Parsing, vol. 2, pp. 695– 719. find
23. Padro, Lluis and Lluis Marquez (1998). On the Evaluation and Comparison of Taggers: the Effect of Noise in Testing Corpora. In Proceedings of ACL-COLING-98. San Francisco, CA. find
24. Taylor, Ann, Mitchell Marcus and Beatrice Santorini (2003). The Penn Treebank: An Overview. In Anne Abeillé (ed.), Treebanks: Building and using syntactically annotated corpora, Dordrecht: Kluwer, chap. 1, pp. 5–22. find
25. Torres Martins, André Filipe, Dipanjan Das, Noah A. Smith and Eric P. Xing (2008). Stacking Dependency Parsers. In Proceedings of EMNLP-08. Honolulu, Hawaii, pp. 157–166. find
26. Zavrel, Jakub, Walter Daelemans and Jorn Veensta (1997). Resolving PP attachment Ambiguities with Memory-Based Learning. In Proceedings of CoNLL-97. Madrid. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/dickinson-2009-correcting-dependency-annotation-errors/