Automatic Detection and Correction of Errors in Dependency Treebanks

What it is about

Treebanks contain annotation errors even when they are widely used, and the authors show this for the Penn Treebank conversion used in the CoNLL shared tasks. Their method looks for disagreement: a token is suspicious when two different parsers agree on a head or label that differs from the gold, and a third parser confirms the candidate correction. The rule is strict, so precision is high, but recall is limited. In an experiment with 1,350 injected erroneous tokens, the method recovered 619 of them (45.9%). The authors estimate that the corpus contains errors on more than 1% of its tokens. A different detection method finds mostly other errors, so the two are complementary.

Ideas

• I1. tools/treeagree: add a disagreement check in which en and gparse both disagree with a gold head and agree on another; count candidates per 1,000 sentences on EWT train and measure precision on 100 hand-checked candidates.
• I2. Recall test as in the paper: inject 1,350 random head errors into a copy of EWT train kept outside the repo under $MOVA_DATA, retrain en, and count how many the check recovers; run twice to confirm determinism.
• I3. annot pipeline: re-check or drop the sentences that the disagreement check flags before the corpus/ silver is used to train en; measure LAS on EWT dev (cv), not on test.

For Mova

Touches 20. Syntax: the Universal Dependencies frame and 37. Register and variation. Mova already trains on silver trees, so the check protects the training data directly. Measure the candidate rate (I1) before any retraining, since it costs nothing.

Topics: Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{volokh2011automatic,
  title = {{Automatic Detection and Correction of Errors in Dependency Treebanks}},
  author = {Volokh, Alexander and Neumann, Günter},
  booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},
  year = {2011},
  url = {https://aclanthology.org/P11-2060/}
}

References

10 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mitchell P. Marcus, Beatrice Santorini and Mary Ann Marcinkiewicz , 1993. Building a Large Annotated Corpus of English: The Penn Treebank. In Computational Lingustics, vol. 19, pp. 313-330. find our card
2. Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluis Marquez and Joakim Nivre. The CoNLL-2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies. In Proceedings of the 12th Conference on Computational Natural Language Learning (CoNLL-2008), 2008 find our card
3. Sabine Buchholz and Erwin Marsi, 2006. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of CONLL-X, pages 149–164, New York. find our card
4. Sabine Buchholz and Darren Green, 2006. Quality control of treebanks: documenting, converting, patching. In LREC 2006 workshop on Quality assurance and quality measurement for language and speech resources. find
5. Markus Dickinson and W. Detmar Meurers, 2005. Prune Diseased Branches to Get Healthy Trees! How to Find Erroneous Local Trees in a Treebank and Why It Matters. In Proceedings of the Fourth Workshop on Treebanks and Linguistic Theories, pp. 41—52 find
6. Adriane Boyd, Markus Dickinson and Detmar Meurers, 2008. On Detecting Errors in Dependency Treebanks. In Research on Language and Computation, vol. 6, pp. 113-137. find
7. Markus Dickinson and Detmar Meurers, 2003. Detecting inconsistencies in treebanks. In Proceedings of TLT 2003 find
8. van Halteren, H. (2000). The detection of inconsistency in manually tagged text. In A. Abeillé, T. Brants, and H. Uszkoreit (Eds.), Proceedings of the Second Workshop on Linguistically Interpreted Corpora (LINC-00), Luxembourg. [9 R. McDonald, F. Pereira, K. Ribarov, and J. Haji˘c . 2005. Non-projective Dependency Parsing using Spanning Tree Algorithms. In Proc. of HLT/EMNLP 2005. find our card
9. Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gulsen Eryigit, Sandra Kubler, Svetoslav Marinov and Erwin Marsi. 2007. MaltParser: A Language-Independent System for Data-Driven Dependency Parsing, Natural Language Engineering Journal, 13, pp. 99-135. find
10. Joakim Nivre and Ryan McDonald, 2008. Integrating GraphBased and Transition-Based Dependency Parsers. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2011-volokh-automatic-detection-correction-errors-dependency/