Mova0.5 pre-release⌕
☰
home / docs / papers / English / Errator: a Tool to Help Detect Annotation Errors in the Universal Dependencies Project

Errator: a Tool to Help Detect Annotation Errors in the Universal Dependencies Project

What it is about

Errator finds likely annotation errors by the annotation variation principle: if the same word sequence occurs in several sentences and is annotated differently each time, at least one annotation is probably wrong. It finds maximal repeats with a generalized suffix tree, aligns each repeat to its tags, heads or tokenization (the alignment handles non-concatenative tokenization, where one written form maps to several tokens or the reverse), and flags the repeats whose annotations differ. It comes with a web interface where annotators can filter ambiguous cases. On the French UD treebanks it flags 2,617 possible segmentation errors, 25,527 PoS errors and 48,986 dependency errors. Correcting and harmonizing the French treebanks against the French Treebank improves a PoS tagger on out-of-domain French, which shows that part of the drop comes from annotation inconsistency. The paper reports the counts of flags but no precision figure for them.

Ideas

For Mova

The chapters are 07. Punctuation, capital letters, abbreviations and numbers and 20-syntax-overview.md. The code is treeagree for the gate and annot for the silver labels. Errator's method is cheap and deterministic, so it fits Mova's rule of measurement before claims; report the precision of the flags before any correction is trusted.

Topics: Universal Dependencies · Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{wisniewski2018errator,
  title = {{Errator: a Tool to Help Detect Annotation Errors in the Universal Dependencies Project}},
  author = {Wisniewski, Guillaume},
  booktitle = {Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)},
  year = {2018},
  url = {https://aclanthology.org/L18-1711/}
}

References

11 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aufrant, L., Wisniewski, G., and Yvon, F. (2017). Limsi@conll’17: Ud shared task. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 163–173, Vancouver, Canada, August. Association for Computational Linguistics. emea-fr-dev 00104 Pour les patients atteints d’ un SCA , la dose initiale recommandée est un bolus intraveineux de 0,1 mg / kg suivi d’ une perfusion de 0,25 mg / kg / h . ADP DET NOUN VERB ADP DET NOUN PUNCT DET NOUN ADJ VERB AUX DET NOUN ADJ ADP NUM NOUN ADP NOUN VERB ADP DET NOUN ADP NUM NOUN ADP NOUN ADP NOUN PUNCT Pour les patients atteints d’ un SCA , la dose initiale recommandée est : ADP DET NOUN VERB ADP DET NOUN PUNCT DET NOUN ADJ VERB VERB PUNCT emea-fr-dev 00510 frwiki 50.1000 00767 - Michel Roussin , reconnu coupable de complicité et recel de corruption , a été condamné ˋ a 4 ans de prison avec sursis et une amende de 50 000 euros . PUNCT PROPN PROPN PUNCT AUX NOUN ADP NOUN CCONJ NOUN ADP NOUN PUNCT AUX AUX VERB ADP NUM NOUN ADP NOUN ADP NOUN CCONJ DET NOUN ADP DET NOUN PUNCT - Louise-Yvonne Casetta , trésoriˋere occulte de le RPR , a été reconnue coupable de complicité et recel de corruption , et a été condamnée ˋ a 20 mois de prison avec sursis et 10 000 euros d’ amende . PUNCT PROPN PROPN PUNCT NOUN ADJ ADP DET PROPN PUNCT AUX AUX AUX ADJ ADP NOUN CCONJ NOUN ADP NOUN PUNCT CCONJ AUX AUX VERB ADP NUM NOUN ADP NOUN ADP NOUN CCONJ DET NOUN ADP NOUN PUNCT frwiki 50.1000 00773 emea-fr-dev 00228 Les saignements majeurs se sont produits le plus fréquemment ˋa le site de ponction ( voir Tableau 3 ) . DET NOUN ADJ PRON AUX VERB DET ADV ADV ADP DET NOUN ADP NOUN PUNCT VERB PROPN NUM PUNCT PUNCT Les saignements majeurs se sont produits le plus fréquemment ˋa le site de ponction ( voir Tableau 5 ) . DET NOUN ADJ PRON AUX VERB DET ADV ADV ADP DET NOUN ADP NOUN PUNCT VERB NOUN NUM PUNCT PUNCT emea-fr-dev 00258 Table 4: Example of errors in PoS annotation found by E RRATOR on the French Sequoia corpus of the UD project. Words in blue and red appears in two different sentence, but the red part has not the same annotation. find our card
  2. Bosco, C., Montemagni, S., and Simi, M. (2013). Converting italian treebanks: Towards an italian stanford dependency treebank. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 61–69, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  3. Boyd, A., Dickinson, M., and Meurers, W. D. (2008). On detecting errors in dependency treebanks. Research on Language and Computation, 6(2):113–137, Oct. find
  4. Dickinson, M. and Meurers, W. D. (2003). Detecting errors in part-of-speech annotation. In Proceedings of the Tenth Conference on European Chapter of the Association for Computational Linguistics - Volume 1, EACL ’03, pages 107–114, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  5. Dickinson, M. and Meurers, W. D. (2005). Detecting errors in discontinuous structural annotation. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 322–329, Ann Arbor, Michigan, June. Association for Computational Linguistics. find
  6. Gusfield, D. (1997). Algorithms on Strings, Trees, and Sequences: Computer Science and Computational Biology. Cambridge University Press, New York, NY, USA. find
  7. Lipenkova, J. and Souček, M. (2014). Converting russian dependency treebank to stanford typed dependencies representation. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 143– 147, Gothenburg, Sweden, April. Association for Computational Linguistics. find
  8. van Halteren, H. (2000). The detection of inconsistency in manually tagged text. In Proceedings of the COLING-2000 Workshop on Linguistically Interpreted Corpora, pages 48–55, Centre Universitaire, Luxembourg, August. International Committee on Computational Linguistics. find
  9. Vilares, D. and Gómez-Rodrı́guez, C. (2017). A non-projective greedy dependency parser with bidirectional lstms. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 152–162, Vancouver, Canada, August. Association for Computational Linguistics. find our card
  10. Wisniewski, G., Pécheux, N., Gahbiche-Braham, S., and Yvon, F. (2014). Cross-lingual part-of-speech tagging through ambiguous learning. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1779–1785, Doha, Qatar, October. Association for Computational Linguistics. 7. find
  11. Language Resource References Nivre, J., Agić, Ž., Ahrenberg, L., et al. (2017). Universal Dependencies 2.0. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics, Charles University, Prague. train test % error FTB corrected FTB corrected FTB original UD corrected UD original 2.96% 4.06% 6.49% 6.70% UD original UD original UD corrected FTB corrected FTB original 4.51% 4.53% 5.93% 6.78% FTB original FTB original FTB corrected UD corrected UD original 3.17% 3.88% 6.91% 7.01% UD corrected UD corrected UD original FTB corrected FTB original 4.42% 4.60% 5.71% 6.75% Table 5: Comparison of the performance of a state-of-the-art PoS tagger on two different French dataset before and after the errors detected by E RRATOR have been corrected. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ud-2018-wisniewski-errator-tool-help-detect-annotation/