Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction

Summary

Grammatical error correction (GEC) systems were usually judged by a single overall score, and because system output carries no error labels, per-type performance could only be estimated through recall. The paper introduces ERRANT, a toolkit that extracts edits from a pair of original and corrected sentences using a linguistically informed alignment and then assigns each edit an error type. Typing is done by a set of roughly fifty deterministic rules over automatically obtained features — part-of-speech tags, lemmas, a dependency parse and a word list — so it needs no training data and is not tied to any particular corpus. The scheme has 25 main categories combined with three operations (missing, replaced, unnecessary), allowing evaluation at several levels of detail. Expert raters judged the vast majority of assigned types as good or acceptable. Applying ERRANT to all CoNLL-2014 shared-task submissions gave the first detailed per-type comparison, showing that a system weaker overall can still lead in specific categories. The tool also helps standardise existing GEC corpora and reduce annotation effort.

How Mova uses it

• en/seeds/errors/errant-taxonomy.md: a seed card summarising the ERRANT taxonomy and its ordered verb rules (ORTH → SPELL → same lemma → FORM → TENSE → SVA), and mapping which error types leave a visible trace in a UD tree; each such type became its own error-rule card.
• Rules in the expert system (engine en/src/expert.rs) cite ERRANT types as their category: a/an choice (DET; en/seeds/errors/a-an-choice.md), adverb between verb and object (WO; adverb-placement.md), coordinated-subject agreement (VERB:SVA; coordinated-subject-agreement.md), determiner–noun number (DET, NOUN:NUM), double comparative (ADJ:FORM), missing article (M:DET), missing copula (M:VERB), passive auxiliary form (VERB:FORM).
• The design principle carried over is ERRANT's own: deterministic, explainable rules over POS, lemma and parse features, no learned classifier.

Effectiveness in Mova

Not measured separately as an idea. The rules categorised with ERRANT types are checked on UD English EWT 2.18, e.g. "a before a vowel-initial noun" 363 hits / 6 flags (all six real text errors), "an before a consonant" 198 / 3 (abbreviations), coordinated subject with singular verb 394 / 5, adverb between verb and object 21 / 3 (all heavy objects). Its role is a taxonomy and design reference for the error-rule section.

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1074

@inproceedings{bryant2017automatic,
  title = {{Automatic Annotation and Evaluation of Error Types for Grammatical Error Correction}},
  author = {Bryant, Christopher and Felice, Mariano and Briscoe, Ted},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1074},
  url = {https://doi.org/10.18653/v1/p17-1074}
}

References

22 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Christopher Bryant and Hwee Tou Ng. 2015. How far are we from fully automatic high quality grammatical error correction? In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Lon… find
2. Daniel Dahlmeier and Hwee Tou Ng. 2012. Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, Montréal, Canada,… find
3. Robert Dale and Adam Kilgarriff. 2011. find
4. Helping Our Own: The HOO 2011 pilot shared task. In Proceedings of the 13th European Workshop on Natural Language Generation. Association for Computational Linguistics, Stroudsburg, PA, USA, ENLG ’11, pages 242–249. http://dl.acm.org/citation.cfm?id=2187681.2187725. \#\# Conclusion Bradley Efron and R… ↗
5. In this paper, we described ERRANT, a grammatical ERRor ANnotation Toolkit designed to au-Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. ACL, Baltimore, Marylan… find
6. Mariano Felice and Ted Briscoe. 2015. Towards a standard evaluation method for grammatical error detection and correction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational… find
7. Justus J. Randolph. 2005. Free-marginal multirater kappa: An alternative to Fleiss’ fixed-marginal multirater kappa. Joensuu University Learning and Instruction Symposium http://files.eric.ed.gov/fulltext/ED490661.pdf. ↗
8. Mariano Felice, Christopher Bryant, and Ted Briscoe. 2016. Automatic extraction of learner errors in ESL sentences using linguistically enhanced alignments. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. find
9. The COLING 2016 Organizing Committee, Osaka, Japan, pages 825–835. http://aclweb.org/anthology/C16-1079. ↗
10. Marek Rei and Helen Yannakoudakis. 2016. Compositional sequence labeling models for error detection in learner writing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, page… find
11. Mariano Felice, Zheng Yuan, Øistein E. Andersen, Helen Yannakoudakis, and Ekaterina Kochmar. 2014. Grammatical error correction using hybrid systems and type filtering. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. Association for Computational… find
12. Alla Rozovskaya, Kai-Wei Chang, Mark Sammons, Dan Roth, and Nizar Habash. 2014. The Illinois-Columbia system in the CoNLL-2014 shared task. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. Association for Computational Linguistics, Baltimore, Maryl… find
13. Roman Grundkiewicz, Marcin Junczys-Dowmunt, and Edward Gillian. 2015. Human evaluation of grammatical error correction systems. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 461–470. http:… find
14. Alla Rozovskaya and Dan Roth. 2016. Grammatical error correction: Machine translation and classifiers. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Berlin, Germany, pages 2205–2215. http://aclweb.org/anthology/… ↗
15. Anubhav Gupta. 2014. Grammatical error detection using tagger disagreement. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. Association for Computational Linguistics, Baltimore, Maryland, pages 49–52. http://www.aclweb.org/anthology/W14-1706. ↗
16. Keisuke Sakaguchi, Courtney Napoles, Matt Post, and Joel Tetreault. 2016. Reassessing the goals of grammatical error correction: Fluency instead of grammaticality. Transactions of the Association for Computational Linguistics 4:169–182. https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/ article/vi… ↗
17. Marcin Junczys-Dowmunt and Roman Grundkiewicz. 2014. The AMU system in the CoNLL-2014 shared task: Grammatical error correction by data-intensive and feature-rich statistical machine translation. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task. Ass… find
18. Ben Swanson and Elif Yamangil. 2012. Correction detection and error type selection as an ESL educational aid. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics,… find
19. Courtney Napoles, Keisuke Sakaguchi, Matt Post, and Joel Tetreault. 2015. Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volum… find
20. Huichao Xue and Rebecca Hwa. 2014. Improved correction detection in revised ESL sentences. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Baltimore, Maryland, pages 599–604. http://www.ac… ↗
21. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics:… find
22. Morphology Tier Other Part Of Speech Complete list of valid error code combinations Token Tier A Type Adjective Adverb Conjunction Determiner Noun Particle Preposition Pronoun Punctuation Verb Contraction Morphology Orthography Other Spelling Word Order Adjective Form Noun Inflection Noun Number Nou… find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/bryant-2017-automatic-annotation-evaluation-error-types/