Universal Dependencies for Learner English

Summary

Much of the English written today is produced by non-native speakers, yet before this work there was no open syntactic treebank of learner English. The authors built the Treebank of Learner English (TLE) from essays of the FCE corpus written by learners with ten different native languages. POS tags and UD trees were assigned manually, and every sentence was reviewed by further annotators. Since FCE already marks errors and corrections, each sentence is annotated twice: as written and in its corrected form. For ungrammatical constructions the authors propose a "literal reading" principle: the syntax describes what the learner actually wrote, not what they presumably meant. Inter-annotator agreement shows this guideline yields consistent annotation. Tagging and parsing experiments on the paired original/corrected sentences indicate that grammatical errors hurt parsers less than earlier estimates suggested. The resource enables both empirical study of learner syntax and automatic processing of ungrammatical text.

How Mova uses it

• en/seeds/errors/literal-annotation-learner.md: the literal-reading principle is adopted for Mova's annotation — a sentence with errors gets the tree of its actual words; spelling errors use UD's Typo=Yes/CorrectForm mechanism, while grammatical errors (agreement, verb form) are left visible in the tree. This is what lets agreement rules double as grammar-error detectors, and it is used to catch LLM annotators that silently "fix" the text.
• en/seeds/errors/missing-copula.md: rule en.errors.missing-copula (nominal/adjectival predicate with a subject but no copula, e.g. "He very happy"), justified by the literal-reading analysis.
• en/seeds/errors/missing-article.md: rule en.errors.bare-count-noun (singular count noun as subject/object without determiner); TLE statistics (ten L1s, average errors per sentence) are cited as background.

Effectiveness in Mova

Not measured separately. The rules derived with its help are checked against gold treebanks: on UD English EWT 2.18 the missing-copula rule fired 3,727 times with 44 flags (mostly telegraphic review style and genuinely missing is), and the bare-count-noun rule 953 / 42 (mostly real missing articles and telegraphic style); on GUM 3,011 / 37 and 512 / 15 respectively. The paper's main role is the literal-annotation principle that the whole error-rule section relies on.

👨‍🔬💥

Topics: Universal Dependencies

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1070 · arXiv 1605.04278

@inproceedings{berzak2016universal,
  title = {{Universal Dependencies for Learner English}},
  author = {Berzak, Yevgeni and Kenney, Jessica and Spadine, Carolyn and Wang, Jing Xian and Lam, Lucia and Mori, Keiko Sophie and Garza, Sebastian and Katz, Boris},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1070},
  eprint = {1605.04278},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p16-1070}
}

References

18 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Aoife Cahill, Binod Gyawali, and James V Bruno. 2014. Self-training for parsing learner text. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages, pages 66–73. find
2. Ryan T McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith B Hall, Slav Petrov, Hao Zhang, Oscar Täckström, et al. 2013. Universal dependency annotation for multilingual parsing. In ACL (2), pages 92–97. Citeseer. find
3. Jacob Cohen. 1960. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 20(1):37. ↗
4. Ryo Nagata, Edward Whittaker, and Vera Shein-man. 2011. Creating a manually error-tagged and shallow-parsed learner corpus. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 1210–1219. Association for Computational Linguistics. find
5. David Crystal. 2003. English as a global language. Ernst Klett Sprachen. ↗
6. Marie-Catherine De Marneffe and Christopher D Manning. 2008. Stanford typed dependencies manual. Technical report, Technical report, Stanford University. find
7. Marie-Catherine De Marneffe, Timothy Dozat, Natalia Silveira, Katri Haverinen, Filip Ginter, Joakim Nivre, and Christopher D Manning. 2014. Universal stanford dependencies: A cross-linguistic typology. In Proceedings of LREC, pages 4585–4592. find our card
8. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The conll-2014 shared task on grammatical error correction. In CoNLL Shared Task, pages 1–14. find
9. Ana Dıaz-Negrillo, Detmar Meurers, Salvador Valera, and Holger Wunsch. 2010. Towards interlanguage pos annotation for effective learner corpora in sla and flt. Language Forum, 36(1–2):139–154. find
10. Diane Nicholls. 2003. The cambridge learner corpus: Error coding and analysis for lexicography and elt. In Proceedings of the Corpus Linguistics 2003 conference, pages 572–581. find
11. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC 2016). find our card
12. Marwa Ragheb and Markus Dickinson. 2012. Defining syntax for learner language annotation. In COL-ING (Posters), pages 965–974. find
13. Victoria Rosén and Koenraad De Smedt. 2010. Syntactic annotation of learner corpora. Systematisk, variert, men ikke tilfeldig, pages 120–132. find
14. Alexandr Rosen, Jirka Hana, Barbora Štindlová, and Anna Feldman. 2014. Evaluating and automating the annotation of a learner corpus. Language Resources and Evaluation, 48(1):65–92. ↗
15. Beatrice Santorini. 1990. Part-of-speech tagging guidelines for the penn treebank project (3rd revision). Technical Reports (CIS). find our card
16. Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel R Bowman, Miriam Connor, John Bauer, and Christopher D Manning. 2014. A gold standard dependency corpus for english. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC-2014). find
17. Joel Tetreault, Jennifer Foster, and Martin Chodorow. 2010. Using parse features for preposition selection and error detection. In Proceedings of the acl 2010 conference short papers, pages 353–358. Association for Computational Linguistics. find
18. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In ACL, pages 180–189. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/berzak-2016-universal-dependencies-learner-english/