The Tembusu Treebank: An English Learner Treebank

What it is about

The authors build a treebank from the NTU Corpus of Learner English, about 25,000 sentences of student assignments, of which 4,900 are annotated so far. Each sentence is parsed by the English Resource Grammar (an HPSG grammar) and a human picks the right parse in the ACE/FFTB tool. The grammar can also use "mal-rules" that describe typical errors, and the treebank keeps them in the derivation. The treebank is used to train a new parse-ranking model for the grammar, and on 1,000 learner sentences the model with mal-rules flags 20.1% of sentences as problematic, against 31.5% with the old model. The paper reports that diagnosis precision rises by 18 to 22% and F1 by 6 to 9%.

Ideas

• I1. Error-seed false positives: run the rules in en/seeds/errors/ on a learner sample and on a native sample of equal size, and report flags per 1k sentences for each; the paper's 31.5% versus 20.1% is the warning that a flag rate alone says little, so also hand-check 100 flags for precision.
• I2. Two-step filter: before the error seeds, require that en parses the sentence with a plausible tree (treeagree check gates clean, no fragment root); measure flag precision with and without this filter, the way the paper gains precision by filtering.
• I3. Learner features for the en perceptron in en/src/parse.rs: add an error-rule id as a feature in the transition scorer, train on the ESLSpok learner data, and measure LAS on the ESLSpok test set and on the learner subset of EWT.
• I4. Use the Tembusu set as a test only (its licence must be checked in Data and model licences before any training use): measure en error-detection precision on its annotated mal-rule sentences.

For Mova

The error seeds live in en/seeds/errors/, the parser in en/src/parse.rs and the verb agreement pass in en/src/ud.rs. The chapters that the ideas touch are 13. Verbs (agreement) and 12-pronouns-determiners-numerals.md. Measure first: flag precision on 100 hand-checked learner sentences, because flag counts alone do not show error detection.

Topics: Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/54m4rangupu2

@inproceedings{costa2022tembusu,
  title = {{The Tembusu Treebank: An English Learner Treebank}},
  author = {Costa, Luís Morgado da and Bond, Francis and Winder, Roger V. P.},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/54m4rangupu2},
  url = {https://doi.org/10.63317/54m4rangupu2}
}

References

35 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bender, E. M., Flickinger, D., Oepen, S., Walsh, A., and Baldwin, T. (2004). Arboretum: Using a precision grammar for grammar checking in CALL. In InSTIL/ICALL Symposium 2004. find
2. Berzak, Y., Kenney, J., Spadine, C., Wang, J. X., Lam, L., Mori, K. S., Garza, S., and Katz, B. (2016). Universal Dependencies for learner English. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 737–746, Berlin, Germany, August. Association for Computational Linguistics. find our card
3. Black, E., Abney, S., Flickinger, D., Gdaniec, C., Grishman, R., Harrison, P., Hindle, D., Ingria, R., Jelinek, F., Klavans, J. L., et al. (1991). A procedure for quantitatively comparing the syntactic coverage of English grammars. In Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California, February 19-22, 1991. find
4. Bond, F., Fujita, S., Hashimoto, C., Kasahara, K., Nariyama, S., Nichols, E., Ohtani, A., Tanaka, T., and Amano, S. (2004). The Hinoki treebank. a treebank for text understanding. In International Conference on Natural Language Processing, pages 158–167. Springer. find
5. Bond, F., Fujita, S., and Tanaka, T. (2008). The Hinoki syntactic and semantic treebank of Japanese. Language Resources and Evaluation, 42(2):243–251. find
6. Collins, M. (1997). Three generative, lexicalised models for statistical parsing. In Proceedings of ACL ’98/EACL ’98, page 16–23, USA. Association for Computational Linguistics. find
7. Copestake, A. and Flickinger, D. (2000). An open source grammar development environment and broad-coverage English grammar using HPSG. In Proceedings of the Second Linguistic Resources and Evaluation Conference, pages 591–600, Athens, Greece. find
8. Copestake, A., Flickinger, D., Pollard, C., and Sag, I. A. (2005). Minimal recursion semantics: An introduction. Research on Language and Computation, 3(2-3):281–332. find
9. Dahlmeier, D., Ng, H. T., and Wu, S. M. (2013). Building a large annotated corpus of learner English: The NUS corpus of learner English. In BEA@ NAACL-HLT, pages 22–31. find
10. Flickinger, D. and Yu, J. (2013). Toward more precision in correction of grammatical errors. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning: Shared Task, CoNLL 2013, Sofia, Bulgaria, August 8-9, 2013, pages 68–73. find
11. Flickinger, D., Zhang, Y., and Kordoni, V. (2012). Deepbank. a dynamically annotated treebank of the wall street journal. In Proceedings of the 11th International Workshop on Treebanks and Linguistic Theories, pages 85–96. find
12. Flickinger, D., Goodman, M., and Packard, W. (2016). find
13. UW-Stanford System description for AESW 2016 shared task on grammatical error detection. In Proceedings of the 11th Workshop on Innovative Use of NLP for Building Educational Applications, pages 105–111. find
14. Flickinger, D. (2000). On building a more effcient grammar by exploiting types. Natural Language Engineering (Special Issue on Efficient Processing with HPSG), 6(1):15–28. find
15. Flickinger, D. (2011). Accuracy vs. robustness in grammar engineering. Language from a cognitive perspective: Grammar, usage, and processing, 201:31–50. find
16. Krieger, H.-U. and Schäfer, U. (1994). TDL – a type description language for constraint-based grammars. arXiv preprint cmp-lg/9406018. find
17. Moeljadi, D., Bond, F., and Song, S. (2015). Building an HPSG-based Indonesian resource grammar (indra). ACL-IJCNLP 2015, page 9. find
18. Moeljadi, D. (2017). Building jati: A treebank for indonesian. In Proceedings of The 4th Atma Jaya Conference on Corpus Studies (ConCorps 4), pages 1–9. find
19. Morgado da Costa, L., Winder, R. V. P., Li, S. Y., Liang, B. C. L. T., Mackinnon, J., and Bond, F. (2020). Automated writing support using deep linguistic parsers. In Proceedings of the 12th Conference on Language Resources and Evaluation, Marseille, France, may. European Language Resources Association (ELRA). find
20. Nicholls, D. (2003). The Cambridge learner corpus: Error coding and analysis for lexicography and elt. In Proceedings of the Corpus Linguistics 2003 conference, volume 16, pages 572–581. find
21. Oepen, S., Toutanova, K., Shieber, S. M., Manning, C. D., Flickinger, D., and Brants, T. (2002). The LinGO Redwoods treebank: Motivation and preliminary applications. In COLING 2002: The 17th International Conference on Computational Linguistics: Project Notes. find our card
22. Oepen, S., Flickinger, D., and Bond, F. (2004). Towards holistic grammar engineering and testing — grafting treebank maintenance into the grammar revision cycle. In Beyond Shallow Analyses — Formalisms and Statistical Modelling for Deep Analysis (Workshop at IJCNLP-2004), Hainan Island. find
23. Packard, W. (2015). Full Forest Treebanking. Master’s thesis, University of Washington, USA. find
24. Pollard, C. and Sag, I. A. (1994). Head-Driven Phrase Structure Grammar. The University of Chicago Press. find
25. Ragheb, M. and Dickinson, M. (2012). Defining syntax for learner language annotation. In Proceedings of COLING 2012: Posters, pages 965–974, Mumbai, India, December. find
26. Ragheb, M. and Dickinson, M. (2014). Developing a corpus of syntactically-annotated learner language for English. In Proceedings of the 13th International Workshop on Treebanks and Linguistic Theories (TLT13), pages 292–300, Tübingen, Germany. find
27. Sag, I. A., Wasow, T., Bender, E. M., and Sag, I. A. (1999). Syntactic theory: a formal introduction, volume 2. CSLI Stanford. find
28. Schneider, D. and McCoy, K. F. (1998). Recognizing syntactic errors in the writing of second language learners. In Proceedings of the 17th International Conference on Computational Linguistics - Volume 2, pages 1198–1204, Stroudsburg, PA, USA. Association for Computational Linguistics. find
29. Siegel, M. and Bender, E. M. (2002). Efficient deep processing of Japanese. In Proceedings of the 3rd workshop on Asian language resources and international standardization-Volume 12, pages 1–8. Association for Computational Linguistics. find
30. Siegel, M., Bender, E. M., and Bond, F. (2016). Jacy: An implemented grammar of Japanese. CSLI Publications. find
31. Suppes, P., Liang, T., Macken, E. E., and Flickinger, D. P. (2014). Positive technological and negative pre-test-score effects in a four-year assessment of low socioeconomic status k-8 student learning in computer-based math and language arts courses. Computers & Education, 71:23–32. find
32. Tanaka, T., Bond, F., Oepen, S., and Fujita, S. (2005). High precision treebanking — blazing useful trees using POS information. In Proceedings of the 43rd annual meeting of the Association for Computational Linguistics, pages 330–337. find
33. Toutanova, K., Manning, C. D., Flickinger, D., and Oepen, S. (2005). Stochastic HPSG parse disambiguation using the Redwoods corpus. Research on Language and Computation, 3(1):83–105. find
34. Winder, R. V. P., MacKinnon, J., Li, S. Y., Lin, B., Heah, C., Morgado da Costa, L., Kuribayashi, T., and Bond, F. (2017). NTUCLE: Developing a corpus of learner English to provide writing support for engineering students. In Proceedings of the 4th Workshop on NLP Techniques for Educational Applications (NLPTEA 2017), Taipei, Taiwan. find
35. Zhang, Y. and Krieger, H.-U. (2011). Large-scale corpus-driven PCFG approximation of an HPSG. In Proceedings of the 12th International Conference on Parsing Technologies, pages 198–208, Dublin, Ireland, October. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/bank-2022-costa-tembusu-treebank-english-learner-treebank/