Mova0.5 pre-release⌕
☰
home / docs / papers / A Dependency Treebank of Spoken Second Language English

A Dependency Treebank of Spoken Second Language English

Summary

POS taggers and parsers lose accuracy on text unlike their training data, and second-language (L2) speech is a clear case; a written L2 treebank existed, but no freely available spoken one. The authors build SL2E from utterances of Japanese learners in oral proficiency interviews (NICT JLE corpus), with interviewer turns removed. Sentences are hand-annotated with Penn POS tags and UD dependencies, each by at least two annotators with a third adjudicating disagreements. They then train spaCy models (RoBERTa-base) on L1 data only, on L1 plus L2, and on L1 plus L2 at equal total size. Even a small amount of L2 data improves both tagging and parsing markedly, with the largest gains on spoken L2 (LAS on SL2E rising from 0.876 to about 0.935). The resource is aimed at language-development research and educational NLP.

How Mova uses it

Effectiveness in Mova

Background reading for one seed note. Not measured separately.

👨‍🔬💥

Topics: Treebanks · Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.bea-1.7

@inproceedings{kyle2022dependency,
  title = {{A Dependency Treebank of Spoken Second Language English}},
  author = {Kyle, Kristopher and Eguchi, Masaki and Miller, Aaron and Sither, Theodore},
  booktitle = {Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022)},
  year = {2022},
  doi = {10.18653/v1/2022.bea-1.7},
  url = {https://doi.org/10.18653/v1/2022.bea-1.7}
}

References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Granger, S., & Bestgen, Y. (2014). The use of collocations by intermediate vs. Advanced non-native writers: A bigram-based study. find
  2. International Anthony, L. (2019). AntConc (3.5.8) [Computer software]. Waseda University. Review of Applied Linguistics in Language Teaching, 52(3), 229–252. International Journal of Corpus Linguistics, 15(4), 474–496. https://doi.org/10.1075/ijcl.15.4.02lu ↗
  3. Grant, L., & Ginther, A. (2000). Using computer-tagged linguistic features to describe L2 writing differences. Journal of Second Language Writing, 9(2), 123–145. find
  4. Lu, X. (2011). A corpus-based evaluation of syntactic complexity measures as indices of college-level ESL writers’ language development. TESOL Quarterly, 45(1), 36–62. find
  5. Honnibal, M., Montani, I., Van Landeghem, S., & Boyd, A. (2020). spaCy: Industrial-strength Natural Language Processing in Python [Python]. https://doi.org/10.5281/zenodo.1212303 (Original work published 2014) ↗
  6. Martens, S. (2013). TüNDRA: A web application for treebank search and visualization. 133. find
  7. McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381–392. https://doi.org/10.3758/BRM.42.2.381 ↗
  8. Izumi, E., Uchimoto, K., & Isahara, H. (2004). The NICT JLE Corpus: Exploiting the language learners’ speech database for research and education. International Journal of The Computer, the Internet and Management, 12(2), 119–125. find
  9. McClosky, D., Charniak, E., & Johnson, M. (2006). Reranking and self-training for parser adaptation. 337–344. find our card
  10. Jarvis, S., & Hashimoto, B. J. (2021). How operationalizations of word types affect measures of lexical diversity. International Journal of Learner Corpus Research, 7(1), 163–194. find
  11. Meurers, D., & Dickinson, M. (2017). Evidence and interpretation in language learning research: Opportunities for collaboration with computational linguistics. Language Learning, 67(S1), 66–95. find
  12. Kyle, K. (2021). Natural language processing for learner corpus research. International Journal of Learner Corpus Research, 7(1), 1–16. find
  13. Monarch, R. M. (2021). Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster. find
  14. Kyle, K., & Crossley, S. A. (2017). Assessing syntactic sophistication in L2 writing: A usage-based approach. Language Testing, 34(4), 513–535. find
  15. Nagata, R. & Sakaguchi, K. (2016). Phrase Structure Annotation and Parsing for Learner English. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1837–1847, Berlin, Germany. Association for Computational Linguistics. find
  16. Kyle, K., & Crossley, S. A. (2018). Measuring Syntactic Complexity in L2 Writing Using Fine-Grained Clausal and Phrasal Indices. The Modern Language Journal, 102(2), 333–349. https://doi.org/10.1111/modl.12468 ↗
  17. Kyle, K., Crossley, S. A., & Berger, C. M. (2018). The tool for the automatic analysis of lexical sophistication (TAALES): Version 2.0. Behavior Research Methods, 50(3), 1030–1046. https://doi.org/10.3758/s13428-017-0924-4 ↗
  18. Nivre, J., Marneffe, M.-C. de, Ginter, F., Hajic, J., Manning, C. D., Pyysalo, S., Schuster, S., Tyers, F., & Zeman, D. (2020). Universal Dependencies v2: An Evergrowing Multilingual Treebank Collection. 4034–4043. https://aclanthology.org/2020.lrec-1.497 ↗ our card
  19. Kyle, K., Crossley, S., & Verspoor, M. (2021). Measuring longitudinal writing development using indices of syntactic complexity and sophistication. Studies in Second Language Acquisition, 43(4), 781–812. find
  20. Paquot, M. (2018). Phraseological Competence: A Missing Component in University Entrance Language Tests? Insights From a Study of EFL Learners’ Use of Statistical Collocations. Language Assessment Quarterly, 15(1), 29–43. find
  21. Kyle, K., & Eguchi, M. (2021). Automatically assessing lexical sophistication using word, bigram, and dependency indices. In Perspectives on the L2 Phrasicon: The View from Learner Corpora (pp. 126–151). Multilingual Matters. find
  22. Paquot, M. (2019). The phraseological dimension in interlanguage complexity research. Second Language Research, 35(1), 121–145. find
  23. Polio, C., & Yoon, H. (2018). The reliability and validity of automated tools for examining variation in syntactic complexity across genres. International Journal of Applied Linguistics, 28(1), 165–188. find
  24. Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019). RoBERTa: A Robustly Optimized BERT Pretraining Approach (arXiv:1907.11692). arXiv. http://arxiv.org/abs/1907.11692 ↗
  25. Rubin, R. (2021). Assessing the impact of automatic dependency annotation on the measurement of phraseological complexity in L2 Dutch. International Journal of Learner Corpus Research, 7(1), 131–162. find
  26. Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing. find
  27. Sakaguchi, K., Post, M., and Van Durme, B. (2017). Error-repair Dependency Parsing for Ungrammatical Texts. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 189–195, Vancouver, Canada. Association for Computational Linguistics. find
  28. Santorini, B. (1990). Part-of-speech tagging guidelines for the Penn Treebank project (3rd revision, 2nd printing). Ms., Department of Linguistics, UPenn. Philadelphia, PA. find our card
  29. Silveira, N., Dozat, T., De Marneffe, M.-C., Bowman, S., Connor, M., Bauer, J., & Manning, C. (2014). A Gold Standard Dependency Corpus for English. Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), 2897–2904. find
  30. Yannakoudakis, H., Briscoe, T., & Medlock, B. (2011). A new dataset and method for automatically grading ESOL texts. 180–189. find
  31. Zeldes, A. (2017). The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3), 581–612. find
  32. Zeman, D., Popel, M., Straka, M., Hajic, J., Nivre, J., Ginter, F., Luotolahti, J., Pyysalo, S., Petrov, S., & Potthast, M. (2017). CoNLL 2017 shared task: Multilingual parsing from raw text to universal dependencies. 1–19. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kyle-2022-dependency-treebank-spoken-second-language/