Mova0.5 pre-release⌕
☰
home / docs / papers / Modern Hebrew / Predicting the Proficiency Level of Nonnative Hebrew Authors

Predicting the Proficiency Level of Nonnative Hebrew Authors

What it is about

The authors use the Hebrew Essay Corpus (4000 essays: native speakers and learners with Arabic, French and Russian as L1) to predict three things with XGBoost on about 2300 hand-chosen features, not deep models: whether the author is a native speaker, the author's L1, and the essay's proficiency score. Native versus non-native gets 99% accuracy (six errors in 800 test essays). L1 identification gets 77%. Score prediction gives a mean absolute error of 1.67 points against 1.90 for a median baseline, and the two human raters differ by 2.52 points. The strongest features are text length, lexical richness (root type-token ratio), median word rank, and letter and affix counts; SHAP is used to read them. For Arabic learners, high mean sentence length and overuse of the conjunctive ו stand out. For Russian learners, underuse of the definite article ה is the strongest sign.

Ideas

For Mova

I1 is a rule table for 2. Script and spelling (homophone and final-form letters) and for the transfer notes in 10. Historical layers: Biblical, Mishnaic, Medieval and Modern Hebrew. I2 is a feature for 8. Lexicon and register once the Modern Hebrew reader exists (12. Mova and Modern Hebrew: where it stands, what it would need). I3 is a reporting rule for any measurement; the data itself is NC and is used only for tests.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{nguyen2022predicting,
  title = {{Predicting the Proficiency Level of Nonnative Hebrew Authors}},
  author = {Nguyen, Isabelle and Wintner, Shuly},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  url = {https://aclanthology.org/2022.lrec-1.573/}
}

References

43 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexopoulou, T., Michel, M., Murakami, A., and Meurers, D. (2017). Task effects on linguistic complexity and accuracy: a large-scale learner corpus analysis employing natural language processing techniques. Language Learning, 67(S1):167–209. find
  2. Amorim, E. and Veloso, A. (2017). A multi-aspect analysis of automatic essay scoring for Brazilian Portuguese. In Proceedings of the Student Research Workshop at the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 94–102, Valencia, Spain, April. Association for Computational Linguistics. find
  3. Artstein, R. and Poesio, M. (2008). Inter-Coder Agreement for Computational Linguistics. Computational Linguistics, 34(4):555–596, December. find
  4. Attali, Y. and Burstein, J. (2005). Automated essay scoring with E-rater v.2.0. Technical report, Educational Testing Service (ETS), November. find
  5. Blanchard, D., Tetreault, J., Higgins, D., Cahill, A., and Chodorow, M. (2013). TOEFL11: A corpus of non-native english. ETS Research Report Series, 2013(2):i–15. find
  6. Bulté, B. and Housen, A. (2012). Defining and operationalising L2 complexity. In Alex Housen, et al., editors, Dimensions of L2 performance and proficiency: Complexity,accuracy and fluency in SLA, pages 21–46. John Benjamins, Amsterdam. find
  7. Carroll, J. B. (1964). Language and thought. Reading Improvement, 2(1):80. find
  8. Crossley, S. A., Salsbury, T., and McNamara, D. S. (2012). Predicting the proficiency level of language learners using lexical indices. Language Testing, 29(2):243–263. find
  9. Dasgupta, T., Naskar, A., Dey, L., and Saha, R. (2018). Augmenting textual qualitative features in deep convolution recurrent neural network for automatic essay scoring. In Proceedings of the 5th Workshop on Natural Language Processing Techniques for Educational Applications, pages 93–102, Melbourne, Australia, July. Association for Computational Linguistics. find
  10. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  11. Fabri, R., Gasser, M., Habash, N., Kiraz, G., and Wintner, S. (2014). Linguistic introduction: The orthography, morphology and syntax of Semitic languages. In Imed Zitouni, editor, Natural Language Processing of Semitic Languages, Theory and Applications of Natural Language Processing, pages 3–41. Springer, Berlin Heidelberg. find
  12. Gafni, C., Prior, A., and Wintner, S. (2022). The Hebrew Essay Corpus. In Proceedings of LREC-2022, June. find our card
  13. Goldin, G., Rabinovich, E., and Wintner, S. (2018). Native language identification with user generated content. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3591–3601. Association for Computational Linguistics. find
  14. Guiraud, P. (1954). Les caractères statistiques du vocabulaire: essai de méthodologie. Presses Universitaires de France, Paris. find
  15. Guiraud, P. (1960). Problèmes et méthodes de la statistique linguistique. Presses Universitaires de France, Paris. find
  16. Hawkins, J. A. and Filipović, L. (2012). Criterial Features in L2 English. Specifying the Reference Levels of the Common European Framework. Cambridge University Press, Cambridge. find
  17. Hirao, R., Arai, M., Shimanaka, H., Katsumata, S., and Komachi, M. (2020). Automated essay scoring system for nonnative Japanese learners. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1250–1257, Marseille, France, May. European Language Resources Association. find
  18. Itai, A. and Wintner, S. (2008). Language resources for Hebrew. Language Resources and Evaluation, 42(1):75–98, March. find
  19. Koppel, M., Schler, J., and Zigdon, K. (2005). Automatically determining an anonymous author’s native language. Intelligence and Security Informatics, pages 41–76. find
  20. Krippendorff, K. (2011). Computing Krippendorff’s Alpha-reliability. find
  21. Lüdeling, A., Doolittle, S., Hirschmann, H., Schmidt, K., and Walter, M. (2008). Das Lernerkorpus Falko. Deutsch als Fremdsprache, 45(2):67–73. find
  22. Malmasi, S., Evanini, K., Cahill, A., Tetreault, J., Pugh, R., Hamill, C., Napolitano, D., and Qian, Y. (2017). A report on the 2017 native language identification shared task. In Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, pages 62–75. Association for Computational Linguistics. find
  23. Massung, S. and Zhai, C. (2016). Non-native text analysis: A survey. Natural Language Engineering, 22(2):163–186. find
  24. Mathias, S. and Bhattacharyya, P. (2018). ASAP++: Enriching the ASAP automated essay grading dataset with essay attribute scores. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). find
  25. McCarthy, P. M. and Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior research methods, 42(2):381–392. find
  26. McCarthy, P. M. (2005). An assessment of the range and usefulness of lexical diversity measures and the potential of the measure of textual, lexical diversity (MTLD). Ph.D. thesis, The University of Memphis. find
  27. Nisioi, S., Rabinovich, E., Dinu, L. P., and Wintner, S. (2016). A corpus of native, non-native and translated texts. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC 2016). European Language Resources Association (ELRA), May. find
  28. Ortega, L. (2003). Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing. Applied Linguistics, 24(4):492–518. Oxford University Press. find
  29. Pilán, I., Volodina, E., and Zesch, T. (2016). Predicting proficiency levels in learner writings by transferring a linguistic complexity model from expert-written coursebooks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2101– 2111, Osaka, Japan, December. find
  30. The COLING 2016 Organizing Committee. find
  31. Rabinovich, E. and Wintner, S. (2015). Unsupervised identification of translationese. Transactions of the Association for Computational Linguistics, 3:419– 432. find
  32. Rabinovich, E., Nisioi, S., Ordan, N., and Wintner, S. (2016). On the similarities between native, non-native and translated texts. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, pages 1870–1881, August. find
  33. Rabinovich, E., Ordan, N., and Wintner, S. (2017). Found in translation: Reconstructing phylogenetic language trees from translations. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 530–540. Association for Computational Linguistics, July. find
  34. Scarborough, H. S. (1990). Index of productive syntax. Applied Psycholinguistics, 11:1–22. find
  35. Tetreault, J., Blanchard, D., and Cahill, A. (2013). A report on the first native language identification shared task. In Proceedings of the Eighth Workshop on Building Educational Applications Using NLP. Association for Computational Linguistics, June. find
  36. Uto, M., Xie, Y., and Ueno, M. (2020). Neural automated essay scoring incorporating handcrafted features. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6077– 6088, Barcelona, Spain (Online), December. International Committee on Computational Linguistics. find
  37. Vajjala, S. and Rama, T. (2018). Experiments with universal CEFR classification. find
  38. Volansky, V., Ordan, N., and Wintner, S. (2015). On the features of translationese. Digital Scholarship in the Humanities, 30(1):98–118, April. find
  39. Weiss, Z. (2017). Using measures of linguistic complexity to assess German L2 proficiency in learner corpora under consideration of task-effects. Master’s thesis, Eberhard Karls Universität Tübingen. Wisniewski, K., Schöne, K., Nicolas, L., Vettori, C., Boyd, A., Meurers, D., Abel, A., and Hana, J. find
  40. (2013). MERLIN: An online trilingual learner corpus empirically grounding the european reference levels in authentic learner data. In ICT4LL 2013, 6th edition of the ICT for Language Learning Conference, Florence, 10. find
  41. Wisniewski, K. (2017). The Empirical Validity of the Common European Framework of Reference Scales. An Exemplary Study for the Vocabulary and Fluency Scales in a Language Testing Context. Applied Linguistics, 39(6):933–959, 03. find
  42. Yona, S. and Wintner, S. (2008). A finite-state morphological grammar of Hebrew. Natural Language Engineering, 14(2):173–190, April. find our card
  43. Zesch, T., Wojatzki, M., and Scholten-Akoun, D. (2015). Task-independent features for automated essay grading. In Proceedings of the Tenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 224–232, Denver, Colorado, June. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2022-nguyen-predicting-proficiency-level-nonnative-hebrew/