Predicting Learner Levels for Online Exercises of Hebrew

What it is about

The paper predicts the course level (100 to 300+) of university students of Hebrew from one placement-exam exercise: a scrambled sentence whose words must be put in order. The data are 128 sentences from 38 learners, so data is very sparse. Each learner answer is aligned to the closest of the gold orders, with a cost that mixes Levenshtein distance, longest common subsequence and displacement. Features are then extracted in three groups: intra-token errors (inside a word, with morphological slots), inter-token errors (shifts, transpositions and missing tokens, found by running the edit distance with substitutions turned off) and global sentence measures. A first memory-based classifier (TiMBL) maps each phenomenon to a level, and a second one maps the counts of phenomena to the learner's level. Accuracy is 60.5 percent against a majority baseline of 34.2 percent. Inter-token features matter most (dropping them gives 36.8 percent), and weighted kappa is 0.647.

Ideas

• I1. A deterministic shift detector for word-order exercises: align the learner's words to a gold order with substitutions off, pair the deletions with the insertions, and report transpositions and shifts with their lengths. This is a rule that can be written in Rust in prag for English scrambled-sentence checks, and it is transparent: each flagged shift can be shown to the learner.
• I2. Store the acceptable word orders of an exercise once, with the morphology kept apart from the order, as the paper does (morphology is annotated once for all orders). In world this is a set of orders over one set of frames, held in BTreeSet so that the result is deterministic.
• I3. Report kappa next to accuracy, and use leave-one-out on small sets, because a 60 percent accuracy on 38 learners is not separated from a 34 percent baseline without a second number. This is the measurement rule for any Mova placement or level test.

For Mova

I1 is a rule for prag (word-order checks). I2 is a data design for world. I3 is a measurement rule for any level test, to be stated in 12. Mova and Modern Hebrew: where it stands, what it would need. The paper is about Hebrew courses; Mova would use the same method only for its own test sets, not for any published exam.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{dickinson2012predicting,
  title = {{Predicting Learner Levels for Online Exercises of Hebrew}},
  author = {Dickinson, Markus and Kübler, Sandra and Meyer, Anthony},
  booktitle = {Proceedings of the Seventh Workshop on Building Educational Applications Using NLP},
  year = {2012},
  url = {https://aclanthology.org/W12-2011/}
}

References

21 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dora Alexopoulou, Helen Yannakoudakis, and Ted Briscoe. 2010. From discriminative features to learner grammars: a data driven approach to learner corpora. Talk given at Second Language Research Forum, University of Maryland, October 2010. find
2. Yigal Attali and Jill Burstein. 2006. Automated essay scoring with e-rater v.2. Journal of Technology, Learning, and Assessment, 4(3), February. find
3. Michele Banko and Eric Brill. 2001. Mitigating the paucity-of-data problem: Exploring the effect of training corpus size on classifier performance for natural language processing. In Proceedings of HLT 2001, First International Conference on Human Language Technology Research, pages 253–257, San Diego, CA. find
4. Adriane Boyd. 2010. EAGLE: an error-annotated corpus of beginning learner German. In Proceedings of LREC-10, Valetta, Malta. find
5. Walter Daelemans, Antal van den Bosch, and Jakub Zavrel. 1999. Forgetting exceptions is harmful in language learning. Machine Learning, 34:11–41. ↗
6. Walter Daelemans, Jakub Zavrel, Ko van der Sloot, and Antal van den Bosch. 2010. Timbl: Tilburg memory based learner, version 6.3, reference guide. Technical report, ILK Research Group. Technical Report Series no. 10-01. find
7. Robert Dale and Adam Kilgarriff. 2011. Helping our own: The HOO 2011 pilot shared task. In Proceedings of the Generation Challenges Session at the 13th European Workshop on Natural Language Generation, pages 242–249, Nancy, France, September. find
8. Ana Dı́az-Negrillo and Jesús Fernández-Domı́nguez. 2006. Error tagging systems for learner corpora. Spanish Journal of Applied Linguistics (RESLA), 19:83–102. find
9. Markus Dickinson, Ross Israel, and Sun-Hee Lee. 2011. Developing methodology for Korean particle error detection. In Proceedings of the Sixth Workshop on Innovative Use of NLP for Building Educational Applications, pages 81–86, Portland, OR, June. find
10. Markus Dickinson. 2011. On morphological analysis for learner language, focusing on Russian. Research on Language and Computation, 8(4):273–298. ↗
11. Glenn Fulcher. 1997. An English language placement test: issues in reliability and validity. Language Testing, 14(2):113–138. ↗
12. Yoav Goldberg and Michael Elhadad. 2011. Joint Hebrew segmentation and parsing using a PCFGLA lattice parser. In Proceedings of ACL-HLT, pages 704– 709, Portland, OR, June. find our card
13. Gintarè Grigonytè, João Paulo Cordeiro, Gaël Dias, Rumen Moraliyski, and Pavel Brazdil. 2010. Paraphrase alignment for synonym evidence discovery. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING), pages 403– 411, Beijing, China. find
14. John A. Hawkins and Paula Buttery. 2010. Criterial features in learner corpora: Theory and illustrations. English Profile Journal, 1(1):1–23. ↗
15. John A. Hawkins and Luna Filipović. 2010. Criterial Features in L2 English: Specifying the Reference Levels of the Common European Framework. Cambridge University Press. find
16. Alon Itai and Shuly Wintner. 2008. Language resources for Hebrew. Language Resources and Evaluation, 42(1):75–98. ↗
17. Alla Rozovskaya and Dan Roth. 2011. Algorithm selection and model adaptation for ESL correction tasks. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 924–933, Portland, OR, June. find
18. Khalil Sima’an, Alon Itai, Yoad Winter, Alon Altman, and N. Nativ. 2001. Building a tree-bank of Modern Hebrew text. Traitment Automatique des Langues, 42(2). find
19. Joel Tetreault and Martin Chodorow. 2008. The ups and downs of preposition error detection in ESL writing. In Proceedings of COLING-08, Manchester. find
20. Helen Yannakoudakis, Ted Briscoe, and Ben Medlock. 2011. A new dataset and method for automatically grading ESOL texts. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 180–189, Portland, OR, June. find
21. Shlomo Yona and Shuly Wintner. 2008. A finite-state morphological grammar of Hebrew. Natural Language Engineering, 14(2):173–190. ↗ our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2012-dickinson-predicting-learner-levels-online-exercises/