Generation and Evaluation of English Grammar Multiple-Choice Cloze Exercises

What it is about

The paper fine-tunes Llama 3 8B (4-bit, QLoRA) to write English grammar multiple-choice cloze items (a sentence with a gap, one key, one or more distractors) for 19 grammar topics. Earlier systems used rules or fixed part-of-speech patterns, and the authors find their outputs creative but dull. The training data are the cleaned SC-Ques set, which has about 300k items; after removing near-duplicates only about 30% remains, and the least frequent topic is upsampled with GPT-4 output. Topics are assigned by a spaCy dependency matcher, by the overlap of the topic span with the key span. Two rule-based checks are proposed. Structural compliance checks that the gap is in the right place and that the distractor is homogeneous: same lemma for inflectional topics, or a word from a short admitted list for free-morpheme topics. Diversity is measured with Self-BLEU on 2- to 5-grams. In the evaluation, 50 items per topic were generated. The automatic check reaches 95% F1 against human structure labels (precision 98%, recall 91%). Averaged over topics, 79% of items pass the whole human review and the mean Self-BLEU is 7%. The main error is a meaningless sentence (75% of the errors).

Ideas

• I1. en/src/morph.rs: the homogeneity test of the paper as a lemma check. A distractor is homogeneous when its lemma equals the key's lemma for inflectional topics. Count how often the test passes on the school exercises, and check the failures by hand.
• I2. Structural compliance as a rule set with a measured F1: write the gap-position and key-form checks over the UD tree (en/src/parse.rs), label 100 school items by hand, and report precision and recall against those labels, as the paper does.
• I3. Topic labels by a tree-pattern matcher over UD relations rather than a spaCy matcher: a fixed table of patterns (for example an auxiliary plus past participle for the perfect), applied in order, with the first match winning. The key-span overlap rule is kept as it is.
• I4. A Self-BLEU-like repetition count over 2- to 5-grams for generated school questions, with a fixed threshold, to catch near-duplicate items before they reach the lessons (world/src/knowledge.rs).

For Mova

Exercise generation for the school sits in world/src/knowledge.rs (lessons, exercises and tests). Morphology and the lemma test belong to en/src/morph.rs. Tree patterns for topics belong to en/src/parse.rs and the UD trees of 20. Syntax: the Universal Dependencies frame. Topic chapters are 13. Verbs, 12. Pronouns, determiners and numerals (possessives and quantifiers) and 26. Questions (WH-questions). Measures: lemma-test pass rate, precision and recall of the rule checks against 100 hand-labelled items, and the repetition rate, run twice for determinism.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{donati2024generation,
  title = {{Generation and Evaluation of English Grammar Multiple-Choice Cloze Exercises}},
  author = {Donati, Nicolò and Periani, Matteo and Natale, Paolo Di and Savino, Giuseppe and Torroni, Paolo},
  booktitle = {Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024)},
  year = {2024},
  url = {https://aclanthology.org/2024.clicit-1.39/}
}

References

36 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. H. G. Widdowson, Teaching Language as Communication, Oxford University Press, Oxford, 1978. find
2. H. G. Widdowson, Explorations in Applied Linguistics, Oxford University Press, Oxford, 1979. find
3. W. Gan, Z. Qi, J. Wu, J. C.-W. Lin, Large language models in education: Vision and opportunities, in: 2023 IEEE International Conference on Big Data (BigData), 2023, pp. 4776–4785. doi: 10. 1109/BigData59044.2023.10386291 . find
4. Q. Wang, R. Rose, N. Orita, A. Sugawara, Automated generation of multiple-choice cloze questions for assessing english vocabulary using gpt-turbo 3.5, 2024. URL: https://arxiv.org/abs/2403. 02078. arXiv:2403.02078 . ↗
5. P. Chomphooyod, A. Suchato, N. Tuaycharoen, P. Punyabukkana, English grammar multiple-choice question generation using text-to-text transfer transformer, Computers and Education: Artificial Intelligence 5 (2023) 100158. URL: https://www.sciencedirect.com/science/ article/pii/S2666920X23000371. doi: https://doi.org/10.1016/j.caeai.2023.100158 . ↗
6. Y. Zhu, S. Lu, L. Zheng, J. Guo, W. Zhang, J. Wang, Y. Yu, Texygen: A benchmarking platform for text generation models, 2018. URL: https://arxiv.org/abs/ 1802.01886. arXiv:1802.01886 . ↗
7. D. H. Hymes, On communicative competence, in: J. B. Pride, J. Holmes (Eds.), Sociolinguistics. Selected Readings, Penguin, Harmondsworth, 1972, pp. 269–293. find
8. Q. Liu, Y. Huang, Z. Liu, S. Huang, J. Chen, X. Zhao, G. Lin, Y. Zhou, W. Luo, Sc-ques: A sentence completion question dataset for english as a second language learners, in: C. Frasson, P. Mylonas, C. Trous-sas (Eds.), Augmented Intelligence and Intelligent Tutoring Systems, Springer Nature Switzerland, Cham, 2023, pp. 678–690. find
9. L. F. Bachman, Fundamental Considerations in Language Testing, Oxford University Press, Oxford, 1990. find
10. J. E. Purpura, Assessing Grammar, Cambridge Language Assessment, Cambridge University Press, 2004. find
11. G. Fulcher, G. Fulcher, Practical Language Testing, 1st ed., Routledge, 2010. doi: 10.4324/ 980203767399 . find
12. V.-M. Pho, T. André, A.-L. Ligozat, B. Grau, G. Il-louz, T. François, Multiple choice question corpus analysis for distractor characterization, in: N. Calzolari, K. Choukri, T. Declerck, H. Loftsson, B. Maegaard, J. Mariani, A. Moreno, J. Odijk, S. Piperidis (Eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), European Language Resources Association (ELRA), Reykjavik, Iceland, 2014, pp. 4284– 4291. URL: http://www.lrec-conf.org/proceedings/ lrec2014/pdf/692_Paper.pdf. ↗
13. E. Sumita, F. Sugaya, S. Yamamoto, Measuring non-native speakers’ proficiency of english by using a test with automatically-generated fill-in-the-blank questions (2005). doi: 10.3115/1609829. 1609839 . ↗
14. J. Brown, G. A. Frishkoff, M. Eskénazi, Automatic question generation for vocabulary assessment, in: HLT/EMNLP 2005, Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, 6-8 October 2005, Vancouver, British Columbia, Canada, The Association for Computational Linguistics, 2005, pp. 819–826. URL: https://aclanthology.org/H05-1103/. ↗
15. S. Smith, A. P.V.S, A. Kilgarriff, Gap-fill tests for language learners: Corpus-driven item generation, 2010. URL: https://api.semanticscholar.org/ CorpusID:61531901. ↗
16. M. Majumder, S. K. Saha, A system for generating multiple choice questions: With a novel approach for sentence selection, in: H. Chen, Y. Tseng, Y. Matsumoto, L. Wong (Eds.), Proceedings of the 2nd Workshop on Natural Language Processing Techniques for Educational Applications, NLP-TEA@ACL/IJCNLP, Beijing, China, July 31, 2015, Association for Computational Linguistics, 2015, pp. 64–72. URL: https://doi.org/10.18653/v1/W15-4410. doi: 10.18653/V1/W15-4410 . ↗
17. M. Majumder, S. K. Saha, A system for generating multiple choice questions: With a novel approach for sentence selection, in: H. Chen, Y. Tseng, Y. Matsumoto, L. Wong (Eds.), Proceedings of the 2nd Workshop on Natural Language Processing Techniques for Educational Applications, NLP-TEA@ACL/IJCNLP, Beijing, China, July 31, 2015, Association for Computational Linguistics, 2015, pp. 64–72. URL: https://doi.org/10.18653/v1/W15-4410. doi: 10.18653/V1/W15-4410 . ↗
18. T. Goto, T. Kojiri, T. Watanabe, T. Iwata, T. Yamada, Automatic generation system of multiple-choice cloze questions and its evaluation, Knowledge Management & E-Learning: An International Journal 2 (2010) 210–224. URL: https://api.semanticscholar.org/CorpusID:15482954. ↗
19. J. D. Lafferty, A. McCallum, F. C. N. Pereira, Conditional random fields: Probabilistic models for segmenting and labeling sequence data, in: C. E. Brodley, A. P. Danyluk (Eds.), Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001), Williams College, Williamstown, MA, USA, June 28 - July 1, 2001, Morgan Kaufmann, 2001, pp. 282–289. find
20. S. K. Bitew, J. Deleu, A. S. Doğruöz, C. Develder, T. Demeester, Learning from partially annotated data: Example-aware creation of gap-filling exercises for language learning, in: E. Kochmar, J. Burstein, A. Horbach, R. Laarmann-Quante, N. Madnani, A. Tack, V. Yaneva, Z. Yuan, T. Zesch (Eds.), Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications, BEA@ACL 2023, Toronto, Canada, 13 July 2023, Association for Computational Linguistics, 2023, pp. 598–609. URL: https://doi.org/10. 18653/v1/2023.bea-1.51. doi: 10.18653/V1/2023. BEA-1.51 . ↗
21. Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoyanov, Roberta: A robustly optimized bert pretraining approach, 2019. URL: https://arxiv.org/abs/1907.11692. arXiv:1907.11692 . ↗
22. S. Matsumori, K. Okuoka, R. Shibata, M. Inoue, Y. Fukuchi, M. Imai, Mask and cloze: Automatic open cloze question generation using a masked language model, IEEE Access 11 (2023) 9835–9850. URL: http://dx.doi.org/10.1109/ACCESS.2023.3239005. doi: 10.1109/ access.2023.3239005 . ↗
23. P. Chomphooyod, A. Suchato, N. Tuaycharoen, P. Punyabukkana, English grammar multiple-choice question generation using text-to-text transfer transformer, Comput. Educ. Artif. Intell. 5 (2023) 100158. URL: https://doi.org/10.1016/j.caeai.2023. 100158. doi: 10.1016/J.CAEAI.2023.100158 . ↗
24. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, 2023. URL: https://arxiv.org/ abs/1706.03762. arXiv:1706.03762 . ↗
25. V. Slavuj, L. Nacinovic Prskalo, M. Brkic Bakaric, Automatic generation of language exercises based on a universal methodology: An analysis of possibilities, Bulletin of the Transilvania University of Brasov. Series IV: Philology and Cultural Studies 14 (63) (2022) 29–48. doi: 10.31926/but.pcs.2021. 63.14.2.3 . ↗
26. A. Malafeev, Language exercise generation, International Journal of Conceptual Structures and Smart Applications 2 (2014) 20–35. doi: 10.4018/ IJCSSA.2014070102 . find
27. D. Perrett, D. March, An evidence-based approach to distractor generation in multiple-choice language tests, 2019. doi: 10.13140/RG.2.2.22779. 16165 . ↗
28. K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: P. Isabelle, E. Charniak, D. Lin (Eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002, pp. 311–318. URL: https://aclanthology.org/P02-1040. doi: 10.3115/ 1073083.1073135 . ↗ our card
29. C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, Association for Computational Linguistics, Barcelona, Spain, 2004, pp. 74–81. URL: https://aclanthology.org/W04-1013. ↗
30. S. Banerjee, A. Lavie, METEOR: An automatic metric for MT evaluation with improved correlation with human judgments, in: J. Goldstein, A. Lavie, C.-Y. Lin, C. Voss (Eds.), Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, Association for Computational Linguistics, Ann Arbor, Michigan, 2005, pp. 65–72. URL: https://aclanthology.org/W05-0909. ↗ our card
31. Meta, Introducing Meta Llama 3: The most capable openly available LLM to date, https://ai.meta.com/ blog/meta-llama-3/, April 2024. ↗
32. K. Tirumala, D. Simig, A. Aghajanyan, A. Morcos, D4: Improving llm pretraining via document deduplication and diversification, in: A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, S. Levine (Eds.), Advances in Neural Information Processing Systems, volume 36, Curran Associates, Inc., 2023, pp. 53983–53995. URL: https://proceedings.neurips.cc/paper_files/paper/2023/file/ a8f8cbd7f7a5fb2c837e578c75e5b615-Paper-Datasets_ and_Benchmarks.pdf. ↗
33. O. et al., Gpt-4 technical report, 2024. URL: https://arxiv.org/abs/2303.08774. arXiv:2303.08774 . ↗
34. T. Dettmers, A. Pagnoni, A. Holtzman, L. Zettlemoyer, Qlora: Efficient finetuning of quantized llms, 2023. URL: https://arxiv.org/abs/2305.14314. arXiv:2305.14314 . ↗
35. Z. Fu, W. Lam, A. M.-C. So, B. Shi, A theoretical analysis of the repetition problem in text generation, Proceedings of the AAAI Conference on Artificial Intelligence 35 (2021) 12848–12856. URL: https://ojs.aaai.org/index.php/AAAI/article/ view/17520. doi: 10.1609/aaai.v35i14.17520 . ↗
36. Z. Xu, S. Jain, M. Kankanhalli, Hallucination is inevitable: An innate limitation of large language models, 2024. URL: https://arxiv.org/abs/2401.11817. arXiv:2401.11817 . ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/donati-2024-generation-evaluation-english-grammar-multiple/