Transformer-based Hebrew NLP models for Short Answer Scoring in Biology

What it is about

The paper tests AlephBERT, a Hebrew pretrained model, for analytic scoring of short high-school biology answers. The data are answers from 669 students (grades 10–12, about 25 schools) to three items on smoking and carbon monoxide, anaemia, and travel at high altitude. Each answer gets 10–13 binary labels from a 13-category causal rubric (category 0–12). One classifier per category was fine-tuned and compared with a CNN baseline from earlier work. AlephBERT is significantly better (one-sided Wilcoxon p = 0.016, n = 44 cells). The largest gain is category 9, a specific phrase ("cellular respiration"), from kappa 0.06–0.73 to above 0.90. In a cross-item setting (train on two items, test on the third), performance is not significantly worse than within-item (p = 0.9, n = 32), and the average kappa over categories is above 0.6 for each item. Category 1 fails completely across items because it occurs in only two of the three items.

Ideas

• I1. The rubric as data: write each analytic category as a small table of required fact patterns (for example, CO binds haemoglobin; less oxygen; slower cellular respiration; less ATP). Match lemmas from the UD tree, not surface strings, so every score names the category and the words that fired it.
• I2. Use the cross-item protocol as a test of generalization: fit the rules on two items and test on the third, with a student-disjoint 60/20/20 split. A rule set that fails on the held-out item says which pattern was item-specific.
• I3. Phrase matching in Hebrew needs prefix handling: the one-phrase category only matches if the attached prepositions and conjunctions are stripped first. The paper's largest-gain category is a good test for this.

For Mova

I1 is a rule table for the planned Hebrew reader in 12. Mova and Modern Hebrew: where it stands, what it would need, applied in prag. I2 is the test design for world. I3 goes into Prepositions, prefixes, conjunctions and particles (prefix stripping). The paper does not say whether its student answers are released, so do not assume access to them.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.bea-1.46

@inproceedings{schleifer2023transformer,
  title = {{Transformer-based Hebrew NLP models for Short Answer Scoring in Biology}},
  author = {Schleifer, Abigail Gurin and Klebanov, Beata Beigman and Ariely, Moriah and Alexandron, Giora},
  booktitle = {Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023)},
  year = {2023},
  doi = {10.18653/v1/2023.bea-1.46},
  url = {https://doi.org/10.18653/v1/2023.bea-1.46}
}

References

13 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Moriah Ariely, Tanya Nazaretsky, and Giora Alexandron. 2023. Machine learning and Hebrew NLP for automated assessment of open-ended questions in biology. International Journal of Artificial Intelligence in Education, 33(1):1–34. find
2. Beata Beigman Klebanov and Nitin Madnani. 2021. Automated essay scoring. Synthesis Lectures on Human Language Technologies, 14(5):1–314. find
3. Marie Bexte, Andrea Horbach, and Torsten Zesch. 2022. Similarity-based content scoring-how to make s-bert keep up with bert. In Proceedings of the 17th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2022), pages 118–123. find
4. Aubrey Condor. 2020. Exploring automatic short answer grading as a tool to assist in human rating. In Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part II 21, pages 74–79. Springer. find
5. Stefan Haller, Adina Aldea, Christin Seifert, and Nicola Strisciuglio. 2022. Survey on automated short answer grading with deep learning: From word embeddings to transformers. arXiv preprint arXiv:2204.03503. ↗
6. Zhaohui Li, Yajur Tomar, and Rebecca J. Passonneau. 2021. A semantic feature-wise transformation relation network for automatic short answer grading. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6030–6040, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
7. Omar Nael, Youssef ELmanyalawy, and Nada Sharaf. 2022. AraScore: A deep learning-based system for Arabic short answer scoring. Array, 13:100109. find
8. Haruki Oka, Hung Tuan Nguyen, Cuong Tuan Nguyen, Masaki Nakagawa, and Tsunenori Ishioka. 2022. Fully automated short answer scoring of the trial tests for common entrance examinations for japanese university. In Artificial Intelligence in Education: 23rd International Conference, AIED 2022, Durham, UK, July 27–31, 2022, Proceedings, Part I, pages 180–192. Springer. find
9. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703– 1714, Online. Association for Computational Linguistics. find
10. Johannes Schneider, Robin Richner, and Micha Riser. 2023. Towards trustworthy autograding of short, multi-lingual, multi-type answers. International Journal of Artificial Intelligence in Education, 33(1):88–118. find
11. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Greenfeld, and Reut Tsarfaty. 2022. AlephBERT: Language model pre-training and evaluation from sub-word to sentence level. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 46–56, Dublin, Ireland. Association for Computational Linguistics. find our card
12. Chul Sung, Tejas Dhamecha, Swarnadeep Saha, Tengfei Ma, Vinay Reddy, and Rishi Arora. 2019a. Pre-training bert on domain resources for short answer grading. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6071–6075. find
13. Chul Sung, Tejas Indulal Dhamecha, and Nirmal Mukhi. 2019b. Improving short answer grading using transformer-based pre-training. In Artificial Intelligence in Education: 20th International Conference, AIED 2019, Chicago, IL, USA, June 25-29, 2019, Proceedings, Part I 20, pages 469–481. Springer. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2023-schleifer-transformer-based-hebrew-nlp-models/