HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark

What it is about

HeQ is an extractive question-answering benchmark in the SQuAD style for Modern Hebrew: 30,147 question–answer pairs from Hebrew Wikipedia and from the Geektime tech news site. Crowd workers were trained one by one, and each question was graded as rejected, verified, good or gold (gold means it needs inference). The central point is the metric: a Hebrew clitic can make a whole answer one word, so a correct span can score 0 under word-level F1. The authors propose TLNLS, a token-level normalised Levenshtein similarity over whitespace tokens, and show it separates verified-correct from verified-wrong spans far better than F1 (positives 0.73 vs 0.58; negatives 0.09 vs 0.02). Among the models, multilingual BERT is best (TLNLS 78.2 on the test set), and ChatGPT is the worst. Going from 1.7K to the full training set adds 11.4 TLNLS points.

Ideas

• I1. Use a TLNLS-style span score in Mova's own Hebrew answer tests: split on whitespace, take for each gold token its best normalised similarity to a predicted token, and average over the longer span. Report it next to F1, never alone.
• I2. Gate any span metric with a negative control: a set of wrong spans that a model actually produced, which the metric must score low. A scorer that cannot show red is not a gate.
• I3. For dates and numbers the character similarity gives partial credit to "1948" vs "1921" (0.5). The paper falls back to plain F1 when digits fill more than half the span; adopt the same rule.

For Mova

I1–I3 go into the span-scoring helper of the test harness in tools/hbo and into the Modern Hebrew test plan in 12. Mova and Modern Hebrew: where it stands, what it would need. The boundary problem behind them is the clitic split discussed in Nouns, adjectives, pronouns and numerals.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.915

@inproceedings{cohen2023heq,
  title = {{HeQ: a Large and Diverse Hebrew Reading Comprehension Benchmark}},
  author = {Cohen, Amir DN and Merhav, Hilla and Goldberg, Yoav and Tsarfaty, Reut},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.915},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.915}
}

References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454–470. find
2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
3. Yanai Elazar, Nora Kassner, Shauli Ravfogel, Amir Feder, Abhilasha Ravichander, Marius Mosbach, Yonatan Belinkov, Hinrich Schütze, and Yoav Goldberg. 2022. Measuring Causal Effects of Data Statistics on Language Model’s ‘Factual’ Predictions. find
4. Jianfeng Gao, Michel Galley, and Lihong Li. 2018. Neural approaches to conversational AI. In 41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018, pages 1371–1374. Association for Computing Machinery, Inc. find
5. Eylon Guetta, Avi Shmidman, Shaltiel Shmidman, Cheyn Shmuel Shmidman, Joshua Guedalia, Moshe Koppel, Dan Bareket, Amit Seker, and Reut Tsarfaty. 2022. Large Pre-Trained Models with Extra-Large Vocabularies: A Contrastive Analysis of Hebrew BERT Models and a New One to Outperform Them All. Technical report. find our card
6. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Retrieval Augmented Language Model Pre-Training. In Proceedings of the 37th International Conference on Machine Learning, pages 3929–3938. PMLR. find
7. Lynette Hirschman, Marc Light, Eric Breck, and John D. Burger. 1999. Deep read: A reading comprehension system. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 325–332, College Park, Maryland, USA. Association for Computational Linguistics. find
8. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In EMNLP 2017 - Conference on Empirical Methods in Natural Language Processing, Proceedings, pages 2021–2031. Association for Computational Linguistics (ACL). find our card
9. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving Pre-training by Representing and Predicting Spans. Transactions of the Association for Computational Linguistics, 8:64–77. find
10. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics. find
11. Omri Keren and Omer Levy. 2021. ParaShoot: A Hebrew question answering dataset. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 106–112, Punta Cana, Dominican Republic. Association for Computational Linguistics. find our card
12. Miyoung Ko, Jinhyuk Lee, Hyunjae Kim, Gangwoo Kim, and Jaewoo Kang. 2020. Look at the first sentence: Position bias in question answering. In EMNLP 2020 - 2020 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pages 1109–1121. Association for Computational Linguistics (ACL). find
13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466. find
14. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. find
15. Amir More, Amit Seker, Victoria Basmova, and Reut Tsarfaty. 2019. Joint transition-based models for morpho-syntactic parsing: Parsing strategies for MRLs and a case study from Modern Hebrew. Transactions of the Association for Computational Linguistics, 7:33–48. find our card
16. Gonzalo Navarro. 2001. A guided tour to approximate string matching. ACM Computing Surveys, 33(1):31– 88. find
17. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018a. Know what you don’t know: Unanswerable questions for SQuAD. In ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), volume 2, pages 784–789. find
18. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018b. Know what you don’t know: Unanswerable questions for SQuAD. In ACL 2018 - 56th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference (Long Papers), volume 2, pages 784–789. find
19. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016a. SQuad: 100,000+ questions for machine comprehension of text. In EMNLP 2016 - Conference on Empirical Methods in Natural Language Processing, Proceedings, pages 2383–2392. find
20. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016b. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. find
21. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. find
22. Marc-Antoine Rondeau and T. J. Hazen. 2018. Systematic error analysis of the Stanford question answering dataset. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 12–20, Melbourne, Australia. Association for Computational Linguistics. find
23. Sebastian Ruder and Avirup Sil. 2021. Multi-Domain Multilingual Question Answering. In EMNLP 2021 -2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 17– 21. find
24. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Greenfeld, and Reut Tsarfaty. 2022. AlephBERT: Language model pre-training and evaluation from sub-word to sentence level. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 46–56, Dublin, Ireland. Association for Computational Linguistics. find our card
25. Kazutoshi Shinoda, Saku Sugawara, and Akiko Aizawa. 2022. Look to the right: Mitigating relative position bias in extractive question answering. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 418–425, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
26. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200, Vancouver, Canada. Association for Computational Linguistics. find
27. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. LIMA: Less Is More for Alignment. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/findings-2023-cohen-heq-large-diverse-hebrew-reading/