ParaShoot: A Hebrew Question Answering Dataset

What it is about

ParaShoot is the first SQuAD-style question answering dataset for modern Hebrew: 3038 (paragraph, question, answer span) examples. Paragraphs come from random Hebrew Wikipedia articles, and Prolific annotators wrote 3–5 questions per paragraph with minimal answer spans. The split is by article: 1792 training, 221 validation and 1025 test questions. The annotation cost was under $800. A manual check of 50 validation examples found 70% minimal spans, 28% too long and 2% too short, which puts the ceiling near 84 F1. Baselines on the test set: mBERT 56.1 F1 (32.0 EM), AlephBERT 49.6 (26.0), HeBERT 36.7 (18.2). Error analysis of 50 AlephBERT predictions: about 46% had no overlap with the annotated answer, and 22% had an answer that was a substring of the prediction. Question words (what, which, who, when, where, how, how much, why) cover about 87% of questions. Hebrew morphology often makes the question and the answer differ in form (a prefix or an inflection).

Ideas

• I1. Lemma-level matching: the paper's example shows the same noun as a prefixed form in the question and as a bare word in the answer. Score answers after stripping prefixes with the segmenter, and report the exact-match and the lemma-match scores side by side, on a small sample first.
• I2. Question-type rules: the eight question words of Table 3 map to answer roles (who to agent, when to time, where to place, how much to quantity). Build this as a rule table in prag, English first, and test whether world finds the answer role in the text.
• I3. Use the paper's error categories (exact, model inside annotation, annotation inside model, partial overlap, zero overlap) as the audit template for Mova's own answers, so that every error is put in one category.
• I4. Article-level split as a held-out habit: hold out whole documents, not sentences, for any Hebrew reading test in Mova.

For Mova

I2 belongs in prag and world, with the Hebrew version later in 12. Mova and Modern Hebrew: where it stands, what it would need. I1 is a measure for tools/hbo once it can read questions, and goes into 12. Mova and Modern Hebrew: where it stands, what it would need. I3 is a note for the docs on evaluation. I4 is a rule for any Hebrew reading test; the test set itself is not to be used before the release test.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.mrqa-1.11 · arXiv 2109.11314

@inproceedings{keren2021parashoot,
  title = {{ParaShoot: A Hebrew Question Answering Dataset}},
  author = {Keren, Omri and Levy, Omer},
  booktitle = {Proceedings of the 3rd Workshop on Machine Reading for Question Answering},
  year = {2021},
  doi = {10.18653/v1/2021.mrqa-1.11},
  eprint = {2109.11314},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.mrqa-1.11}
}

References

9 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

2. Joakim Nivre, Daniel Zeman, Filip Ginter, and Francis Tyers. 2017. Universal Dependencies. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts, Valencia, Spain. Association for Computational Linguistics. find
3. Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. 2020. A monolingual approach to contextualized word embeddings for mid-resource languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714, Online. Association for Computational Linguistics. find
4. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. find
5. Reut Tsarfaty, Djamé Seddah, Yoav Goldberg, Sandra Kuebler, Yannick Versley, Marie Candito, Jennifer Foster, Ines Rehbein, and Lamia Tounsi. 2010. Statistical parsing of morphologically rich languages (SPMRL) what, how and whither. In Proceedings of the NAACL HLT 2010 First Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 1–12, Los Angeles, CA, USA. Association for Computational Linguistics. find
6. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find
7. Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. 2021. Few-shot question answering by pretraining span selection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 3066–3079. Association for Computational Linguistics. find
8. Djamé Seddah, Reut Tsarfaty, Sandra Kübler, Marie Candito, Jinho D. Choi, Richárd Farkas, Jennifer Foster, Iakes Goenaga, Koldo Gojenola Gal-letebeitia, Yoav Goldberg, Spence Green, Nizar Habash, Marco Kuhlmann, Wolfgang Maier, Joakim Nivre, Adam Przepiórkowski, Ryan Roth, Wolfgang Seeker, Yannick Versley, Veronika Vincze, Marcin Woliński, Alina Wróblewska, and Eric Villemonte de la Clergerie. 2013. Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceedings of the Fourth Workshop on Statistical Parsing of Morphologically-Rich Languages, pages 146–182, Seattle, Washington, USA. Association for Computational Linguistics. find
9. Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert: A hebrew large pretrained language model to start-off your hebrew nlp application with. arXiv preprint arXiv:2104.04052. ↗ our card
10. Reut Tsarfaty, Dan Bareket, Stav Klein, and Amit Seker. 2020. From SPMRL to NMRL: What did we learn (and unlearn) in a decade of parsing find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ud-2021-keren-parashoot-hebrew-question-answering-dataset/