Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative Comprehension

Summary

FairytaleQA is a question-answering dataset for narrative comprehension aimed at the reading level of kindergarten to eighth-grade children. Its questions were written by education experts following an evidence-based framework from reading research, so that each question targets a specific narrative element. The dataset contains 10,580 questions over 278 child-friendly stories, labelled with seven types (character, setting, action, feeling, causal relationship, outcome resolution, prediction) and as explicit (answer present in the text) or implicit (answer must be inferred). The authors show that existing QA models perform unevenly across these fine-grained types, which makes the labels useful for diagnosing reading skills. They also use the data for question generation in the education domain and report that a generator trained on it asks higher-quality and more diverse questions.

How Mova uses it

• world/src/ftqa.rs: loader for the FairytaleQA CSV (copy from Hugging Face, Apache-2.0 per its card); it reconstructs full stories from per-question sections and splits sentences. Used only for evaluation, never for training.
• world read-qa: answers questions by picking the sentence with the highest weighted overlap of content lemmas, then extracting a constituent by question type (who → agent, where → place, why → cause, what did X do → clause).
• world read-events: an event graph (verb with participants; next/previous, cause, shared participant links) with a transparent traversal policy per question type, learned on the validation split and measured on test.
• world read-feel: "feeling" questions answered from explicit emotion words near the event or by inference through first-level links (e.g. death → loss → sadness), later extended with ATOMIC-2020 reactions.
• The seven question types and the explicit/implicit split are adopted verbatim as the evaluation taxonomy.

Effectiveness in Mova

FairytaleQA is the benchmark rather than a method, so the numbers below are Mova results measured on it (ROUGE-L):

• Sentence + constituent by question type (read-qa, 23 stories, 1007 questions): 0.181 vs 0.192 for returning the whole retrieved sentence; for "where" questions 0.277 vs 0.202.
• Event graph (read-events): test 0.199 — first result above the whole-sentence baseline (0.192); "what happened" 0.264 vs 0.244, "what" 0.210 vs 0.182.
• Feelings (read-feel): feel questions 0.037 → 0.141; with the event graph overall 0.212; with ATOMIC-2020 feelings overall 0.215.
• A context graph of entities (v1) gave no gain: valid 0.182 → 0.184, test 0.181 → 0.178.

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.34 · arXiv 2203.13947

@inproceedings{xu2022fantastic,
  title = {{Fantastic Questions and Where to Find Them: FairytaleQA – An Authentic Dataset for Narrative Comprehension}},
  author = {Xu, Ying and Wang, Dakuo and Yu, Mo and Ritchie, Daniel and Yao, Bingsheng and Wu, Tongshuang and Zhang, Zheng and Li, Toby Jia-Jun and Bradford, Nora and Sun, Branda and Hoang, Tran Bao and Sang, Yisi and Hou, Yufang and Ma, Xiaojuan and Yang, Diyi and Peng, Nanyun and Yu, Zhou and Warschauer, Mark},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.34},
  eprint = {2203.13947},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.acl-long.34}
}

References

37 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Young-Suk Grace Kim. 2017. Why the simple view of reading is not simplistic: Unpacking component skills of reading using a direct and indirect effect model of reading (dier). Scientific Studies of Reading, 21(4):310–333. find
2. Julie Alonzo, Deni Basaraba, Gerald Tindal, and Ronald S Carriveau. 2009. They read, but how well do they understand? an empirical look at the nuances of measuring reading comprehension. Assessment for Effective Intervention, 35(1):34–44. find
3. Jacopo Amidei, Paul Piwek, and Alistair Willis. 2018. Rethinking the agreement in human evaluation tasks. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3318–3329, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
4. Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindi-enst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint arXiv:1610.00956. — [arXiv:1610.00956](https://arxiv.org/abs/1610.00956) find
5. Faeze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Mrinmaya Sachan, and Snigdha Chaturvedi. 2021. " let your characters tell their story": A dataset for character-centric narrative understanding. arXiv preprint arXiv:2109.05438. find
6. Bidyut Das, Mukta Majumder, Santanu Phadikar, and Arif Ahmed Sekh. 2021. Automatic question generation and answer assessment: a survey. Research and Practice in Technology Enhanced Learning, 16(1):1– 15. find
7. Carolyn A Denton, Mischa Enos, Mary J York, David J Francis, Marcia A Barnes, Paulina A Kulesz, Jack M Fletcher, and Suzanne Carter. 2015. Text-processing differences in adolescent adequate and poor comprehenders reading accessible and challenging narrative and informational text. Reading Research Q… find
8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
9. Jindrich Klufa. 2015. Multiple choice question tests– advantages and disadvantages. In 3rd International Conference on Education and Modern Educational Technologies (EMET), pages 39–42. find
10. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328. find our card
11. Wojciech Kryściński, Nazneen Rajani, Divyansh Agarwal, Caiming Xiong, and Dragomir Radev. 2021. Booksum: A collection of datasets for long-form narrative summarization. arXiv preprint arXiv:2105.08209. find
12. Ghader Kurdi, Jared Leo, Bijan Parsia, Uli Sattler, and Salam Al-Emari. 2020. A systematic review of automatic question generation for educational purposes. International Journal of Artificial Intelligence in Education, 30(1):121–204. find
13. Faisal Ladhak, Bryan Li, Yaser Al-Onaizan, and Kathleen McKeown. 2020. Exploring content selection in summarization of novel chapters. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5043–5054. find
14. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv:1704.04683. find our card
15. Yash Kumar Lal, Nathanael Chambers, Raymond Mooney, and Niranjan Balasubramanian. 2021. Tellmewhy: A dataset for answering why-questions in narratives. arXiv preprint arXiv:2106.06132. find our card
16. David J Francis, Jack M Fletcher, Hugh W Catts, and J Bruce Tomblin. 2005. Dimensions affecting the assessment of reading comprehension. In Children’s reading comprehension and assessment, pages 387– 412. Routledge. find
17. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. find
18. Anna S Gellert and Carsten Elbro. 2013. Cloze tests may be quick, but are they dirty? development and preliminary validation of a cloze test of reading comprehension. Journal of Psychoeducational Assessment, 31(1):16–28. find
19. Meghan D Liebfreund. 2021. Cognitive and motivational predictors of narrative and informational text comprehension. Reading Psychology, 42(2):177– 196. find
20. Peter Goldie. 2003. One’s remembered past: Narrative thinking, emotion, and the external perspective. Philosophical Papers, 32(3):301–319. find
21. Julie S Lynch, Paul Van Den Broek, Kathleen E Kremer, Panayiota Kendeou, Mary Jane White, and Elizabeth P Lorch. 2008. The development of narrative comprehension and its relation to other early reading skills. Reading Psychology, 29(4):327–365. find
22. Martha H Head, John E Readence, and Ray R Buss. 1989. An examination of summary writing as a measure of reading comprehension. Literacy Research and Instruction, 28(4):1–11. find
23. Nancy A Martin and Rick Brownell. 2011. Expressive one-word picture vocabulary test-4 (EOWPVT-4). Academic Therapy Publications. find
24. Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and evaluation framework for deeper understanding of commonsense stories. arXiv preprint arXiv:1604.01696. find
25. Xiangyang Mou, Chenghao Yang, Mo Yu, Bingsheng Yao, Xiaoxiao Guo, Saloni Potdar, and Hui Su. 2021. Narrative question answering with cutting-edge open-domain qa techniques: A comprehensive study. arXiv preprint arXiv:2106.03826. question type distribution learning and event-centric summarization. As… find
26. Tricia A Zucker, Laura M Justice, Shayne B Piasta, and Joan N Kaderavek. 2010. Preschool teachers’ literal and inferential questions and children’s responses during whole-class shared reading. Early Childhood Research Quarterly, 25(1):65–83. find
27. Ezgi Çetinkaya Özdemir and Hayati Akyol. 2019. The development of a reading comprehension test. Universal Journal of Educational Research, 7(2):563– 570. find
28. Alison H Paris and Scott G Paris. 2003. Assessing narrative comprehension in young children. Reading Research Quarterly, 38(1):36–76. find
29. Taffy E Raphael. 1986. Teaching question answer relationships, revisited. The reading teacher, 39(6):516– 522. find
30. Paula Roberts and Helena Priest. 2006. Reliability and validity in research. Nursing standard, 20(44):41–46. find
31. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. find
32. Matthew Sims, Jong Ho Park, and David Bamman. 2019. Literary event detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3623–3634. find
33. Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard Hovy. 2017. Large-scale cloze test dataset created by teachers. arXiv preprint arXiv:1711.03225. find
34. Ying Xu, Dakuo Wang, Penelope Collins, Hyelim Lee, and Mark Warschauer. 2021. Same benefits, different communication patterns: Comparing children’s reading with a conversational agent vs. a human partner. Computers & Education, 161:104059. find
35. Bingsheng Yao, Dakuo Wang, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Mo Yu, and Ying Xu. 2022. It is ai’s turn to ask humans a question: Question-answer pair generation for children’s story books. Association for Computational Linguistics. find our card
36. Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, and Toby Jia-Jun Li. 2022. Storybuddy: A human-ai collaborative agent for parent-child interactive storytelling with flexible parent involvement. In Proceedings of the 2022 CHI Conference on Human Fac… find
37. Zhenjie Zhao, Yufang Hou, Dakuo Wang, Mo Yu, Chengzhong Liu, and Xiaojuan Ma. 2022. Educational question generation of children storybooks via find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/xu-2022-fantastic-questions-where-find-them/