It is AI's Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books

What it is about

The paper generates question–answer pairs for children's storybooks the way a teacher would ask them, using FairytaleQA (278 classic fairy tales, 10,580 QA pairs written by education experts and labelled with seven narrative elements from Paris & Paris: character, setting, feeling, action, causal relationship, outcome resolution, prediction; the experts almost never ask yes/no questions). The pipeline has three steps: heuristic answer extraction (named entities and noun chunks for character, setting and feeling; PropBank SRL subject–verb–object spans around action verbs for action, causal, outcome and prediction), BART question generation fine-tuned on FairytaleQA (better than with NarrativeQA added), and a DistilBERT ranker trained to tell expert pairs from generated ones, keeping the top N per section. Against a two-step QG/QA baseline and the PAQ generator it scores higher on ROUGE-L-based MAP@N (N = 1, 3, 5, 10) and in human ratings, and it drives a prototype story-telling app.

Ideas

• I1. Generate course questions from Mova's own frames by narrative element, with the paper's split of answer sources: character → agents of event frames; setting → place and when:; feeling → express/evaluate frames and feeling adjectives; action → event frames; causal → cause links; outcome → the last frames of the tale; prediction → cond/want frames. Deterministic, no generator model, and each question carries its element label. Use them to measure the student per element and to see which elements the lessons never test.
• I2. Compare the element distribution of Mova's tale courses with the experts' distribution in FairytaleQA (counts only, no text carried over), and add questions where Mova's courses are thin (feelings and predictions are the usual gaps of template generators). This fits the rule that books feed the courses with analysis after each tale.
• I3. Follow the experts' choice and keep yes/no questions rare in the tale courses: they are guessable at 50% and test less. Report the share of yes/no questions per course next to the score.

For Mova

36. Discourse relations and coherence §8.2 (narrative structure), 30-pragmatics-overview.md §5. Code: world/src/lessons.rs, world/src/school.rs, prag/src/frames.rs (frames as answer sources); docs/school-making.md. Measures: student accuracy per narrative element, element coverage per course, share of yes/no questions. The card xu-2022-fantastic-questions-where-find-them in papers/ describes FairytaleQA itself.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.54

@inproceedings{yao2022it,
  title = {{It is AI's Turn to Ask Humans a Question: Question-Answer Pair Generation for Children's Story Books}},
  author = {Yao, Bingsheng and Wang, Dakuo and Wu, Tongshuang and Zhang, Zheng and Li, Toby Jia-Jun and Yu, Mo and Xu, Ying},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.54},
  url = {https://doi.org/10.18653/v1/2022.acl-long.54}
}

References

38 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 889–898. find
2. Hao Cheng, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2020. Probabilistic assumptions matter: Improved models for distantly-supervised document-level question answering. arXiv preprint arXiv:2005.01898. find
3. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197. find
4. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. arXiv preprint arXiv:1705.00106. find
5. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. find
6. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. find
7. Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. 2021. Paq: 65 million probably-asked questions and what you can do with them. arXiv preprint arXiv:2102.07033. find
8. Yu Li, Baolin Peng, Yelong Shen, Yi Mao, Lars Liden, Zhou Yu, and Jianfeng Gao. 2021. Knowledge-grounded dialogue generation with a unified knowledge representation. arXiv preprint arXiv:2112.07924. find
9. David Lindberg, Fred Popowich, John Nesbit, and Phil Winne. 2013. Generating natural language questions to support learning on-line. In Proceedings of the 14th European Workshop on Natural Language Generation, pages 105–114. find
10. Michael Heilman and Noah A Smith. 2009. Question generation via overgenerating transformations and ranking. Technical report, Carnegie-Mellon Univ Pittsburgh pa language technologies insT. find
11. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301. find
12. Jack Mostow and Wei Chen. 2009. Generating instruction automatically for the reading strategy of self-questioning. In AIED, pages 465–472. find
13. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551. find
14. Xiangyang Mou, Chenghao Yang, Mo Yu, Bingsheng Yao, Xiaoxiao Guo, Saloni Potdar, and Hui Su. 2021. Narrative question answering with cutting-edge open-domain qa techniques: A comprehensive study. Transactions of the Association for Computational Linguistics, 9:1032–1046. find
15. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906. find
16. Preksha Nema and Mitesh M Khapra. 2018. Towards a better metric for evaluating question generation systems. arXiv preprint arXiv:1808.10192. find
17. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328. find our card
18. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. find
19. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistic… find
20. Igor Labutov, Sumit Basu, and Lucy Vanderwende. 2015. Deep questions without deep understanding. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. M… find
21. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational linguistics, 31(1):71– 106. find our card
22. Alison H Paris and Scott G Paris. 2003. Assessing narrative comprehension in young children. Reading Research Quarterly, 38(1):36–76. find
23. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. find
24. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. find
25. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108. human-ai collaborative agent for parent-child interactive storytelling with flexible parent involvement. In Proceedings… find
26. Thomas Scialom, Benjamin Piwowarski, and Jacopo Staiano. 2019. Self-attention architectures for answer-agnostic neural question generation. In Proceedings of the 57th annual meeting of the Association for Computational Linguistics, pages 6027– 6032. find
27. Zhenjie Zhao, Yufang Hou, Dakuo Wang, Mo Yu, Chengzhong Liu, and Xiaojuan Ma. 2022. Educational question generation of children storybooks via question type distribution learning and event-centric summarization. Association for Computational Linguistics. find
28. Siamak Shakeri, Cicero Nogueira dos Santos, Henry Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. 2020. End-to-end synthetic data generation for domain adaptation of question answering systems. arXiv preprint arXiv:2010.06028. find
29. Qingyu Zhou, Nan Yang, Furu Wei, Chuanqi Tan, Hangbo Bao, and Ming Zhou. 2017. Neural question generation from text: A preliminary study. In National CCF Conference on Natural Language Processing and Chinese Computing, pages 662–671. Springer. find
30. Lynn Snyder, Donna Caccamise, and Barbara Wise. 2005. The assessment of reading comprehension: Considerations and cautions. Topics in Language Disorders, 25(1):33–50. find
31. Duyu Tang, Nan Duan, Tao Qin, Zhao Yan, and Ming Zhou. 2017. Question answering and question generation as dual tasks. arXiv preprint arXiv:1706.02027. — [arXiv:1706.02027](https://arxiv.org/abs/1706.02027) find
32. Tong Wang, Xingdi Yuan, and Adam Trischler. 2017. A joint model for question answering and question generation. arXiv preprint arXiv:1706.01450. — [arXiv:1706.01450](https://arxiv.org/abs/1706.01450) find
33. Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2019. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. arXiv preprint arXiv:1912.09637. find
34. Ying Xu, Dakuo Wang, Penelope Collins, Hyelim Lee, and Mark Warschauer. 2021. Same benefits, different communication patterns: Comparing children’s reading with a conversational agent vs. a human partner. Computers & Education, 161:104059. find
35. Ying Xu, Dakuo Wang, Mo Yu, Daniel Ritchie, Bingsheng Yao, Tongshuang Wu, Zheng Zhang, Toby Jia-Jun Li, Nora Bradford, Branda Sun, Tran Bao Hoang, Yisi Sang, Yufang Hou, Xiaojuan Ma, Diyi Yang, Nanyun Peng, Zhou Yu, and Mark Warschauer. 2022. Fantastic questions and where to find them: FairytaleQA –… find our card
36. Xuchen Yao, Gosse Bouma, and Yi Zhang. 2012. Semantics-based question generation and implementation. Dialogue & Discourse, 3(2):11–42. find
37. Xuchen Yao and Yi Zhang. 2010. Question generation with minimal recursion semantics. In Proceedings of QG2010: The Third Workshop on Question Generation, pages 68–75. Citeseer. find
38. Zheng Zhang, Ying Xu, Yanhao Wang, Bingsheng Yao, Daniel Ritchie, Tongshuang Wu, Mo Yu, Dakuo Wang, and Toby Jia-Jun Li. 2022. Storybuddy: A Narrative Element Definition Example Character Ask test takers to identify the character of the story or describe characteristics of characters Q: Who does Cas… find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/yao-2022-it-is-ai-s-turn/