Inquisitive Question Generation for High Level Text Comprehension

What it is about

INQUISITIVE is a set of about 19K questions that crowd workers asked while reading the first five sentences of 1,500 news articles one sentence at a time, each question tied to the span it is about and checked not to be answered already by the earlier text. The questions are far more often why and how than in SQuAD, QuAC or NewsQA. In a hand-analysed sample, 38.7% ask for causes (often triggered by an evaluative adjective or adverb, or by an event verb), 21.6% for elaboration (including missing arguments: «Who got snubbed?»), 12.6% for definitions of terms, 10% for background, 8.1% for instances of a set («Which ones?»), and 4.5% for what happens next. Readers rated about 70% of the questions as likely to be answered later in the article. A GPT-2 model fine-tuned on the data generates plausible questions, best when it sees the context and the span.

Ideas

• I1. Give the implicit-QUD step of world/src/qud.rs typed triggers taken from the paper's categories, instead of the three generic templates (what did X do / what was X like / what happened next): an evaluative adjective or adverb or an unexplained event → a why QUD on that frame; a frame with an empty core role → «who/what was V-ed?»; a plural or set noun with no instances yet → «which ones?»; a rare noun → «what is an N?». Each candidate keeps the givenness check already there. Measure the share of later sentences that attach to an accommodated QUD, on tales, against the current templates.
• I2. Use the same triggers to make reading questions for the school courses from the text itself: a why question at each evaluative or causal span whose answer frame appears later in the lesson (a cause link exists) is answerable and checkable; one with no later answer is a background question and should not be asked. Count how many such questions the frame reader can answer with world knowledge diagnose.
• I3. The cause share (39%) supports making Ask::Cause the best-served question type: check in the course statistics how many questions are why and how often the student fails at «nothing remembered matched» for them.

For Mova

36. Discourse relations and coherence §9 and §11.5 (questions under discussion, the QUD stack), 26-questions.md. Code: world/src/qud.rs (typed implicit QUDs), world/src/knowledge.rs (question making and diagnosis), prag/src/frames.rs (cause links, empty roles). Measures: QUD attachment rate on tales; answerable share of generated why-questions.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.530

@inproceedings{ko2020inquisitive,
  title = {{Inquisitive Question Generation for High Level Text Comprehension}},
  author = {Ko, Wei-Jen and Chen, Te-yuan and Huang, Yiyan and Durrett, Greg and Li, Junyi Jessy},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.530},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.530}
}

References

46 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. ↗
2. Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What do you mean exactly?: Analyzing clarification questions in CQA. In Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval, pages 345–348. find
3. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184. find
4. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the Association for Computational Linguistics, to appear. find
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186. find
6. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference for Learning Representations. find
7. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466. find
8. Xinya Du and Claire Cardie. 2017. Identifying where to focus in reading comprehension for neural question generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2067–2073. find
9. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119. find
10. Xinya Du, Junru Shao, and Claire Cardie. 2017. Learning to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1342– 1352. find
11. Junyi Jessy Li, Bridget O’Daniel, Yi Wu, Wenli Zhao, and Ani Nenkova. 2016b. Improving the annotation of sentence specificity. In Proceedings of the Tenth International Conference on Language Resources and Evaluation, pages 3921–3927. find
12. Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. ↗
13. Marcus Errico, John April, Andrew Asch, Lynnette Khalfani, Miriam A. Smith, and Xochiti R. Ybarra. 1997. The evolution of the summary news lead. Media History Monographs, 1. find
14. George Loewenstein. 1994. The psychology of curiosity: A review and reinterpretation. Psychological bulletin, 116(1):75–98. find
15. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations, pages 55–60. find
16. Nicholas FitzGerald, Julian Michael, Luheng He, and Luke Zettlemoyer. 2018. Large-scale QA-SRL parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2051–2060. find our card
17. Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
18. Donna Harman and Mark Liberman. 1993. TIPSTER Complete LDC93T3A. Linguistic Data Consortium. find
19. Andrew McKinlay and Katja Markert. 2011. Modelling entity instantiations. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 268–274. find
20. Vasileios Hatzivassiloglou and Janyce M Wiebe. 2000. Effects of adjective orientation and gradability on sentence subjectivity. In Proceedings of the 18th International Conference on Computational Linguistics. find
21. Karen Huang, Michael Yeomans, Alison Wood Brooks, Julia Minson, and Francesca Gino. 2017. It doesn’t hurt to ask: Question-asking increases liking. Journal of personality and social psychology, 113(3):430–452. find
22. Jamie J Jirout. 2011. Curiosity and the development of question generation skills. In 2011 AAAI Fall Symposium Series. find
23. Mao Nakanishi, Tetsunori Kobayashi, and Yoshihiko Hayashi. 2019. Towards answer-unaware conversational question generation. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 63–71. find
24. Paul Piwek and Kristy Elizabeth Boyer. 2012. Varieties of question generation: Introduction to this special issue. Dialogue & Discourse, 3(2):1–9. find
25. Peng Qi, Yuhao Zhang, and Christopher D. Manning. 2020. Stay hungry, stay focused: Generating informative and specific questions in information-seeking conversations. arXiv preprint arXiv:2004.14530. ↗
26. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. In OpenAI Technical Report. find
27. Philip Bachman. 2017. NewsQA: A Machine Comprehension Dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200. find
28. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784–789. find
29. Lucy Vanderwende. 2007. Answering and questioning for machine reading. In AAAI Spring Symposium: Machine Reading. find
30. Pranav Rajpurkar, Konstantin Lopyrev Jian Zhang, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. find
31. Sudha Rao and Hal Daumé III. 2018. Learning to ask good questions: Ranking clarification questions using neural expected value of perfect information. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2737–2746. find
32. Sudha Rao and Hal Daumé III. 2019. Answer-based adversarial training for generating clarification questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 143– 155. find
33. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. find
34. Vasile Rus, Brendan Wyse, Paul Piwek, Mihai Lintean, Svetlana Stoyanchev, and Cristian Moldovan. 2010. The first question generation shared task evaluation challenge. In Proceedings of the 6th International Natural Language Generation Conference. find
35. Thomas Scialom and Jacopo Staiano. 2019. Ask to learn: A study on curiosity-driven question generation. arXiv preprint arXiv:1911.03350. ↗
36. Linfeng Song, Zhiguo Wang, Wael Hamza, Yue Zhang, and Daniel Gildea. 2018. Leveraging context information for natural question generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 569–574. find
37. Sandeep Subramanian, Tong Wang, Xingdi Yuanv Saizheng Zhang, Adam Trischler, and Yoshua Bengio. 2018. Neural models for key phrase extraction and question generation. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 78–88. find
38. Lucy Vanderwende. 2008. The importance of being important: Question generation. In Proceedings of the 1st Workshop on the Question Generation Shared Task Evaluation Challenge. find
39. Siyuan Wang, Zhongyu Wei, Zhihao Fan, Yang Liu, and Xuanjing Huang. 2019. A multi-agent communication framework for question-worthy phrase extraction and question generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7168–7175. find
40. Matthijs Westera, Laia Mayol, and Hannah Rohde. 2020. TED-Q: TED talks and the questions they evoke. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 1118– 1127. find
41. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics, 3:283–297. find
42. Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. 2018. Visual curiosity: Learning to ask questions to learn visual recognition. In Conference on Robot Learning, pages 63–80. find
43. Yi Yang, Wen-Tau Yih, and Christopher Meek. 2015. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2013–2018. find
44. Xingdi Yuan, Tong Wang, Caglar Gulcehre, Alessandro Sordoni, Philip Bachman, Saizheng Zhang, Sandeep Subramanian, and Adam Trischler. 2017. Machine comprehension by text-to-text neural question generation. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 15–25. find
45. Yao Zhao, Xiaochuan Ni, Yuanyuan Ding, and Qifa Ke. 2018. Paragraph-level neural question generation with maxout pointer and gated self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3901–3910. find
46. Long Zhou, Wenpeng Hu, Jiajun Zhang, and Chengqing Zong. 2017. Neural system combination for machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 378–384. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, and Kaheer Suleman *Figure 4: Instructions for question collection.* *Figure 5: Turk interface for collecting questions.* find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ko-2020-inquisitive-question-generation-high-level/