PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations

What it is about

The dataset has 6,873 question-answer pairs in teacher-student conversations about Fandom wiki topics. For each student question the teacher marks literal answer spans, pragmatic spans that answer the follow-up the student is likely to ask, and a paraphrased final answer. Crowdworkers choose topics of mutual interest and teachers pass a background-knowledge test, which the authors use as incentive alignment. A metric scores the pragmatic spans only after removing the literal ones, F_prag = F1(predicted pragmatic minus literal, gold pragmatic minus literal), so copying the literal answer scores zero. A Fusion-in-Decoder baseline with DPR retrieval reaches a dev F_prag of 6.11 with five passages, against a literal F1 of 11.48, and the authors say that it recovers less than 90% of the pragmatic information. Of the pragmatic spans, 41% answer likely follow-up questions and 25% give relevant background.

Ideas

• I1. A literal-subtracted metric for Mova's question answering in world/src/qa.rs: score only the facts an answer adds beyond the literal answer. Negative control: an answer that repeats the literal fact must score zero.
• I2. A table of follow-up questions per question type, keyed on the pattern of 26. Questions: after a yes/no question about a fact, the likely follow-up asks "in what form" or "where". Build it from the school lessons, not from the test set.
• I3. The paper's question-type rule (find a WH-word, or an auxiliary, and count three words to the right) is a deterministic classifier. Implement it in world/src/qud.rs and report the distribution per lesson set.
• I4. Use the background test as a gate for teaching: the teacher answers only about topics the model has coverage of, and the rest is marked as "I do not know".

For Mova

The chapters are 30. Pragmatics and discourse: overview, 31. Speech acts and 38. Dialogue, discourse markers and reported speech. The code is world/src/qa.rs and world/src/qud.rs. The measure is the literal-subtracted score on a held-out lesson set, with a negative control that repeats the literal answer, run twice to check determinism.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.385

@inproceedings{qi2023pragmaticqa,
  title = {{PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations}},
  author = {Qi, Peng and Du, Nina and Manning, Christopher and Huang, Jing},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.385},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.385}
}

References

36 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Vaibhav Adlakha, Shehzaad Dhuliawala, Kaheer Suleman, Harm de Vries, and Siva Reddy. 2021. TopiOCQA: Open-domain conversational question answeringwith topic switching. arXiv preprint arXiv:2110.00768. ↗
2. James F Allen and C Raymond Perrault. 1980. Analyzing intention in utterances. Artificial intelligence, 15(3):143–178. find
3. Raviteja Anantha, Svitlana Vakulenko, Zhucheng Tu, Shayne Longpre, Stephen Pulman, and Srinivas Chappidi. 2021. Open-domain question answering goes conversational via question rewriting. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 520–534, Online. Association for Computational Linguistics. find
4. Nicholas Asher and Alex Lascarides. 1998. Questions in dialogue. Linguistics and Philosophy, 23(3):237– 309. find
5. Nicholas Asher and Alex Lascarides. 2003. Logics of conversation. Cambridge University Press. find
6. Nicoletta Bersia, Barbara Di Eugenio, Leonardo Lesmo, and Pietro Torasso. 1986. The overanswering mechanism in the Fido system. In Cybernetics and Systems ’86: Proceedings of the Eighth European Meeting on Cybernetics and Systems Research, organized by the Austrian Society for Cybernetic Studies, held at the University of Vienna, pages 823–830. Springer. find
7. Brant A. Cheikes and Bonnie L. Webber. 1989. Elements of a computational model of cooperative response generation. In Speech and Natural Language: Proceedings of a Workshop Held at Philadelphia, Pennsylvania, February 21-23, 1989. find
8. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879, Vancouver, Canada. Association for Computational Linguistics. find
9. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2174–2184, Brussels, Belgium. Association for Computational Linguistics. find
10. Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018. Pragmatically informative image captioning with character-level inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 439–443, New Orleans, Louisiana. Association for Computational Linguistics. find
11. Harm de Vries, Dzmitry Bahdanau, and Christopher Manning. 2020. Towards ecologically valid research on language user interfaces. arXiv preprint arXiv:2007.14435. ↗
12. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018. Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241. ↗
13. Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. find
14. Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018. Speaker-follower models for vision-and-language navigation. Advances in Neural Information Processing Systems, 31. find
15. H. P. Grice. 1975. Logic and conversation. In Peter Cole and Jerry L. Morgan, editors, Speech Acts: Syntax and Semantics Volume 3, pages 41–58. Academic Press, New York. find
16. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 874–880, Online. Association for Computational Linguistics. find
17. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics. find
18. S Jerrold Kaplan. 1982. Cooperative responses from a portable natural language query system. Artificial Intelligence, 19(2):165–187. find
19. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. find
20. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In International Conference on Learning Representations. find
21. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466. find
22. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online. Association for Computational Linguistics. find
23. Alexander Miller, Will Feng, Dhruv Batra, Antoine Bordes, Adam Fisch, Jiasen Lu, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 79–84, Copenhagen, Denmark. Association for Computational Linguistics. find
24. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783– 5797, Online. Association for Computational Linguistics. find
25. Will Monroe, Robert X.D. Hawkins, Noah D. Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics, 5:325–338. find
26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. find
27. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266. find
28. Pedro Rodriguez, Paul Crook, Seungwhan Moon, and Zhiguang Wang. 2020. Information seeking in the spirit of learning: A dataset for conversational curiosity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8153–8172, Online. Association for Computational Linguistics. find
29. Jon Scott Stevens, Anton Benz, Sebastian Reuße, and Ralf Klabunde. 2016. Pragmatic question answering: A game-theoretic approach. Data & Knowledge Engineering, 106:52–69. find
30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
31. Nelson F Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating verifiability in generative search engines. arXiv preprint arXiv:2304.09848. ↗
32. Wolfgang Wahlster, Heinz Marburger, Anthony Jame-son, and Stephan Busemann. 1983. Over-answering yes-no questions: Extended responses in a nl interface to a vision system. In Proceedings of the Eighth international joint conference on Artificial intelligence-Volume 2, pages 643–646. find
33. Daniel Marcu. 1998. A surface-based approach to identifying discourse markers and elementary textual units in unrestricted texts. In Discourse Relations and Discourse Markers. find
34. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find
35. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics. find
36. Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34. Genre Communities Anime Books Comics Games Lifestyle Movies Music TV Examples C Soul Eater, One Piece, Studio Ghibli H. P. Lovecraft, Wizard of Oz, The Maze Runner Sonic the Hedgehog, Batman, Peanuts Comics Halo, Fallout, The Legend of Zelda Olympics, The Formula 1, LEGO Pixar, Harry Potter, The Matrix Lady Gaga, ‘Cats’ Musical, Taylor Swift Doom Patrol, Game of Thrones, Doctor Who find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/qi-2023-pragmaticqa-dataset-pragmatic-question-answering/