CREPE: Open-Domain Question Answering with False Presuppositions

What it is about

CREPE has 8,400 «explain like I'm five» questions from Reddit. Annotators, guided by the most upvoted answer, mark whether the question rests on a false presupposition and, if so, write the presupposition and a correction. About 25% of real information-seeking questions contain one. A sample of 50 shows the kinds: a false predicate (30%, «how is current stored in power plants?»), false properties (22%), false causal links between facts (22%), false clauses (14%, «if water has to be 100° to become steam…»), false existentials (6%) and over-generalisations (4%). The corrective answers usually start with a lexical signal: negation in 70% of signalled cases, or actually, just, really, «you are mistaken». Baselines with retrieval and pretrained models detect the explicit cases moderately well but fail on implicit ones. The main bottleneck is retrieving evidence that contradicts the presupposition, not reading it.

Ideas

• I1. The taxonomy shows which presuppositions of a school question Mova's student can check against its memory: the content of an «if»/«since» clause in the question (cond frames), the asked frame itself for why-questions (Ask::Cause presupposes the event happened), and existentials from definites. Add these three to the presupposition check proposed for world/src/knowledge.rs (see the card on Kim et al. 2021). In world knowledge diagnose, list the questions whose presupposed frame contradicts a remembered negated frame.
• I2. Recognise corrective replies in dialogue by their opening signal. A reply that starts with negation or with actually / in fact / it's not that and repeats a frame of the previous question is act=disagree aimed at the question's presupposition. This gives the prag act labeller a concrete feature (prag/src/feats.rs: first word is actually + previous act is ask), measurable as disagree F1 in prag eval.
• I3. For the absurdity matrix and lesson writing: questions in the courses could deliberately include a few false-presupposition items, each with the correction as the right option. This tests whether the student refuses to accommodate, as a negative control for I1.

For Mova

33. Implicature and presupposition §6–7, 26. Questions; world/src/knowledge.rs, world knowledge diagnose, prag/src/feats.rs; measures: count of flagged questions per course, disagree F1 in prag eval.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.583 · arXiv 2211.17257

@inproceedings{yu2023crepe,
  title = {{CREPE: Open-Domain Question Answering with False Presuppositions}},
  author = {Yu, Xinyan and Min, Sewon and Zettlemoyer, Luke and Hajishirzi, Hannaneh},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.583},
  eprint = {2211.17257},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.acl-long.583}
}

References

45 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Akari Asai and Eunsol Choi. 2021. Challenges in information-seeking QA: Unanswerable questions and paragraph retrieval. In Proceedings of the Association for Computational Linguistics. find
2. FairScale authors. 2021. Fairscale: A general purpose modular pytorch library for high performance and large scale training. https://github.com/f acebookresearch/fairscale. ↗
3. David I. Beaver, Bart Geurts, and Kristie Denlinger. 2014. Presupposition. The Stanford Encyclopedia of Philosophy. find
4. Rahul Bhagat and Eduard Hovy. 2013. Squibs: What is a paraphrase? Computational Linguistics. find
5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Proceedings of Advances in Neural Information Processing Systems. find
6. Isabel Cachola, Eric Holgate, Daniel Preoţiuc-Pietro, and Junyi Jessy Li. 2018. Expressively vulgar: The socio-dynamics of vulgarity and its effects on sentiment analysis in social media. In Proceedings of International Conference on Computational Linguistics. In Proceedings of Empirical Methods in Natural Language Processing. find
7. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find
8. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find
9. Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In Proceedings of the International Conference on Learning Representations. find
10. Marie Duží and Martina Číhalová. 2015. Questions, answers, and presuppositions. Computación y Sistemas. find
11. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics. find
12. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The paraphrase database. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find our card
13. Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. In Proceedings of the European Chapter of the Association for Computational Linguistics. find
14. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the Association for Computational Linguistics. find our card
15. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data. find
16. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint. find
17. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find
18. Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. 2018. QuAC: Question answering in context. find
19. S. Jerrold Kaplan. 1978. Indirect responses to loaded questions. In Theoretical Issues in Natural Language Processing-2. find
20. Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist invented the lightbulb? presupposition verification for question-answering. In Proceedings of the Association for Computational Linguistics. find our card
21. Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021. Hurdles to progress in long-form question answering. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find
22. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics. find
23. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the Association for Computational Linguistics. Association for Computational Linguistics. find
24. Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021. Question and answer test-train overlap in open-domain question answering datasets. In Proceedings of the European Chapter of the Association for Computational Linguistics. biguous open-domain questions. In Proceedings of Empirical Methods in Natural Language Processing. find
25. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. 2021. Webgpt: Browser-assisted question-answering with human feedback. arXiv preprint. find
26. Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the Association for Computational Linguistics. find our card
27. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find
28. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation. find our card
29. Christopher Potts. 2015. Presupposition and implicature. The handbook of contemporary semantic theory. find
30. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matus-sière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. find
31. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research. find
32. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining approach. arXiv preprint. find
33. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of Empirical Methods in Natural Language Processing. find
34. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In Proceedings of the International Conference on Learning Representations. find
35. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering am-Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the Association for Computational Linguistics. find
36. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of Empirical Methods in Natural Language Processing. find
37. Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In Proceedings of Empirical Methods in Natural Language Processing. find
38. Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the Association for Computational Linguistics. find
39. John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Uribe, Liam Fedus, Luke Metz, et al. 2022. ChatGPT: Optimizing language models for dialogue. find
40. Robert Stalnaker, Milton K Munitz, and Peter Unger. 1977. Pragmatic presuppositions. In Proceedings of the Texas conference on per˜ formatives, presuppositions, and implicatures. find
41. Peter F Strawson. 1950. On referring. Mind. find
42. Ellen M. Voorhees and Dawn M. Tice. 2000. The TREC-8 question answering track. In Proceedings of the Language Resources and Evaluation Conference. find
43. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics. find
44. Fangyuan Xu, Junyi Jessy Li, and Eunsol Choi. 2022. How do we answer complex questions: Discourse structure of long-form answers. In Proceedings of the Association for Computational Linguistics. find
45. Michael Zhang and Eunsol Choi. 2021. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7371– 7387, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/yu-2023-crepe-open-domain-question-answering/