Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering

What it is about

The paper studies unanswerable questions in Natural Questions. In a hand analysis of 100 of them, 30% are unanswerable because a presupposition of the question cannot be verified in the linked document («What is the stock symbol for Mars candy?» presupposes that there is one). Most such presuppositions come from the wh-word (19 of 30) and the definite article (10 of 30). In a user study, answers that deny the failed presupposition («…because we could not verify that any linguist invented the lightbulb») were preferred over a bare «unanswerable», a random related sentence, and an open-domain system's answer. The authors split the task into three steps. A rule-based generator over parse trees turns triggers into presuppositions: wh-word → «some X did Y», definite → existence and uniqueness, possessive → «A has B», factive verb → complement, temporal clause → event, counterfactual. It is 83% valid; errors come from compounds like «bachelor's degree» and from embedding under believe. Verification against the document uses NLI or fact-checking models, with only modest gains (macro-F1 0.60 vs 0.44 for majority). Template explanations follow.

Ideas

• I1. Add a presupposition check to the SLM student. Before choosing among options, world/src/knowledge.rs generates the question's presuppositions with the frames reader: the wh-existential («What did Isaac ask?» → Isaac asked something), definites and possessives in the question. It then looks for a remembered frame that supports each one. world knowledge diagnose should report "presupposition not in the lesson" as its own step. The Torah diagnosis already lists questions about quotes that the lesson text does not hold; this step would count them automatically instead of hiding them among guesses (step 3).
• I2. Generate question presuppositions by reusing prag/src/infer.rs (which already knows factives) and the Asked structure of prag::frames. The wh-word becomes an indefinite filler in the asked role. Guard against the paper's two error types: no possessive presupposition for compound-like genitives («bachelor's degree», «children's book»), and none under a non-factive embedder (believe, think, say).
• I3. In the dialogue layer (world/src/talk.rs), when a member is asked about something not in its memory, answer with the denied presupposition («I don't know of any X that…») and its reason, rather than a bare «I don't know». Test with a question whose presupposition is true (must not trigger) and one where it fails.

For Mova

33. Implicature and presupposition §6–7 (triggers, projection, accommodation), 26. Questions; world/src/knowledge.rs and world knowledge diagnose (Watching the SLM think: the diagnosis of its answers), prag/src/infer.rs, world/src/talk.rs; measures: the share of test questions flagged "presupposition unsupported" per course, and whether excluding them changes accuracy on the rest (it should not).

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-long.304 · arXiv 2101.00391

@inproceedings{kim2021which,
  title = {{Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering}},
  author = {Kim, Najoung and Pavlick, Ellie and Ayan, Burcu Karagol and Ramachandran, Deepak},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-long.304},
  eprint = {2101.00391},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.acl-long.304}
}

References

36 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Márta Abrusán. 2011. Presuppositional and negative islands: a semantic account. Natural Language Semantics, 19(3):257–321. find
2. Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 268–284, Online. Association for Computational Linguistics. find
3. Akari Asai and Eunsol Choi. 2020. Challenges in information seeking QA: Unanswerable questions and paragraph retrieval. arXiv:2010.11915. ↗
4. Seohyun Back, Sai Chetan Chinthakindi, Akhil Kedia, Haejun Lee, and Jaegul Choo. 2020. NeurQuRI: Neural question requirement inspector for answerability prediction in machine reading comprehension. In International Conference on Learning Representations. find
5. David Beaver. 1997. Presupposition. In Handbook of Logic and Language, pages 939–1008. Elsevier. find
6. Andre Cianflone, Yulan Feng, Jad Kabbara, and Jackie Chi Kit Cheung. 2018. Let’s do it “again”: A first computational approach to detecting adverbial presupposition triggers. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2747– 2755, Melbourne, Australia. Association for Computational Linguistics. find our card
7. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 845–855, Melbourne, Australia. Association for Computational Linguistics. find
8. David Clausen and Christopher D. Manning. 2009. Presupposed content and entailments in natural language inference. In Proceedings of the 2009 Workshop on Applied Textual Inference (TextInfer), pages 70–73, Suntec, Singapore. Association for Computational Linguistics. find
9. Elizabeth Coppock and David Beaver. 2012. Weak uniqueness: The only difference between definites and indefinites. In Semantics and Linguistic Theory, volume 22, pages 527–544. find
10. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558–3567, Florence, Italy. Association for Computational Linguistics. find
11. Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did Aristotle use a laptop? A question answering benchmark with implicit reasoning strategies. arXiv:2101.02235. ↗
12. Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, AKBC ’13, page 25–30, New York, NY, USA. Association for Computing Machinery. find
13. Minghao Hu, Furu Wei, Yuxing Peng, Zhen Huang, Nan Yang, and Dongsheng Li. 2019. Read+ verify: Machine reading comprehension with unanswerable questions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6529– 6537. find
14. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. find our card
15. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769– 6781, Online. Association for Computational Linguistics. find
16. Lauri Karttunen. 2016. Presupposition: What went wrong? In Semantics and Linguistic Theory, volume 26, pages 705–731. find
17. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452–466. find
18. Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2020. QED: A framework and dataset for explanations in question answering. arXiv:2009.06354. ↗
19. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A Lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations. find
20. Stephen C. Levinson. 1983. Pragmatics, Cambridge Textbooks in Linguistics, pages 181–184, 194. Cambridge University Press. find
21. Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020. Fine-grained fact verification with kernel graph attention network. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7342–7351, Online. Association for Computational Linguistics. find
22. Galina Tremper and Anette Frank. 2011. Extending fine-grained semantic relation classification to presupposition relations between verbs. Bochumer Linguistische Arbeitsberichte. find
23. Randall Munroe. 2020. Learning new things from Google. https://twitter.com/xkcd/status/ 1333529967079120896. Accessed: 2021-02-01. ↗
24. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. find
25. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 433–440, Sydney, Australia. Association for Computational Linguistics. find our card
26. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics. find
27. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784– 789, Melbourne, Australia. Association for Computational Linguistics. ↗
28. Mark Yatskar. 2019. A qualitative comparison of CoQA, SQuAD 2.0 and QuAC. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2318–2323, Minneapolis, Minnesota. Association for Computational Linguistics. find
29. Rob A. Van der Sandt. 1992. Presupposition projection as anaphora resolution. Journal of Semantics, 9(4):333–377. find
30. Philippe Schlenker. 2008. Presupposition projection: Explanatory strategies. Theoretical Linguistics, 34(3):287–316. find
31. Bernhard Schwarz and Alexandra Simonenko. 2018. Decomposing universal projection in questions. In Sinn und Bedeutung 22, volume 22, pages 361–374. find
32. Florian Schwarz. 2016. Experimental work in presupposition and presupposition projection. Annual Review of Linguistics, 2(1):273–292. find
33. Mandy Simons. 2013. Presupposing. Pragmatics of Speech Actions, pages 143–172. find
34. Peter F. Strawson. 1950. 59(235):320–344. On referring. find
35. Mind, Zhuosheng Zhang, Junjie Yang, and Hai Zhao. 2020. Retrospective reader for machine reading comprehension. arXiv:2001.09694. ↗
36. Haichao Zhu, Li Dong, Furu Wei, Wenhui Wang, Bing Qin, and Ting Liu. 2019. Learning to ask unanswerable questions for machine reading comprehension. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4238–4248, Florence, Italy. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kim-2021-which-linguist-invented-lightbulb-presupposition/