NOPE: A Corpus of Naturally-Occurring Presuppositions in English

What it is about

NOPE collects about 1,280 corpus passages, each with a presupposition trigger of one of ten types: change of state, aspectual verbs (begin, stop), clause-embedding verbs (know, realize, say), clefts, comparatives, embedded questions, implicatives (manage to, fail to), numeric determiners (both), re- verbs, and temporal adverbial clauses (before, after). Experts write a negated version of each trigger sentence and a sentence spelling out the presupposition; crowd workers then rate how likely that presupposition is given the passage, and the ratings are mapped to NLI labels. Projection out of negation holds for most items. It is near-categorical for the hard triggers (clefts, both, temporal adverbs: about 95% stay entailed), moderate for change of state, re- verbs and aspectual verbs (about 73–79%), and low for clause-embedding verbs (31%, since many are not factive). There is also gradience and context-sensitivity. NLI transformers handle the typical cases but miss the exceptions where context cancels the presupposition; a minimal-pair adversarial set checks for lexical-overlap shortcuts.

Ideas

• I1. Extend prag/src/infer.rs with the trigger types Mova lacks. re- verbs («rebuilt» → it was built before; check the lemma without the prefix exists as a verb). Implicatives («managed to X» → X happened and was hard; «failed to X» → not X, and X was tried). Temporal clauses («V1 before/after S2» → S2 happened; advcl with mark before/after). Embedded questions («knows where X is» → X is somewhere). Each needs a tree pattern, an explanation and a negative control.
• I2. Give every trigger a strength class from these projection rates: hard (cleft, both, temporal clause), firm (change of state, aspectual, re-), soft (clause-embedding, implicatives). Print it in the explanation, and let the student in world/src/knowledge.rs count an accommodated hard presupposition as a fact but a soft one as a weak hint.
• I3. Separate factive from non-factive clause embedders before projecting anything. say and think must never produce «the complement is true». Add a negative-control test to infer that «X said that p» and «X didn't think that p» yield no presupposition of p.
• I4. Copy the adversarial minimal-pair idea for Mova's own tests. For each presupposition the inference module produces, write a near-miss («There are three sides of town» for «both sides of town») that it must not produce. This guards against string matching.

For Mova

33. Implicature and presupposition §6.1 (triggers), §6.2 (what Mova computes), §7.1 (projection); prag/src/infer.rs, world/src/knowledge.rs; measure: unit tests per trigger and environment, plus the count of inferences produced on the tale silver before and after (manually checked sample).

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.conll-1.28 · arXiv 2109.06987

@inproceedings{parrish2021nope,
  title = {{NOPE: A Corpus of Naturally-Occurring Presuppositions in English}},
  author = {Parrish, Alicia and Schuster, Sebastian and Warstadt, Alex and Agha, Omar and Lee, Soo-Hwan and Zhao, Zhuoye and Bowman, Samuel R. and Linzen, Tal},
  booktitle = {Proceedings of the 25th Conference on Computational Natural Language Learning},
  year = {2021},
  doi = {10.18653/v1/2021.conll-1.28},
  eprint = {2109.06987},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.conll-1.28}
}

References

66 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Márta Abrusán. 2011. Predicting the presuppositions of soft triggers. Linguistics and philosophy, 34(6):491–535. find
2. Dorit Abusch. 2002. Lexical alternatives as a source of pragmatic presuppositions. In Semantics and linguistic theory, volume 12, pages 1–19. find
3. Dorit Abusch. 2010. Presupposition triggering from alternatives. Journal of Semantics, 27(1):37–80. find
4. Dorit Abusch and Mats Rooth. 2004. Empty-domain effects for presuppositional and nonpresuppositional determiners. Context dependency in the analysis of linguistic meaning. find
5. David I. Beaver. 1997. Presupposition. In Handbook of logic and language, pages 939–1008. Elsevier. find
6. David I. Beaver. 2001. Presupposition and assertion in dynamic semantics, volume 29. CSLI publications Stanford. find
7. David I. Beaver and Cleo Condoravdi. 2003. A uniform analysis of ‘before’ and ‘after’. In Semantics and Linguistic Theory, volume 13, pages 37–54. find
8. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and D. Christopher Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 632– –642. find
9. Emmanuel Chemla. 2009. Similarity: towards a uni-fed account of scalar implicatures, free choice permission and presupposition projection. Ms. LSCP & MIT. Also similar in content to Chapter 5 of my dissertation. find
10. Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020. Uncertain natural language inference. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8772–8779, Online. Association for Computational Linguistics. find
11. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics. find
12. Chris Cummins, Patrícia Amaral, and Napoleon Katsos. 2012. Experimental investigations of the typology of presupposition triggers. HUMANA. MENTE Journal of Philosophical Studies, 5(23):1–15. find
13. Chris Cummins, Patricia Amaral, and Napoleon Katsos. 2013. Backgrounding and accomodation of presuppositions: An experimental approach. In Proceedings of Sinn und Bedeutung, volume 17, pages 201– 218. find
14. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In J. Quiñonero-Candela, I. Dagan, B. Magnini, and F. D’Alché-Buc, editors, Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classifcation, and Recognising Tectual Entailment, pages 177–190. Springer. find
15. Mark Davies. 2008. The corpus of contemporary american english: 450 million words, 1990-present. find
16. Marie-Catherine de Marneffe, Mandy Simons, and Judith Tonhauser. 2019. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107–124. find our card
17. Judith Degen and Judith Tonhauser. 2021. Prior beliefs modulate projection. Open Mind, pages 1–12. find
18. Nanjiang Jiang and Marie-Catherine de Marneffe. 2019. Evaluating BERT for natural language inference: A case study on the CommitmentBank. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6086– 6091, Hong Kong, China. Association for Computational Linguistics. find
19. Lauri Karttunen. 1971. Implicative verbs. Language, pages 340–358. find
20. Judy Delin. 1992. Properties of it-cleft presupposition. Journal of semantics, 9(4):289–306. find
21. Judy Delin. 1995. Presupposition and shared knowledge in it-clefts. Language and Cognitive Processes, 10(2):97–120. find
22. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. find
23. Lauri Karttunen. 1973. Presuppositions of compound sentences. Linguistic inquiry, 4(2):169–193. find
24. Lauri Karttunen. 1974. Presupposition and linguistic context. find
25. Lauri Karttunen. 1977. Syntax and semantics of questions. Linguistics and philosophy, 1(1):3–44. find
26. Lauri Karttunen and Stanley Peters. 1979. Conventional implicature. In Presupposition, pages 1–56. Brill. find
27. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. 2020. Evaluating models’ local decision boundaries via contrast sets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1307–1323. find our card
28. Jeroen Groenendijk and Martin Stokhof. 1991. Dynamic predicate logic. Linguistics and philosophy, pages 39–100. find
29. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. DeBERTa: Decoding-enhanced BERT with disentangled attention. arXiv preprint arXiv:2006.03654. ↗
30. Irene Heim. 1983. On the projection problem for presuppositions. Formal semantics–the essential readings, pages 249–260. find
31. Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, and Deepak Ramachandran. 2021. Which linguist invented the lightbulb? presupposition verifcation for question-answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3932–3945. Association for Computational Linguistics. find
32. Shalom Lappin and Tanya Reinhart. 1988. Presuppositional effects of strong determiners: a processing account. Linguistics, 26(6):1021–1038. find
33. Beth Levin. 1993. English verb classes and alternations: A preliminary investigation. University of Chicago press. find
34. Stephen Levinson. 1983. Pragmatics. Cambridge University Press. find
35. Irene Heim. 1992. Presupposition projection and the semantics of attitude verbs. Journal of semantics, 9(3):183–221. find
36. Orvokki Tellervo Heinamaki. 1974. Semantics of English temporal connectives. The University of Texas at Austin. find
37. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692. ↗
38. Matthew Honnibal, Ines Montani, Sofe Van Lan-deghem, and Adriane Boyd. 2020. spaCy: Industrial-strength natural language processing in python. find
39. Taylor Mahler. 2020. The social component of the projection behavior of clausal complement contents. In Proceedings of the Linguistic Society of America, volume 5, pages 777–791. find
40. Alex Warstadt, Suvrat Bhooshan, and Paloma Jeretic, Adina Williams. 2020. Are natural language inference models IMPPRESsive? Learning IMPlicature and PRESupposition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8690–8705, Online. Association for Computational Linguistics. find our card
41. Alec Marantz. 2009. Roots, re-, and affected agents: can roots pull the agent under little v. In workshop Roots: Word Formation from the Perspective of “Core Lexical Elements”, Stuttgart. find
42. Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computational Linguistics. and VERifcation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819. find our card
43. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). find
44. Judith Tonhauser, David I. Beaver, and Judith Degen. 2018. How Projective is Projective Content? Gradience in Projectivity and At-issueness. Journal of Semantics, 35(3):495–542. find
45. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885–4901. find
46. Christopher Potts. 2015. Presupposition and implicature. The handbook of contemporary semantic theory, 2:168–202. find
47. Judith Tonhauser and Judith Degen. 2020. Which predicates are factive? an empirical investigation. find
48. Amos Tversky and Daniel Kahneman. 1981. The framing of decisions and the psychology of choice. Science, 211(4481):453–458. find
49. Wataru Uegaki. 2019. The semantics of question-embedding predicates. Language and Linguistics Compass, 13(1):e12308. find
50. Ellen F. Prince. 1986. On the syntactic marking of presupposed open propositions. In Proceedings of the 22nd Annual Meeting of the Chicago Linguistic Society. find
51. Shiva Upadhye, Leon Bergen, and Andrew Kehler. 2020. Predicting reference: What do language models learn about discourse models? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 977–982, Online. Association for Computational Linguistics. find
52. Jacopo Romoli. 2015. The presuppositions of soft triggers are obligatory scalar implicatures. Journal of semantics, 32(2):173–219. find
53. Rob A. van der Sandt. 1992. Presupposition projection as anaphora resolution. Journal of Semantics, 9(4):333–377. find
54. Alexis Ross and Ellie Pavlick. 2019. How well do nli models capture verb veridicality? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2230–2240. find our card
55. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. find
56. Philippe Schlenker. 2021. Triggering presuppositions. Glossa: a journal of general linguistics, 6(1). find
57. Sebastian Schuster, Yuxing Chen, and Judith Degen. 2020. Harnessing the linguistic signal to predict scalar inferences. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5387–5403, Online. Association for Computational Linguistics. find our card
58. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics. find
59. Mandy Simons. 2001. On the conversational basis of some presuppositions. In Semantics and Linguistic Theory, volume 11, pages 431–448. find
60. Stephen Wechsler. 1989. Accomplishments and the prefx rei-. In North East Linguistics Society, volume 19, page 29. find
61. Mandy Simons, Judith Tonhauser, David I. Beaver, and Craige Roberts. 2010. What projects and why. In Semantics and linguistic theory, volume 20, pages 309–327. find
62. Scott Soames. 1982. How presuppositions are inherited: A solution to the projection problem. Linguistic inquiry, 13(3):483–545. find
63. Aaron Steven White, Pushpendre Rastogi, Kevin Duh, and Benjamin Van Durme. 2017. Inference is everything: Recasting semantic resources into a uni-fed evaluation framework. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 996–1005, Taipei, Taiwan. Asian Federation of Natural Language Processing. find
64. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for Fact Extraction Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4717–4724, Brussels, Belgium. Association for Computational Linguistics. find
65. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. find
66. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/parrish-2021-nope-corpus-naturally-occurring-presuppositions/