How well do NLI models capture verb veridicality?

What it is about

From MultiNLI sentences with a verb + to/that complement the authors build 1,498 sentences over 137 verbs, each paired twice with its complement as hypothesis: once as is and once with the main verb negated («He knows / does not know that the answer is 5» → «The answer is 5»). Three crowd ratings on a −2…+2 scale per pair give each verb in context a «soft signature», compared with the lexical signatures of Karttunen and Nairn et al. (eight classes: +/+ know, realize, notice, discover; +/− manage, begin, dare; −/+ forget to, fail to; −/○ refuse, decline; ○/− attempt; +/○ show, confirm, help; ○/+ suspect, predict; ○/○ try, hope, say, think, want). On average verbs follow their class, but people show a strong veridicality bias: non-veridical ○/○ verbs (indicate, prefer, think) are read like +/− in context. The signature class explains little of the per-sentence variation (R² ≈ 0.11) where the verb itself explains much more (≈ 0.72). BERT fine-tuned on MNLI follows the same patterns with an even stronger bias towards treating complements as true, and is sensitive to negation and to to vs that complements.

Ideas

• I1. Use the paper's verb–signature table as the seed of the implication signatures proposed for Mova's frame factuality (see rudinger-2018-neural-models-factuality I1): our own list built from these published class memberships, stored in prag/data with the signature and the complement type (to vs that). Extend prag/src/infer.rs, which today knows factives like know, realize, regret, forget and the scale try < manage, to all eight classes with negation flipping as in the table.
• I2. Keep two readings for reported content in the student: strict (signature-based: «The hunter thought the wolf was asleep» does not make the wolf asleep) and invited (the human bias: what a narrator or trusted character says is taken as true unless contradicted). Measure on the courses which reading gives more right answers, and keep the strict one as the explanation's default label («the text only says the hunter thought so»).
• I3. Build a minimal-pair test from tale sentences the way the paper builds its pairs: for each verb-complement sentence, a negated copy, and check that the frame reader's factuality flips or stays according to the signature. This is a gate with a negative control (a ○/○ verb must not change the complement's status under negation).

For Mova

33. Implicature and presupposition §6 (presupposition triggers, factives), 23-complementation.md (to- and that-complements), 27-negation.md. Code: prag/src/infer.rs, prag/src/frames.rs, prag/data (signature list), world/src/knowledge.rs (strict vs invited reading). Measures: minimal-pair gate; course test counts under each reading.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1228

@inproceedings{ross2019how,
  title = {{How well do NLI models capture verb veridicality?}},
  author = {Ross, Alexis and Pavlick, Ellie},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1228},
  url = {https://doi.org/10.18653/v1/d19-1228}
}

References

33 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gabor Angeli and Christopher D Manning. 2014. Nat-uralli: Natural logic inference for common sense Linguistics: Human Language Technologies, pages 752–757. reasoning. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 534–545. find
2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics. find
3. Ignacio Cases, Clemens Rosenbaum, Matthew Riemer, Atticus Geiger, Tim Klinger, Alex Tamkin, Olivia Li, Sandhini Agarwal, Joshua D Greene, Dan Jurafsky, et al. 2019. Recursive routing networks: Learning to compose modules for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1. find
4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. find
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
6. Ingrid Falk and Fabienne Martin. 2017. Towards an inferential lexicon of event selecting predicates for French. In IWCS 2017 - 12th International Conference on Computational Semantics - Long papers. find
7. Michael L Geis and Arnold M Zwicky. 1971. On invited inferences. Linguistic inquiry, 2(4):561–566. find
8. Lauri Karttunen. 1971. Implicative verbs. Language, pages 340–358. find
9. Lauri Karttunen. 2012. Simple and phrasal implicatives. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pages 124–131, Montréal, Canada. Association for Computational Linguistics. find
10. Lauri Karttunen, Stanley Peters, Annie Zaenen, and Cleo Condoravdi. 2014. The chameleon-like nature of evaluative adjectives. Empirical Issues in Syntax and Semantics, 10:233–250. find
11. Bill MacCartney. 2009. Natural language inference. Ph.D. thesis, Citeseer. find
12. Bill MacCartney and Christopher D Manning. 2009. Natural language inference. Citeseer. find
13. Christopher D Manning. 2006. Local textual inference: its hard to circumscribe, but you know it when you see it–and nlp needs it. find
14. Marie-Catherine de Marneffe, Christopher D Manning, and Christopher Potts. 2012. Did it happen? the pragmatic complexity of veridicality assessment. Computational linguistics, 38(2):301–333. find
15. R Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. arXiv preprint arXiv:1902.01007. ↗ our card
16. Rowan Nairn, Cleo Condoravdi, and Lauri Karttunen. 2006. Computing relative polarity for textual inference. In Proceedings of the fifth international workshop on inference in computational semantics (icos-5). find our card
17. Ellie Pavlick and Chris Callison-Burch. 2016. Tense manages to predict implicative behavior in verbs. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2225–2229. find
18. Kyle Richardson and Jonas Kuhn. 2012. Light textual inference for semantic parsing. In Proceedings of COLING 2012: Posters, pages 1007–1018, Mumbai, India. find
19. The COLING 2012 Organizing Committee. find
20. Rachel Rudinger, Aaron Steven White, and Benjamin Van Durme. 2018. Neural models of factuality. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 731–744, New Orleans, Louisiana. Association for Computational Linguistics. find our card
21. Vı́ctor Manuel Sánchez Valencia. 1991. Studies on natural logic and categorial grammar. VM Sanchez Valencia. find
22. Paul Kiparsky and Carol Kiparsky. 1968. Fact. Linguistics Club, Indiana University. find
23. Roser Saurı́ and James Pustejovsky. 2012. Are you sure that this happened? assessing the factuality degree of events in text. Computational Linguistics, 38(2):261–299. find
24. Amnon Lotan, Asher Stern, and Ido Dagan. 2013. Truthteller: Annotating predicate truth. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Mandy Simons, David Beaver, Craige Roberts, and Judith Tonhauser. 2017. The best question: Explaining the projection behavior of factives. Discourse Processes, 54(3):187–206. find
25. Mandy Simons, Judith Tonhauser, David Beaver, and Craige Roberts. 2010. What projects and why. In Semantics and linguistic theory, volume 20, pages 309–327. find
26. Gabriel Stanovsky, Judith Eckle-Kohler, Yevgeniy Puzikov, Ido Dagan, and Iryna Gurevych. 2017. Integrating deep linguistic features in factuality prediction over unified datasets. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 352–357. find
27. Judith Tonhauser, David I Beaver, and Judith Degen. 2018. How projective is projective content? gradience in projectivity and at-issueness. find
28. Matthijs Westera and Gemma Boleda. 2019. Dont blame distributional semantics it cant do entailment. In Proceedings of the 13th International Conference on Computational Semantics. find
29. Aaron Steven White and Kyle Rawlins. 2018. The role of veridicality and factivity in clause selection. In Proceedings of the 48th Annual Meeting of the North East Linguistic Society, to appear. Amherst, MA: GLSA Publications. find
30. Aaron Steven White, Rachel Rudinger, Kyle Rawlins, and Benjamin Van Durme. 2018. Lexicosyntactic inference in neural models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4717–4724, Brussels, Belgium. Association for Computational Linguistics. find
31. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. find
32. Annie Zaenen, Lauri Karttunen, and Richard Crouch. 2005. Local textual inference: can it be defined or circumscribed? In Proceedings of the ACL workshop on empirical modeling of semantic equivalence and entailment, pages 31–36. Association for Computational Linguistics. find
33. Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379–395. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ross-2019-how-well-do-nli-models/