Mova0.5 pre-release⌕
☰
home / docs / papers / English / Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation

Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation

What it is about

The question is whether neural NLI models learn how negation reverses lexical entailment: "dance" entails "move", but "not move" does not entail "not dance". The authors build MoNLI, 2,678 SNLI-style pairs in which one noun is replaced by a WordNet hypernym or hyponym. In 1,202 pairs the substitution is under "not" (NMoNLI), in 1,476 it is not (PMoNLI). They test four models: CBOW, BiLSTM, ESIM and BERT. Behaviourally, models trained on SNLI match on PMoNLI but fall well below chance on NMoNLI, and BERT is nearly always wrong there. Fine-tuning on NMoNLI, with the substituted words kept disjoint between training and test, lets all models solve the task, and BERT keeps its SNLI accuracy. Structurally, linear probes find the lexical relation at most positions, and interchange interventions at one BERT position reproduce the causal behaviour of a four-line monotonicity algorithm on subsets of 98, 63, 47 and 37 pairs. The authors conclude that BERT "at least partially" embeds the theory. The intervention search used about 7 million runs at layer 3.

Ideas

For Mova

The chapters are 27. Negation (downward monotone "not", "no", "hardly") and 19. The lexicon: vocabulary, multiword expressions and register (hypernyms and hyponyms). The code is prag for polarity and entailment, world/src/knowledge.rs for items, and en/src/dict.rs for any hypernym table (grep before use, since none is confirmed here). The measure: accuracy on PMoNLI and NMoNLI separately, run twice for determinism, against a negative control in which negation is removed and the labels are swapped.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.blackboxnlp-1.16

@inproceedings{geiger2020neural,
  title = {{Neural Natural Language Inference Models Partially Embed Theories of Lexical Entailment and Negation}},
  author = {Geiger, Atticus and Richardson, Kyle and Potts, Christopher},
  booktitle = {Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP},
  year = {2020},
  doi = {10.18653/v1/2020.blackboxnlp-1.16},
  url = {https://doi.org/10.18653/v1/2020.blackboxnlp-1.16}
}

References

47 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. 2018. Systematic generalization: What is required and can it be learned? In In Proceedings of the 6th International Conference on Learning Representations, Beijing. find
  2. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics, 7:49–72. find
  3. Johan van Benthem. 2008. A brief history of natural logic. In Logic, Navya-Nyaya and Applications: Homage to Bimal Matilal. find
  4. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. find
  5. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, and Hui Jiang. 2016. Enhancing and combining sequential and tree LSTM for natural language inference. CoRR, abs/1609.06038. ↗
  6. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computational Linguistics. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  8. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Database. MIT Press, Cambridge, MA. find
  9. Jerry A. Fodor and Zenon W. Pylyshyn. 1988. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1):3–71. find
  10. Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. 2019. Posing fair generalization tasks for natural language inference. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4485–4495, Hong Kong, China. Association for Computational Linguistics. find
  11. Mario Giulianelli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 240–248, Brussels, Belgium. Association for Computational Linguistics. find
  12. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking NLI systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655, Melbourne, Australia. Association for Computational Linguistics. find our card
  13. Emily Goodwin, Koustuv Sinha, and Timothy J. O’Donnell. 2020. Probing linguistic systematicity. find
  14. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. find
  15. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733–2743, Hong Kong, China. Association for Computational Linguistics. find
  16. Hai Hu, Qi Chen, and Larry Moss. 2019a. Natural language inference with monotonicity. In Proceedings of the 13th International Conference on Computational Semantics - Short Papers, pages 8– 15, Gothenburg, Sweden. Association for Computational Linguistics. find
  17. Hai Hu, Qi Chen, Kyle Richardson, Atreyee Mukherjee, Lawrence S. Moss, and Sandra Kübler. 2019b. MonaLog: A lightweight system for natural language inference based on monotonicity. ArXiv, abs/1910.08772. ↗
  18. Dieuwke Hupkes, Sanne Bouwmeester, and Raquel Fernández. 2018. Analysing the potential of seq-to-seq models for incremental interpretation in task-oriented dialogue. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 165–174, Brussels, Belgium. Association for Computational Linguistics. find
  19. Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2019. Compositionality decomposed: how do neural networks generalise? find
  20. Thomas Icard, Lawrence Moss, and William Tune. 2017. A monotonicity calculus and its completeness. In Proceedings of the 15th Meeting on the Mathematics of Language, pages 75–87, London, UK. Association for Computational Linguistics. find
  21. Thomas F. Icard. 2012. Inclusion and exclusion in natural language. Studia Logica, 100(4):705–725. find
  22. Thomas F. Icard. 2017. From programs to causal models. In Proceedings of the 21st Amsterdam Colloquium, pages 35–44. University of Amsterdam. find
  23. Thomas F. Icard and Lawrence S. Moss. 2013. Recent progress on monotonicity. Linguistic Issues in Language Technology, 9(7):1–31. find
  24. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. CoRR, abs/1707.07328. ↗ our card
  25. Brenden M. Lake and Marco Baroni. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2879–2888. PMLR. find
  26. Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2171–2179, Minneapolis, Minnesota. Association for Computational Linguistics. find
  27. Bill MacCartney. 2009. Natural Language Inference. Ph.D. thesis, Stanford University. find
  28. Bill MacCartney and Christopher D. Manning. 2008. Modeling semantic containment and exclusion in natural language inference. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 521–528, Manchester, UK. Coling 2008 Organizing Committee. find
  29. Bill MacCartney and Christopher D. Manning. 2009. An extended model of natural logic. In Proceedings of the Eight International Conference on Computational Semantics, pages 140–156, Tilburg, The Netherlands. Association for Computational Linguistics. find
  30. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  31. Yixin Nie, Yicheng Wang, and Mohit Bansal. 2019a. Analyzing compositionality-sensitivity of NLI models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6867–6874. find
  32. Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2019b. Adversarial NLI: A new benchmark for natural language understanding. find
  33. Ellie Pavlick and Chris Callison-Burch. 2016. Most “babies” are “little” and most “problems” are “huge”: Compositional entailment in adjective-nouns. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2164–2173, Berlin, Germany. Association for Computational Linguistics. find
  34. Judea Pearl. 2001. Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAI’01, page 411–420, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  35. Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499–1509, Brussels, Belgium. Association for Computational Linguistics. find
  36. Kyle Richardson, Hai Hu, Lawrence S. Moss, and Ashish Sabharwal. 2019. Probing natural language inference models through semantic fragments. find
  37. Vı́ctor Sánchez-Valencia. 1991. Studies in Natural Logic and Categorial Grammar. Ph.D. thesis, University of Amsterdam. find
  38. Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2019. olmpics – on what language model pre-training captures. find
  39. David Marr. 1982. Vision: A Computational Investigation into the Human Representation and Processing of Visual Information. Henry Holt and Co., Inc., New York, NY, USA. find
  40. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593– 4601, Florence, Italy. Association for Computational Linguistics. find
  41. Lawrence S Moss. 2009. Natural logic and semantics. In Proceedings of the 18th Amsterdam Colloquium: Revised Selected Papers, pages 71–80, Berlin. University of Amsterdam, Springer. find
  42. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Causal mediation analysis for interpreting neural nlp: The case of gender bias. find
  43. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112–1122. Association for Computational Linguistics. find
  44. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771. ↗
  45. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, and Kentaro Inui. 2020. Do neural models learn systematicity of monotonicity inference in natural language? find
  46. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019a. Can neural networks understand monotonicity reasoning? In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 31–40, Florence, Italy. Association for Computational Linguistics. find
  47. Hitomi Yanaka, Koji Mineshima, Daisuke Bekki, Kentaro Inui, Satoshi Sekine, Lasha Abzianidze, and Johan Bos. 2019b. HELP: A dataset for identifying shortcomings of neural models in monotonicity reasoning. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 250–255, Minneapolis, Minnesota. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/geiger-2020-neural-natural-language-inference-models/