Breaking NLI Systems with Sentences that Require Simple Lexical Inferences

What it is about

SNLI-trained NLI systems look strong, but the authors show they fail on very simple lexical inferences. They take SNLI training premises and replace one word with a synonym, hypernym, co-hyponym or antonym (for example "champagne" for "wine" gives entailment, "saxophone" for "guitar" gives contradiction). The replacement words all occur in the SNLI training set and in the embeddings. The new test set has 8,193 pairs, 7,164 of them contradictions, and crowd workers confirmed the labels. Human accuracy is estimated at 94.1%. Three models trained on SNLI lose 22 to 33 points of accuracy on it (Decomposable Attention from 84.7% to 51.9%, ESIM from 87.9% to 65.6%). Accuracy depends on how often the replacement word appears in the training contradictions: 40.2% for words never seen and 98.5% for words seen 100 times or more. A WordNet baseline, which classifies each pair from the WordNet relation between the two words, is included. The authors warn that the set is biased towards contradiction and should only supplement the standard test.

Ideas

• I1. Substitution test for Mova: take school-lesson sentences, replace one word by a hypernym, synonym or antonym from en/src/dict.rs, and generate pairs whose labels follow from the rule. Add a negative control in which the word is replaced by an unrelated one, which must come out neutral.
• I2. Frequency table: report Mova's accuracy by how often each replacement word occurs in the lesson corpus, as the paper does. This shows whether the lexicon answers depend on coverage.
• I3. Label-bias check: when the answer set is small, the label distribution can flatter the score. Report it for the lexical-inference questions in world/src/qa.rs, as the paper warns.
• I4. Implement the paper's WordNet baseline exactly as described (hyponym gives entailment, hypernym gives neutral, antonym or a shared hypernym within two edges gives contradiction, other gives other) in world/ as a control for lexical-inference questions, and measure it beside Mova.

For Mova

The chapters are 19. The lexicon: vocabulary, multiword expressions and register and 17. Derivation II: suffixes for the word relations, and 27. Negation for contradiction. The code is en/src/dict.rs and world/src/knowledge.rs. Measure on the substitution set with the negative control, run twice.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-2103 · arXiv 1805.02266

@inproceedings{glockner2018breaking,
  title = {{Breaking NLI Systems with Sentences that Require Simple Lexical Inferences}},
  author = {Glockner, Max and Shwartz, Vered and Goldberg, Yoav},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-2103},
  eprint = {1805.02266},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p18-2103}
}

References

20 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and D. Christopher Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 632–642. https://doi.org/10.18653/v1/D15-1075. ↗
3. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics pages 159–174. find
4. Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). volume 2, pages 130–136. find
5. Samuel R Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 1466–1477. find
6. Yixin Nie and Mohit Bansal. 2017. Shortcut-stacked sentence encoders for multi-domain inference. arXiv preprint arXiv:1708.02312 . ↗
7. Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2249–2255. https://aclweb.org/anthology/D16-1244. ↗
8. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Diana Inkpen, and Si Wei. 2018. Neural natural language inference models enhanced with external knowledge. In The 56th Annual Meeting of the Association for Computational Linguistics (ACL). Melbourne, Australia. find
9. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vancouver, Canada, pages 1657–1668. find
10. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, Doha, Qatar, pages 1532–1543. http://www.aclweb.org/anthology/D14-1162. ↗
11. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing textual entailment: Models and applications. Synthesis Lectures on Human Language Technologies 6(4):1–220. find
12. Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018. Hypothesis Only Baselines in Natural Language Inference. In Joint Conference on Lexical and Computational Semantics (StarSem). find
13. Christiane Fellbaum. 1998. WordNet. Wiley Online Library. find
14. Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomas Kocisky, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In International Conference on Learning Representations (ICLR). find
15. Yichen Gong, Heng Luo, and Jian Zhang. 2018. Natural language inference over interaction space. In International Conference on Learning Representations (ICLR). find
16. Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Sen Wang, and Chengqi Zhang. 2018. Reinforced self-attention network: a hybrid of hard and soft attention for sequence modeling. arXiv preprint arXiv:1801.10296 . ↗
17. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). New Orleans, Louisiana. find
18. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In The 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). New Orleans, Louisiana. find
19. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Copenhagen, Denmark, pages 2021–2031. https://www.aclweb.org/anthology/D17-1215. ↗ our card
20. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics 2:67–78. find
21. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. SciTail: A textual entailment dataset from science question answering. In The Thirty-Second AAAI find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/glockner-2018-breaking-nli-systems-sentences-that/