EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference

What it is about

EQUATE is a benchmark of five test sets for quantitative reasoning in entailment: RTE-Quant (166 pairs from RTE2–4), NewsNLI (968 crowdsourced pairs from news summaries), RedditNLI (250 expert-labelled headline pairs), a synthetic stress test (7,500 pairs) and AwpNLI (722 algebra word problems turned into pairs). Nine published NLI models are tested on it. On average they gain little over the majority-class baseline; the neural models do best on verbal aspects (RTE-Quant, NewsNLI), not on numbers. The authors' symbolic baseline, Q-REAS, extracts quantity mentions, keeps the unit-compatible pairs, and checks by an integer linear program whether a hypothesis quantity can be built from premise quantities. Against the best NLI model it gains 24.2 points on numerical tests and loses 8.1 points on verbal ones. On a perturbed set where one number is changed, GPT falls from 72.04% to 51.18%, while Q-REAS scores 98.1%.

Ideas

• I1. A deterministic quantity parser over UD trees: value, unit, approximator ("about", "roughly"), governing verb and direction of change, read from nummod, amod and nmod edges of the output of en/src/parse.rs. Start with the paper's approximator list as a BTreeMap table.
• I2. Unit compatibility by the paper's three rules (string match, lexical relation from the dictionary in en/src/dict.rs, and a person-noun rule). Log which rule fired, so a wrong comparison names its cause.
• I3. Number-perturbation control: change one number in each hypothesis (79 to 80) and require the verdict to flip. A verdict that does not flip is a lexical shortcut.
• I4. Report each set against the majority baseline separately, with the difference column, not only an average.

For Mova

Chapters: 07. Punctuation, capital letters, abbreviations and numbers and 12. Pronouns, determiners and numerals (numerals, approximators). Code: en/src/tok.rs for number normalisation, prag/src/infer.rs for the entailment decision, world/src/qa.rs for school questions with numbers. Measure: accuracy per set against the majority baseline, run twice, and the flip rate under perturbation.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/k19-1033 · arXiv 1901.03735

@inproceedings{ravichander2019equate,
  title = {{EQUATE: A Benchmark Evaluation Framework for Quantitative Reasoning in Natural Language Inference}},
  author = {Ravichander, Abhilasha and Naik, Aakanksha and Rose, Carolyn and Hovy, Eduard},
  booktitle = {Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)},
  year = {2019},
  doi = {10.18653/v1/k19-1033},
  eprint = {1901.03735},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/k19-1033}
}

References

41 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

2. Peter Clark. 2018. What knowledge is needed to solve the rte5 textual entailment challenge? arXiv preprint arXiv:1806.03561. ↗
3. Gabor Angeli and Christopher D Manning. 2014. Na-turalli: Natural logic inference for common sense reasoning. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 534–545. find
4. Cleo Condoravdi, Dick Crouch, Valeria De Paiva, Reinhard Stolle, and Daniel G Bobrow. 2003. Entailment, intensionality and text understanding. In Proceedings of the HLT-NAACL 2003 workshop on Text meaning-Volume 9, pages 38–45. Association for Computational Linguistics. find
5. Jorge Balazs, Edison Marrese-Taylor, Pablo Loyola, and Yutaka Matsuo. 2017. Refining raw sentence representations for textual entailment recognition via attention. In Proceedings of the 2nd Workshop on Evaluating Vector Space Representations for NLP, pages 51–55, Copenhagen, Denmark. Association for Computational Linguistics. find
6. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics. find
7. Jon Barwise and Robin Cooper. 1981. Generalized quantifiers and natural language. In Philosophy, Language, and Artificial Intelligence, pages 241– 301. Springer. find
8. Robin Cooper, Dick Crouch, Jan Van Eijck, Chris Fox, Johan Van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, et al. 1996. Using the framework. Technical report. find
9. Luisa Bentivogli, Elena Cabrio, Ido Dagan, Danilo Giampiccolo, Medea Lo Leggio, and Bernardo Magnini. 2010. Building textual entailment specialized data sets: a methodology for isolating linguistic phenomena relevant to inference. In Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC’10), Valletta, Malta. European Languages Resources Association (EL-RA). find
10. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. The fourth pascal recognizing textual entailment challenge. Journal of Natural Language Engineering. find
11. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment, pages 177– 190. Springer. find
12. Johan Bos and Katja Markert. 2005. Recognising textual entailment with logical inference. In Proceedings of the conference on Human Language Technology and Empirical Methods in Natural Language Processing, pages 628–635. Association for Computational Linguistics. find our card
13. Marie-Catherine De Marneffe, Anna N. Rafferty, and Christopher D. Manning. 2008. Finding contradictions in text. In Proceedings of ACL-08: HLT, pages 1039–1047, Columbus, Ohio. Association for Computational Linguistics. find
14. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computational Linguistics. find
15. Stanislas Dehaene. 2011. The number sense: How the mind creates mathematics. OUP USA. find
16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
17. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017a. Enhanced lstm for natural language inference. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1657–1668, Vancouver, Canada. Association for Computational Linguistics. find
18. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. CoRR, abs/1903.00161. ↗ our card
19. Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017b. Recurrent neural network-based sentence encoder with gated attention for natural language inference. In Proceedings of the 2nd Workshop on Evaluating Vector Space Ruth B Ekstrom, Diran Dermen, and Harry Horace Harman. 1976. Manual for kit of factor-referenced cognitive tests, volume 102. Educational Testing Service Princeton, NJ. find
20. Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597. find
21. Michael C Frank, Daniel L Everett, Evelina Fedorenko, and Edward Gibson. 2008. Number as a cognitive technology: Evidence from pirahã language and cognition. Cognition, 108(3):819–824. Yaroslav Fyodorov. A natural logic inference system. Citeseer. find
22. Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014a. Learning to automatically solve algebra word problems. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 271–281. find
23. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1–9. Association for Computational Linguistics. find
24. Nate Kushman, Luke S. Zettlemoyer, Regina Barzilay, and Yoav Artzi. 2014b. Learning to automatically solve algebra word problems. In ACL. find
25. Oren Glickman, Ido Dagan, and Moshe Koppel. 2005. Web based probabilistic textual entailment. find
26. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 650–655, Melbourne, Australia. Association for Computational Linguistics. find our card
27. Stephen C Levinson. 2001. Pragmatics. In International Encyclopedia of Social and Behavioral Sciences: Vol. 17, pages 11948–11954. Pergamon. find
28. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. arXiv preprint arXiv:1803.02324. ↗
29. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017b. Program induction by rationale generation: Learning to solve and explain algebraic word problems. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 158–167, Vancouver, Canada. Association for Computational Linguistics. find
30. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017a. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146. ↗
31. Aria Haghighi, Andrew Ng, and Christopher Manning. 2005. Robust textual inference via graph matching. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing. find
32. Bill MacCartney. 2009. Natural language inference. Stanford University. find
33. Sanda Harabagiu and Andrew Hickl. 2006. Methods for using textual entailment in open-domain question answering. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics, pages 905–912. Association for Computational Linguistics. find
34. Prodromos Malakasiotis and Ion Androutsopoulos. 2007. Learning textual entailment using svms and string similarity measures. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 42–47. Association for Computational Linguistics. find
35. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693–1701. find
36. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, Roberto Zamparelli, et al. 2014. A sick cure for the evaluation of compositional distributional semantic models. find
37. Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 523–533. find our card
38. Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018). find
39. Danqing Huang, Shuming Shi, Chin-Yew Lin, and Jian Yin. 2017. Learning fine-grained expressions to solve math word problems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 805–814. find
40. Arindam Mitra and Chitta Baral. 2016. Learning to use formulas to solve simple arithmetic problems. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 2144–2153. find our card
41. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. find
42. Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ravichander-2019-equate-benchmark-evaluation-framework-quantitative/