How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks

What it is about

The paper runs strong readers on corrupted versions of popular benchmarks: questions randomly reassigned (passage-only) or passages replaced by gibberish that keeps the candidate answers in random places (question-only). On bAbI, passage-only models exceed 50% on 14 of 20 tasks and match the full model on tasks 3, 13, 16 and 20; a question-only model reaches 91% on task 18 against 93% for the full one. On the Children's Book Test, question-only models come close to the full model for names and nouns and beat it for verbs and prepositions, and using only the last of the 20 passage sentences works as well as all of them. Who-did-What passage-only gets over 50%; CNN (with anonymised entities) and SQuAD hold up best. The recommendation is to report such partial-input baselines with every new benchmark result.

Ideas

• I1. Report Mova's bAbI results (world/src/babi.rs) and each course with the paper's three columns: full, question-only (lesson replaced), passage-only (questions shuffled within the lesson), and Δ(min) = full minus the better partial score. A task or course with small Δ(min) does not show reading. Same partial-input idea as huang-2019-cosmos-qa-machine-reading-comprehension I1, here with the shuffled-question variant, which also catches defaults like «answer with the most salient character».
• I2. A last-sentence ablation for the courses: keep only the lesson sentence with the highest overlap to the question (or only the last k sentences). If the student scores the same, the questions test sentence lookup, not integration across the lesson; use the result to choose which courses can show gains from coreference and links between frames.
• I3. Keep the negative controls in the course report permanently (not one-off), so every claimed gain is shown next to them, in line with Mova's rule that a gate which cannot show red is not a gate.

For Mova

30. Pragmatics and discourse: overview §9 (measurements). Code: world/src/babi.rs, world/src/knowledge.rs, world/src/report.rs, world/src/school.rs. Measures: full / question-only / passage-only / Δ(min) per bAbI task and per course; last-sentence ablation scores.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1546 · arXiv 1808.04926

@inproceedings{kaushik2018how,
  title = {{How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks}},
  author = {Kaushik, Divyansh and Lipton, Zachary C.},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1546},
  eprint = {1808.04926},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d18-1546}
}

References

33 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D Manning. 2014. Modeling biological processes for reading comprehension. In Empirical Methods in Natural Language Processing (EMNLP). find
2. Eric Breck, Marc Light, Gideon Mann, Ellen Riloff, Brianne Brown, and Pranav Anand. 2001. Looking under the hood: Tools for diagnosing your question answering engine. In Association for Computational Linguistics (ACL) Workshop on Open-Domain Question Answering. find
3. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Association for Computational Linguistics (ACL). find
4. Peter Clark and Oren Etzioni. 2016. My computer is an honor student but how intelligent is it? standardized tests as a measure of ai. AI Magazine, 37(1):5–12. find
5. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Association for Computational Linguistics (ACL). find
6. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In Association for Computational Linguistics (ACL). find our card
7. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In Computer Vision and Pattern Recognition (CVPR). find
8. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In North American Chapter of the Association for Computational Linguistics (NAACL): Human Language Technologies. find
9. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS). find
10. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The Goldilocks principle: Reading children’s books with explicit memory representations. In International Conference on Learning Representations (ICLR). find
11. Lynette Hirschman, Marc Light, Eric Breck, and John D Burger. 1999. Deep read: A reading comprehension system. In Association for Computational Linguistics on Computational Linguistics (ACL). find
12. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Association for Computational Linguistics (ACL). find
13. Rudolf Kadlec, Martin Schmid, Ondřej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In Association for Computational Linguistics (ACL). find
14. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In Empirical Methods in Natural Language Processing (EMNLP). find our card
15. Moontae Lee, Xiaodong He, Wen-tau Yih, Jianfeng Gao, Li Deng, and Paul Smolensky. 2016. Reasoning in vector space: An exploratory study of question answering. In International Conference on Learning Representations (ICLR). find
16. Zachary C Lipton and Jacob Steinhardt. 2018. Troubling trends in machine learning scholarship. In International Conference on Machine Learning (ICML) Machine Learning Debates Workshop. find
17. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
18. Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Richard Sutcliffe, Corina Forascu, and Caroline Sporleder. 2011. Overview of qa4mre at clef 2011: Question answering for machine reading evaluation. find
19. A. Poliak, J. Naradowsky, A. Haldar, R. Rudinger, and B. Van Durme. 2018. Hypothesis only baselines in natural language inference. In Joint Conference on Lexical and Computational Semantics (*SEM). find
20. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find
21. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find our card
22. D Sculley, Jasper Snoek, Alex Wiltschko, and Ali Rahimi. 2018. Winner’s curse? on pace, progress, and empirical rigor. In International Conference on Learning Representations (ICLR) Workshop Track. find
23. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems (NIPS). find
24. Richard Sutcliffe, Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Corina Forascu, Yassine Benajiba, and Petya Osenova. 2013. Overview of qa4mre main task at clef 2013. find
25. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Workshop on Representation Learning for NLP (RepL4NLP). find
26. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. find
27. Mengqiu Wang, Noah A Smith, and Teruko Mitamura. 2007. What is the jeopardy model? a quasi-synchronous grammar for qa. In Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). find
28. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Empirical Methods in Natural Language Processing (EMNLP). find
29. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. In International Conference on Learning Representations (ICLR). find our card
30. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The lambada dataset: Word prediction requiring a broad discourse context. In Association for Computational Linguistics (ACL). find
31. Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Richard Sutcliffe, Caroline Sporleder, Corina Forascu, Yassine Benajiba, and Petya Osenova. 2012. Overview of qa4mre at clef 2012: Question answering for machine reading evaluation. find
32. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In Empirical Methods in Natural Language Processing (EMNLP). find
33. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. Qanet: Combining local convolution with global self-attention for reading comprehension. In International Conference on Learning Representations (ICLR). find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kaushik-2018-how-much-reading-does-reading/