Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences

What it is about

MultiRC has about 6,000 verified multi-sentence questions (9,872 collected) on 871 paragraphs from seven domains, fiction (Gutenberg, MCTest children's stories, movie plots) being the largest. Each question needs on average 2.4 sentences, often not adjacent. A crowd step verifies multi-sentenceness: the question is paired with each needed sentence alone, and if any single sentence lets workers answer it, it is dropped — a pilot found that half of MCTest's «multiple» questions were answerable from one sentence. Answer options vary in number (median 5) with several right ones (median 2), so each option must be judged on its own; only 35% of right options appear verbatim in the text (and 26% of wrong ones do). Coreference is the most frequent phenomenon needed. Humans reach 86.4 F1; IR and logistic-regression baselines and a strong reader stay far below.

Ideas

• I1. Judge every option on its own: the student outputs, per option, supported / contradicted / unsupported, with the frames behind each verdict, before choosing one. This turns the diagnosis steps «right option unsupported» and «right option outscored» into per-option facts and allows «select all that apply» questions; score with the paper's per-option F1 alongside accuracy.
• I2. Verify multi-sentenceness automatically, the paper's step 2 done with the student instead of crowd workers: present the question with each lesson sentence alone; if any single sentence gives the right answer, the question is single-sentence. Report the student's accuracy separately on the two kinds; only multi-sentence questions can show gains from coreference and links between frames.
• I3. On the multi-sentence subset, measure what coreference buys: the student with the frame reader's pronoun resolver on and off (PRAG_COREF), and with the coref sieve. Coreference was the most frequent need in MultiRC, so this is where a resolver change should show in course numbers.

For Mova

35. Reference and coreference §10 (the resolver inside the frame reader), 36-discourse-relations.md. Code: world/src/knowledge.rs (per-option verdicts, single-sentence runs), prag/src/frames.rs (PRAG_COREF), coref/src/sieve.rs. Measures: per-option F1, accuracy on single- vs multi-sentence questions, coreference on/off difference on the multi-sentence subset.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1023

@inproceedings{khashabi2018looking,
  title = {{Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences}},
  author = {Khashabi, Daniel and Chaturvedi, Snigdha and Roth, Michael and Upadhyay, Shyam and Roth, Dan},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1023},
  url = {https://doi.org/10.18653/v1/n18-1023}
}

References

35 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Lynette Hirschman, Marc Light, Eric Breck, and John D. Burger. 1999. Deep read: A reading comprehension system. In 27th Annual Meeting of the Association for Computational Linguistics, ACL 1999. http://www. aclweb.org/anthology/P99-1042. find
2. David Bamman, Brendan O’Connor, and Noah A. Smith. 2013. Learning latent personas of film characters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL 2013, Volume 1: Long Papers. pages 352–361. http://aclweb.org/ anthology/P/P13/P13-1035.pdf. ↗
3. Nancy Ide, Collin F. Baker, Christiane Fellbaum, Charles J. Fillmore, and Rebecca J. Passonneau. 2008. MASC: the manually annotated sub-corpus of american english. In Proceedings of the International Conference on Language Resources and Evaluation, LREC 2008. http://www.lrec-conf.org/proceedings/ lr… ↗
4. Virginia W. Berninger, William Nagy, and Scott Beers. 2011. Child writers construction and reconstruction of single sentences and construction of multi-sentence texts: Contributions of syntax and transcription to translation. Reading and writing 24(2):151–182. find
5. Nancy Ide and Keith Suderman. 2006. Integrating linguistic resources: The american national corpus model. In Proceedings of the Fifth International Conference on Language Resources and Evaluation, LREC 2006. pages 621–624. http://www.lrec-conf.org/ proceedings/lrec2006/pdf/560\_pdf.pdf. ↗
6. Ronald Brachman, David Gunning, Selmer Bringsjord, Michael Genesereth, Lynette Hirschman, and Lisa Ferro. 2005. Selected grand challenges in cognitive science. Technical report, MITRE Technical Report 05-1218. find
7. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, Volume 1: Long Papers. http://aclweb.org/anthology/P/P16/ P16-122… ↗
8. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017. pages 2021– 2031. https://aclanthology.info/papers/ D17-1215/d17-1215. ↗ our card
9. Peter Clark and Oren Etzioni. 2016. My computer is an honor student - but how intelligent is it? standardized tests as a measure of AI. AI Magazine 37(1):5– 12. http://www.aaai.org/ojs/index.php/ aimagazine/article/view/2636. ↗
10. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Volume 1: Long Papers. pages 1601–161… find
11. Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter D. Turney, and Daniel Khashabi. 2016. Combining retrieval, statistics, and inference to answer elementary science questions. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence. pages 2580–2586. ht… find
12. Aniruddha Kembhavi, Min Joon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017. pages 537… — [doi:10.1109/cvpr.2017.571](https://doi.org/10.1109/cvpr.2017.571) find
13. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzoto. 2013. Recognizing Textual Entailment: Models and Applications. Morgan and Claypool. find
14. Ernest Davis. 2014. The limitations of standardized science tests as benchmarks for artificial intelligence research: Position paper. CoRR abs/1411.1629. http://arxiv. org/abs/1411.1629. — [arXiv:1411.1629](https://arxiv.org/abs/1411.1629) find
15. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, and Dan Roth. 2016. Question answering via integer programming over semi-structured knowledge. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI 2016. pages 1145–1152. http://… find
16. Bert F. Green Jr., Alice K. Wolf, Carol Chomsky, and Kenneth Laughery. 1961. Baseball: An automatic question-answerer. In Papers Presented at the May 9-11, 1961, Western Joint IRE-AIEE-ACM Computer Conference. ACM, IRE-AIEE-ACM ’61 (Western), pages 219–224. https://doi.org/10.1145/ 1460690.1460714. find
17. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Dan Roth. 2018a. Question answering as global reasoning over semantic abstractions. In Proceedings of The Thirty-Second AAAI Conference on Artificial Intelligence, AAAI 2018 (to appear). http://cogcomp.org/page/ publication\_view/824. ↗
18. Peter H. Greene. 1959. An approach to computers that perceive, learn, and reason. In Papers Presented at the the March 3-5, 1959, Western Joint Computer Conference. ACM, IRE-AIEE-ACM ’59 (Western), pages 181–186. https://doi.org/10.1145/ 1457838.1457870. find
19. Daniel Khashabi, Mark Sammons, Ben Zhou, Tom Redman, Christos Christodoulopoulos, Vivek Srikumar, Nicholas Rizzolo, Lev Ratinov, Guanheng Luo, Quang Do, Chen-Tse Tsai, Subhro Roy, Stephen Mayhew, Zhilli Feng, John Wieting, Xiaodong Yu, Yangqiu Song, Shashank Gupta, Shyam Upadhyay, Naveen Arivazhagan… find
20. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015. pages 169… find
21. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. 2017. RACE: large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017. pages 785– 794. https://aclanthology.info/papers/ D17-1… ↗ our card
22. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. CoRR abs/1611.09830. http://arxiv.org/abs/ 1611.09830. ↗
23. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. CoRR abs/1502.05698. http://arxiv.org/abs/1502.05698. ↗ our card
24. J. McCarthy. 1976. An example for natural language understanding and the AI problems it raises. Available at http://jmc.stanford.edu/articles/ mrhug/mrhug.pdf. ↗
25. J. Wieting, M. Bansal, K. Gimpel, K. Livescu, and D. Roth. 2015. From paraphrase database to compositional paraphrase model and back. TACL 3:345–358. find
26. Elizabeth Merkhofer, John Henderson, David Bloom, Laura Strickhart, and Guido Zarrella. 2018. Mitre at semeval-2018 task 11: Commonsense reasoning without commonsense knowledge. In Proceedings of the International Workshop on Semantic Evaluation (SemEval-2018). New Orleans, LA, USA. find
27. Terry Winograd. 1980. Extended inference modes in reasoning by computer systems. Artificial Intelligence 13:5–26. find
28. Yi Yang, Wen-tau Yih, and Christopher Meek. 2015. Wikiqa: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015. pages 2013–2018. http://aclweb. org/anthology/D/D15/D15-1237.pdf. find
29. Sewon Min, Min Joon Seo, and Hannaneh Hajishirzi. 2017. Question answering through transfer learning from large fine-grained supervision data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Volume 2: Short Papers. pages 510–517. https://doi.org/… find
30. Karthik Narasimhan and Regina Barzilay. 2015. Machine comprehension with discourse relations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Langua… find
31. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR abs/1611.09268. http:// arxiv.org/abs/1611.09268. find
32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016. pages 2383– 2392. http://aclweb.org/anthology/D/D16/ D16-1264.… ↗
33. Raymond Reiter. 1976. A semantically guided deductive system for automatic theorem proving. IEEE Transactions on Computers 25:328–334. https://doi.org/10. 1109/TC.1976.1674613. find
34. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013. pages 193– 203. http://aclweb.org/anthology/D/D1… ↗ our card
35. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In International Conference on Learning Representations, ICLR 2017. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/khashabi-2018-looking-beyond-surface-challenge-set/