Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning

What it is about

Cosmos QA has 35,588 four-option questions on 21,886 paragraphs from personal-narrative blogs, written by crowd workers to need commonsense «between the lines»: likely causes of events, what happened before or after, plausible facts about people and things, and counterfactuals («What would have happened if…?»). Over 83% of right answers are not stated in the paragraph, and 93.8% of questions need contextual commonsense (motivation, reaction, pre/post-condition, temporal, situational facts, counterfactuals). «None of the above» is used as an option: questions answerable without the paragraph had their right answer replaced by it, and in 70% of the others it replaced the weakest distractor. BERT with multiway attention reaches 68.4% against 94% for humans; word-overlap readers do far worse because the right answer often overlaps less with the text. A partial-input ablation shows that removing the question hurts little, since question types are few and guessable from the options.

Ideas

• I1. Partial-input controls for every course, after the paper's ablation: run the student with the lesson removed (question + options only) and with the question removed (lesson + options only). Questions still answered right without the lesson measure prior or option bias, not reading; report them per course and subtract them when claiming a gain. A deterministic, cheap gate that can go red.
• I2. Handle «none of the above» options explicitly: when no option has frame support (the diagnosis step «no option supported», 29.4% on the Torah course) and such an option exists, choose it with the reason «no remembered frame supports A–C»; when one option is supported, never choose it. Count first how many course questions carry such an option.
• I3. Use the absurdity matrix (world/src/domains.rs) to demote options a person would not choose under common sense (leaving a baby alone at home), the error class that made a third of the paper's model errors. Only as a tie-breaker between options with equal frame support, with the score shown in the trace; measure on the «right option outscored» cases.

For Mova

33. Implicature and presupposition (inference beyond the text), 36-discourse-relations.md §8, §11. Code: world/src/knowledge.rs (partial-input runs, none-of-the-above), world/src/domains.rs, world/src/report.rs. Measures: course scores with and without lesson and question, diagnosis step shares, «right option outscored» count.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1243 · arXiv 1909.00277

@inproceedings{huang2019cosmos,
  title = {{Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning}},
  author = {Huang, Lifu and Bras, Ronan Le and Bhagavatula, Chandra and Choi, Yejin},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1243},
  eprint = {1909.00277},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d19-1243}
}

References

47 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72. find our card
2. Kevin Burton, Akshay Java, Ian Soboroff, et al. 2009. The icwsm 2009 spinn3r dataset. In Proceedings of ICWSM 2009. find
3. Zheng Cai, Lifu Tu, and Kevin Gimpel. 2017. Pay attention to the ending: Strong neural baselines for the roc story cloze task. In Proceedings of ACL 2017, pages 616–622. find
4. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In Proceedings of ACL 2016, pages 2358–2367. find
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
6. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In Proceedings of ACL 2017, pages 1832–1846. find
7. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. find
8. Andrew Gordon and Reid Swanson. 2009. Identifying personal stories in millions of weblog entries. In Third International Conference on Weblogs and Social Media, Data Challenge Workshop. find
9. Rudolf Kadlec, Martin Schmid, Ondřej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In Proceedings of ACL 2016, pages 908–918. find
10. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gáabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Proceedings of TACL 2018, pages 317–328. find our card
11. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In Proceedings of EMNLP 2017, pages 785–794. find our card
12. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
13. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out. find
14. Sewon Min, Minjoon Seo, and Hannaneh Hajishirzi. 2017. Question answering through transfer learning from large fine-grained supervision data. In Proceedings of ACL 2017, pages 510–517. find
15. Bhavana Dalvi Mishra, Lifu Huang, Niket Tandon, Wen-tau Yih, and Peter Clark. 2018. Tracking state changes in procedural text: a challenge dataset and models for process paragraph comprehension. arXiv preprint arXiv:1805.06975. find
16. Nasrin Mostafazadeh, Michael Roth, Annie Louis, Nathanael Chambers, and James Allen. 2017. find
17. Lsdsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46–51. find
18. Andrew S Gordon. 2016. Commonsense interpretation of triangle behavior. In Proceedings of AAAI 2016. find
19. Peter Norvig. 1987. A Unified Theory of Inference for Text Understanding. Ph.D. thesis, EECS Department, University of California, Berkeley. find
20. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of NAACL 2018, pages 107–112. find
21. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018. Mcscript: A novel dataset for assessing machine comprehension using script knowledge. In Proceedings of LREC 2018. find
22. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of NIPS 2015, pages 1693–1701. find
23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL 2002, pages 311–318. find our card
24. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301. find
25. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of ACL 2018, pages 328–339. find
26. Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI. find
27. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, page 8. find
28. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: the winograd schema challenge. In Proceedings of EMNLP 2012, pages 777–789. find
29. Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you dont know: Unanswerable questions for squad. In Proceedings of ACL 2018, pages 784–789. find
30. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP 2016, pages 2383–2392. find
31. Hannah Rashkin, Antoine Bosselut, Maarten Sap, Kevin Knight, and Yejin Choi. 2018. Modeling naive psychology of characters in simple commonsense stories. In Proceedings of ACL 2018, pages 2289–2299. find
32. Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. Coqa: A conversational question answering challenge. arXiv preprint arXiv:1808.07042. find
33. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of EMNLP 2013, pages 193–203. find our card
34. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series. find
35. Maarten Sap, Ronan LeBras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2018. Atomic: An atlas of machine commonsense for if-then reasoning. arXiv preprint arXiv:1811.00146. find
36. Liang Wang, Meng Sun, Wei Zhao, Kewei Shen, and Jingming Liu. 2018a. Yuanfudao at semeval-2018 task 11: Three-way attention and relational knowledge for commonsense machine comprehension. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 758–762. find
37. Shuohang Wang, Mo Yu, Jing Jiang, and Shiyu Chang. 2018b. A co-matching model for multi-choice reading comprehension. In Proceedings of ACL 2018, pages 746–751. find
38. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of EMNLP 2018, pages 93–104. find
39. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885. find
40. Shuailiang Zhang, Hai Zhao, Yuwei Wu, Zhuosheng Zhang, Xi Zhou, and Xiang Zhou. 2019a. Dual co-matching network for multi-choice reading comprehension. arXiv preprint arXiv:1901.09381. — [arXiv:1901.09381](https://arxiv.org/abs/1901.09381) find
41. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019b. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
42. Haichao Zhu, Furu Wei, Bing Qin, and Ting Liu. 2018. Hierarchical attention flow for multiple-choice reading comprehension. In Proceedings of AAAI 2018. find
43. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728. find
44. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603. find
45. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2018. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937. find
46. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191–200. find
47. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of CVPR 2015, pages 4566–4575. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/huang-2019-cosmos-qa-machine-reading-comprehension/