RACE: Large-scale ReAding Comprehension Dataset From Examinations

What it is about

RACE collects about 28,000 passages and 100,000 four-option questions from English exams for Chinese middle-school (RACE-M, ages 12–15) and high-school (RACE-H) students, written by teachers; options are human-written sentences, often not in the passage. On 500 annotated questions, only 15.8% are word matching, 19.2% paraphrase, 33.4% single-sentence reasoning, 25.8% multi-sentence reasoning and 5.8% insufficient or ambiguous — far more reasoning than SQuAD or CNN. Reasoning questions are further split into detail reasoning, whole-picture understanding, passage summarisation («The main idea is…»), attitude analysis («The author's attitude toward X is…») and world knowledge (arithmetic, dates). A sliding-window word-matching baseline gets 32%, neural readers about 44%, crowd workers 73%, and the ceiling (questions with an unambiguous answer) 94.5%.

Ideas

• I1. Add the sliding-window baseline of Richardson et al. (the one used here: TF-IDF overlap of question + option with every window of the lesson) to Mova's course reports as the floor every course must beat, per course and per question type. It costs nothing, is deterministic, and shows where the student is no better than word overlap.
• I2. Label each course question with the paper's difficulty classes by rule: word match (all content lemmas of question+right option in one sentence), paraphrase (in one sentence after synonym/lemma mapping), single-sentence, multi-sentence (frames from two or more sentences needed — the decision trace shows how many frames supported the answer), insufficient (no supporting frame at all). Report the student's accuracy per class next to world knowledge diagnose; the «insufficient» share is the ceiling the Torah diagnosis already hit (quotes not in the lesson).
• I3. Route the two whole-text question types instead of matching them to one frame: «main idea» options compared against the most salient agent and the frames with the most links, «attitude» options against the polarity of the evaluation and express frames of that character (prag polarity/act=evaluate). Count first how many course questions are of these shapes.

For Mova

30. Pragmatics and discourse: overview §9 (measurements), 32-politeness-and-hedging.md (attitude), 36-discourse-relations.md. Code: world/src/knowledge.rs (diagnosis, question classes), world/src/report.rs (baseline column), world/src/school.rs. Measures: student minus window-baseline per course, accuracy per difficulty class, share of insufficient questions.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d17-1082 · arXiv 1704.04683

@inproceedings{lai2017race,
  title = {{RACE: Large-scale ReAding Comprehension Dataset From Examinations}},
  author = {Lai, Guokun and Xie, Qizhe and Liu, Hanxiao and Yang, Yiming and Hovy, Eduard},
  booktitle = {Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  doi = {10.18653/v1/d17-1082},
  eprint = {1704.04683},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d17-1082}
}

References

19 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint arXiv:1610.00956 . — [arXiv:1610.00956](https://arxiv.org/abs/1610.00956) find
2. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. arXiv preprint arXiv:1606.02858 . find
3. Bhuwan Dhingra, Hanxiao Liu, William W Cohen, and Ruslan Salakhutdinov. 2016. Gated-attention readers for text comprehension. arXiv preprint arXiv:1606.01549 . find
4. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. pages 1693– 1701. find
5. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301 . find
6. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. arXiv preprint arXiv:1603.01547 . find
7. Daniel Khashabi, Tushar Khot, Ashish Sabharwal, Peter Clark, Oren Etzioni, and Dan Roth. 2016. Question answering via integer programming over semi-structured knowledge. arXiv preprint arXiv:1604.06076 . — [arXiv:1604.06076](https://arxiv.org/abs/1604.06076) find
8. Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1. Association for Computational Lingu… find
9. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268 . find
10. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. arXiv preprint arXiv:1608.05457 . find
11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics. Association for Computational Linguistics, pages 311–318. find our card
12. Anselmo Peñas, Yusuke Miyao, Álvaro Rodrigo, Eduard H Hovy, and Noriko Kando. 2014. Overview of clef qa entrance exams task 2014. In CLEF (Working Notes). pages 1194–1200. find
13. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In EMNLP. volume 14, pages 1532– 1543. find
14. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250 . find
15. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP. volume 3, page 4. find our card
16. Álvaro Rodrigo, Anselmo Peñas, Yusuke Miyao, Eduard H Hovy, and Noriko Kando. 2015. Overview of clef qa entrance exams task 2015. In CLEF (Working Notes). find
17. Hideyuki Shibuki, Kotaro Sakamoto, Yoshinobu Kano, Teruko Mitamura, Madoka Ishioroshi, Kelly Y Itakura, Di Wang, Tatsunori Mori, and Noriko Kando. 2014. Overview of the ntcir-11 qa-lab task. In NTCIR. find
18. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830 . find
19. Zhilin Yang, Junjie Hu, Ruslan Salakhutdinov, and William W Cohen. 2017. Semi-supervised qa with generative domain-adaptive nets. arXiv preprint arXiv:1702.02206 . find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lai-2017-race-large-scale-reading-comprehension/