Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning

What it is about

Quoref has over 24K span-answer questions on 4.7K Wikipedia paragraphs (about 40% film plot summaries, which mention many characters), written by crowd workers so that answering needs linking two mentions of one entity. To keep out questions answerable by surface cues, a BERT model trained on SQuAD ran in the background and a question was accepted only if that model got it wrong. In a sample of 100, 78% truly need coreference: pronominal 69%, nominal 54%, both 32%, commonsense 10%; about 10% of answers are several spans. Questions often name the target by a description instead of a name («the person who is defied by her daughter» → Hippolyta, from «Hippolyta was granted a daughter, Diana … Diana defies her mother»). The best model reached 70.5 F1 against 93.4 for humans.

Ideas

• I1. Count, in world knowledge diagnose, which course questions can be answered only through a resolved pronoun or nominal (the remembered frame that matches the question has the answer role filled by a mention the resolver linked, not by a name). Report the share right among those separately. It tells how much of the student's score depends on Ctx resolution, and gives the coref work a course-level yardstick next to GUM CoNLL F1.
• I2. Read questions that refer to an entity through a description: a relative clause on the person/character/animal who … in the question is itself a frame to be matched («who is defied by her daughter» → find X with a daughter-frame and a defy frame whose patient is X). This needs the kin inverse (daughter of X ⇔ X is mother) from the possessive and kin nouns; the parser side exists (world/src/ans.rs handles possessives). Measure on course questions of the form «What is the name of the … who …».
• I3. Use the adversarial filter idea to grade Mova's own course questions: mark as «shallow» every question that a trivial baseline (the option sharing most words with one lesson sentence, or the nearest name to a question word) answers right, and report the student on deep questions only. A gain that is only on shallow questions is not reading.

For Mova

35. Reference and coreference §10 (the resolver inside the frame reader), Watching the SLM think: the diagnosis of its answers (the diagnosis steps). Code: world/src/knowledge.rs (diagnose split), prag/src/frames.rs (Ctx, relative clauses in questions), world/src/ans.rs. Measures: course test counts on pronoun-dependent and deep questions; Quoref dev only as an outside test set, given the Wikipedia licence.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 1908.05803

@misc{dasigi2019quoref,
  title = {{Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning}},
  author = {Dasigi, Pradeep and Liu, Nelson F. and Marasovic, Ana and Smith, Noah A. and Gardner, Matt},
  year = {2019},
  eprint = {1908.05803},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1908.05803}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Rahul Aralikatte and Anders Søgaard. 2019. Model-based annotation of coreference. arXiv:1906.10724. ↗
2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. MS MARCO: A human generated machine reading comprehension dataset. In NeurIPS. find
3. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proc. of EMNLP. find our card
4. Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Proc. of ACL. find
5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL. find
6. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proc. of NAACL. find our card
7. Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proc. of Workshop for NLP Open Source Software. find
8. Abbas Ghaddar and Phillippe Langlais. 2016. Wiki-Coref: An English coreference-annotated corpus of Wikipedia articles. In Proc. of LREC. find our card
9. Anupam Guha, Mohit Iyyer, Danny Bouman, and Jordan Boyd-Graber. 2015. Removing the training wheels: A coreference dataset that entertains humans and challenges computers. In Proc. of NAACL. find
10. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A Smith. 2018. Annotation artifacts in natural language inference data. In Proc. of NAACL. find
11. Mandar S. Joshi, Eunsol Choi, Daniel S. Weld, and Luke S. Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. of ACL. find
12. Divyansh Kaushik and Zachary Chase Lipton. 2018. How much reading does reading comprehension require? A critical investigation of popular benchmarks. In Proc. of EMNLP. find our card
13. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red-field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural Questions: A benchmark for question answering research. In TACL. find
14. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. arXiv:1906.08237. ↗
15. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proc. of EMNLP. find our card
16. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. 2018. QANet: Combining local convolution with global self-attention for reading comprehension. In Proc. of ICLR. find
17. Nelson F. Liu, Roy Schwartz, and Noah A. Smith. 2019. Inoculation by fine-tuning: A method for analyzing challenge datasets. In Proc. of NAACL. find
18. Courtney Napoles, Matthew R. Gormley, and Benjamin Van Durme. 2012. Annotated Gigaword. Technical report. find
19. Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proc. of NAACL. find
20. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the ARRAU corpus. In Proc. of CRAC Workshop. find
21. Sameer S. Pradhan, Lance Ramshaw, Ralph Weischedel, Jessica MacBride, and Linnea Micciulla. 2007. Unrestricted coreference: Identifying entities and events in OntoNotes. In Proc. of ICSC. find
22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. of EMNLP. find
23. Marta Recasens, Eduard Hovy, and M Antònia Martı́. 2011. Identity, non-identity, and near-identity: Addressing the complexity of coreference. Lingua, 121(6):1138–1152. find
24. Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proc. of EMNLP. find our card
25. Mark D. Smucker, Charles L. A. Clarke, and Gordon V. Cormack. 2009. Experiments with clueweb09: Relevance feedback and web tracks. In Proc. of TREC. find
26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NeurIPS. find
27. Yannick Versley. 2008. Vagueness and referential ambiguity in a large-scale annotated corpus. Research on Language and Computation, 6(3):333–353. find
28. Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. Proc. of ICCV. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/dasigi-2019-quoref-reading-comprehension-dataset-questions/