The NarrativeQA Reading Comprehension Challenge

What it is about

NarrativeQA pairs 1,567 long stories (half Gutenberg books, half movie scripts) with plot summaries from Wikipedia; annotators who saw only the summary wrote about 30 question–answer pairs per story in their own words (46,765 in all, answers 4.7 tokens on average, a second reference answer per question). Because the questions come from summaries, answering from the full story requires finding and integrating scattered passages, not matching one sentence. Question first words: what 38%, who 23%, why 10%, how 9%, where 8%; about 15% ask what happened before, after or together with an event, and many ask relations between characters («How is Oscar related to Dana?»). On summaries, span and attention models beat IR baselines but stay far below humans; on full stories, a retrieve-then-read pipeline over 200-word chunks performs only slightly above simple baselines. Entities are replaced by permuted markers so models cannot learn specific names.

Ideas

• I1. A name-permutation control for the SLM student, after the paper's entity markers: rename the characters of a tale consistently (fox → Ziv, crow → Pell) in both lesson and questions; the student's score must stay the same, because it should read rather than recall. A drop shows reliance on stored knowledge about animals or names. Cheap, deterministic, and a gate that can go red.
• I2. Summary-written questions as a harder test for the books course: let the teacher write a short summary of a tale and questions from the summary only (silver, for measuring), then ask the student on the full tale text. These questions use the summary's words, not the tale's, which tests the frame matching across paraphrase that world knowledge diagnose step 2 («nothing remembered matched») counts. Report it separately from the existing courses.
• I3. Character-relation questions («How is the miller related to the princess?», «Who is X's father?») are a frequent type here; check how many such questions the courses have and whether world/src/family.rs relations are filled from tale text (appositions, «his daughter», «the king's son») before adding rules.
• I4. For long books, retrieve before reading by frames, not by word chunks: select the frames whose entities and verb lemmas the question shares, in story order, and answer from them; the paper's chunk retrieval was the weak step. Measure on the books course whether answers come from the right part of the story (the decision trace names the frame).

For Mova

35. Reference and coreference (characters across a story), 36-discourse-relations.md §8 (narrative structure). Code: world/src/knowledge.rs, world/src/family.rs, world/src/bookans.rs, world/src/index.rs. Measures: score difference under name permutation (should be 0), accuracy on summary-written questions, diagnosis step shares on the books course.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00023

@article{kocisku2018narrativeqa,
  title = {{The NarrativeQA Reading Comprehension Challenge}},
  author = {Kočiský, Tomáš and Schwarz, Jonathan and Blunsom, Phil and Dyer, Chris and Hermann, Karl Moritz and Melis, Gábor and Grefenstette, Edward},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00023},
  url = {https://doi.org/10.1162/tacl_a_00023}
}

References

33 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. CoRR, arXiv:1610.00956. — [arXiv:1610.00956](https://arxiv.org/abs/1610.00956) find
2. David Bamman, Brendan O’Connor, and Noah A. Smith. 2014a. Learning latent personas of film characters. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, page 352. find
3. David Bamman, Ted Underwood, and Noah A. Smith. 2014b. A Bayesian mixed effects model of literary character. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 370–379. find
4. John B. Black and Robert Wilensky. 1979. An evaluation of story grammars. Cognitive Science, 3(3):213–229. find
5. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3(Jan):993–1022. find
6. Nathanael Chambers and Dan Jurafsky. 2009. Unsupervised learning of narrative schemas and their participants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2, ACL… find
7. Snigdha Chaturvedi, Mohit Iyyer, and Hal Daumé III. 2017. Unsupervised learning of evolving relationships between literary characters. In Association for the Advancement of Artificial Intelligence. find
8. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/Daily Mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. find
9. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the EMNLP 2011 Workshop on Statistical Machine Translation, pages 85–91. find
10. Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. arXiv:1704.05179v2. find
11. Lea Frermann and György Szarvas. 2017. Inducing semantic micro-clusters from deep multi-view representations of novels. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1874–1884. find
12. Philip John Gorinski and Mirella Lapata. 2015. Movie script summarization as graph-based scene extraction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1066–1076, May–June. find
13. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1693–1701. find
14. Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2016. WikiReading: A novel large-scale language understanding task over wikipedia. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguis… find
15. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children’s books with explicit memory representations. In Proceedings of ICLR. find
16. Mohit Iyyer, Anupam Guha, Snigdha Chaturvedi, Jordan Boyd-Graber, and Hal Daumé III. 2016. Feuding families and former friends: Unsupervised learning for dynamic fictional relationships. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguis… find
17. Rudolf Kadlec, Martin Schmid, Ondřej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 908–918. find
18. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Proc. ACL Workshop on Text Summarization Branches Out. find
19. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, arXiv:1611.09268. find
20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 311–318. find our card
21. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Processing of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543. find
22. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP). find
23. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 193– 203. find our card
24. Roger C. Schank and Robert P. Abelson. 1977. Scripts, Plans, Goals and Understanding: an Inquiry into Human Knowledge Structures. find
25. L. Erlbaum, Hillsdale, NJ. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv:1611.01603. find
26. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of Conference on the Advances in Neural Information Processing Systems, pages 3104–3112. find
27. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. MovieQA: Understanding stories in movies through question-answering. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4631– 4640. find
28. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. NewsQA: A machine comprehension dataset. CoRR, arXiv:1611.09830. find
29. Shuohang Wang and Jing Jiang. 2016. Machine comprehension using match-LSTM and answer pointer. arXiv:1608.07905. find
30. Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistcs, pages 189–198. find
31. Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. FastQA: A simple and efficient neural architecture for question answering. CoRR, arXiv:1703.04816v1. find
32. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2015. Towards AI-complete question answering: A set of prerequisite toy tasks. CoRR, arXiv:1502.05698. find our card
33. R. Wilensky. 1978. Why John married Mary: Understanding stories involving recurring goals. Cognitive Science, 2(3):235–266. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kocisky-2018-narrativeqa-reading-comprehension-challenge/