Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds

What it is about

TextWorldsQA is generated, not collected: five simulated worlds (a university, a workplace with meetings, a shopping/chores world and others), each with entity classes and unary-to-ternary relations; every simulation step adds or changes facts and is verbalised as a first-person sentence with varied referring expressions («the meeting I last added», «John's new office is GHC122»), and questions about the current state are interleaved, answered from the underlying knowledge base. Questions are grouped by complexity: single entity/single relation, multiple entities/single relation, two relations, three relations (e.g. «Which undergraduates are enrolled in courses I teach?»). Logistic regression, memory networks, BiDAF, DrQA and seq2seq are evaluated within a world and across worlds: performance falls with every added relation, and drops sharply on unseen worlds, showing that end-to-end readers do not generalise to new relations and vocabulary.

Ideas

• I1. A generated test bed for the frame reader and the student, built the paper's way: a small Rust simulator of a tale world (characters, things, places; give, take, move, lose, find) that writes the story sentence by sentence with pronouns and definite descriptions and asks current-state questions with known answers. Gold comes from the simulator, so no teacher is needed; vary the referring expressions to test the resolver. Report accuracy by number of relations composed, as the paper does.
• I2. State updates: «the key is now in the box» replaces the earlier location. Questions about the current state must take the latest frame that sets the property (inertia until a later frame terminates it, see sarathy-2020-reasoning-requirements-indirect-speech-act I5). Test it on generated stories with two or more updates of one property.
• I3. Two-relation questions as joins over frames («Who ate the cake the fox stole?» = agent of eat whose patient is the patient of the fox's steal), answered by matching two frames that share an entity, with both frames printed. Count how many course questions need two frames (the diagnosis trace shows it) before investing.

For Mova

35. Reference and coreference (definite descriptions), 36-discourse-relations.md §11.4 (story time). Code: world/src/story.rs, world/src/state.rs, world/src/babi.rs (an existing generated-task harness), world/src/knowledge.rs. Measures: accuracy on generated stories by number of relations and by number of updates, course questions needing two frames.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1077

@inproceedings{labutov2018multi,
  title = {{Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds}},
  author = {Labutov, Igor and Yang, Bishan and Prakash, Anusha and Azaria, Amos},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1077},
  url = {https://doi.org/10.18653/v1/p18-1077}
}

References

42 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. David Ahn, Valentin Jijkoun, Gilad Mishne, Karin Müller, Maarten de Rijke, Stefan Schlobach, M Voorhees, and L Buckland. 2004. Using wikipedia at the trec qa track. In TREC. Citeseer. find
2. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301. find
3. Amos Azaria and Jason Hong. 2016. Recommender system with personality. In RecSys, pages 207–210. find
4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find
5. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In ACL. find
6. Michele Banko, Eric Brill, Susan Dumais, and Jimmy Lin. 2002. Askmsr: Question answering using the worldwide web. In Proceedings of 2002 AAAI Spring Symposium on Mining Answers from Texts and Knowledge Bases, pages 7–9. find
7. Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. In ACL. find
8. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018a. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In NAACL. find our card
9. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1533–1544. find
10. Daniel Khashabi, Tushar Khot Ashish Sabharwal, and Dan Roth. 2018b. Question answering as global reasoning over semantic abstractions. In AAAI. find
11. Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the cnn/daily mail reading comprehension task. In ACL. find
12. Tushar Khot, Ashish Sabharwal, and Peter Clark. 2017. Answering complex questions using open information extraction. In ACL. find
13. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL. find
14. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
15. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP. find
16. Tomáš Kočiskỳ, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2017. The narrativeqa reading comprehension challenge. arXiv preprint arXiv:1712.07040. find our card
17. Peter Clark. 2015. Elementary school science and math tests as a driver for ai: take the aristo challenge! In AAAI, pages 4019–4021. find
18. Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers for text comprehension. In ACL. find
19. Ankit Kumar, Ozan Irsoy, Peter Ondruska, Mohit Iyyer, James Bradbury, Ishaan Gulrajani, Victor Zhong, Romain Paulus, and Richard Socher. 2016. Ask me anything: Dynamic memory networks for natural language processing. In International Conference on Machine Learning, pages 1378–1387. find
20. Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. find
21. Tom Kwiatkowski, Eunsol Choi, Yoav Artzi, and Luke Zettlemoyer. 2013. Scaling semantic parsers with on-the-fly ontology matching. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1545–1556. find
22. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. In EMNLP. find our card
23. Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440–2448. find
24. Moontae Lee, Xiaodong He, Wen-tau Yih, Jianfeng Gao, Li Deng, and Paul Smolensky. 2015. Reasoning in vector space: An exploratory study of question answering. arXiv preprint arXiv:1511.06426. — [arXiv:1511.06426](https://arxiv.org/abs/1511.06426) find
25. Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: A machine comprehension dataset. arXiv preprint arXiv:1611.09830. find
26. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869. find
27. Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. arXiv preprint arXiv:1606.03126. find
28. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2015. Towards ai-complete question answering: A set of prerequisite toy tasks. arXiv preprint arXiv:1502.05698. find our card
29. Nick Montfort. 2005. Twisty Little Passages: an approach to interactive fiction. Mit Press. find
30. Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dynamic coattention networks for question answering. In ICLR. find
31. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. find
32. Scott Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In ACL. find
33. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. arXiv preprint arXiv:1608.05457. find
34. John M Zelle and Raymond J Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, pages 1050–1055. find
35. Baolin Peng, Zhengdong Lu, Hang Li, and Kam-Fai Wong. 2015. Towards neural network-based reasoning. arXiv preprint arXiv:1508.05508. — [arXiv:1508.05508](https://arxiv.org/abs/1508.05508) find
36. Luke S Zettlemoyer and Michael Collins. 2012. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. arXiv preprint arXiv:1207.1420. find
37. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In EMNLP. find
38. Will Y Zou, Richard Socher, Daniel Cer, and Christopher D Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1393–1398. find
39. Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193–203. find our card
40. Mrinmaya Sachan, Kumar Dubey, Eric Xing, and Matthew Richardson. 2015. Learning answer-entailing structures for machine comprehension. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing… find
41. Mrinmaya Sachan and Eric Xing. 2016. Machine comprehension using rich semantic representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 486–492. find
42. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/labutov-2018-multi-relational-question-answering-narratives/