Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks

Summary

The paper proposes bAbI, a suite of 20 synthetic question-answering tasks meant to work like unit tests for text understanding and reasoning. Stories are produced by a simple world simulator, similar to a text adventure, in which characters move between locations and pick up, drop or hand over objects. Each task targets one skill: retrieving one, two or three supporting facts, argument relations, yes/no questions, counting, lists, negation, coreference, time reasoning, basic deduction and induction, positional and size reasoning, path finding and agents' motivations. Answers are single words or short lists, so scoring is unambiguous, and the data is noise-free so a human can answer everything. As an illustration, the authors extend Memory Networks with adaptive numbers of memory hops, multi-word answers and better sentence representations; the improved models solve many but not all tasks, and a structured-SVM pipeline with external NLP tools fails several. The value of the suite is that passing or failing a given task points to a specific missing ability.

How Mova uses it

• world/src/babi.rs — the world babi command solves all 20 tasks with a reader built on the en parse tree, verb classes, and a first-level relation block, without training on the questions.
• Relations of space, size and time — the relation seed (inverses such as north/south, plane vectors, transitivity of bigger/smaller, fits inside = smaller than, order of parts of the day) cites bAbI tasks 4, 14, 17, 18 and 19; facts come from the story, inferences come from this shared seed.
• world babi-learn — agents' motivations (hungry → kitchen, etc.) are learned automatically from the training split and written back as a seed.
• world babi-contrast — contrast variants in the spirit of Gardner et al. 2020 (new names, places, objects, synonym verbs) test that the reader does not depend on bAbI's fixed vocabulary.

Effectiveness in Mova

From the "Ideas from papers through the bench" table in Mova's development notes: test 99.8%, 20/20 tasks ≥ 95%; training split 99.7% (not tuned to). Motivations were learned from the training split (375/375). Majority voting in induction was worse than using the freshest example (97.6% vs 99.5%). On contrast variants accuracy first dropped to 74.6%, exposing parser defects (attic tagged ADJ, Taras lemmatised as "tara", Following as a verb, is as a main verb) and two lexicon gaps (hurry, abandon); after fixing them contrast accuracy reached 99.7%, 20/20, with no drop on the original (99.8%).

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 1502.05698

@misc{weston2015ai,
  title = {{Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks}},
  author = {Weston, Jason and Bordes, Antoine and Chopra, Sumit and Rush, Alexander M. and Merriënboer, Bart van and Joulin, Armand and Mikolov, Tomas},
  year = {2015},
  eprint = {1502.05698},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1502.05698}
}

References

34 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bache, K. and Lichman, M. UCI machine learning repository, http://archive.ics.uci.edu/ml. Berant, Jonathan, Chou, Andrew, Frostig, Roy, and Liang, Percy. Semantic parsing on freebase from question-answer pairs. In EMNLP, pp. 1533–1544, 2013. ↗
2. Berant, Jonathan, Srikumar, Vivek, Chen, Pei-Chun, Huang, Brad, Manning, Christopher D, Vander Linden, Abby, Harding, Brittany, and Clark, Peter. Modeling biological processes for reading comprehension. In Proc. EMNLP, 2014. find
3. Bordes, Antoine, Usunier, Nicolas, Collobert, Ronan, and Weston, Jason. Towards understanding situated natural language. In AISTATS, 2010. find
4. Bordes, Antoine, Usunier, Nicolas, Chopra, Sumit, and Weston, Jason. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075, 2015. find
5. Chen, David L and Mooney, Raymond J. Learning to interpret natural language navigation instructions from observations. San Francisco, CA, pp. 859–865, 2011. find
6. Collobert, Ronan, Weston, Jason, Bottou, Léon, Karlen, Michael, Kavukcuoglu, Koray, and Kuksa, Pavel. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537, 2011. find
7. Curtis, Jon, Matthews, Gavin, and Baxter, David. On the effective use of cyc in a question answering system. In IJCAI Workshop on Knowledge and Reasoning for Answering Questions, pp. 61–70, 2005. find
8. Fader, Anthony, Zettlemoyer, Luke, and Etzioni, Oren. Paraphrase-driven learning for open question answering. In ACL, pp. 1608–1618, 2013. find
9. Fader, Anthony, Zettlemoyer, Luke, and Etzioni, Oren. Open question answering over curated and extracted knowledge bases. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 1156–1165. ACM, 2014. find
10. Graves, Alex, Wayne, Greg, and Danihelka, Ivo. arXiv:1410.5401, 2014. find
11. Halevy, Alon, Norvig, Peter, and Pereira, Fernando. The unreasonable effectiveness of data. Intelligent Systems, IEEE, 24(2):8–12, 2009. find
12. Hermann, Karl Moritz, Kočiský, Tomáš, Grefenstette, Edward, Espeholt, Lasse, Kay, Will, Suleyman, Mustafa, and Blunsom, Phil. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS), 2015. URL http://arxiv.org/abs/1506.03340. ↗
13. Hill, Felix, Bordes, Antoine, Chopra, Sumit, and Weston, Jason. The goldilocks principle: Reading children’s books with explicit memory representation s. arXiv preprint arXiv:1511.02301, 2015. find
14. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
15. Kumar, Ankit, Irsoy, Ozan, Su, Jonathan, Bradbury, James, English, Robert, Pierce, B̃rian, Ondruska, Peter, Gulrajani, Ishaan, and Socher, Richard. Ask me anything: Dynamic memory networks for natural language processing. http://arxiv.org/abs/1506.07285, 2015. 2013. URL Neural turing machines. arXiv… ↗
16. Liang, Percy. Lambda dependency-based compositional semantics. arXiv preprint arXiv:1309.4408, 2013. find
17. Liang, Percy, Jordan, Michael I, and Klein, Dan. Learning dependency-based compositional semantics. Computational Linguistics, 39(2):389–446, 2013. find
18. Minsky, Marvin and Papert, Seymour. Perceptron: an introduction to computational geometry. The MIT Press, Cambridge, expanded edition, 19:88, 1969. find
19. Montfort, Nick. Twisty Little Passages: an approach to interactive fiction. Mit Press, 2005. find
20. Müller, K-R, Smola, Alex J, Rätsch, Gunnar, Schölkopf, Bernhard, Kohlmorgen, Jens, and Vapnik, Vladimir. Predicting time series with support vector machines. In Artificial Neural NetworksI-CANN’97, pp. 999–1004. Springer, 1997. find
21. Ng, Andrew Y, Jordan, Michael I, Weiss, Yair, et al. On spectral clustering: Analysis and an algorithm. Advances in neural information processing systems, 2:849–856, 2002. find
22. Peng, Baolin, Lu, Zhengdong, Li, Hang, and Wong, Kam-Fai. Towards neural network-based reasoning. arXiv preprint arXiv:1508.05508, 2015. — [arXiv:1508.05508](https://arxiv.org/abs/1508.05508) find
23. Raghunathan, Karthik, Lee, Heeyoung, Rangarajan, Sudarshan, Chambers, Nathanael, Surdeanu, Mihai, Jurafsky, Dan, and Manning, Christopher. A multi-pass sieve for coreference resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 492–501. Associatio… find our card
24. Richardson, Matthew, Burges, Christopher JC, and Renshaw, Erin. Mctest: A challenge dataset for the open-domain machine comprehension of text. In EMNLP, pp. 193–203, 2013. find our card
25. Rumelhart, David E, Hinton, Geoffrey E, and Williams, Ronald J. Learning internal representations by error propagation. Technical report, DTIC Document, 1985. find
26. Soon, Wee Meng, Ng, Hwee Tou, and Lim, Daniel Chung Yong. A machine learning approach to coreference resolution of noun phrases. Computational linguistics, 27(4):521–544, 2001. find our card
27. Sukhbaatar, Sainbayar, Szlam, Arthur, Weston, Jason, and Fergus, Rob. End-to-end memory networks. Proceedings of NIPS, 2015. find
28. Sutskever, Ilya, Vinyals, Oriol, and Le, Quoc VV. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pp. 3104–3112, 2014. find
29. UzZaman, Naushad, Llorens, Hector, Allen, James, Derczynski, Leon, Verhagen, Marc, and Pustejovsky, James. Tempeval-3: Evaluating events, time expressions, and temporal relations. arXiv preprint arXiv:1206.5333, 2012. find
30. Weston, Jason, Chopra, Sumit, and Bordes, Antoine. Memory networks. CoRR, abs/1410.3916, 2014. find
31. Winograd, Terry. Understanding natural language. Cognitive psychology, 3(1):1–191, 1972. find
32. Yao, Xuchen, Berant, Jonathan, and Van Durme, Benjamin. Freebase qa: Information extraction or semantic parsing? ACL 2014, pp. 82, 2014. find
33. Yu, Mo, Gormley, Matthew R, and Dredze, Mark. Factor-based compositional embedding models. NIPS 2014 workshop on Learning Semantics, 2014. find
34. Zhu, Xiaojin, Ghahramani, Zoubin, Lafferty, John, et al. Semi-supervised learning using gaussian fields and harmonic functions. In ICML, volume 3, pp. 912–919, 2003. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/weston-2015-towards-ai-complete-question-answering/