Mova0.5 pre-release⌕
☰
home / docs / papers / StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts

StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts

Summary

The paper argues that near-perfect scores on the spatial tasks of bAbI (tasks 17 and 19) overstate what models can do, because bAbI uses a handful of fixed phrasings, needs only one or two reasoning steps, and most test pairs also occur in training. StepGame is a question-answering benchmark in which agents are placed on a grid and the model must say where one agent is relative to another, choosing among eight directions plus "overlap". Descriptions of each relation come from crowdsourced templates, so the language is much more varied, and chains of reasoning range from 1 to 10 steps; test stories also contain distracting sentences. Models are trained on shorter chains and evaluated on longer ones to test generalisation. The authors also propose TP-MANN, a recurrent memory network based on tensor-product representations, which beats the neural baselines, although all models lose accuracy quickly as the number of steps grows.

How Mova uses it

Effectiveness in Mova

From the "Ideas from papers through the bench" table in Mova's development notes: 96.3% on the full test set (k=1 98.2% → k=10 94.7%), deterministic. The suspicion detector found 2 phrasings that are contradictory in the data (one sentence carries 2–3 different labels, a generator defect); without them accuracy is 98.0–99.6% on all k, so long chains barely degrade because composition is exact. The headline number remains 96.3% on the full test; 11,553 questions excluded from the "clean" figure are listed with reasons. For comparison, the paper reports far lower neural results (e.g. TP-MANN averaging about 53% for k=1–5), though setups differ (Mova trains its reader only on level 1 and composes symbolically).

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2204.08292

@misc{shi2022stepgame,
  title = {{StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts}},
  author = {Shi, Zhengxiang and Zhang, Qiang and Lipani, Aldo},
  year = {2022},
  eprint = {2204.08292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.08292}
}

References

39 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anderson, P.; Wu, Q.; Teney, D.; Bruce, J.; Johnson, M.; Sünderhauf, N.; Reid, I. D.; Gould, S.; and van den Hengel, A. 2018. Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments. In 2018 IEEE Conference on Computer Vision and Pattern Recogniti… find
  2. IEEE Computer Society. Battaglia, P. W.; Hamrick, J. B.; Bapst, V.; Sanchez-Gonzalez, A.; Zambaldi, V.; Malinowski, M.; Tacchetti, A.; Raposo, D.; Santoro, A.; Faulkner, R.; et al. 2018. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261. find
  3. Bisk, Y.; Shih, K. J.; Choi, Y.; and Marcu, D. 2018. Learning Interpretable Spatial Operations in a Rich 3D Blocks World. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI… find
  4. AAAI Press. Chen, C.-H.; Fu, Y.-F.; Cheng, H.-H.; and Lin, S.-D. 2020a. find
  5. Unseen Filler Generalization In Attention-based Natural Language Reasoning Models. In 2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI), 42–51. find
  6. IEEE. Chen, H.; Suhr, A.; Misra, D.; Snavely, N.; and Artzi, Y. 2019. TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. Computer Vision Foundatio… find
  7. Chen, K.; Huang, Q.; Palangi, H.; Smolensky, P.; Forbus, K. D.; and Gao, J. 2020b. Mapping natural-language problems to formal-language solutions using structured neural representations. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Even… find
  8. PMLR. Dehghani, M.; Gouws, S.; Vinyals, O.; Uszkoreit, J.; and Kaiser, L. 2019. Universal Transformers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. find
  9. Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Lo… find
  10. Ding, D.; Hill, F.; Santoro, A.; and Botvinick, M. 2020. Object-based attention for spatio-temporal reasoning: Outperforming neuro-symbolic models with flexible distributed architectures. arXiv preprint arXiv:2012.08508. — [arXiv:2012.08508](https://arxiv.org/abs/2012.08508) find
  11. Gershman, S.; and Tenenbaum, J. B. 2015. Phrase similarity in humans and machines. In Proceedings of the 37th Annual Meeting of the Cognitive Science Society, CogSci 2015, Pasadena, California, USA, July 22-25, 2015. cogni-tivesciencesociety.org. Graves, A.; Wayne, G.; Reynolds, M.; Harley, T.; Dani… find
  12. Nature. Huang, Q.; Smolensky, P.; He, X.; Deng, L.; and Wu, D. 2018. Tensor Product Generation Networks for Deep NLP Modeling. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). Ne… find
  13. Janner, M.; Narasimhan, K.; and Barzilay, R. 2018. Representation Learning for Grounded Spatial Reasoning. Transactions of the Association for Computational Linguistics. find
  14. Kipf, T. N.; and Welling, M. 2017. Semi-Supervised Classification with Graph Convolutional Networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Koster, R.; Chadwick, M. J.; Chen, Y.; Berron, D.; Banino, A.;… find
  15. Kruijff, G.-J. M.; Zender, H.; Jensfelt, P.; and Christensen, H. I. 2007. Situated dialogue and spatial organization: What, where. . . and why? International Journal of Advanced Robotic Systems. find
  16. Kumaran, D.; and McClelland, J. L. 2012. Generalization through the recurrent interaction of episodic memories: a model of the hippocampal system. Psychological review. find
  17. Landsiedel, C.; Rieser, V.; Walter, M.; and Wollherr, D. 2017. A review of spatial reasoning and interaction for real-world robotics. find
  18. Advanced Robotics. Le, H.; Tran, T.; and Venkatesh, S. 2020. Self-Attentive Associative Memory. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, 5682–5691. find
  19. PMLR. Liu, F.; and Perez, J. 2017. Gated End-to-End Memory Networks. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Valencia, Spain: Association for Computational Linguistics. find
  20. Mirzaee, R.; Faghihi, H. R.; Ning, Q.; and Kordjamshidi, P. 2021. SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 4582–4598. find our card
  21. Palm, R. B.; Paquet, U.; and Winther, O. 2018. Recurrent Relational Networks. In Bengio, S.; Wallach, H. M.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; and Garnett, R., eds., Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, Ne… find
  22. Petruck, M. R. L.; and Ellsworth, M. J. 2018. Representing Spatial Relations in FrameNet. In Proceedings of the First International Workshop on Spatial Language Understanding, 41–45. New Orleans: Association for Computational Linguistics. find
  23. Pustejovsky, J. 1989. Language and Spatial Cognition. Computational Linguistics, 15(3). find
  24. Pustejovsky, J.; Kordjamshidi, P.; Moens, M.-F.; Levine, A.; Dworman, S.; and Yocum, Z. 2015. SemEval-2015 Task 8: SpaceEval. In Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015). Denver, Colorado: Association for Computational Linguistics. find
  25. Santoro, A.; Raposo, D.; Barrett, D. G. T.; Malinowski, M.; Pascanu, R.; Battaglia, P. W.; and Lillicrap, T. 2017. A simple neural network module for relational reasoning. In Guyon, I.; von Luxburg, U.; Bengio, S.; Wallach, H. M.; Fergus, R.; Vishwanathan, S. V. N.; and Garnett, R., eds., Advances i… find
  26. Schapiro, A. C.; Turk-Browne, N. B.; Botvinick, M. M.; and Norman, K. A. 2017. Complementary learning systems within the hippocampus: a neural network modelling approach to reconciling episodic memory with statistical learning. Philosophical Transactions of the Royal Society B: Biological Sciences. find
  27. Schlag, I.; Munkhdalai, T.; and Schmidhuber, J. 2021. Learning Associative Inference Using Fast Weight Memory. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. find
  28. Schlag, I.; and Schmidhuber, J. 2018. Learning to Reason with Third Order Tensor Products. In Bengio, S.; Wallach, H. M.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; and Garnett, R., eds., Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Sys… find
  29. Smolensky, P. 1990. Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2): 159–216. find
  30. Sukhbaatar, S.; Szlam, A.; Weston, J.; and Fergus, R. 2015. End-To-End Memory Networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, 2440–2448. find
  31. Talmor, A.; and Berant, J. 2018. The Web as a Knowledge-Base for Answering Complex Questions. In Walker, M. A.; Ji, H.; and Stent, A., eds., Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018,… find
  32. Tan, H.; and Bansal, M. 2018. Source-Target Inference Models for Spatial Instruction Understanding. In McIlraith, S. A.; and Weinberger, K. Q., eds., Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (I… find
  33. AAAI Press. Tversky, B. 2019. Mind in motion: How action shapes thought. Hachette UK. van Aken, B.; Winter, B.; Löser, A.; and Gers, F. A. 2019. How Does BERT Answer Questions?: A Layer-Wise Analysis of Transformer Representations. In Proceedings of the 28th ACM International Conference on Informat… find
  34. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems. Velickovic, P.; Cucurull, G.; Casanova, A… find
  35. OpenReview.net. Vogel, A.; and Jurafsky, D. 2010. Learning to Follow Navigational Directions. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, 806–814. Uppsala, Sweden: Association for Computational Linguistics. find
  36. Welbl, J.; Stenetorp, P.; and Riedel, S. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6: 287–302. find
  37. Weston, J.; Bordes, A.; Chopra, S.; and Mikolov, T. 2016. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. In Bengio, Y.; and LeCun, Y., eds., 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedin… find our card
  38. Yang, T.-Y.; Lan, A.; and Narasimhan, K. 2020. Robust and Interpretable Grounding of Spatial References with Relation Networks. In Findings of the Association for Computational Linguistics: EMNLP 2020, 1908–1923. Online: Association for Computational Linguistics. find
  39. Yang, Z.; Qi, P.; Zhang, S.; Bengio, Y.; Cohen, W. W.; Salakhutdinov, R.; and Manning, C. D. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - Nov… find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/shi-2022-stepgame-new-benchmark-robust-multi/