Mova0.5 pre-release⌕
☰
home / docs / papers / A Corpus for Reasoning About Natural Language Grounded in Photographs (NLVR2)

A Corpus for Reasoning About Natural Language Grounded in Photographs (NLVR2)

Summary

The paper introduces NLVR2, a dataset for visual reasoning in which a model receives a pair of real photographs and an English sentence and must decide whether the sentence is true of the pair. It follows the earlier NLVR dataset, which used synthetic images of coloured geometric shapes, and moves to natural photographs to get richer visual and linguistic variety. Sentences were written by crowdworkers who were shown contrasting image pairs, which encourages statements involving counting, comparison, quantifiers, spatial relations and set reasoning. The collection procedure is designed to reduce simple language-only biases. Baseline and then state-of-the-art vision-and-language models perform well below human accuracy, showing that compositional grounded reasoning was still hard.

How Mova uses it

Effectiveness in Mova

Not measured separately, and Mova does not evaluate on NLVR or NLVR2. The scene seed it informs is part of the SpartQA-human reader, which scores 59.3% on test (majority baseline 34.9%) per Mova's development notes.

👨‍🔬💥

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 1811.00491

@misc{suhr2018corpus,
  title = {{A Corpus for Reasoning About Natural Language Grounded in Photographs (NLVR2)}},
  author = {Suhr, Alane and Zhou, Stephanie and Zhang, Ally and Zhang, Iris and Bai, Huajun and Artzi, Yoav},
  year = {2018},
  eprint = {1811.00491},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1811.00491}
}

References

65 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Manoj Acharya, Kushal Kafle, and Christopher Kanan. 2019. TallyQA: Answering complex counting questions. In AAAI Conference on Artificial Intelligence. find
  2. Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don’t just assume; look and answer: Overcoming priors for visual question answering. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4971–4980. find
  3. Aishwarya Agrawal, Aniruddha Kembhavi, Dhruv Batra, and Devi Parikh. 2017. C-VQA: A compositional split of the visual question answering (VQA) v1.0 dataset. CoRR, abs/1704.08243. — [arXiv:1704.08243](https://arxiv.org/abs/1704.08243) find
  4. Malihe Alikhani and Matthew Stone. 2019. "Caption" as a coherence relation: Evidence and implications. In Proceedings of the Workshop on Shortcomings in Vision and Language. find
  5. Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton van den Hengel. 2018. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In IEEE Conference on Computer Vision and Pattern R… find
  6. Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Neural module networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 39–48. find
  7. Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In IEEE International Conference on Computer Vision, pages 2425–2433. find
  8. Yonatan Bisk, Daniel Marcu, and William Wong. 2016. Towards a dataset for human computer communication via grounded language acquisition. In Proceedings of the AAAI Workshop on Symbiotic Cognitive Systems. find
  9. Howard Chen, Alane Suhr, Dipendra Misra, Noah Snavely, and Yoav Artzi. 2019. Touchdown: Natural language navigation and spatial reasoning in visual street environments. In IEEE Conference on Computer Vision and Pattern Recognition. find
  10. Wenhu Chen, Aurélien Lucchi, and Thomas Hofmann. 2016. Bootstrap, review, decode: Using out-of-domain textual data to improve image captioning. CoRR, abs/1611.05321. — [arXiv:1611.05321](https://arxiv.org/abs/1611.05321) find
  11. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325. find
  12. Volkan Cirik, Taylor Berg-Kirkpatrick, and Louis-Philippe Morency. 2018. Using syntax to ground referring expressions in natural images. In AAAI Conference on Artificial Intelligence. find
  13. Anne Cocos, Aaron Masino, Ting Qian, Ellie Pavlick, and Chris Callison-Burch. 2015. Effectively crowdsourcing radiology report annotations. In Proceedings of the Sixth International Workshop on Health Text Mining and Information Analysis, pages 109– 114. find
  14. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José M.F. Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1080–1089. find
  15. Jia Deng, Olga Russakovsky, Jonathan Krause, Michael S. Bernstein, Alex Berg, and Li Fei-Fei. 2014. Scalable multi-label annotation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 3099–3102. find
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page… find
  17. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14:179–211. find
  18. Francis Ferraro, Nasrin Mostafazadeh, Ting-Hao Huang, Lucy Vanderwende, Jacob Devlin, Michel Galley, and Margaret Mitchell. 2015. A survey of current datasets for vision and language research. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 207– 213. find
  19. Nicholas FitzGerald, Yoav Artzi, and Luke Zettlemoyer. 2013. Learning distributions over logical forms for referring expression generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1914–1925. find
  20. Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. 2018. Detectron. https://github.com/facebookresearch/detectron. ↗
  21. Omer Goldman, Veronica Latcinnik, Ehud Nave, Amir Globerson, and Jonathan Berant. 2018. Weakly supervised semantic parsing with abstract examples. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1809– 1819. find
  22. Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in visual question answering. In IEEE Conference on Computer Vision and Pattern Recognition, pages 6325–6334. find
  23. Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. 2017. Mask R-CNN. In IEEE International Conference on Computer Vision, pages 2980–2988. find
  24. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778. find
  25. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9. find
  26. Hexiang Hu, Ishan Misra, and Laurens van der Maaten. 2019. Binary image selection (BISON): Interpretable evaluation of visual grounding. CoRR, abs/1901.06595. — [arXiv:1901.06595](https://arxiv.org/abs/1901.06595) find
  27. Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. 2018. Explainable neural computation via stack neural module networks. In European Conference on Computer Vision. find
  28. Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In IEEE International Conference on Computer Vision, pages 804–813. find
  29. Drew A. Hudson and Christopher D. Manning. 2018. Compositional attention networks for machine reasoning. In Proceedings of the International Conference on Learning Representations. find
  30. Drew A. Hudson and Christopher D. Manning. 2019. GQA: a new dataset for compositional question answering over real-world images. In IEEE Conference on Computer Vision and Pattern Recognition. find
  31. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017a. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1988–1997. find
  32. Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C. Lawrence Zitnick, and Ross B. Girshick. 2017b. Inferring and executing programs for visual reasoning. In IEEE International Conference on Computer Vision, pages 3008–3017. find
  33. Kushal Kafle and Christopher Kanan. 2017. An analysis of visual question answering algorithms. In Proceedings of the IEEE International Conference on Computer Vision, pages 1965–1973. find
  34. Samira Ebrahimi Kahou, Adam Atkinson, Vincent Michalski, Ákos Kádár, Adam Trischler, and Yoshua Bengio. 2018. FigureQA: An annotated figure dataset for visual reasoning. In Proceedings of the International Conference on Learning Representations. find
  35. Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. ReferItGame: Referring to objects in photographs of natural scenes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 787–798. find
  36. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations. find
  37. Alexander Kuhnle and Ann A. Copestake. 2017. ShapeWorld - a new test methodology for multimodal language understanding. CoRR, abs/1704.04517. — [arXiv:1704.04517](https://arxiv.org/abs/1704.04517) find
  38. J. Richard Landis and Gary Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33 1:159–74. find
  39. Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. 2018. TVQA: Localized, compositional video question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1369–1379. Association for Computational Linguistics. find
  40. Yining Li, Chen Huang, Xiaoou Tang, and Chen Change Loy. 2017. Learning to disambiguate by asking discriminative questions. In IEEE International Conference on Computer Vision, pages 3439–3448. find
  41. Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common objects in context. In European Conference on Computer Vision. find
  42. Matthew MacMahon, Brian Stankiewics, and Benjamin Kuipers. 2006. Walk the talk: Connecting language, knowledge, action in route instructions. In Proceedings of the National Conference on Artificial Intelligence. find
  43. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In IEEE Conference on Computer Vision and Pattern Recognition, pages 11–20. find
  44. Cynthia Matuszek, Nicholas FitzGerald, Luke S. Zettlemoyer, Liefeng Bo, and Dieter Fox. 2012. A joint model of language and perception for grounded attribute learning. In Proceedings of the International Conference on Machine Learning. find
  45. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. CoRR, abs/1301.3781. find
  46. George A. Miller. 1993. WordNet: A lexical database for English. In Proceedings of the Workshop on Human Language Technology, pages 409–409. find our card
  47. Dipendra Misra, Andrew Bennett, Valts Blukis, Eyvind Niklasson, Max Shatkhin, and Yoav Artzi. 2018. Mapping instructions to actions in 3D environments with visual goal prediction. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find
  48. Margaret Mitchell, Kees van Deemter, and Ehud Reiter. 2010. Natural reference to objects in a visual domain. In Proceedings of the International Natural Language Generation Conference. find
  49. Juan Pavez, Hector Alllende, and Hector Allende-Cid. 2018. Working memory networks: Augmenting memory networks with a relational reasoning module. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1000–1009. find
  50. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1532–1543. find
  51. Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. 2018. FiLM: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence. find
  52. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. 2015. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 115(3):211–252. find
  53. Adam Santoro, David Raposo, David G.T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural network module for relational reasoning. In Advances in Neural Information Processing Systems, pages 4967–4976. find
  54. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556. find
  55. Joseph Suarez, Justin Johnson, and Fei-Fei Li. 2018. DDRprog: A CLEVR differentiable dynamic reasoning programmer. CoRR, abs/1803.11361. — [arXiv:1803.11361](https://arxiv.org/abs/1803.11361) find
  56. Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. A corpus of natural language for visual reasoning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 217–223. find
  57. Hao Tan and Mohit Bansal. 2018. Object ordering with bidirectional matchings for visual reasoning. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 444–451. find
  58. Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. 2018. Talk the Walk: Navigating New York City through grounded dialogue. CoRR, abs/1807.03367. — [arXiv:1807.03367](https://arxiv.org/abs/1807.03367) find
  59. Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019. Visual entailment: A novel task for fine-grained image understanding. CoRR, abs/1901.06706. — [arXiv:1901.06706](https://arxiv.org/abs/1901.06706) find
  60. Robert Guangyu Yang, Igor Ganichev, Xiao Jing Wang, Jonathon Shlens, and David Sussillo. 2018. A dataset and architecture for visual reasoning with a working memory. In European Conference on Computer Vision. find
  61. Yiqun Yao, Jiaming Xu, Feng Wang, and Bo Xu. 2018. Cascaded mutual modulation for visual reasoning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 975–980. Association for Computational Linguistics. find
  62. Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural-symbolic VQA: Disentangling reasoning from vision and language understanding. In Advances in Neural Information Processing Systems, pages 1031–1042. find
  63. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In IEEE Conference on Computer Vision and Pattern Recognition. find
  64. Peng Zhang, Yash Goyal, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2016. Yin and yang: Balancing and answering binary visual questions. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5014–5022. find
  65. C. Lawrence Zitnick and Devi Parikh. 2013. Bringing semantics into focus using visual abstraction. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3009–3016. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/suhr-2019-corpus-reasoning-natural-language-grounded/