GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning

What it is about

GRICE is a synthetic, bAbI-style dialogue dataset. In a small simulated world of agents, objects and rooms, Alice asks questions, and Bob answers with replies that carry one of five implicature types. Relevance: «Did you see the apples?» — «There is a basket in the dining room». Strengthening: «All of them are there». Limiting: «Some are» → not all. Ignorance: «in the kitchen or the bedroom» → the speaker does not know which. Close-but: «Did you put them there?» — «I was in the living room» → no. Dialogues are generated by a temporal And-Or grammar with synonym variation, pronouns and elliptical follow-ups («What about the apples?»). There are two tasks: choose the explicit paraphrase of an answer among distractors (wrong coreferent, irrelevant true fact, wrong fact, near miss) and answer bAbI-like questions about the world, including belief questions. Memory networks do well on recovery (97% R@1) but only 67% on reasoning against human 97.5%; an explicit recovery module adds about 5 points.

Ideas

• I1. Add the ignorance implicature to prag/src/infer.rs. A disjunction «X or Y» in an assertion (not under a question or a conditional) implicates that the speaker does not know which disjunct holds, with the or dependency as the trigger in the explanation. The scale «or < and» is already in SCALES; this is the clausal (epistemic) half that is missing.
• I2. Teach the QUD stack (world/src/qud.rs) the close-but and relevance answers to yes/no questions. If the question asks «did X do P at L?» and the answer places X at a different location L′ at that time, then the answer is «no», with the explanation «X was at L′, not L». This needs only the place role of frames and a disjointness check on locations, which bAbI already uses in world/src/babi.rs.
• I3. Build our own generator in the GRICE style on top of the bAbI world code. Generate question–indirect answer pairs with each of the five implicature types, each labelled with the intended explicit form. This gives a fully licensed, red-capable test set for infer and qud: an answer with no implicature must yield no inference. Measure recovery accuracy per type.
• I4. The distractor recipe (wrong coreferent, irrelevant but true fact, contradicting fact, near miss) is a good template for the school-course option sets. It also maps onto the failure steps of world knowledge diagnose (step 4 vs step 5): one could report the student's errors by distractor type.

For Mova

33. Implicature and presupposition §3–5 (kinds of implicature, scalar and other Quantity inferences) and §10 (the inference module), 38. Dialogue, discourse markers and reported speech §9 (questions and answers in dialogue); prag/src/infer.rs, world/src/qud.rs, world/src/babi.rs; measures: per-type recovery accuracy on the generated set, with a no-implicature control.

Topics: Implicature

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-acl.182

@inproceedings{zheng2021grice,
  title = {{GRICE: A Grammar-based Dataset for Recovering Implicature and Conversational rEasoning}},
  author = {Zheng, Zilong and Qiu, Shuwen and Fan, Lifeng and Zhu, Yixin and Zhu, Song-Chun},
  booktitle = {Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.182},
  url = {https://doi.org/10.18653/v1/2021.findings-acl.182}
}

References

72 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gregory Bateson. 2000. Steps to an ecology of mind: Collected essays in anthropology, psychiatry, evolution, and epistemology. University of Chicago Press. find
2. Robert H Bonczek, Clyde W Holsapple, and Andrew B Whinston. 1979. Computer-based support of organizational decision making. Decision Sciences, 10(2):268–291. find
3. Robert H Bonczek, Clyde W Holsapple, and Andrew B Whinston. 1981. A generalized decision support system using predicate calculus and network data base management. Operations Research, 29(2):263–281. find
4. Emma Borg. 2009. On three theories of implicature: Default theory, relevance theory and minimalism. International Review of Pragmatics, 1(1):63–83. find
5. Mark Edmonds, Feng Kubricht, James, Colin Summers, Yixin Zhu, Brandon Rothrock, Song-Chun Zhu, and Hongjing Lu. 2018. Human causal transfer: Challenges for deep reinforcement learning. In Proceedings of the Annual Meeting of the Cognitive Science Society (CogSci). find
6. Mark Edmonds, Xiaojian Ma, Siyuan Qi, Yixin Zhu, Hongjing Lu, and Song-Chun Zhu. 2020. Theory-based causal transfer: Integrating instance-level induction and abstract-level structure learning. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI). find
7. Mark Edmonds, Siyuan Qi, Yixin Zhu, James Kubricht, Song-Chun Zhu, and Hongjing Lu. 2019b. Decomposing human causal learning: Bottom-up associative learning and top-down schema reasoning. In Proceedings of the Annual Meeting of the Cognitive Science Society (CogSci). find
8. Henry Y Chen, Ethan Zhou, and Jinho D Choi. 2017. Robust coreference resolution and entity linking on dialogues: Character identification on tv show transcripts. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL). find
9. Victor Escandell. 1996. Introducción a la pragmática. Ariel Linguistica. españa. 6ta. find
10. Leyang Cui, Yu Wu, Shujie Liu, Yue Zhang, and Ming Zhou. 2020. Mutual: A dataset for multi-turn dialogue reasoning. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
11. Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. find
12. Robert Dale and Ehud Reiter. 1995. Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive science, 19(2):233–263. find our card
13. Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
14. Wayne A Davis. 2016. Implicature. In Irregular Negatives, Implicatures, and Idioms, pages 51–84. Springer. find
15. Anita Fetzer. 2017. Context. In The Oxford handbook of pragmatics. Oxford University Press. find
16. Noah D Goodman and Michael C Frank. 2016. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(11):818–829. find
17. Noah D Goodman and Andreas Stuhlmüller. 2013. Knowledge and implicature: Modeling language understanding as social cognition. Topics in cognitive science, 5(1):173–184. find
18. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. find
19. Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. 2017. Tracking the world state with recurrent entity networks. International Conference on Learning Representations (ICLR). find
20. Gerard De Melo and Mohit Bansal. 2013. Good, great, excellent: Global inference of semantic intensities. Transactions of the Association for Computational Linguistics (TACL), 1:279–290. find
21. Julia Linn Bell Hirschberg. 1985. A theory of scalar implicature. University of Pennsylvania. find
22. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 4171–4186. Association for Computational Linguistics. find
23. Yan Huang. 2017. The Oxford handbook of pragmatics. Oxford University Press. find
24. Mark Edmonds, Feng Gao, Hangxin Liu, Xu Xie, Siyuan Qi, Brandon Rothrock, Yixin Zhu, Ying Nian Wu, Hongjing Lu, and Song-Chun Zhu. 2019a. A tale of two explanations: Enhancing human trust by explaining robot behavior. Science Robotics, 4(37). find
25. Laurence R Horn and Gregory L Ward. 2004. The handbook of pragmatics. Wiley Online Library. find
26. Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan, and Adina Williams. 2020. Are natural language inference models imppressive? learning implicature and presupposition. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find our card
27. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. ↗
28. Kepa Korta and John Perry. 2020. Pragmatics. In Edward N. Zalta, editor, The Stanford Encyclopedia of Philosophy, spring 2020 edition. Metaphysics Research Lab, Stanford University. find
29. Satwik Kottur, José M. F. Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2018. Visual coreference resolution in visual dialog using neural module networks. In Proceedings of European Conference on Computer Vision (ECCV). find
30. Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338. find
31. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In International Conference on the Principles of Knowledge Representation and Reasoning. find
32. Stephen C Levinson. 1985. Pragmatics. Cambridge Univ. Press. find
33. Stephen C Levinson. 1995. Interactional biases in human thinking. In Social intelligence and interaction, pages 221–260. Cambridge University Press. find
34. Qing Li, Siyuan Huang, Yining Hong, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. 2021. A hint from arithmetic: On systematic generalization of perception, syntax, and semantics. arXiv preprint arXiv:2103.01403. ↗
35. Ryan Lowe, Nissan Pow, Iulian Vlad Serban, and Joelle Pineau. 2015. The ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems. In Annual Meeting of the Special Interest Group on Discourse and Dialogue. find
36. Marie-Catherine de Marneffe, Christopher D Manning, and Christopher Potts. 2010. Was it good? it was provocative. learning the meaning of scalar adjectives. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find our card
37. Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019. Opendialkg: Explainable conversational reasoning with attention-based walks over knowledge graphs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
38. Aida Nematzadeh, Kaylee Burns, Erin Grant, Alison Gopnik, and Tom Griffiths. 2018. Evaluating theory of mind in question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
39. Judea Pearl. 1984. Intelligent search strategies for computer problem solving. Addision Wesley. find
40. Siyuan Qi, Baoxiong Jia, and Song-Chun Zhu. 2020. Generalized earley parser: Bridging symbolic grammars and sequence data for future prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). find
41. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
42. Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics (TACL), 7:249–266. find
43. Bertrand Russell. 1903. The Principles of Mathematics. Cambridge University Press. find
44. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI). find
45. Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. 2017. A simple neural network module for relational reasoning. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). find
46. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
47. Rebecca Saxe. 2006. Uniquely human social cognition. Current opinion in neurobiology, 16(2):235–239. find
48. Heung-Yeung Shum, Xiao-dong He, and Di Li. 2018. From eliza to xiaoice: challenges and opportunities with social chatbots. Frontiers of Information Technology & Electronic Engineering, 19(1):10–26. find
49. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI). find
50. Dan Sperber and Deirdre Wilson. 1986. Relevance: Communication and cognition. Harvard University Press Cambridge, MA. find
51. Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge dataset and models for dialogue-based reading comprehension. Transactions of the Association for Computational Linguistics (TACL), 7:217–231. find
52. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
53. Kewei Tu, Maria Pavlovskaia, and Song-Chun Zhu. 2013. Unsupervised structure learning of stochastic and-or grammars. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). find
54. Tony Veale and Yanfen Hao. 2010. Detecting ironic intent in creative comparisons. In European Conference on Artificial Intelligence (ECAI). find
55. Ellen M Voorhees et al. 1999. The trec-8 question answering track report. In Trec. find
56. Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. 2016. Towards ai-complete question answering: A set of prerequisite toy tasks. International Conference on Learning Representations (ICLR). find our card
57. Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. International Conference on Learning Representations (ICLR). find
58. Ludwig Wittgenstein. 1953. Philosophical investigations. Philosophische Untersuchungen. Macmillan. find
59. Ludwig Wittgenstein. 1969. The blue and brown books, volume 958. Blackwell Oxford. find
60. Qi Wu, Peng Wang, Chunhua Shen, Ian Reid, and Anton Van Den Hengel. 2018. Are you talking to me? reasoned visual dialog generation through adversarial learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
61. Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. 2017. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
62. Chi Zhang, Baoxiong Jia, Song-Chun Zhu, and Yixin Zhu. 2021b. Abstract spatial-temporal reasoning via probabilistic abduction and execution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
63. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL). find
64. Wenhe Zhang, Chi Zhang, Yixin Zhu, and Song-Chun Zhu. 2020. Machine number sense: A dataset of visual arithmetic problems for abstract and relational reasoning. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI). find
65. Zilong Zheng, Wenguan Wang, Siyuan Qi, and Song-Chun Zhu. 2019. Reasoning visual dialogs with structural and partial observations. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
66. Song-Chun Zhu and David Mumford. 2007. A stochastic grammar of images. Foundations and Trends® in Computer Graphics and Vision, 2(4):259–362. find
67. Yixin Zhu, Tao Gao, Lifeng Fan, Siyuan Huang, Mark Edmonds, Hangxin Liu, Feng Gao, Chi Zhang, Siyuan Qi, Ying Nian Wu, Josh Tenenbaum, and Song-Chun Zhu. 2020. Dark, beyond deep: A paradigm shift to cognitive ai with humanlike common sense. Engineering, 6(3):310–345. find
68. Sirui Xie, Xiaojian Ma, Peiyu Yu, Yixin Zhu, Ying Nian Wu, and Song-Chun Zhu. 2021. Halma: Humanlike abstraction learning meets affordance in rapid problem solving. arXiv preprint arXiv:2102.11344. ↗
69. Tom Young, Erik Cambria, Iti Chaturvedi, Hao Zhou, Subham Biswas, and Minlie Huang. 2018. Augmenting end-to-end dialogue systems with commonsense knowledge. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI). find
70. Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. 2019a. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
71. Chi Zhang, Baoxiong Jia, Mark Edmonds, Song-Chun Zhu, and Yixin Zhu. 2021a. Acre: Abstract causal reasoning beyond covariation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
72. Chi Zhang, Baoxiong Jia, Feng Gao, Yixin Zhu, Hongjing Lu, and Song-Chun Zhu. 2019b. Learning perceptual inference by contrasting. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS). find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/zheng-2021-grice-grammar-based-dataset-recovering/