Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answering

What it is about

The case is over-informative answering: asked «Do you have iced tea?» in a café that has iced coffee, soda and Chardonnay, people answer «No, but we have iced coffee». A probabilistic RSA model explains this: a questioner picks a question to serve a goal (a decision problem), and the respondent infers the goal from the question and adds the alternative that is most useful for it; in a human study 43% of explanations for such answers appeal to the asker's goal and 20% to goal-relevant features. The paper keeps this model as a scaffold and lets an LLM (GPT-4o-mini, also Qwen-2.5-32B) fill single parts: utilities of options given a goal, truth of candidate answers, proposals of alternative answers, questions and goals, or larger chunks up to one chain-of-thought prompt. On 30 vignettes, models with the scaffold and LLM modules match or beat the hand-specified model in fit to human answer types (Jensen–Shannon divergence), while one monolithic prompt fits worse. LLMs are good at proposing and at rating utilities from world knowledge, and less reliable at the truth of open-ended answers.

Ideas

• I1. The paper's division of labour is Mova's: the LLM fills tables once, the symbolic model decides at run time. Apply it to over-informative answers in world/src/talk.rs and in the student's explanations: when the true answer to a yes/no question is no, add the remembered alternative that fills the same role and is closest in category («No, the fox did not get the grapes; he went away hungry»; «No, the crow had cheese, not bread»). Closeness from the existing category tables (global ontology levels), not from an LLM call at answer time; a utility table per category, if needed, written once by the teacher and reviewed.
• I2. Add the paper's game as a small exact case in prag/src/rsa.rs: questions and answers over a handful of options, goals as decision problems with integer utilities, respondent L1 choosing the answer with the highest expected utility for the inferred goal. Test: with a goal that prefers cold drinks, «no, but iced coffee» wins over «no, but Chardonnay»; with a uniform goal, the plain «no» is not worse (negative control).
• I3. When the teacher is used to fill a table (utilities, alternatives), keep its proposals as data with provenance and check a sample by hand: the paper finds LLM proposals plausible but narrower than human ones (less social goals), which matters for tales where goals are often social.

For Mova

33. Implicature and presupposition §9 (RSA), 38-dialogue-and-discourse-markers.md §9 (questions and answers). Code: world/src/talk.rs, world/src/knowledge.rs (explanations), prag/src/rsa.rs. Measures: hand-computed RSA tests; share of no answers in a scripted dialogue test that carry a relevant alternative.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2506.01474

@inproceedings{tsvilodub2025integrating,
  title = {{Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answering}},
  author = {Tsvilodub, Polina and Hawkins, Robert D. and Franke, Michael},
  booktitle = {Proceedings of the Society for Computation in Linguistics 2025},
  year = {2025},
  eprint = {2506.01474},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.01474}
}

References

28 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Noah D Goodman and Andreas Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org. Accessed: 2025-1-30. ↗
2. Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Ger-stenberg, and Noah Goodman. 2024. STar-GATE: Teaching language models to ask clarifying questions. In First Conference on Language Modeling. find
3. Jeroen Antonius Gerardus Groenendijk and Martin Johan Bastiaan Stokhof. 1984. Studies on the Semantics of Questions and the Pragmatics of Answers. Ph.D. thesis, Univ. Amsterdam. find
4. Sebastian Bader, Pascal Hitzler, and Steffen Hoell-dobler. 2004. The integration of connectionism and first-order knowledge representation and reasoning as a challenge for artificial intelligence. Preprint, arXiv:cs/0408069. ↗
5. Auli Hakulinen. 2001. Minimal and non-minimal answers to yes-no questions. Pragmatics, 11(1):1–15. find
6. Anna Bavaresco, Raffaella Bernardi, Leonardo Berto-lazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael CL Hamblin. 1973. Questions in Montague English. Foundations of Language, 10(1):41–53. find
7. Robert D. Hawkins and Noah D. Goodman. 2017. Why do you ask? The informational dynamics of questions and answers. PsyArXiv. find
8. Lea Löhn, Niklas Kiehne, Alexander Ljapunov, and Wolf-Tilo Balke. 2024. Is machine psychology here? On requirements for using human psychological tests on large language models. In Proceedings of the 17th International Natural Language Generation Conference, pages 230–242, Tokyo, Japan. Association for Computational Linguistics. find
9. Robert D. Hawkins, Andreas Stuhlmüller, Judith Degen, and Noah D. Goodman. 2015. Why do you ask? Good questions provoke informative answers. Cognitive Science. Robert D. Hawkins, Polina Tsvilodub, Claire Augusta Bergey, Noah D. Goodman, and Michael Franke. to appear. Relevant answers to polar questions. Philosophical Transactions B. find
10. Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. 2021. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. Advances in Neural Information Processing Systems, 34:25192– 25204. find
11. Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D. Goodman. 2023. Solving math word problems by combining language models with symbolic solvers. arXiv preprint arXiv:2304.09102. ↗
12. Kathryn Pruitt and Floris Roelofsen. 2011. Disjuntive questions: Prosody, syntax, and semantics. Handout, Göttingen. find
13. Ben Prystawski, Paul Thibodeau, Christopher Potts, and Noah Goodman. 2023. Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45. find
14. Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023. A fine-grained comparison of pragmatic language understanding in humans and language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4194–4213, Toronto, Canada. Association for Computational Linguistics. find our card
15. Anselm Rothe, Brenden M Lake, and Todd Gureckis. 2017. Question asking as program generation. Advances in neural information processing systems, 30. find
16. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12). find
17. Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rocktäschel, and Edward Grefenstette. 2023. The goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicature resolution by LLMs. find
18. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bham-bri, Lucas Saldyt, and Anil Murthy. 2024. LLMs can’t plan, but can help planning in LLM-Modulo frameworks. Preprint, arXiv:2402.01817. ↗
19. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Preprint, arXiv:2302.04761. ↗
20. Alexander K Lew, Michael Henry Tessler, Vikash K Mansinghka, and Joshua B Tenenbaum. 2020. Leveraging unstructured statistical knowledge in a probabilistic language of thought. In Proceedings of the annual conference of the cognitive science society. find
21. Richard Shiffrin and Melanie Mitchell. 2023. Probing the psychology of AI models. Proceedings of the National Academy of Sciences, 120(10):e2300963120. find
22. Alexander K. Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K. Mansinghka. 2023. Sequential Monte Carlo steering of large language models using probabilistic programs. Preprint, arXiv:2306.03081. ↗
23. Jon Scott Stevens, Anton Benz, Sebastian Reuße, and Ralf Klabunde. 2016. Pragmatic question answering: A game-theoretic approach. Data & Knowledge Engineering, 106:52–69. find
24. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33:9459– 9474. find
25. Theodore R Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. 2023. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427. ↗
26. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. Preprint, arXiv:2307.03172. ↗
27. Polina Tsvilodub, Michael Franke, and Fausto Carcassi. 2024a. Cognitive modeling with scaffolded LLMs: A case study of referential expression generation. In ICML 2024 Workshop on LLMs and Cognition. find
28. Qwen Team. 2024. Qwen2.5: A party of foundation models. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/tsvilodub-2025-integrating-neural-symbolic-components-model/