Towards Pragmatic Production Strategies for Natural Language Generation Tasks

What it is about

This position paper proposes a framework for NLG as decision-making. A speaker chooses an utterance to reach a communicative goal, balancing production costs (encoding and realisation), comprehension costs (the hearer's processing and decoding) and utility. The notions are defined formally and applied in two case studies: a visually grounded reference game, where speaker and listener hold distributions over candidate images, and abstractive summarisation, where utility is the audience's accuracy in answering questions about the document. The author argues that Rational Speech Acts models and utility-based decoding such as minimum Bayes risk point in this direction, and reads Grice's maxims as cost trade-offs. No computational experiments are reported, and the paper says so in its limitations.

Ideas

• I1. Make the costs countable: realisation cost is the word count of a candidate; processing cost the number of tree arcs the hearer must attach (from the Mova tree); decoding cost the number of frame slots left unresolved. Each is a deterministic count, and utility is a lookup over a fixed question set.
• I2. Use these costs to choose among the realisations of one frame set in en/src/lin.rs; compare the choice with the current default order.
• I3. Treat the maxim readings as predictions: an answer with extra information should gain utility only when the extra slots answer a likely question. Check this on the quiz questions in world/src/qa.rs.
• I4. Negative control: a random choice among the same candidates must score lower on utility than the cost-based choice.

For Mova

Chapters: 30. Pragmatics and discourse: overview, 33. Implicature and presupposition. Code: prag/src/rsa.rs (the RSA module), en/src/lin.rs (candidates), world/src/qa.rs (questions). Measure: utility of the chosen realisations on the quiz questions, against the random-choice control; two runs must match. These are hypotheses until measured.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.emnlp-main.544 · arXiv 2210.12828

@inproceedings{giulianelli2022pragmatic,
  title = {{Towards Pragmatic Production Strategies for Natural Language Generation Tasks}},
  author = {Giulianelli, Mario},
  booktitle = {Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.544},
  eprint = {2210.12828},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.emnlp-main.544}
}

References

32 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Michele Banko, Vibhu O Mittal, and Michael J Witbrock. 2000. Headline generation based on statistical translation. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 318–325. find
2. Susan E. Brennan and Herbert H. Clark. 1996. Conceptual pacts and lexical choice in conversation. Journal of Experimental Psychology: Learning, Memory, and Cognition, 22:1482–1493. find
3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc. find
4. Sarah Brown-Schmidt, Si On Yoon, and Rachel Anna Ryskin. 2015. People as contexts in conversation. In Psychology of learning and motivation, volume 62, pages 59–99. Elsevier. find
5. Herbert H. Clark and Edward F. Schaefer. 1989. Contributing to discourse. Cognitive Science, 13(2):259– 294. find
6. Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22(1):1– 39. find
7. Louise Connell and Dermot Lynott. 2014. Principles of representation: Why you can’t represent the same concept twice. Topics in Cognitive Science. find
8. Gerald DeJong. 1982. An overview of the FRUMP system. Strategies for Natural Language Processing, 113:149–176. find
9. Harold P Edmundson. 1969. New methods in automatic extracting. Journal of the ACM (JACM), 16(2):264– 285. find
10. Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. find
11. Michael Franke, Gerhard Jäger, and Robert van Rooij. 2010. Vagueness, signaling and bounded rationality. In JSAI international symposium on artificial intelligence, pages 45–59. Springer. find
12. Vaibhava Goel and William J Byrne. 2000. Minimum Bayes-risk automatic speech recognition. Computer Speech and Language, 14(2):115–135. find
13. Herbert P Grice. 1975. Logic and conversation. In Speech acts, pages 41–58. Brill. find
14. Janosch Haber, Tim Baumgärtner, Ece Takmaz, Lieke Gelderloos, Elia Bruni, and Raquel Fernández. 2019. The PhotoBook dataset: Building common ground through visually-grounded dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1895–1910. find
15. Robert D Hawkins, Michael C Frank, and Noah D Goodman. 2020. Characterizing the dynamics of learning in repeated reference games. Cognitive science, 44(6):e12845. find
16. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The Curious Case of Neural Text Degeneration. In International Conference on Learning Representations. find
17. T Florian Jaeger and Harry Tily. 2011. On language ‘utility’: Processing complexity and communicative efficiency. Wiley Interdisciplinary Reviews: Cognitive Science, 2(3):323–335. find
18. Emiel Krahmer and Kees van Deemter. 2012. Computational generation of referring expressions: A survey. Computational Linguistics, 38(1):173–218. find
19. Robert M Krauss and Sidney Weinheimer. 1964. Changes in reference phrases as a function of frequency of usage in social interaction: A preliminary study. Psychonomic Science, 1(1):113–114. find
20. Shankar Kumar and Bill Byrne. 2002. Minimum Bayes-risk word alignments of bilingual texts. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 140–147. find
21. Shankar Kumar and William Byrne. 2004. Minimum Bayes-risk decoding for statistical machine translation. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 169–176. find
22. Hans Peter Luhn. 1958. The automatic creation of literature abstracts. IBM Journal of Research and Development, 2(2):159–165. find
23. David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and Brain Sciences, 1(4):515–526. find
24. Raj Reddy. 1977. Speech understanding systems: A summary of results of the five-year research effort at Carnegie Mellon University. find
25. Ehud Reiter and Robert Dale. 1997. Building applied natural language generation systems. Natural Language Engineering, 3(1):57–87. find
26. Claude E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423. find
27. Todd Shore, Theofronia Androulakaki, and Gabriel Skantze. 2018. KTH tangrams: A dataset for research on alignment and conceptual pacts in task-oriented dialogue. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
28. Herbert A Simon. 1990. Bounded rationality. In Utility and probability, pages 15–18. Springer. find
29. Ted Sumers, Robert Hawkins, Mark K Ho, and Tom Griffiths. 2021. Extending rational models of communication from beliefs to actions. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 43. find
30. Michael Tomasello. 2005. Constructing a language: A usage-based theory of language acquisition. Harvard university press. find
31. Noga Zaslavsky, Jennifer Hu, and Roger P. Levy. 2021. A Rate–Distortion view of human pragmatic reasoning? In Proceedings of the Society for Computation in Linguistics 2021, pages 347–348, Online. Association for Computational Linguistics. find
32. G.K. Zipf. 1949. Human Behavior and the Principle of Least Effort: An Introduction to Human Ecology. Addison-Wesley Press. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/giulianelli-2022-pragmatic-production-strategies-natural-language/