Pragmatically Informative Text Generation

What it is about

Conditional generators often under-inform: a restaurant description may leave out attributes of its input meaning representation. The paper applies two pragmatic methods from the Rational Speech Acts framework to generation. The reconstructor method uses a listener that tries to recover the input from the output and rescores candidates. The distractor method makes the speaker, token by token, prefer words that separate the true input from a distractor. On the E2E restaurant task the reconstructor-based speaker raises BLEU from 66.52 to 68.60 over its base model and beats the best E2E challenge result on BLEU, NIST and CIDEr. On CNN/Daily Mail, both methods add 0.2 to 0.5 ROUGE points. A coverage ratio (how often each input value appears in the output) shows that much of the gain comes from mentioning more attributes.

Ideas

• I1. Coverage ratio as a metric: the fraction of input frame values found in the output, after lemmatisation with en/src/morph.rs, reported per attribute type beside BLEU.
• I2. Listener from parse-back: parse each candidate of en/src/lin.rs with the Mova pipeline, compare its frames with the input frames, and use the overlap as the listener term. The weight is a sweep parameter, tuned on dev only.
• I3. Distractor frames by masking: for a fact frame in the school world, build a distractor by masking one attribute, as the paper masks one meaning-representation attribute, and prefer answers that separate the two.
• I4. Negative control: a listener over shuffled frames must lower the coverage; determinism check by running twice.

For Mova

Chapters: 33. Implicature and presupposition (the quantity maxim), 30. Pragmatics and discourse: overview. Code: prag/src/rsa.rs (the RSA module), world/src/knowledge.rs (school facts), en/src/lin.rs (candidates). Measure: coverage ratio per attribute type on the dev lessons; the test set is used once per release.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1410 · arXiv 1904.01301

@inproceedings{shen2019pragmatically,
  title = {{Pragmatically Informative Text Generation}},
  author = {Shen, Sheng and Fried, Daniel and Andreas, Jacob and Klein, Dan},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1410},
  eprint = {1904.01301},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n19-1410}
}

References

38 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
2. Michael C Frank, Noah D Goodman, Peter Lai, and Joshua B Tenenbaum. 2009. Informative communication in word production and word learning. In Proceedings of the 31st annual conference of the cognitive science society, pages 1228–1233. find
3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. ↗
4. Daniel Fried, Jacob Andreas, and Dan Klein. 2018. Unified pragmatic models for generating and following instructions. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics. find
5. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics, pages 1662–1675, New Orleans, Louisiana. Association for Computational Linguistics. find
6. Dave Golland, Percy Liang, and Dan Klein. 2010. A game-theoretic approach to generating spatial descriptions. In Proceedings of the 2010 conference on Empirical Methods in Natural Language Processing, pages 410–419. Association for Computational Linguistics. find
7. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the Annual Meeting of the Association for Computational Linguistics. find
8. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 484– 494, Berlin, Germany. Association for Computational Linguistics. find
9. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 1724–1734, Doha, Qatar. Association for Computational Linguistics. find
10. Reuben Cohn-Gordon, Noah Goodman, and Chris Potts. 2018. Pragmatically informative image captioning with character-level reference. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics. find
11. George Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the second international conference on Human Language Technology Research, pages 138–145. Morgan Kaufmann Publishers Inc. find
12. Ondrej Dušek and Filip Jurčı́ček. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In ACL. find
13. Ondřej Dušek, Jekaterina Novikova, and Verena Rieser. 2018. Findings of the E2E NLG challenge. In Proceedings of the International Conference on Natural Language Generation. find
14. Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. find
15. Heng Gong. 2018. Technical report for E2E NLG challenge. In E2E NLG Challenge System Descriptions. find
16. Herbert P Grice. 1970. Logic and conversation. find
17. Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems. find
18. Laurence Horn. 1984. Toward a new taxonomy for pragmatic inference: Q-based and r-based implicature. Meaning, form, and use in context: Linguistic applications, 11:42. find
19. Juraj Juraska, Panagiotis Karagiannis, Kevin Bowden, and Marilyn Walker. 2018. A deep ensemble model with slot alignment for sequence-to-sequence natural language generation. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics, pages 152–162, New Orleans, Louisiana. Association for Computational Linguistics. find
20. Alon Lavie and Abhaya Agarwal. 2007. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228–231, Prague, Czech Republic. Association for Computational Linguistics. find
21. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL-04 Workshop. find
22. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. 2015. Generation and comprehension of unambiguous object descriptions. arXiv preprint arXiv:1511.02283. ↗
23. Will Monroe, Robert XD Hawkins, Noah D Goodman, and Christopher Potts. 2017. Colors in context: A pragmatic neural model for grounded language understanding. Transactions of the Association for Computational Linguistics. find
24. Will Monroe, Jennifer Hu, Andrew Jong, and Christopher Potts. 2018. Generating bilingual pragmatic color references. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics. find
25. Will Monroe and Christopher Potts. 2015. Learning in the Rational Speech Acts model. In Proceedings of 20th Amsterdam Colloquium, Amsterdam. ILLC. find
26. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proceedings of the Meeting of the Association for the Advancement of Artificial Intelligence. find
27. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In SIGNLL Conference on Computational Natural Language Learning, pages 280– 290, Berlin, Germany. Association for Computational Linguistics. find
28. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics. find
29. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. find
30. Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image description evaluation. In Proceedings of the Conference on Computer Vision and Pattern Recognition. find
31. Biao Zhang, Jing Yang, Qian Lin, and Jinsong Su. 2018. Attention regularized sequence-to-sequence learning for E2E NLG challenge. In E2E NLG Challenge System Descriptions. find
32. Ani Nenkova and Kathleen McKeown. 2011. Automatic summarization. Foundations and Trends in Information Retrieval, 5(2–3):103–233. find
33. Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the Annual SIGdial Meeting on Discourse and Dialogue. find
34. Naho Orita, Eliana Vornov, Naomi Feldman, and Hal Daumé III. 2015. Why discourse affects speakers’ choice of referring expressions. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), volume 1, pages 1639– 1649. find
35. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, pages 311–318. Association for Computational Linguistics. find our card
36. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proceedings of the International Conference on Learning Representations, volume abs/1705.04304. ↗
37. Yevgeniy Puzikov and Iryna Gurevych. 2018. E2E NLG challenge: Neural models vs. templates. In Proceedings of the International Conference on Natural Language Generation. find
38. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379–389, Lisbon, Portugal. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/shen-2019-pragmatically-informative-text-generation/