Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories

What it is about

A speaker must refer to an object in an image whose category the speaker does not know, for example a bus when only cars were seen in training. A neural literal speaker trained on known categories tends to name the wrong category. The paper adds a pragmatic decoder in the style of Rational Speech Acts: the listener keeps a belief over the object's category, and the speaker chooses each word by a score combining the literal speaker and the listener, with weights that favour words not yet used. Uncertainty is set to uniform over the categories. Six medium-frequency categories (cat, horse, cup, bottle, bus, train) are held out from the RefCOCO training data. In Table 1, the share of expressions that contain a wrong category noun falls for all six categories: for example from 39.8% to 16.6% for bottle and from 62.7% to 33.2% for cup. The share of expressions with no noun at all rises, from 27.5% to 56.2% for bottle. The reference-resolution test uses a listener that knows the true categories, to avoid training the evaluator on the same mistakes. The results are mixed across the two test sets; the authors call them positive evidence for communicative success.

Ideas

• I1. prag/src/rsa.rs: a greedy word-level pragmatic speaker with two fixed weights, one for the literal score and one for the listener score, and a penalty that grows for words already used in the sentence. Keep the score table in a BTreeMap and check that two runs give the same text.
• I2. Hedging for unknown words: when a noun in the input is outside the dictionary (en/src/dict.rs), prefer a general word ("thing") plus attributes, and count how often a wrong category name is produced for such objects in the school texts.
• I3. Evaluate with a listener that knows the true categories (a rule-based checker such as coref/src/eval.rs style scoring, not a model trained on the same text), as the paper does. This avoids the circularity the authors describe. Report resolution accuracy with the reference speaker as a negative control.
• I4. Build a zero-shot split for the school lessons: hold out one set of object words from the training lessons, and test whether the answer avoids them. Split by lesson, not by sentence, and report on the held-out lessons only.

For Mova

The pragmatic speaker belongs with prag/src/rsa.rs and prag/src/roles.rs. Unknown-word handling relates to en/src/dict.rs and 19. The lexicon: vocabulary, multiword expressions and register. Reference and the choice of noun belong in 35. Reference and coreference, and the evaluation idea fits coref/src/eval.rs. Measures: the wrong-category rate and the no-noun rate on held-out lessons, the resolution accuracy with a listener that knows the categories, run twice for determinism.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1063

@inproceedings{zarries2019know,
  title = {{Know What You Don't Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories}},
  author = {Zarrieß, Sina and Schlangen, David},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1063},
  url = {https://doi.org/10.18653/v1/p19-1063}
}

References

23 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Emiel Krahmer and Kees Van Deemter. 2012. Computational generation of referring expressions: A survey. Computational Linguistics, 38(1):173–218. find
2. Harsh Agrawal, Karan Desai, Xinlei Chen, Rishabh Jain, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2018. nocaps: novel object captioning at scale. arXiv preprint arXiv:1812.08658. ↗
3. Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73. find
4. Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided open vocabulary image captioning with constrained beam search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 936–945, Copenhagen, Denmark. Association for Computational Linguistics. find
5. Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. 2009. Learning to detect unseen object classes by between-class attribute transfer. In IEEE Computer Vision and Pattern Recognition, pages 951–958. IEEE. find
6. Jacob Andreas and Dan Klein. 2016. Reasoning about pragmatics with neural listeners and speakers. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1173–1182, Austin, Texas. Association for Computational Linguistics. find
7. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C.Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision – ECCV 2014, volume 8693, pages 740–755. Springer International Publishing. find
8. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. ↗
9. Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L. Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous object descriptions. In CVPR 2016. find
10. Herbert H. Clark and Deanna Wilkes-Gibbs. 1986. Referring as a collaborative process. Cognition, 22:1– 39. find
11. Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556. ↗
12. Reuben Cohn-Gordon, Noah Goodman, and Christopher Potts. 2018. Pragmatically informative image captioning with character-level inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 439–443. Association for Computational Linguistics. find
13. Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. 2013. Zero-shot learning through cross-modal transfer. In Advances in neural information processing systems, pages 935–943. find
14. Ramakrishna Vedantam, Samy Bengio, Kevin Murphy, Devi Parikh, and Gal Chechik. 2017. Context-aware captions from context-agnostic supervision. In Computer Vision and Pattern Recognition (CVPR), volume 3. find
15. Robert Dale and Ehud Reiter. 1995. Computational interpretations of the gricean maxims in the generation of referring expressions. Cognitive Science, 19(2):233–263. find our card
16. J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09. find
17. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Computer Vision and Pattern Recognition. find
18. Rui Fang, Malcolm Doering, and Joyce Y Chai. 2014. Collaborative models for referring expression generation in situated dialogue. In Twenty-Eighth AAAI Conference on Artificial Intelligence. find
19. Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016. Modeling Context in Referring Expressions, pages 69–85. Springer International Publishing, Cham. find
20. Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998–998. find
21. Licheng Yu, Hao Tan, Mohit Bansal, and Tamara L Berg. 2017. A joint speakerlistener-reinforcer model for referring expressions. In Computer Vision and Pattern Recognition (CVPR), volume 2. find
22. Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Darrell. 2016. Deep compositional captioning: Describing novel object categories without paired training data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–10. find
23. Sina Zarrieß and David Schlangen. 2018. Decoding strategies for neural referring expression generation. Proceedings of INLG 2018. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/zarrie-2019-know-what-you-don-t/