Referring Expressions with Rational Speech Act Framework: A Probabilistic Approach

What it is about

The paper generates referring expressions ("the dog with the frisbee") for objects in photographs with the Rational Speech Act (RSA) model instead of a neural speaker. A scene graph (object types, attributes, relations) is first extracted with pre-trained Detectron2 and Graph R-CNN, and ordinal relations (left, right, second from left) are added by sorting objects by x-coordinate. The speaker adds one descriptor at a time until the listener's entropy over objects falls below a threshold or four descriptors are used. On RefCOCO and RefCOCO+ the expressions are checked by human choice among boxes. The method does not beat an end-to-end neural model trained on full data, and the authors trace many failures (boxes with no match, or not highlighted) to the vision step rather than to the pragmatic one. The tables are hard to align in the text read here, so the card quotes only one clear figure: an adjusted accuracy of 52.54% on RefCOCO+.

Ideas

• I1. prag/src/rsa.rs: a deterministic version of the speaker. At each step choose the descriptor that most reduces the set of candidate referents, with ties broken by the descriptor string in BTreeMap order, and stop when one candidate remains or after a fixed length. Compare length and success with the sampling version on the school lessons.
• I2. world thresholds: the paper fixes one threshold for attributes and one for relations by hand, and says it would learn them from data. Fit each threshold on a training split and report it on a held-out split (cv), never on the test set.
• I3. Ordinal descriptors as a rule in the story world: sort the same-kind objects by position in the text (first, second, last) and add "second from left"-style descriptors. Test on the school scene questions, with a negative control where the order is shuffled.
• I4. Keep the trace: store the descriptor sequence and the entropy after each step for each referring expression. This makes each choice explainable (the "why" the project needs), and the trace is the input to an error table like the paper's.

For Mova

The speaker and listener belong with prag/src/rsa.rs and prag/src/infer.rs. The scene and facts are in world/src/knowledge.rs and world/src/events.rs. Referring expressions and the choice of descriptor relate to 35. Reference and coreference and to the quantity implicature in 33. Implicature and presupposition. Measures: expression length and success on the school scene questions, with the random-order control, and the four error categories on a held-out scene set.

Topics: Speech act theory

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2205.07795

@misc{le2022referring,
  title = {{Referring Expressions with Rational Speech Act Framework: A Probabilistic Approach}},
  author = {Le, Hieu and Daryanto, Taufiq and Zhafransyah, Fabian and Wijaya, Derry and Coppock, Elizabeth and Chin, Sang},
  year = {2022},
  eprint = {2205.07795},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.07795}
}

References

18 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. W. Monroe and C. Potts, “Learning in the Rational Speech Acts model,” CoRR, vol. abs/1510.06807, 2015. ↗
2. S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg, “ReferItGame: Referring to objects in photographs of natural scenes,” in Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), (Doha, Qatar), pp. 787–798, ACL, Oct. 2014. find
3. M. C. Frank and N. D. Goodman, “Predicting pragmatic reasoning in language games,” Science, vol. 336, no. 6084, p. 998, 2012. find
4. J. Degen, R. X. D. Hawkins, C. Graf, E. Kreiss, and N. D. Goodman, “When redundancy is rational: A Bayesian approach to ’overinformative’ referring expressions,” CoRR, vol. abs/1903.08237, 2019. ↗
5. J. Andreas and D. Klein, “Reasoning about pragmatics with neural listeners and speakers,” in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, (Austin, Texas), pp. 1173–1182, ACL, Nov. 2016. find
6. R. Cohn-Gordon, N. D. Goodman, and C. Potts, “Pragmatically informative image captioning with character-level reference,” CoRR, vol. abs/1804.05417, 2018. ↗
7. J. Yang, J. Lu, S. Lee, D. Batra, and D. Parikh, “Graph R-CNN for scene graph generation,” CoRR, vol. abs/1808.00191, 2018. ↗
8. Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2.” https://github.com/facebookresearch/detectron2, 2019. ↗
9. L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg, “Modeling context in referring expressions,” CoRR, vol. abs/1608.00272, 2016. ↗
10. G. Scontras, M. H. Tessler, and M. Franke, “Probabilistic language understanding: An introduction to the rational speech act framework,” 2018. find
11. S. Ren, K. He, R. Girshick, and J. Sun, “Faster R-CNN: Towards real-time object detection with region proposal networks,” in Advances in Neural Information Processing Systems (NIPS), 2015. find
12. W. Abdulla, “Mask R-CNN for object detection and instance segmentation on Keras and TensorFlow.” https://github.com/matterport/Mask_RCNN, 2017. ↗
13. R. Luo and G. Shakhnarovich, “Comprehension-guided referring expressions,” CoRR, vol. abs/1701.03439, 2017. ↗
14. D. Lassiter and N. Goodman, “Adjectival vagueness in a Bayesian model of interpretation,” Synthese, vol. 10, pp. 3801–3836, 2017. find
15. D. Edgington, “The philosophical problem of vagueness,” Legal Theory, vol. 7, pp. 371–378, 2001. find
16. L. Yu, H. Tan, M. Bansal, and T. L. Berg, “A joint speaker-listener-reinforcer model for referring expressions,” 2017. find
17. J. Kim, H. Ko, and J. Wu, “CoNAN: A complementary neighboring-based attention network for referring expression generation,” in Proceedings of the 28th International Conference on Computational Linguistics, (Barcelona, Spain (Online)), pp. 1952– 1962, International Committee on Computational Linguistics, Dec. 2020. find
18. P. Joshi, S. Santy, A. Budhiraja, K. Bali, and M. Choudhury, “The state and fate of linguistic diversity and inclusion in the NLP world,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, (Online), pp. 6282–6293, Association for Computational Linguistics, July 2020. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/le-2022-referring-expressions-rational-speech-act/