AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference

What it is about

The paper asks whether coreference models react to pronoun ambiguity as people do. It builds AmbiCoref, a corpus of minimal sentence pairs: an ambiguous and an unambiguous sentence differ only in one verb phrase (for example "Abby told Brittney that she upset Courtney" against "hugged"). The templates come from four psycholinguistic effects (experiencer constraints for objects and subjects, implicit causality, transfer of possession) and use VerbNet classes; the corpus has over 96,000 sentences. In a crowdsourced study of 625 sentences with three judgements each, unambiguous items get the intended referent (likely or definitely) for 83.2%, 91.9%, 85.8% and 68.3% of cases in the four types, while ambiguous items show a clear shift toward uncertainty. Five models (three CoreNLP variants, SpanBERT, NeuralCoref 4.0) are compared by the Earth Mover's Distance between their cluster-case distributions for matched pairs. Four of the five hardly change (mean EMD 0.6 to 4.0); SpanBERT changes most (11.7).

Ideas

• I1. Minimal-pair set for Mova: generate ambiguous and unambiguous pairs in coref/ from the same templates (verb class decides the constraint), and check that coref/src/sieve.rs changes its output between the pair. A sieve that outputs the same cluster for both members is missing the verb's effect.
• I2. Use the paper's distribution, not the final decision, as a rule output: the sieve can report a confidence per case (A, B, singleton, other) and "abstain" when two cases are close. Log this in prag as a clarification trigger, explainable per rule.
• I3. Human-judgement check for the rules: a small annotation of 50 pairs by the operator, with the confidence scale as in the paper, to fix which template types Mova should treat as ambiguous before any gain is claimed.

For Mova

Code: coref/src/sieve.rs (case output and abstention), coref/src/eval.rs (the minimal-pair measure), and prag for a clarification response. Chapters: 35. Reference and coreference, 33. Implicature and presupposition. Measure: the EMD-style shift between ambiguous and unambiguous pairs, run twice, with a shuffled-verb negative control that should show no shift.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-eacl.75 · arXiv 2302.00762

@inproceedings{yuan2023ambicoref,
  title = {{AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference}},
  author = {Yuan, Yuewei and Malaviya, Chaitanya and Yatskar, Mark},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-eacl.75},
  eprint = {2302.00762},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-eacl.75}
}

References

28 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Laura Aina and Tal Linzen. 2021. The language model understood the prompt was ambiguous: Probing syntactic uncertainty through generation. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 42– 57, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
2. Alfonso Caramazza, Ellen Grober, Catherine Garvey, and Jack Yates. 1977. Comprehension of anaphoric pronouns. Journal of Verbal Learning and Verbal Behavior, 16(5):601–609. find
3. Kevin Clark and Christopher D Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1405–1415. find
4. Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262, Austin, Texas. Association for Computational Linguistics. find
5. Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295–302, Online. Association for Computational Linguistics. find
6. Anca Dumitrache, Lora Aroyo, and Chris Welty. 2019. A crowdsourced frame disambiguation corpus with ambiguity. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2164–2170, Minneapolis, Minnesota. Association for Computational Linguistics. find
7. Meiqi Guo, Mingda Zhang, Siva Reddy, and Malihe Alikhani. 2021. Abg-coqa: Clarifying ambiguity in conversational question answering. In 3rd Conference on Automated Knowledge Base Construction. find
8. Sarah L Haywood, Martin J Pickering, and Holly P Branigan. 2005. Do speakers avoid ambiguities during dialogue? Psychological Science, 16(5):362– 366. find
9. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering. Transactions of the Association for Computational Linguistics, 9:962–977. find
10. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. find
11. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. find
12. H. Rohde and A. Kehler. 2014. Grammatical and information-structural influences on pronoun production. Language, Cognition and Neuroscience, 29(8):912–927. find
13. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. find our card
14. Hinrich Schutze. 1995. Ambiguity in language learning: computational and cognitive models. Stanford University. find
15. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Maryland. Association for Computational Linguistics. find
16. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783– 5797, Online. Association for Computational Linguistics. find
17. Ofir Pele and Michael Werman. 2009. Fast and robust earth mover’s distances. In 2009 IEEE 12th International Conference on Computer Vision, pages 460–467. IEEE. find
18. Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. Cognition, 122(3):280–291. find
19. Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 76–83, Ann Arbor, Michigan. Association for Computational Linguistics. find
20. Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1778–1789, Minneapolis, Minnesota. Association for Computational Linguistics. find
21. Karin Kipper Schuler. 2005. VerbNet: A broad-coverage, comprehensive verb lexicon. University of Pennsylvania. find
22. F. Springston. 1976. Verb-derived constraints in the comprehension of anaphoric pronouns. Paper presented at the Eastern Psychological Association (1976). find
23. Shiva Upadhye, Leon Bergen, and Andrew Kehler. 2020. Predicting reference: What do language models learn about discourse models? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 977–982, Online. Association for Computational Linguistics. find
24. Thomas Wasow, Amy Perfors, and David Beaver. 2003. The puzzle of ambiguity. find
25. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23. find
26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. find
27. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. find
28. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguistics. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/yuan-2023-ambicoref-evaluating-human-model-sensitivity/