Recognition of They/Them as Singular Personal Pronouns in Coreference Resolution

What it is about

The paper tests whether coreference systems resolve singular personal "they" (a named person) as a singular referent. It builds WinoNB schemas: about 4,077 templates adapted by hand from Winogender, WinoBias, WSC and DPR, in which one referent is a group of people and the other a named individual, filled with 15 names. Each pair differs in a small edit so that the correct antecedent flips. Five systems are tested: Clark and Manning, two variants of the Lee et al. end-to-end model, BERT and SpanBERT. On the "all" row of Table 2, accuracy on singular personal "they" is 0.008 to 0.145 for the five systems, against 0.846 to 0.988 for plural "they". For the singular generic case ("someone") accuracy is 0.011 to 0.618. The three OntoNotes-only models resolve "they" almost never as singular. BERT-base does best on the personal case (about 3 times the average), which the authors link to exposure to singular "they" in pre-training. A hand check of 100 sampled OntoNotes sentences found no singular personal "they".

Ideas

• I1. Add a WinoNB-style set to coref/ as a measurement: a template generator with the group and named-person slots, a singular-personal, singular-generic and plural split, and the accuracy reported per split. Keep the template list in a seed file, in BTreeMap order, so each run is identical.
• I2. Name balance as a control: report accuracy by name group (AFAB, AMAB, neutral) as the paper does, and treat a gap as a bias signal for coref/src/sieve.rs, since a rule-based sieve has no learned name prior but can still use gender lists.
• I3. Negative control: swap the singular "they" for a plural group noun in the same template and check that the score drops. A sieve that scores both the same is not reading the number.

For Mova

The sieve is coref/src/sieve.rs, with mention features in coref/src/mention.rs. The gold split is by number and by the pronoun form. The chapter is 12. Pronouns, determiners and numerals (the they-pronoun and number of pronouns) and 35. Reference and coreference. Measure: the WinoNB-style accuracy for the sieve, split by the three cases, run twice. Keep the set separate from any release test set.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.naacl-main.250

@inproceedings{baumler2022recognition,
  title = {{Recognition of They/Them as Singular Personal Pronouns in Coreference Resolution}},
  author = {Baumler, Connor and Rudinger, Rachel},
  booktitle = {Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2022},
  doi = {10.18653/v1/2022.naacl-main.250},
  url = {https://doi.org/10.18653/v1/2022.naacl-main.250}
}

References

18 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: Al-locative versus representational harms in machine learning. In SIGCIS Conference. find
2. Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454– 5476, Online. Association for Computational Linguistics. find
3. Yang Trista Cao and Hal Daumé III. 2020. Toward gender-inclusive coreference resolution. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4568–4595, Online. Association for Computational Linguistics. find
4. Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262, Austin, Texas. Association for Computational Linguistics. find
5. Sunipa Dev, Masoud Monajatipoor, Anaelia Ovalle, Arjun Subramonian, Jeff Phillips, and Kai-Wei Chang. 2021. Harms of gender exclusivity and challenges in non-binary representation in language technologies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1968–1994, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
6. Julie Foertsch and Morton Ann Gernsbacher. 1997. In search of gender neutrality: Is singular they a cognitively efficient substitute for generic he? Psychological science, 8(2):106–111. find
7. Milton Friedman. 1937. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200):675–701. find
8. Milton Friedman. 1940. A comparison of alternative tests of significance for the problem of m rankings. The Annals of Mathematical Statistics, 11(1):86–92. find
9. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. find
10. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. find
11. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics. find
12. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. find
13. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguistics. find our card
14. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789, Jeju Island, Korea. Association for Computational Linguistics. find
15. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 8–14, New Orleans, Louisiana. Association for Computational Linguistics. find
16. Thomas Wolf. 2017. State-of-the-art neural coreference resolution for chatbots. Blog post. find
17. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics. find
18. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/baumler-2022-recognition-they-them-as-singular/