Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns

What it is about

GAP is 8,908 pronoun–name pairs (4,454 contexts) from Wikipedia, each with a gendered pronoun and two names of the same gender, either of which (or neither) could be the referent, with no other compatible mention in between. Contexts are extracted by three patterns (both names before the pronoun in one sentence; pronoun in an initial subordinate clause or prepositional possessive; all in one sentence) and balanced for gender (1:1, though Wikipedia gives 9:1 masculine), for which name is right, and for whether the page's own entity is involved. Three raters per example agree well (kappa 0.74). Off-the-shelf resolvers trained on OntoNotes (rule-based Stanford, three neural ones) do poorly and are biased: feminine pronouns are resolved worse. Simple cue baselines help separate the factors: token distance and «topical entity» are weak, syntactic distance and especially parallelism (same grammatical role as the pronoun) are strong; a Transformer-attention baseline combined with parallelism is best, at 66.9 F1 overall in the snippet setting, more with the page title.

Ideas

• I1. Use GAP (if its licence permits evaluation) as a targeted test of the pronoun sieve where gender cannot decide, which is exactly where Hobbs order alone must work: report F1 overall and masculine vs feminine, and compare with the paper's parallelism and syntactic-distance baselines. Mova does not guess gender from names, so a gender gap in its scores would point to the noun lists or to animacy evidence.
• I2. Add a parallelism preference to the pronoun sieve and to the frame reader's salience: a subject pronoun prefers a subject antecedent, an object pronoun an object one (the +35 parallelism weight of Lappin & Leass, already noted in coref — SLM coreference: a sieve of rules with a reason for every link «What next»). Measure per-pronoun precision for he/she/him/her on GUM dev and on GAP.
• I3. The page-entity effect has a tale analogue: the title character of a tale is the default referent of an otherwise ambiguous he/she at paragraph start. Give the title character a standing salience bonus in prag/src/frames.rs Ctx and measure frame bench coref on tale lines.

For Mova

35. Reference and coreference §2 (salience), §8 (algorithms), §9. Code: coref/src/sieve.rs (pronoun sieve, parallelism), prag/src/frames.rs (salience). Measures: GAP F1 by gender; per-pronoun precision on GUM dev; frame bench coref F1.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00240

@article{webster2018mind,
  title = {{Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns}},
  author = {Webster, Kellie and Recasens, Marta and Axelrod, Vera and Baldridge, Jason},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00240},
  url = {https://doi.org/10.1162/tacl_a_00240}
}

References

54 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

2. David Bamman and Noah A. Smith. 2014. Unsupervised discovery of biographical structure from text. Transactions of the ACL, 2:363–376. ↗
3. Mohit Bansal and Dan Klein. 2012. Coreference semantics from web features. In Proceedings of ACL, pages 389–398, Jeju Island, Korea. find
4. Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. Transactions of the ACL, 2:477–490. ↗
5. David Bean and Ellen Riloff. 2004. Unsupervised learning of contextual role knowledge for coreference resolution. In Proceedings of HLT-NAACL, pages 297–304, Boston, Massachusetts. find
6. Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard S. Zemel. 2012. Fairness through awareness. In Proceedings of ITCS, pages 214–226. ↗
7. Joseph L. Fleiss, Bruce Levin, and Myunghee Cho Paik. 2003. The Measurement of Interrater Agreement, 3 edition. John Wiley and Sons Inc. find
8. Shane Bergsma and Dekang Lin. 2006. Bootstrapping path-based pronoun resolution. In Proceedings of ACL, pages 33–40, Sydney, Australia. find
9. Pierre Geurts, Damien Ernst, and Louis Wehenkel. 2006. Extremely randomized trees. Machine Learning, 63(1):3–42. ↗
10. Steven Bird and Edward Loper. 2004. NLTK: The natural language toolkit. In Proceedings of the ACL 2004 on Interactive Poster and Demonstration Sessions, Barcelona, Spain. find
11. Abbas Ghaddar and Philippe Langlais. 2016a. Coreference in Wikipedia: Main Concept Resolution. In Proceedings of CoNLL, pages 229–238, Berlin, Germany. find
12. Tolga Bolukbasi, Kai-Wei Chang, James Y. Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? Debiasing word embeddings. In Proceedings of NIPS, pages 4349–4357, Barcelona, Spain. find
13. Abbas Ghaddar and Philippe Langlais. 2016b. WikiCoref: An English coreference-annotated corpus of Wikipedia articles. In Proceedings of LREC, Portorož, Slovenia. find our card
14. Mateusz Buda. 2017. A systematic study of the class imbalance problem in convolutional neural networks. Master’s thesis, KTH Royal Institute of Technology. ↗
15. Eduardo Graells-Garrido, Mounia Lalmas, and Filippo Menczer. 2015. First women, second sex: Gender bias in Wikipedia. In Proceedings of the 26th ACM Conference on Social Media, pages 165–174, Guzelyurt, Northern Cyprus. ↗
16. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. ↗
17. Ralph Grishman and Beth Sundheim. 1996. Message Understanding Conference 6: A brief history. In Proceedings of COLING, pages 466–471, Copenhagen, Denmark. find
18. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of ACL: HLT, pages 789–797, Columbus, Ohio. find
19. Liane Guillou. 2012. Improving pronoun translation for statistical machine translation. In Proceedings of the Student Research Workshop at the 13th Conference of the EACL, pages 1–10, Avignon, France. find
20. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of ACL, pages 1405–1415, Beijing, China. find
21. Ido Dagan and Alon Itai. 1990. Automatic processing of large corpora for the resolution of anaphora references. In Proceedings of COL-ING, pages 330–332, Helsinki, Finland. find
22. Moritz Hardt. 2014. How big data is unfair: Understanding unintended sources of unfairness in data driven decision making. https://medium.com/@mrtz/how-bigdata-is-unfair-9aa544d739de. ↗
23. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of EMNLP, pages 1971–1982, Seattle, Washington. find our card
24. Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of opportunity in supervised learning. In Proceedings of NIPS, pages 3323–3331, Barcelona, Spain. ↗
25. Kotaro Nakayama. 2008. Wikipedia mining for triple extraction enhanced by co-reference resolution. In Proceedings of the ISWC Workshop on Social Data on the Web, Berlin, Germany. find
26. Andrew Kehler, Douglas Appelt, Lara Taylor, and Aleksandr Simma. 2004. The (non)utility of predicate-argument frequencies for pronoun interpretation. In Proceedings of HLT-NAACL, pages 289–296, Boston, Massachusetts. find
27. Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. 2008. Discrimination-aware data mining. In Proceedings of KDD, pages 560–568, Las Vegas, Nevada. find
28. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. ↗ our card
29. Haoruo Peng, Daniel Khashabi, and Dan Roth. 2015. Solving hard coreference problems. In Proceedings of NAACL, pages 809–819, Denver, Colorado. find
30. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of EMNLP, pages 188–197, Copenhagen, Denmark. find our card
31. Simone Paolo Ponzetto and Michael Strube. 2006. Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution. In Proceedings of HLT-NAACL, pages 192–199, New York, New York. find
32. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of KR, pages 552–561, Rome, Italy. find
33. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of CoNLL: Shared Task, pages 1–40, Jeju, Republic of Korea. find our card
34. Shasha Liao and Ralph Grishman. 2010. Large corpus-based semantic feature extraction for pronoun coreference. In Proceedings of the 2nd Workshop on NLP Challenges in the Information Explosion Era (NLPIX), pages 60–68, Beijing, August. find
35. Sameer Pradhan, Lance Ramshaw, Ralph Weischedel, Jessica MacBride, and Linnea Micciulla. 2007. Unrestricted coreference: Identifying entities and events in OntoNotes. In Proceedings of ICSC, pages 446–453, Irvine, California. find
36. Pierre Lison, Jörg Tiedemann, and Milen Kouylekov. 2018. OpenSubtitles2018: Statistical rescoring of sentence alignments in large, noisy parallel corpora. In Proceedings of LREC, pages 1742–1748, Miyazaki, Japan. find
37. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The Winograd Schema Challenge. In Proceedings of EMNLP-CoNLL, pages 777–789, Jeju, Republic of Korea. find
38. Quan Liu, Hui Jiang, Zhen-Hua Ling, Xiaodan Zhu, Si Wei, and Yu Hu. 2017. Combing context and commonsense knowledge through neural networks for solving Winograd schema problems. In Proceedings of AAAI, pages 315–321, San Francisco, California. find
39. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of NAACL, New Orleans, Louisiana. find
40. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of ACL System Demonstrations, pages 55–60, Baltimore, Maryland. find
41. Hee Jung Ryu, Margaret Mitchell, and Hartwig Adam. 2017. Improving smiling detection with race and gender diversity. ArXiv e-prints v2 https://arxiv.org/ abs/1712.00193. ↗
42. Leora Morgenstern, Ernest Davis, and Charles L. Ortiz. 2016. Planning, executing, and evaluating the Winograd Schema Challenge. AI Magazine, 37(1):50–54. ↗
43. Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D. Sculley. 2017. find
44. Claudia Wagner, Eduardo Graells-Garrido, David Garcia, and Filippo Menczer. 2016. Women through the glass ceiling: Gender assymetrics in Wikipedia. EPJ Data Science, 5. No classification without representation: Assessing geodiversity issues in open data sets for the developing world. In Proceedings of the NIPS Workshop on Machine Learning for the Developing World, Long Beach, California. find
45. Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning global features for coreference resolution. In Proceedings of NAACL-HLT, pages 994–1004, San Diego, California. find
46. Michael Skirpan and Micha Gorelick. 2017. The authority of “fair" in machine learning. In Proceedings of the KDD Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML), Halifax, Nova Scotia. ↗
47. Xiaofeng Yang, Jian Su, and Chew Lim Tan. 2005. Improving pronoun resolution using statistics-based semantic compatibility information. In Proceedings of ACL, pages 165–172, Ann Arbor, Michigan. find
48. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL-IJCNLP, pages 656–664, Singapore. find our card
49. Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. 2017. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of WWW, pages 1171–1180, Perth, Australia. ↗
50. Antonio Torralba and Alexei A. Efros. 2011. Unbiased look at dataset bias. In Proceedings of CVPR 2011, pages 1521–1528, Colorado Springs, Colorado. find
51. Trieu H. Trinh and Quoc V. Le. 2018. A simple method for commonsense reasoning. ArXiv e-prints v1 https:/arxiv.org/ abs/1806.02847. ↗
52. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men also like shopping: Reducing gender bias amplification using corpus-level constraints. In Proceedings of EMNLP, pages 2941–2951, Copenhagen, Denmark. find
53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS, pages 6000–6010, Long Beach, California. find
54. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of NAACL, New Orleans, Louisiana. ↗
55. Elena Voita, Pavel Serdyukov, Rico Sennrich, and Ivan Titov. 2018. Context-aware neural machine translation learns anaphora resolution. In Proceedings of ACL, Melbourne, Australia. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/webster-2018-mind-gap-balanced-corpus-gendered/