The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution

What it is about

KnowRef is a set of 8,724 natural Winograd-like passages: two person noun phrases, then one connective (but, because, although, when, so …), then a pronoun that agrees with both («Paul helped Lionel hide when he was pursued by the authorities»). It is mined by filters: sentence length, exactly one connective cluster, exactly two person NPs before it and none repeated after it, no pronoun before it. Training labels come free: keep sentences whose two names differ in gender, read the answer off the pronoun's gender (86% right on a check), then replace one name so that the gender cue disappears. The test set (Wikipedia and subtitles) is labelled by humans with high agreement. Rule-based, feature-based and neural resolvers all do far worse than people, and a «consistency» test (swap the two names; a model using context should keep pointing at the same position) shows they mostly follow gender and number. Training on name-swapped copies ("antecedent switching") helps, also on GAP.

Ideas

• I1. Harvest knowledge for hard pronouns from the easy ones, as the paper harvests labels: in the tale corpus, find «A verb1 B, connective, pronoun verb2 …» where A and B differ in gender (from Mova's own evidence: gendered nouns, earlier he/she, titles — not names), so the pronoun's antecedent is known. Count, per (verb1, role of the antecedent in verb1, connective, verb2), how often the subject of verb2 is verb1's agent or patient. These counts become a deterministic preference used only when gender and number leave two candidates. Put the table next to the pronoun sieve in coref/src/sieve.rs and in Ctx; print the count that decided. Measure on a held-out set of the same harvested cases with the cue removed, and he/she precision on GUM dev.
• I2. Add the consistency gate: for every two-person test sentence, swap the two names (or nouns) and check that the resolver points to the same position. Mova's resolvers do not use names for gender, so they should pass by construction; a failure means a hidden lexical cue. A negative control that can go red: a resolver that prefers the first alphabetical name must fail it.
• I3. Build a small hard pronoun set from the tales with the paper's filters (two animate NPs before one connective, a pronoun after, no repeated NP), labelled by the teacher, and report coref precision on it separately from GUM. It measures exactly the cases the sieve now resolves by Hobbs order alone.

For Mova

35. Reference and coreference §2 (salience), §8 (algorithms), §10 (the resolver inside the frame reader). Code: coref/src/sieve.rs (pronoun), prag/src/frames.rs (Ctx), world/src/events.rs (verb statistics from books). Measures: precision on the harvested held-out set, he/she/they precision on GUM dev, the swap-consistency gate.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1386 · arXiv 1811.01747

@inproceedings{emami2019knowref,
  title = {{The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution}},
  author = {Emami, Ali and Trichelair, Paul and Trischler, Adam and Suleman, Kaheer and Schulz, Hannes and Cheung, Jackie Chi Kit},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1386},
  eprint = {1811.01747},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1386}
}

References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dan Bailey, Amelia Harrison, Yuliya Lierler, Vladimir Lifschitz, and Julian Michael. 2015. The winograd schema challenge and reasoning about correlation. In Working Notes of the Symposium on Logical Formalizations of Commonsense Reasoning. find
2. David Bean and Ellen Riloff. 2004. Unsupervised learning of contextual role knowledge for coreference resolution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004. find
3. Shane Bergsma and Dekang Lin. 2006. Bootstrapping path-based pronoun resolution. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. find
4. Kevin Clark and Christopher D Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing. find
5. Kevin Clark and Christopher D Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. find
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
7. George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In LREC. find
8. Greg Durrett, David Hall, and Dan Klein. 2013. Decentralized entity-level modeling for coreference resolution. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics. find
9. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. find our card
10. Ralph Grishman and Beth Sundheim. 1996. Message understanding conference-6: A brief history. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics, volume 1. find
11. Jerry R Hobbs. 1977. SIGART Bulletin. Pronoun resolution. find
12. ACM Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task. In Proceedings of the fifteenth conference on computational natural language learning: Shared task. find our card
13. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. find our card
14. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
15. Hector J Levesque, Ernest Davis, and Leora Morgenstern. 2011. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. find
16. Quan Liu, Hui Jiang, Andrew Evdokimov, Zhen-Hua Ling, Xiaodan Zhu, Si Wei, and Yu Hu. 2016. Probabilistic reasoning via deep learning: Neural association models. arXiv preprint arXiv:1603.07704. ↗
17. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association of Computational Linguistics. find
18. Karthik Raghunathan, Heeyoung Lee, Sudarshan Rangarajan, Nathanael Chambers, Mihai Surdeanu, Dan Jurafsky, and Christopher Manning. 2010. A multi-pass sieve for coreference resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find our card
19. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. find
20. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: the winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. find
21. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
22. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology. find
23. Andrew McCallum and Ben Wellner. 2005. Conditional models of identity uncertainty with application to noun coreference. In Advances in neural information processing systems. find
24. Paul Trichelair, Ali Emami, Jackie Chi Kit Cheung, Adam Trischler, Kaheer Suleman, and Fernando Diaz. 2018. On the evaluation of common-sense reasoning in natural language understanding. The NeurIPS Workshop on Critiquing and Correcting Trends in Machine Learning. find
25. JF McCarthy. 1995. Using decision trees for coreference resolution. In Proc. 14th International Joint Conf. on Artificial Intelligence. find
26. Trieu H Trinh and Quoc V Le. 2018. A simple method for commonsense reasoning. arXiv preprint arXiv:1806.02847. ↗
27. Thomas S Morton. 2000. Coreference for nlp applications. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics. find
28. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the gap: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. find our card
29. Haoruo Peng, Daniel Khashabi, and Dan Roth. 2015. Solving hard coreference problems. Urbana. find
30. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task. find our card
31. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task. find our card
32. Sam Wiseman, Alexander M Rush, and Stuart M Shieber. 2016. Learning global features for coreference resolution. In Proceedings of NAACL-HLT. find
33. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In NAACL-HLT. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/emami-2019-knowref-coreference-corpus-removing-gender/