On Generalization in Coreference Resolution

What it is about

The paper asks whether coreference models trained on one dataset transfer to others. It consolidates eight datasets (OntoNotes, LitBank, PreCo as training sets; Character Identification, WikiCoref, Quiz Bowl, GAP and Winograd-style WSC as evaluation or analysis sets) and trains a long-document entity-ranking model with a Longformer encoder. Single-dataset models transfer poorly, for example the LitBank model is weak on OntoNotes-style data. Two changes help: adding about 30K to 60K pseudo-singletons (silver mentions from a mention detector, to match the datasets that annotate singletons), and joint training on a sampled mixture (1K documents per epoch from OntoNotes and PreCo). The joint model with pseudo-singletons reaches a macro average of 70.0 CoNLL F1 over the eight datasets (Table 2), against 59.1 for the OntoNotes-only baseline, and the abstract reports a 9.5-point macro gain. It sets a new state of the art on LitBank (79.3).

Ideas

• I1. Make the dataset-agreement question a measurement in Mova: coref/src/eval.rs should report a per-dataset table, not one number, and a macro average over several corpora, so that a gain on one corpus cannot hide a loss on another.
• I2. Pseudo-singletons as a rule: in coref/src/mention.rs, unclustered noun phrases and names could be emitted as silver singleton mentions, with the count of added mentions logged, so that CoNLL scores are not inflated by singletons (the paper cites Kübler and Zhekova on this).
• I3. Balanced sampling as a table: when Mova builds a training mix over corpora (school texts, news, dialogue), sample per corpus with a fixed quota per pass, in BTreeMap order, and test whether the quota changes out-of-domain F1 in coref/src/corefud.rs.

For Mova

Relevant to coref/src/sieve.rs and coref/src/eval.rs (per-dataset reporting), and to coref/src/corefud.rs for multi-corpus CorefUD. The neural model is not for Mova; the findings to take are the singleton-annotation gap and the per-dataset macro average. Measure: run the sieve on two corpora, report the macro average and the per-corpus numbers, run twice, and check that the two runs are identical.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.crac-1.12 · arXiv 2109.09667

@inproceedings{toshniwal2021generalization,
  title = {{On Generalization in Coreference Resolution}},
  author = {Toshniwal, Shubham and Xia, Patrick and Wiseman, Sam and Livescu, Karen and Gimpel, Kevin},
  booktitle = {Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference},
  year = {2021},
  doi = {10.18653/v1/2021.crac-1.12},
  eprint = {2109.09667},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.crac-1.12}
}

References

39 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-Order Coreference Resolution with Coarse-to-Fine Inference. In NAACL-HLT. find
2. Berfin Aktaş, Veronika Solopova, Annalena Kohnert, and Manfred Stede. 2020. Adapting Coreference Resolution to Twitter Conversations. In Findings of EMNLP. find
3. Rahul Aralikatte, Heather Lent, Ana Valeria Gonzalez, Daniel Herschcovich, Chen Qiu, Anders Sandholm, Michael Ringaard, and Anders Søgaard. 2019. Rewarding Coreference Resolvers for Being Consistent with World Knowledge. In EMNLP-IJCNLP. find
4. David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An Annotated Dataset of Coreference in English Literature. In LREC. find our card
5. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer. arXiv:2004.05150. ↗
6. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A Large-scale Dataset in Preschool Vocabulary for Coreference Resolution. In EMNLP. find our card
7. Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2018. Neural Models for Reasoning over Multiple Mentions Using Coreference. In NAACL-HLT. find
8. Yifan Gao, Piji Li, Irwin King, and Michael R. Lyu. 2019. Interconnected Question Generation with Coreference Alignment and Conversation Flow Modeling. In ACL. find
9. Abbas Ghaddar and Phillippe Langlais. 2016. Wiki-Coref: An English Coreference-annotated Corpus of Wikipedia Articles. In LREC. find our card
10. Anupam Guha, Mohit Iyyer, Danny Bouman, and Jordan Boyd-Graber. 2015. Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers. In NAACL-HLT. find
11. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving Pre-training by Representing and Predicting Spans. TACL, 8. find
12. Ben Kantor and Amir Globerson. 2019. Coreference Resolution with Entity Equalization. In ACL. find
13. Catherine Kobus, Josep Crego, and Jean Senellart. 2017. Domain Control for Neural Machine Translation. In RANLP. find
14. Sandra Kübler and Desislava Zhekova. 2011. Singletons and Coreference Resolution Evaluation. In Proceedings of the International Conference Recent Advances in Natural Language Processing 2011. find our card
15. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end Neural Coreference Resolution. In EMNLP. find our card
16. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning. find
17. Manling Li, Alireza Zareian, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare Voss, Daniel Napierski, and Marjorie Freedman. 2020. GAIA: A Fine-grained Multimedia Knowledge Extraction System. In ACL: System Demonstrations. find
18. Nafise Sadat Moosavi and Michael Strube. 2018. Using Linguistic Features to Improve the Generalization Capability of Neural Coreference Resolvers. In EMNLP. find our card
19. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora Resolution with the ARRAU Corpus. In Proceedings of the First Workshop on Computational Models of Reference, Anaphora and Coreference. find
20. Corbèn Poot and Andreas van Cranenburgh. 2020. A Benchmark of Rule-Based and Neural Coreference Resolution in Dutch Novels and News. In CRAC. find
21. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task. find our card
22. Marta Recasens, Marie-Catherine de Marneffe, and Christopher Potts. 2013. The Life and Death of Discourse Entities: Identifying Singleton Mentions. In NAACL-HLT. find
23. Sara Stymne, Miryam de Lhoneux, Aaron Smith, and Joakim Nivre. 2018. Parser Training with Heterogeneous Treebanks. In ACL. find
24. Sanjay Subramanian and Dan Roth. 2019. Improving Generalization in Coreference Resolution via Adversarial Training. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics. find
25. Swabha Swayamdipta, Sam Thomson, Kenton Lee, Luke Zettlemoyer, Chris Dyer, and Noah A. Smith. 2018. Syntactic Scaffolds for Semantic Structures. In EMNLP. find
26. Xu Tan, Jiale Chen, Di He, Yingce Xia, Tao Qin, and Tie-Yan Liu. 2019. Multilingual Neural Machine Translation with Language Clustering. In EMNLP-IJCNLP. find
27. Shubham Toshniwal, Sam Wiseman, Allyson Ettinger, Karen Livescu, and Kevin Gimpel. 2020. Learning to Ignore: Long Document Coreference with Bounded Memory Neural Networks. In EMNLP. find
28. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A Balanced Corpus of Gendered Ambiguous Pronouns. TACL, 6. find our card
29. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. OntoNotes Release 5.0. Linguistic Data Consortium. find
30. Sam Wiseman, Alexander M. Rush, and Stuart Shieber. 2016. Antecedent Prediction Without a Pipeline. In Proceedings of the Workshop on Coreference Resolution Beyond OntoNotes (CORBON). find
31. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference Resolution as Query-based Span Prediction. In ACL. find
32. Patrick Xia, João Sedoc, and Benjamin Van Durme. 2020. Incremental Neural Coreference Resolution in Constant Memory. In EMNLP. find our card
33. Liyan Xu and Jinho D. Choi. 2020. Revealing the Myth of Higher-Order Inference in Coreference Resolution. In EMNLP. find
34. Jian Bo Yang, Qi Mao, Qiao Liang Xiang, Ivor Wai-Hung Tsang, Kian Ming Adam Chai, and Hai Leong Chieu. 2012. Domain Adaptation for Coreference Resolution: An Adaptive Ensemble Approach. In EMNLP. find
35. Juntao Yu, Alexandra Uma, and Massimo Poesio. 2020. A Cluster Ranking Model for Full Anaphora Resolution. In LREC. find
36. Rui Zhang, Cícero Nogueira dos Santos, Michihiro Yasunaga, Bing Xiang, and Dragomir Radev. 2018. Neural Coreference Resolution with Deep Biaffine Attention by Joint Mention Detection and Mention Clustering. In ACL. find
37. Shanheng Zhao and Hwee Tou Ng. 2014. Domain Adaptation with Active Learning for Coreference Resolution. In Proceedings of the 5th International Workshop on Health Text Mining and Information Analysis (Louhi). find
38. Ethan Zhou and Jinho D. Choi. 2018. They Exist! Introducing Plural Mentions to Coreference Resolution and Entity Linking. In Proceedings of the 27th International Conference on Computational Linguistics. find
39. Yilun Zhu, Sameer Pradhan, and Amir Zeldes. 2021. OntoGUM: Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres. In ACL. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/toshniwal-2021-generalization-coreference-resolution/