Anatomy of OntoGUM—Adapting GUM to the OntoNotes Scheme to Evaluate Robustness of SOTA Coreference Algorithms

What it is about

GUM annotates more coreference than OntoNotes, so the authors convert it automatically to the OntoNotes scheme, using GUM's gold UD trees, in a fixed order of steps: remove bridging and cataphora (an expletive or cataphoric pronoun becomes a singleton), contract verbal mentions to the verb, merge appositions into one mention, remove common-noun compound modifiers, remove copular predicates (detected by a cop on the second span's head), remove nested i-within-i mentions, split chains at indefinite mentions (definite = pronoun, possessed head, proper noun, or definite determiner), and finally drop singletons. The result, OntoGUM, has 168 documents in 12 genres (~150K tokens, 19,378 mentions, 4,471 chains). A SpanBERT end-to-end model trained on OntoNotes loses about 15 points (19%) on OntoGUM and the rule-based Stanford dcoref over 18 points (30%); both are worst on textbook text and genres with few pronouns and no speaker information, and best on vlogs with gold speakers.

Ideas

• I1. Write the conversion as a Rust output mode of coref resolve (GUM-style chains → OntoNotes-style chains by the same ordered steps, all from the UD tree) and score Mova's sieve on OntoGUM test. That gives a direct comparison with the published rule-based dcoref and SpanBERT figures per genre, which the CRAC comparison in coref — SLM coreference: a sieve of rules with a reason for every link lacks («the settings differ»). Evaluation only (NC data).
• I2. Report coref per GUM genre and look hardest at textbook and academic: those are the genres closest to Mova's school lessons and the worst for both systems in the paper. If they lag, collect the errors there first (it/its to sections and events is the likely leader).
• I3. Reuse the paper's definiteness rule (pronoun, possessed head, proper noun, or det with a definite lemma) as the single definition in coref/src/mention.rs and in the frame reader, so both crates agree on what «definite» means; check where they differ today.

For Mova

35. Reference and coreference §7 (annotation schemes), §9.3 (results). Code: coref/src/corefud.rs and coref/src/eval.rs (OntoNotes-style output and scoring), coref/src/mention.rs. Measures: CoNLL F1 on OntoGUM test per genre against dcoref and SpanBERT; per-genre CoNLL F1 on GUM dev.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.crac-1.15 · arXiv 2110.05727

@inproceedings{zhu2021anatomy,
  title = {{Anatomy of OntoGUM—Adapting GUM to the OntoNotes Scheme to Evaluate Robustness of SOTA Coreference Algorithms}},
  author = {Zhu, Yilun and Pradhan, Sameer and Zeldes, Amir},
  booktitle = {Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference},
  year = {2021},
  doi = {10.18653/v1/2021.crac-1.15},
  eprint = {2110.05727},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.crac-1.15}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proceedings of EMNLP 2018, pages 172–181, Brussels. find our card
2. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of ACL-IJCNLP 2015, Long Papers, pages 1405–1415, Beijing, China. find
3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL 2019, pages 4171–4186, Minneapolis, MN. find
4. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of EMNLP 2013, pages 1971–1982, Seattle, WA. find our card
5. Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. TACL, 2:477–490. find
6. Hongliang Fei, Xu Li, Dingcheng Li, and Ping Li. 2019. End-to-end deep reinforcement learning based coreference resolution. In Proceedings of ACL 2019, pages 660–665, Florence, Italy. find
7. Abbas Ghaddar and Phillippe Langlais. 2016. Wiki-Coref: An English coreference-annotated corpus of Wikipedia articles. In Proceedings of LREC 2016), pages 136–142, Portorož, Slovenia. find our card
8. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019a. SpanBERT: Improving pre-training by representing and predicting spans. CoRR, abs/1907.10529. ↗
9. Mandar Joshi, Omer Levy, Daniel S. Weld, and Luke Zettlemoyer. 2019b. BERT for coreference resolution: Baselines and analysis. In Proceedings of EMNLP-IJCNLP 2019, pages 5803–5808, Hong Kong, China. find
10. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. find our card
11. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of EMNLP 2017, pages 188– 197, Copenhagen, Denmark. find our card
12. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of NAACL 2018, Short Papers, pages 687–692, New Orleans, LA. find
13. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262, Austin, Texas. find
14. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL 2014 System Demonstrations, pages 55–60. find
15. Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of ACL 2016, Long Papers, pages 643–653, Berlin. find
16. Nafise Sadat Moosavi and Michael Strube. 2017. Lexical features in coreference resolution: To be used with caution. In Proceedings of ACL 2017, Short Papers, pages 14–19, Vancouver, Canada. find our card
17. Marie-Catherine de Marneffe, Christopher D. Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal Dependencies. Computational Linguistics, 47(2):255–308. find our card
18. Nafise Sadat Moosavi and Michael Strube. 2018. Using linguistic features to improve the generalization capability of neural coreference resolvers. In Proceedings of EMNLP 2018, pages 193–203, Brussels. find our card
19. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the ARRAU corpus. In Proceedings of CRAC 2018, pages 11–22, New Orleans, LA. find
20. Massimo Poesio, Roland Stuckardt, and Yannick Versley. 2016. Anaphora resolution. Springer. find
21. Yilun Zhu, Sameer Pradhan, and Amir Zeldes. 2021. OntoGUM: Evaluating contextualized SOTA coreference resolution on 12 more genres. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 461–467, Online. Association for Computational Linguistics. ↗
22. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of ACL 2014, pages 30–35, Baltimore, MD. find our card
23. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of CoNLL 2013, pages 143–152, Sofia. find
24. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. find our card
25. Marta Recasens, Marie-Catherine de Marneffe, and Christopher Potts. 2013. The life and death of discourse entities: Identifying singleton mentions. In Proceedings of NAACL 2013, pages 627–633, Atlanta, Georgia. find
26. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. TACL, pages 605–617. find our card
27. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A large training corpus for enhanced processing. In Joseph Olive, Caitlin Christianson, and John McCary, editors, Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation. Springer. find
28. Sam Wiseman, Alexander M. Rush, Stuart Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of ACL-IJCNLP 2015, Long Papers, pages 1416–1426, Beijing, China. find
29. Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning global features for coreference resolution. In Proceedings of NAACL 2016, pages 994–1004, San Diego, CA. find
30. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/zhu-2021-anatomy-ontogum-adapting-gum-ontonotes/