Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation

Summary

Coreference metrics such as MUC, B-cubed and CEAF were originally defined for gold mentions, and different research groups adapted them to predicted mentions in incompatible ways, often by adding or removing mentions artificially so that the key and response partitions line up. This made published scores hard to compare. The paper states how each metric should be computed directly on partitions with predicted mentions, without such manipulations: a mention present in only one partition simply contributes nothing where it has no counterpart. It walks through worked examples and describes a reference implementation of the scorer that became the standard tool for CoNLL-style evaluation. The aim is a single, transparent definition so that results across systems and papers are comparable.

How Mova uses it

• coref/src/score.rs — the MUC, B-cubed, CEAF-e and CEAF-m formulas for predicted mentions follow this paper: no mention manipulation, unmatched mentions just do not contribute; CoNLL F1 is the mean of MUC, B-cubed and CEAF-e, and multi-document totals sum numerators and denominators.
• The worked example from the paper's Section 4 (key {a,b,c}{d,e,f,g}, response {a,b}{c,d}{f,g,h,i}) is a unit test of the Rust scorer, checking MUC, B-cubed, CEAF-e and CEAF-m values. The test documents that the paper's printed B-cubed F1 of 0.46 comes from rounded precision and recall; the exact value is 0.4545.

Effectiveness in Mova

Not an idea that changes scores, so not measured as a gain. Its role is correctness of the measuring instrument: the scorer reproduces the paper's worked example, and every reported coreference result (for example CoNLL F1 73.04 with singletons and 60.31 without on GUM test, gold trees) depends on it.

👨‍🔬💥

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-2006

@inproceedings{pradhan2014scoring,
  title = {{Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation}},
  author = {Pradhan, Sameer and Luo, Xiaoqiang and Recasens, Marta and Hovy, Eduard and Ng, Vincent and Strube, Michael},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-2006},
  url = {https://doi.org/10.3115/v1/p14-2006}
}

References

24 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of LREC, pages 563–566. find our card
2. Jie Cai and Michael Strube. 2010. Evaluation metrics for end-to-end coreference resolution systems. In Proceedings of SIGDIAL, pages 28–36. find
3. Chen Chen and Vincent Ng. 2013. Linguistically aware coreference evaluation metrics. In Proceedings of the Sixth IJCNLP, pages 1366–1374, Nagoya, Japan, October. find
4. Nancy Chinchor and Beth Sundheim. 2003. Message understanding conference (MUC) 6. In LDC2003T13. find
5. Nancy Chinchor. 2001. Message understanding conference (MUC) 7. In LDC2001T02. find
6. George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extraction (ACE) program-tasks, data, and evaluation. In Proceedings of LREC. find
7. Lynette Hirschman and Nancy Chinchor. 1997. Coreference task definition (v3.0, 13 jul 97). In Proceedings of the 7th Message Understanding Conference. find
8. Gordana Ilic Holen. 2013. Critical reflections on evaluation practices in coreference resolution. In Proceedings of the NAACL-HLT Student Research Workshop, pages 1–7, Atlanta, Georgia, June. find
9. Daniel Jurafsky and James H. Martin. 2008. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Prentice Hall. Second Edition. find
10. Xiaoqiang Luo, Sameer Pradhan, Marta Recasens, and Eduard Hovy. 2014. An extension of BLANC to system mentions. In Proceedings of ACL, Baltimore, Maryland, June. find
11. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of HLT-EMNLP, pages 25–32. find our card
12. Sameer Pradhan, Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2007. OntoNotes: A Unified Relational Semantic Representation. International Journal of Semantic Computing, 1(4):405–419. find
13. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of CoNLL: Shared Task, pages 1–27. find our card
14. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of CoNLL: Shared Task, pages 1–40. find our card
15. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of EMNLP, pages 968–977. find
16. Altaf Rahman and Vincent Ng. 2011. Coreference resolution with world knowledge. In Proceedings of ACL, pages 814–824. find
17. Marta Recasens and Eduard Hovy. 2011. BLANC: Implementing the Rand Index for coreference evaluation. Natural Language Engineering, 17(4):485– 510. find
18. Marta Recasens, Lluı́s Màrquez, Emili Sapena, M. Antònia Martı́, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. Semeval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of SemEval, pages 1–8. find
19. Marta Recasens, Marie-Catherine de Marneffe, and Chris Potts. 2013. The life and death of discourse entities: Identifying singleton mentions. In Proceedings of NAACL-HLT, pages 627–633. find
20. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL-IJCNLP, pages 656–664. find our card
21. William F. Styler, Steven Bethard an Sean Finan, Martha Palmer, Sameer Pradhan, Piet C de Groen, Brad Erickson, Timothy Miller, Chen Lin, Guergana Savova, and James Pustejovsky. 2014. Temporal annotation in the clinical domain. Transactions of Computational Linguistics, 2(April):143–154. find
22. Ozlem Uzuner, Andreea Bodnari, Shuying Shen, Tyler Forbush, John Pestian, and Brett R South. 2012. Evaluating the state of the art in coreference resolution for electronic medical records. Journal of American Medical Informatics Association, 19(5), September. find
23. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference, pages 45–52. find our card
24. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A large training corpus for enhanced processing. In Joseph Olive, Caitlin Christianson, and John McCary, editors, Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation. Springer. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/pradhan-2014-scoring-coreference-partitions-predicted-mentions/