On Coreference Resolution Performance Metrics
Summary
The paper criticises existing coreference metrics, notably MUC (which ignores singletons and rewards over-merging) and the ACE value, and proposes CEAF (Constrained Entity-Alignment F-measure). CEAF finds the best one-to-one alignment between key and response entities, computed with the Kuhn-Munkres (Hungarian) algorithm, under a chosen entity-similarity function. Recall and precision are the total similarity of the alignment normalised by the self-similarity of the key and of the response respectively. Two similarity functions give the mention-based variant (number of shared mentions) and the entity-based variant (a Dice-like overlap per entity pair). The paper shows that CEAF behaves more sensibly than MUC on degenerate outputs such as putting all mentions in one cluster. CEAF later became one of the three components of the CoNLL coreference score.
How Mova uses it
coref/src/score.rs— Mova's own Rust scorer implements CEAF-e (φ4, entity-based) and CEAF-m (φ3, mention-based), with the optimal entity alignment found by Kuhn-Munkres, computed separately per connected component of the overlap graph.- CEAF-e is one of the three terms of the CoNLL F1 reported for the coref crate (average of MUC, B³ and CEAF-e).
- Unit tests check hand-computed CEAF values and the degenerate cases (one giant cluster, all singletons).
Effectiveness in Mova
It is an evaluation metric, not a modelling idea, so it has no "gain" of its own. It is part of every coref measurement, e.g. the default system's CEAF-e F1 on GUM test is 74.68 with singletons and 52.89 without on gold trees (the development notes of coref).
👨🔬💥
Cite
@inproceedings{luo2005coreference,
title = {{On Coreference Resolution Performance Metrics}},
author = {Luo, Xiaoqiang},
booktitle = {Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing},
year = {2005},
url = {https://aclanthology.org/H05-1004/}
}References
- Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the Linguistic Coreference Workshop at The First International Conference on Language Resources and Evaluation (LREC’98), pages 563–566. find our card
- Egon Balas, Donald Miller, Joseph Pekny, and Paolo Toth. 1991. A parallel shortest augmenting path algorithm for the assignment problem. Journal of the ACM (JACM), 38(4). find
- Francois Bourgeois and Jean-Claude Lassalle. 1971. An extension of the munkres algorithm for the assignment problem to rectangular matrices. Communications of the ACM, 14(12). find
- R. Fletcher. 1987. Practical Methods of Optimization. John Wiley and Sons. find
- Anshul Gupta and Lexing Ying. 1999. Algorithms for finding maximum matchings in bipartite graphs. Technical Report RC 21576 (97320), IBM T.J. Watson Research Center, October. find
- H.W. Kuhn. 1955. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(83). find
- Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proc. of ACL. find
- MUC-6. 1995. Proceedings of the Sixth Message Understanding Conference(MUC-6), San Francisco, CA. Morgan Kaufmann. find
- MUC-7. 1998. Proceedings of the Seventh Message Understanding Conference(MUC-7). find
- J. Munkres. 1957. Algorithms for the assignment and transportation problems. Journal of SIAM, 5:32–38. find
- NIST. 2003a. The ACE evaluation plan. www.nist.gov/speech/tests/ace/index.htm. find
- NIST. 2003b. Proceedings of ACE’03 workshop. Booklet, Alexandria, VA, September. find
- M. Vilain, J. Burger, J. Aberdeen, D. Connolly, , and L. Hirschman. 1995. A model-theoretic coreference scoring scheme. In In Proc. of MUC6, pages 45–52. find our card
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/luo-2005-coreference-resolution-performance-metrics/