Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric

What it is about

The paper shows a flaw shared by B³, CEAF and BLANC, the «mention identification effect»: because they count common mentions rather than links, a system gains recall and precision merely by putting gold mentions into its output, even when those mentions are linked wrongly; experiments on CoNLL outputs that add completely wrong entities of gold mentions raise B³ and CEAF scores. MUC is immune but weak (it cannot tell merging two singletons from merging two big entities, and it favours over-merging), so averaging the three (the CoNLL score) averages unreliable numbers. The proposed LEA weighs each key entity by its importance (its size) and scores how well it is resolved as the fraction of its coreference links (n(n−1)/2) found in the response; precision swaps key and response; singletons count through a self-link. LEA ranks systems more sensibly and was added to the official CoNLL scorer.

Ideas

• I1. Add LEA to coref/src/score.rs next to MUC, B³, CEAF-e/m and CoNLL F1, with self-links for singletons, and give it the same «checks that can show red»: gold against itself = 1; all-separate output gets LEA recall 0 on non-singleton entities; one giant cluster gets low precision. Report it in coref eval in both settings (with and without singletons).
• I2. Read Mova's «with singletons» figures with the paper's warning: a large part of the gap between with and without singletons (73.0 vs 60.3 on gold trees) is mention identification, not linking. LEA without that effect gives one number to track sieve changes; check whether past dev decisions (e.g. disabling strict_c) would have gone the same way under LEA.
• I3. Use LEA's importance slot for a task-oriented score: weight each gold entity by how often the school courses ask about it (or by being a character in a tale), so coreference work is measured where the student needs it. A small extension of the same scorer.

For Mova

35. Reference and coreference §7 and §9.3 (annotation, results). Code: coref/src/score.rs, coref/src/eval.rs. Measures: LEA on GUM dev/test alongside CoNLL F1; agreement of past dev decisions under LEA.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1060

@inproceedings{moosavi2016which,
  title = {{Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric}},
  author = {Moosavi, Nafise Sadat and Strube, Michael},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1060},
  url = {https://doi.org/10.18653/v1/p16-1060}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 Shared Task: Modeling unrestricted coreference in OntoNotes. In Proceedings of the Shared Task of the 15th Conference on Computational Natural Language Learning, Portland, Oreg., 23–24 June 2011, pages 1–27. find our card
2. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain, 28–30 May 1998, pages 563–566. find our card
3. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Beijing, China, 26–31 July 2015, pages 1405–1415. find
4. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Shared Task of the 16th Conference on Computational Natural Language Learning, Jeju Island, Korea, 12–14 July 2012, pages 1–40. find our card
5. Pascal Denis and Jason Baldridge. 2009a. Global joint models for coreference resolution and named entity classification. Procesamiento del Lenguaje Natural, (42):87–96. find
6. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Baltimore, Md., 22–27 June 2014, pages 30– 35. find our card
7. Pascal Denis and Jason Baldridge. 2009b. Global joint models for coreference resolution and named entity classification. Procesamiento del Lenguaje Natural, 42:87–96, March. find
8. Gordana Ilic Holen. 2013. Critical reflections on evaluation practices in coreference resolution. In Proceedings of the 2013 NAACL HLT Student Research Workshop, Atlanta, Georgia, 9-14 June 2013, pages 1–7. find
9. William R. Rand. 1971. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(336):846–850. find
10. Marta Recasens and Eduard Hovy. 2011. BLANC: Implementing the Rand index for coreference evaluation. Natural Language Engineering, 17(4):485– 510. find
11. Xiaoqiang Luo and Sameer Pradhan. 2016. Evaluation metrics. In M. Poesio, R. Stuckardt, and Y. Versley, editors, Anaphora Resolution: Algorithms, Resources, and Applications. Springer. To appear. find
12. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of the Joint Conference of the 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing, Singapore, 2–7 August 2009, pages 656–664. find our card
13. Xiaoqiang Luo, Sameer Pradhan, Marta Recasens, and Eduard Hovy. 2014. An extension of BLANC to system mentions. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 24–29, Baltimore, Maryland, June. Association for Computational Linguistics. find
14. Roland Stuckhardt. 2003. Coreference-based summarization and question answering: A case for high precision anaphor resolution. In Proceedings of the 2003 International Symposium on Reference Resolution and Its Applications to Question Answering and Summarization, Venice, Italy, 23–24 June 2003, pages 33–42. find
15. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of the Human Language Technology Conference and the 2005 Conference on Empirical Methods in Natural Language Processing, Vancouver, B.C., Canada, 6–8 October 2005, pages 25–32. find our card
16. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics, 3:405–418. find
17. Don Tuggener. 2014. Coreference resolution evaluation for higher level applications. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, Volume 2: Short Papers, Gothenburg, Sweden, 26–30 April 2014, pages 231–235. find
18. Eric W. Noreen. 1989. Computer Intensive Methods for Hypothesis Testing: An Introduction. Wiley, New York, N.Y. find
19. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference (MUC-6), pages 45–52, San Mateo, Cal. Morgan Kaufmann. find our card
20. Haoruo Peng, Kai-Wei Chang, and Dan Roth. 2015. A joint framework for coreference resolution and mention head detection. In Proceedings of the 19th Conference on Computational Natural Language Learning, Beijing, China, 30–31 July 2015, pages 12–21. find
21. Sam Wiseman, Alexander M. Rush, Stuart Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Beijing, China, 26–31 July 2015, pages 1416–1426. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/moosavi-2016-which-coreference-evaluation-metric-do/