Singletons and Coreference Resolution Evaluation

What it is about

The paper asks how singletons (mentions that are never coreferent with another mention) change coreference scores. It uses the English data of SemEval-2010 Task 1 and CoNLL-2011, which differ in whether singletons are annotated, and the two official scorers of those tasks. The system is UBIU, a memory-based mention-pair model. Four settings are compared: singletons kept or removed in the gold standard, and kept or removed in the system output. Singletons in the gold standard raise the scores, and the authors show that the metrics behave very differently. MUC ignores singletons; B3, CEAF and BLANC react to them, and the SemEval and CoNLL scorers differ in how they treat singletons in the system output. A trivial baseline that puts every mention in its own chain already reaches high CEAF and B3 scores on the SemEval English data (the paper's Table 1, singleton column: CEAF F1 71.2, B3 F1 83.2, MUC F1 0.0). The conclusion is that scores are not comparable unless the singleton policy is stated and matched.

Ideas

• I1. Make the singleton policy an explicit setting in coref/src/eval.rs: report the four settings (gold with or without singletons, system with or without) side by side, and refuse to compare two runs whose settings differ.
• I2. Keep a trivial baseline in every evaluation table: all-singletons and all-in-one, as in the paper, so that a score is read against what a system that links nothing would get.
• I3. Twinless mentions: when the sieve proposes a mention that is not in the gold set, log it as a count, not as a silent drop, so that a gain from fewer mentions is visible.

For Mova

Code: coref/src/eval.rs (scorer settings and the baseline rows), coref/src/mention.rs (mention extraction that over-generates NPs). Chapter: 35. Reference and coreference. Measure: the sieve's CoNLL-style scores under the four singleton settings, with the all-singletons row, run twice to check the output is identical.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{kubler2011singletons,
  title = {{Singletons and Coreference Resolution Evaluation}},
  author = {Kübler, Sandra and Zhekova, Desislava},
  booktitle = {Proceedings of the International Conference Recent Advances in Natural Language Processing 2011},
  year = {2011},
  url = {https://aclanthology.org/R11-1036/}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the LREC Workshop on Linguistic Coreference, Granada, Spain. find our card
2. Walter Daelemans, Jakub Zavrel, Ko van der Sloot, and Antal van den Bosch. 2007. TiMBL: Tilburg memory based learner – version 6.1 – reference guide. Technical Report ILK 07-07, ILK-CL, Tilburg University. find
3. Pascal Denis and Jason Baldridge. 2007. Joint determination of anaphoricity and coreference resolution using integer programming. In Proceedings of HLT-NAACL 2007, Rochester, NY. find
4. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: The 90% solution. In Proceedings of HLT-NAACL, New York, NY. find
5. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proceedings of ACL, Barcelona, Spain. find
6. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of HLT-EMNLP, Vancouver, Canada. find our card
7. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of CoNLL 2011, Portland, OR. find our card
8. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of EMNLP, Singapore. find
9. Altaf Rahman and Vincent Ng. 2011. Narrowing the modeling gap: A cluster-ranking approach to coreference resolution. JAIR, 40:469–521. find
10. Marta Recasens and Eduard Hovy. 2011. BLANC: Implementing the Rand Index for coreference evaluation. Natural Language Engineering. find
11. Marta Recasens, Lluı̀s Márquez, Emili Sapena, M. Antònia Martı́, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. SemEval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of SemEval, Uppsala, Sweden. find
12. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL-AFNLP, Singapore. find our card
13. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of MUC-6, Columbia, MD. find our card
14. Holger Wunsch. 2009. Rule-Based and Memory-Based Pronoun Resolution for German: A Comparison and Assessment of Data Sources. Ph.D. thesis, Universität Tübingen. find
15. Desislava Zhekova and Sandra Kübler. 2010. UBIU: A language-independent system for coreference resolution. In Proceedings of SemEval, Uppsala, Sweden. find
16. Desislava Zhekova and Sandra Kübler. 2011. UBIU: A Robust System for Resolving Unrestricted Coreference. In Proceedings of CoNLL: Shared Task, Portland, OR. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kubler-2011-singletons-coreference-resolution-evaluation/