Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art

What it is about

The paper asks why published coreference scores differ so much between the MUC and ACE data sets. It uses the RECONCILE platform: a classifier with about 61 features, an averaged perceptron, and single-link clustering. Three subtasks are measured by giving the resolver more information. Gold named entities help only by about 0.5 to 3 points. Perfect coreference elements (CEs) help a great deal, and the authors estimate that better CE detection could add 10 to 20 MUC F points. Knowing anaphoricity also helps. The second part splits anaphors into nine resolution classes: proper names and common nouns, each with exact string match, partial match or no match to the antecedent, and three pronoun types. The average MUC-RC scores are 0.89 for proper names with exact match, 0.21 for proper names with no match, 0.18 for common nouns with no match, 0.80 for gendered third-person pronouns and 0.48 for ungendered ones. A corpus's predicted score from its class mix correlates with the observed score at 0.913 without MUC-6 (0.731 with it).

Ideas

• I1. Define the resolution classes in coref/src/eval.rs as a table keyed by string tests (exact match, shared content word, no overlap, pronoun person and gender). Report a per-class score in every run. The classes can be read from the strings alone, so they are cheap to compute.
• I2. Predict a new corpus's score from its class shares and the per-class averages on training data. Use it as a check only: measure the gap on a held-out corpus, and say so when the gap is large.
• I3. State the CE regime for every number: gold CEs, predicted CEs, or anaphoric-only. The paper shows that gold-CE results can look far better than end-to-end ones.
• I4. Before improving the named-entity tagger, swap in gold NE tags on a dev set and measure the gain. The paper's result suggests the gain will be small, which would save effort.

For Mova

The code is coref/src/eval.rs (classes and per-class MUC-RC) and coref/src/mention.rs (CE extraction). The chapter is 35. Reference and coreference. The measure is the per-class score on the Mova dev split. Negative control: shuffling antecedents within a class must lower that class's score, and a class that does not move under the shuffle is not measuring anything.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{stoyanov2009conundrums,
  title = {{Conundrums in Noun Phrase Coreference Resolution: Making Sense of the State-of-the-Art}},
  author = {Stoyanov, Veselin and Gilbert, Nathan and Cardie, Claire and Riloff, Ellen},
  booktitle = {Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP},
  year = {2009},
  url = {https://aclanthology.org/P09-1074/}
}

References

26 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Bagga and B. Baldwin. 1998. Algorithms for Scoring Coreference Chains. In In Linguistic Coreference Workshop at LREC 1998. find our card
2. Baldridge, J. 2005. The OpenNLP project. http://opennlp.sourceforge.net/. ↗
3. D. Bean and E. Riloff. 2004. Unsupervised Learning of Contextual Role Knowledge for Coreference Resolution. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL 2004). find
4. Eric Bengtson and Dan Roth. 2008. Understanding the Value of Features for Coreference Resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 294–303. Association for Computational Linguistics. find
5. Alexandra Birch, Miles Osborne, and Philipp Koehn. 2008. Predicting Success in Machine Translation. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 745–754. Association for Computational Linguistics. find
6. Donna Byron. 2001. The Uncommon Denominator: A Proposal for Consistent Reporting of Pronoun Resolution Results. Computational Linguistics, 27(4):569–578. find
7. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating Typed Dependency Parses from Phrase Structure Parses. In LREC. find our card
8. J. Finkel, S. Dingare, H. Nguyen, M. Nissim, and C. Manning. 2004. Exploiting Context for Biomedical Entity Recognition: From Syntax to the Web. In Joint Workshop on Natural Language Processing in Biomedicine and its Applications at COLING 2004. find
9. Yoav Freund and Robert E. Schapire. 1999. Large Margin Classification Using the Perceptron Algorithm. In Machine Learning, pages 277–296. find
10. Megumi Kameyama. 1997. Recognizing Referential Links: An Information Extraction Perspective. In Workshop On Operational Factors In Practical Robust Anaphora Resolution For Unrestricted Texts. find
11. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A Mention-Synchronous Coreference Resolution Algorithm Based on the Bell Tree. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics. find
12. X. Luo. 2005. On Coreference Resolution Performance Metrics. In Proceedings of the 2005 Human Language Technology Conference / Conference on Empirical Methods in Natural Language Processing. find our card
13. Xiaoqiang Luo. 2007. Coreference or Not: A Twin Model for Coreference Resolution. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL 2007). find
14. A. McCallum and B. Wellner. 2004. Conditional Models of Identity Uncertainty with Application to Noun Coreference. In 18th Annual Conference on Neural Information Processing Systems. find
15. Ruslan Mitkov. 2002. Anaphora Resolution. Longman, London. find
16. MUC-6. 1995. Coreference Task Definition. In Proceedings of the Sixth Message Understanding Conference (MUC-6), pages 335–344. find
17. MUC-7. 1997. Coreference Task Definition. In Proceedings of the Seventh Message Understanding Conference (MUC-7). find
18. V. Ng and C. Cardie. 2002a. Identifying Anaphoric and Non-Anaphoric Noun Phrases to Improve Coreference Resolution. In Proceedings of the 19th International Conference on Computational Linguistics (COLING 2002). find
19. V. Ng and C. Cardie. 2002b. Improving Machine Learning Approaches to Coreference Resolution. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. find
20. NIST. 2004. The ACE Evaluation Plan. find
21. S. Petrov and D. Klein. 2007. Improved Inference for Unlexicalized Parsing. In Proceedings of the Annual Meeting of the North American Chapter of the Association for Computational Linguistics (HLT/NAACL 2007). find
22. W. Soon, H. Ng, and D. Lim. 2001. A Machine Learning Approach to Coreference of Noun Phrases. Computational Linguistics, 27(4):521–541. find
23. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, Ellen Riloff, David Buttler, and David Hysom. 2009. Reconcile: A Coreference Resolution Research Platform. Computer Science Technical Report, Cornell University, Ithaca, NY. find
24. Kees van Deemter and Rodger Kibble. 2000. On Coreferring: Coreference in MUC and Related Annotation Schemes. Computational Linguistics, 26(4):629–637. find our card
25. M. Vilain, J. Burger, J. Aberdeen, D. Connolly, and L. Hirschman. 1995. A Model-Theoretic Coreference Scoring Theme. In Proceedings of the Sixth Message Understanding Conference (MUC-6). find
26. Xiaofeng Yang, Guodong Zhou, Jian Su, and Chew Lim Tan. 2003. Coreference Resolution Using Competition Learning Approach. In ACL ’03: Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, pages 176–183. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/stoyanov-2009-conundrums-noun-phrase-coreference-resolution/