Mova0.5 pre-release⌕
☰
home / docs / papers / Deterministic Coreference Resolution Based on Entity-Centric, Precision-Ranked Rules

Deterministic Coreference Resolution Based on Entity-Centric, Precision-Ranked Rules

Summary

This journal article is the full description of the Stanford multi-pass sieve coreference system, which won the CoNLL-2011 shared task. Instead of a learned pairwise model, a sequence of deterministic "sieves" is applied from the most precise (speaker identification, exact string match) to the least precise (pronoun resolution), so that high-confidence links are made first. The model is entity-centric: each mention can use attributes pooled over its whole cluster (number, gender, animacy, NER type), and unknown attribute values are compatible with anything. Only the first mention of a cluster is resolved, indefinite mentions are not linked except by exact match, and candidate antecedents are ordered by a Hobbs-like syntactic traversal. The paper analyses the contribution of each sieve and shows that this simple, modular and interpretable design is competitive with machine-learned systems across several corpora.

How Mova uses it

Effectiveness in Mova

Measured on GUM test (UD 2.18, head match), from the development notes of coref. CoNLL F1 of the default sieve (12 sieves): 73.04 with singletons / 60.31 without on gold trees, 67.90 / 54.03 on Mova's own trees. Baselines on gold trees: exact string match only 53.07 / 27.88; pronoun → nearest agreeing mention 57.73 / 32.67. Cumulative sieve contributions (CoNLL F1 without singletons, gold trees): speaker +21.88, exact +20.94, strict_a +5.18, pronoun +6.38. Dev decisions: Hobbs order vs pure recency gave pronoun precision 64.2% vs 55.4% (CoNLL 73.43 / 63.34 vs 72.23 / 61.33); a pronoun window of 1–4 sentences made no difference (±0.1), so 3 was kept as in Lee et al.; dropping indefinites from exact match was worse (−0.4 / −0.9), so the paper's rule was kept. For reference, the CRAC 2025 mBERT baseline on en_gum is 61.7 (settings differ, comparison is only indicative).

👨‍🔬💥

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00152

@article{lee2013deterministic,
  title = {{Deterministic Coreference Resolution Based on Entity-Centric, Precision-Ranked Rules}},
  author = {Lee, Heeyoung and Chang, Angel and Peirsman, Yves and Chambers, Nathanael and Surdeanu, Mihai and Jurafsky, Dan},
  journal = {Computational Linguistics},
  year = {2013},
  doi = {10.1162/coli\_a\_00152},
  url = {https://doi.org/10.1162/coli_a_00152}
}

References

87 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bagga, Amit and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The First International Conference on Language Resources and Evaluation Workshop on Linguistics Coreference, volume 1, pages 563–566, Granada. find our card
  2. Baldwin, Breck. 1995. CogNIAC: A Discourse Processing Engine. University of Pennsylvania Department of Computer and Information Sciences. Ph.D. thesis. find
  3. Baldwin, Breck. 1997. Cogniac: High precision coreference with limited knowledge and linguistic resources. In Proceedings of a Workshop on Operational Factors in Practical, Robust Anaphora Resolution for Unrestricted Texts, pages 38–45, Madrid. find our card
  4. Bansal, Mohit and Dan Klein. 2012. Coreference semantics from web features. In Proceedings of ACL 2012, pages 389–398, Jeju Island. find
  5. Bengtson, Eric and Dan Roth. 2008. Understanding the value of features for coreference resolution. In Proceedings of EMNLP 2008, pages 294–303, Honolulu, HI. find
  6. Bergsma, Shane and Dekang Lin. 2006. Bootstrapping path-based pronoun resolution. In Proceedings of COLING-ACL, pages 33–40, Stroudsburg, PA. find
  7. Bergsma, Shane, Dekang Lin, and Randy Goebel. 2008. Distributional identification of non-referential pronouns. In Proceedings of ACL-HLT 2008, pages 10–18, Columbus, OH. find
  8. Borghesi, Luigi and Chiara Favareto. 1982. Flexible parsing of discretely uttered sentences. In Proceedings of the 9th Conference on Computational Linguistics-Volume 1, pages 37–42, Prague. find
  9. Boyd, Adriane, Whitney Gegg-Harrison, and Donna Byron. 2005. Identifying non-referential it: A machine learning approach incorporating linguistically motivated features. In Proceedings of the ACL Workshop on Feature Engineering for Machine Learning in NLP, pages 40–47, Ann Arbor, MI. find
  10. Brennan, Susan E., Marilyn W. Friedman, and Carl Pollard. 1987. A centering approach to pronouns. In Proceedings of the 25th Annual Meeting on Association for Computational Linguistics, pages 155–162, Stanford, CA. find
  11. Brown, Peter F., Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: parameter estimation. Computational Linguistics, 19(2):263–311. find
  12. Cai, Jie, Eva Mujdricza-Maydt, and Michael Strube. 2011. Unrestricted coreference resolution via global hypergraph partitioning. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 56–60, Portland, OR. find
  13. Cardie, Claire and Kiri Wagstaff. 1999. Noun phrase coreference as clustering. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 82–89, College Park, MD. find
  14. Chang, Kai-Wei, Rajhans Samdani, Alla Rozovskaya, Nick Rizzolo, Mark Sammons, and Dan Roth. 2011. Inference protocols for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 40–44, Portland, OR. find
  15. Chen, Chen and Vincent Ng. 2012. Combining the best of two worlds: A hybrid approach to multilingual coreference resolution. In Proceedings of the CoNLL-2012 Shared Task, pages 56–63, Jeju Island. find
  16. Chiticariu, Laura, Rajasekar Krishnamurthy, Yunyao Li, Frederick Reiss, and Shivakumar Vaithyanathan. 2010. Domain adaptation of rule-based annotators for named-entity recognition tasks. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1,002–1,012, Cambridge, MA. find
  17. Chomsky, Noam. 1981. Lectures on Government and Binding. Mouton de Gruyter, Berlin. find
  18. Collins, Michael and Yoram Singer. 1999. Unsupervised models for named entity classification. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora, pages 100–110, College Park, MD. find
  19. Connolly, Dennis, John D. Burger, and David S. Day. 1994. A machine learning approach to anaphoric reference. In Proceedings of the International Conference on New Methods in Language Processing (NeMLaP-1), pages 255–261, Manchester. find
  20. Corazza, A., R. De Mori, R. Gretter, and G. Satta. 1991. Stochastic context-free grammars for island-driven probabilistic parsing. In Proceedings of Second International Workshop on Parsing Technologies (IWPT 91), pages 210–217, Cancun. find
  21. Culotta, Aron, Michael Wick, Robert Hall, and Andrew McCallum. 2007. First-order probabilistic models for coreference resolution. In Proceedings of HLT-NAACL 2007, pages 81–88, Rochester, NY. find
  22. Daumé III, Hal and Daniel Marcu. 2005. A large-scale exploration of effective global features for a joint entity detection and tracking model. In HLT-EMNLP 2005, pages 97–104, Vancouver. find
  23. de Marneffe, Marie-Catherine and Christopher D. Manning. 2008. The Stanford typed dependencies representation. In Proceedings of COLING Workshop on Cross-framework and Cross-domain Parser Evaluation, pages 1–8, Manchester. find our card
  24. Denis, Pascal and Jason Baldridge. 2007. Joint determination of anaphoricity and coreference resolution using integer programming. In Proceedings of NAACL-HLT 2007, pages 236–243, Rochester, NY. find
  25. Denis, Pascal and Jason Baldridge. 2009. Global joint models for coreference resolution and named entity classification. Procesamiento del Lenguaje Natural, 42:87–96. find
  26. Doddington, George, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The Automatic Content Extraction (ACE) program—Tasks, data, and evaluation. In Proceedings of LREC 2004, pages 837–840, Lisbon. find
  27. Elsner, Micha and Eugene Charniak. 2010. The same-head heuristic for coreference. In Proceedings of ACL 2010 Short Papers, pages 33–37, Uppsala. find
  28. Fernandes, Eraldo, Cicero dos Santos, and Ruy Milidiu. 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In Proceedings of the CoNLL-2012 Shared Task, pages 41–48, Jeju Island. find
  29. Finkel, Jenny Rose, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by Gibbs sampling. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, ACL ’05, pages 363–370, Stroudsburg, PA. find
  30. Finkel, Jenny Rose and Christopher D. Manning. 2008. Enforcing transitivity in coreference resolution. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers, pages 45–48, Columbus, OH. find
  31. Fox, Barbara A. 1993. Discourse Structure and Anaphora: Written and Conversational English. Cambridge University Press. find
  32. Greene, Barbara B. and Gerald M. Rubin. 1971. Automatic Grammatical Tagging of English. Brown University Press. find
  33. Gupta, Surabhi, Matthew Purver, and Dan Jurafsky. 2007. Disambiguating between generic and referential “you” in dialog. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pages 105–108, Prague. find
  34. Haghighi, Aria and Dan Klein. 2007. Unsupervised coreference resolution in a nonparametric Bayesian model. find
  35. In Proceedings of ACL 2007, pages 848–855, Prague. find
  36. Haghighi, Aria and Dan Klein. 2009. Simple coreference resolution with rich syntactic and semantic features. In Proceedings of EMNLP 2009, pages 1,152–1,161, Suntec. find our card
  37. Haghighi, Aria and Dan Klein. 2010. Coreference resolution in a modular, entity-centered model. In Proceedings of HLT-NAACL 2010, pages 385–393, Los Angeles, CA. find
  38. Hobbs, Jerry R. 1978. Resolving pronoun references. Lingua, 44(4):311–338. find our card
  39. Ji, Heng and Dekang Lin. 2009. Gender and animacy knowledge discovery from web-scale n-grams for unsupervised person mention detection. In Proceedings of the Pacific Asia Conference on Language, Information and Computation, pages 220–229, Hong Kong. find
  40. Kehler, Andrew. 1997. Probabilistic coreference in information extraction. In Proceedings of EMNLP 1997, pages 163–173, Providence, RI. find
  41. Klein, Dan and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL ’03, pages 423–430, Stroudsburg, PA. find our card
  42. Klein, Sheldon and Robert F. Simmons. 1963. A computational approach to grammatical coding of English words. Journal of the Association for Computing Machinery, 10(3):334–347. find
  43. Kobdani, Hamidreza, Hinrich Schuetze, Michael Schiehlen, and Hans Kamp. 2011a. Bootstrapping coreference resolution using word associations. In Proceedings of ACL HLT 2011, pages 783–792, Portland, OR. find
  44. Kobdani, Hamidreza, Hinrich Schütze, Michael Schiehlen, and Hans Kamp. 2011b. Bootstrapping coreference resolution using word associations. In Proceedings of ACL, pages 783–792, Portland, OR. find
  45. Kuebler, Sandra, Ryan McDonald, and Joakim Nivre. 2009. Dependency Parsing. Morgan and Claypool Publishers. find
  46. Lappin, Shalom and Herbert Leass. 1994. An algorithm for pronominal anaphora resolution. Computational Linguistics, 20(4):535–561. find our card
  47. Lee, Heeyoung, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of CoNLL 2011: Shared Task, pages 28–34, Portland, OR. find our card
  48. Lee, Heeyoung, Marta Recasens, Angel Chang, Mihai Surdeanu, and Dan Jurafsky. 2012. Joint entity and event coreference resolution across documents. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 489–500, Jeju Island. find
  49. Luo, Xiaoqiang. 2005. On coreference resolution performance metrics. In Proceedings of HLT-EMNLP 2005, pages 25–32, Vancouver. find our card
  50. Luo, Xiaoqiang, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the Bell tree. In Proceedings of ACL 2004, pages 21–26, Barcelona. find
  51. McCallum, Andrew and Ben Wellner. 2004. Conditional models of identity uncertainty with application to noun coreference. In Proceedings of NIPS 2004, pages 905–912, Vancouver. find
  52. McCarthy, Joseph F. and Wendy G. Lehnert. 1995. Using decision trees for coreference resolution. In Proceedings of IJCAI 1995, pages 1,050–1,055, Montréal. find
  53. Morton, Thomas S. 2000. Coreference for NLP applications. In Proceedings of ACL 2000, pages 173–180, Hong Kong. find
  54. Ng, Vincent. 2008. Unsupervised models for coreference resolution. In Proceedings of EMNLP 2008, pages 640–649, Honolulu, HI. find
  55. Ng, Vincent. 2009. Graph-cut-based anaphoricity determination for coreference resolution. In Proceedings of NAACL-HLT 2009, pages 575–583, Boulder, CO. find
  56. Ng, Vincent. 2010. Supervised noun phrase coreference research: The first fifteen years. In Proceedings of ACL, pages 1,396–1,411, Uppsala. find our card
  57. Ng, Vincent and Claire Cardie. 2002a. Identifying anaphoric and non-anaphoric noun phrases to improve coreference resolution. In Proceedings of COLING, pages 1–7, Taipei. find
  58. Ng, Vincent and Claire Cardie. 2002b. Improving machine learning approaches to coreference resolution. In Proceedings of ACL 2002, pages 104–111, Philadelphia, PA. find
  59. Poesio, Massimo, Rahul Mehta, Axel Maroudas, and Janet Hitzeman. 2004a. Learning to resolve bridging references. In Proceedings of ACL, pages 143–150, Barcelona. find
  60. Poesio, Massimo, Olga Uryupina, Renata Vieira, Mijail Alexandrov-Kabadjov, and Rodrigo Goulart. 2004b. Discourse-new detectors for definite description resolution: A survey and a preliminary proposal. In ACL 2004: Workshop on Reference Resolution and its Applications, pages 47–54, Barcelona. find
  61. Poon, Hoifung and Pedro Domingos. 2008. Joint unsupervised coreference resolution with Markov logic. In Proceedings of EMNLP 2008, pages 650–659, Honolulu, HI. find
  62. Pradhan, Sameer, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL), page 1, Jeju Island. find our card
  63. Pradhan, Sameer, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning (CoNLL), pages 1–27, Portland, OR. find our card
  64. Raghunathan, Karthik, Heeyoung Lee, Sudarshan Rangarajan, Nathanael Chambers, Mihai Surdeanu, Dan Jurafsky, and Chris Manning. 2010. A multi-pass sieve for coreference resolution. In Proceedings of EMNLP 2010, pages 492–501, Cambridge, MA. find our card
  65. Rahman, Altaf and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 968–977, Suntec. find
  66. Rand, William M. 1971. Objective criteria for the evaluation of clustering methods. Journal of the American Statistical Association, 66(336):846–850. find
  67. Recasens, Marta and Eduard Hovy. 2010. Coreference resolution across corpora: Languages, coding schemes, and preprocessing information. In Proceedings of ACL 2010, pages 1,423–1,432, Uppsala. find our card
  68. Recasens, Marta, Matthew Can, and Dan Jurafsky. 2013. Same referent, different words: Unsupervised mining of opaque coreferent mentions. In Proceedings of NAACL 2013, pages 897–906, Atlanta. find
  69. Recasens, Marta and Eduard Hovy. 2011. BLANC: Implementing the Rand index for coreference evaluation. Natural Language Engineering, 17(4):485–510. find
  70. Sapena, Emili, Lluı́s Padró, and Jordi Turmo. 2011. Relaxcor participation in CoNLL-shared task on coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 35–39, Portland, OR. find
  71. Shou, Heming and Hai Zhao. 2012. System paper for CoNLL-2012 shared task: Hybrid rule-based algorithm for coreference resolution. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 118–121, Jeju Island. find
  72. Skinner, B. F. 1938. The Behavior of Organisms: An Experimental Analysis. Appleton-Century-Crofts. find
  73. Soon, Wee M., Hwee T. Ng, and Daniel C. Y. Lim. 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27(4):521–544. find our card
  74. Spitkovsky, Valentin I., Hiyan Alshawi, and Daniel Jurafsky. 2010. From baby steps to leapfrog: How “less is more” in unsupervised dependency parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, HLT ’10, pages 751–759, Stroudsburg, PA. find
  75. Stoyanov, Veselin and Jason Eisner. 2012. Easy-first coreference resolution. In Proceedings of COLING 2012, pages 2,519–2,534, Mumbai. find
  76. Uryupina, Olga, Massimo Poesio, Claudio Giuliano, and Kateryna Tymoshenko. 2011a. Disambiguation and filtering methods in using web knowledge for coreference resolution. In FLAIRS Conference, pages 317–322, Palm Beach, FL. find
  77. Uryupina, Olga, Massimo Poesio, Claudio Giuliano, and Kateryna Tymoshenko. 2011b. Disambiguation and filtering methods in using web knowledge for coreference resolution. In Proceedings of FLAIRS, pages 317–322, Palm Beach, FL. find
  78. Vieira, Renata and Massimo Poesio. 2000. An empirically based system for processing definite descriptions. Computational Linguistics, 26(4):539–593. find
  79. Vilain, Marc, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of MUC-6, pages 45–52, Columbia, MD. find our card
  80. Vogel, Adam and Dan Jurafsky. 2012. He Said, She Said: Gender in the ACL Anthology. In ACL Workshop on Rediscovering 50 Years of Discoveries, pages 33–41, Jeju Island. find
  81. Xiong, Hao and Qun Liu. 2012. Ict: System description for CoNLL-2012. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 71–75, Jeju Island. find
  82. Yang, Xiaofeng and Jian Su. 2007. Coreference resolution using semantic relatedness information from automatically discovered patterns. In Proceedings of ACL 2007, pages 525–535, Prague. find
  83. Yang, Xiaofeng, Jian Su, Jun Lang, Chew L. Tan, Ting Liu, and Sheng Li. 2008. An entity-mention model for coreference resolution with inductive logic programming. In Proceedings of ACL-HLT 2008, pages 843–851, Columbus, OH. find
  84. Yang, Xiaofeng, Guodong Zhou, Jian Su, and Chew L. Tan. 2004. An NP-cluster approach to coreference resolution. In Proceedings of COLING 2004, pages 219–225, Geneva. find
  85. Yuan, Bo, Qingcai Chen, Yang Xiang, Xiaolong Wang, Liping Ge, Zengjian Liu, Meng Liao, and Xianbo Si. 2012. A mixed deterministic model for coreference resolution. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 76–82, Jeju Island. find
  86. Zhang, Xiaotian, Chunyang Wu, and Hai Zhao. 2012. Chinese coreference resolution via ordered filtering. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 95–99, Jeju Island. find
  87. Zhou, Guodong and Jian Su. 2004. A high-performance coreference resolution system using a constraint-based multi-agent strategy. In Proceedings of the 16th International Conference on Computational Linguistics (COLING), page 522, Geneva. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lee-2013-deterministic-coreference-resolution-based-entity/