Mova0.5 pre-release⌕
☰
home / docs / papers / English / Error-Driven Analysis of Challenges in Coreference Resolution

Error-Driven Analysis of Challenges in Coreference Resolution

What it is about

The paper gives an automatic classifier of coreference errors that works only on system output, so it applies to link-based and global-inference systems alike. The system output is turned into the gold annotation by a sequence of edits (alter span, split, merge, add and remove mentions, and so on), and each edit is mapped to a linguistically named error type: for example, an entity divided into pieces, a spurious entity, or a non-referring pronoun given an antecedent. The authors apply it to all systems of the CoNLL 2011 shared task plus several public systems, and then to ten top systems, measuring for each error type how much the standard metrics would gain if that error were fixed. Their findings, as stated in the introduction: the best systems are not best on every error type, a large share of span errors comes from superficial parse differences, and the largest loss is on missed entities with few mentions. The tool is released as the Berkeley coreference analyser. I read the abstract, introduction and the error-classification section; I did not check the later result tables.

Ideas

For Mova

Code: coref/src/eval.rs (new error analysis), coref/src/mention.rs (span errors, mention heads), coref/src/sieve.rs (the output to classify). Chapter: 35. Reference and coreference. Measure: the error-type counts on one corpus with the sieve and with gold parses, run twice; the missed-entity count is the first target, as in the paper.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{kummerfeld2013error,
  title = {{Error-Driven Analysis of Challenges in Coreference Resolution}},
  author = {Kummerfeld, Jonathan K. and Klein, Dan},
  booktitle = {Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  url = {https://aclanthology.org/D13-1027/}
}

References

47 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of The First International Conference on Language Resources and Evaluation Workshop on Linguistics Coreference, pages 563–566. find our card
  2. Eric Bengtson and Dan Roth. 2008. Understanding the value of features for coreference resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 294–303. find
  3. Anders Björkelund and Richárd Farkas. 2012. Data-driven multilingual coreference resolution using resolver stacking. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 49–55. find
  4. Anders Björkelund and Pierre Nugues. 2011. Exploring lexicalized features for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 45–50. find
  5. Jie Cai, Eva Mujdricza-Maydt, and Michael Strube. 2011. Unrestricted coreference resolution via global hypergraph partitioning. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 56–60. find
  6. Kai-Wei Chang, Rajhans Samdani, Alla Rozovskaya, Nick Rizzolo, Mark Sammons, and Dan Roth. 2011. Inference protocols for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 40–44. find
  7. Eric Charton and Michel Gagnon. 2011. Poly-co: a multilayer perceptron approach for coreference detection. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 97–101. find
  8. Chen Chen and Vincent Ng. 2012. Combining the best of two worlds: A hybrid approach to multilingual coreference resolution. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 56–63. find
  9. Weipeng Chen, Muyu Zhang, and Bing Qin. 2011. Coreference resolution system using maximum entropy classifier. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 127–130. find
  10. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. find our card
  11. Aria Haghighi and Dan Klein. 2009. Simple coreference resolution with rich syntactic and semantic features. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1152–1161. find our card
  12. Gordana Ilic Holen. 2013. Critical reflections on evaluation practices in coreference resolution. In Proceedings of the 2013 NAACL HLT Student Research Workshop, pages 1–7. find
  13. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: the 90% solution. In Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, pages 57–60. find
  14. Joseph Irwin, Mamoru Komachi, and Yuji Matsumoto. 2011. Narrative schema as world knowledge for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 86–92. find
  15. Manfred Klenner and Don Tuggener. 2011. An incremental model for coreference resolution with restrictive antecedent accessibility. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 81–85. find
  16. Hamidreza Kobdani and Hinrich Schuetze. 2011. Supervised coreference resolution with sucre. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 71– 75. find
  17. Jonathan K. Kummerfeld, Mohit Bansal, David Burkett, and Dan Klein. 2011. Mention detection: Heuristics for the ontonotes annotations. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 102–106. find
  18. Jonathan K. Kummerfeld, David Hall, James R. Curran, and Dan Klein. 2012. Parser showdown at the wall street corral: An empirical investigation of error types in parser output. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1048–1059. find our card
  19. Sobha Lalitha Devi, Pattabhi Rao, Vijay Sundar Ram R, M. C S, and A. A. 2011. Hybrid approach for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 93–96. find
  20. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanfords multi-pass sieve coreference resolution system at the conll-2011 shared task. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 28–34. find our card
  21. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4). find our card
  22. Xinxin Li, Xuan Wang, and Shuhan Qi. 2011. Coreference resolution with loose transitivity constraints. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 107–111. find
  23. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25– 32. find our card
  24. Sebastian Martschat, Jie Cai, Samuel Broscheit, Éva Mújdricza-Maydt, and Michael Strube. 2012. A multigraph model for coreference resolution. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 100–106. find
  25. Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, pages 104–111. find
  26. Cicero Nogueira dos Santos and Davi Lopes Carvalho. 2011. Rule and tree ensembles for unrestricted coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 51–55. find
  27. C. D. Paice and G. D. Husk. 1987. Towards the automatic recognition of anaphoric features in english text: the impersonal pronoun ‘it’. Computer Speech & Language, 2(2):109–132. find
  28. Sameer Pradhan, Lance Ramshaw, Ralph Weischedel, Jessica MacBride, and Linnea Micciulla. 2007. Unrestricted coreference: Identifying entities and events in ontonotes. In Proceedings of the International Conference on Semantic Computing, pages 446–453. find
  29. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In Proceedings of the 15th Conference on Computational Natural Language Learning (CoNLL 2011), pages 1–27. find our card
  30. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40. find our card
  31. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 968–977. find
  32. M. Recasens and E. Hovy. 2011. BLANC: Implementing the rand index for coreference evaluation. Natural Language Engineering, 17:485–510, 9. find
  33. Emili Sapena, Lluı́s Padró, and Jordi Turmo. 2011. Re-laxcor participation in conll shared task on coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 35–39. Lalitha Devi. Sobha, RK. Rao. Pattabhi, R. Vijay Sundar Ram, CS. find
  34. Malarkodi, and A. Akilandeswari. 2011. Hybrid approach for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 93–96. find
  35. Yang Song, Houfeng Wang, and Jing Jiang. 2011. Link type based pre-cluster pair model for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 131–135. find
  36. Veselin Stoyanov, Nathan Gilbert, Claire Cardie, and Ellen Riloff. 2009. Conundrums in noun phrase coreference resolution: making sense of the state-of-the-art. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 656–664. find our card
  37. Veselin Stoyanov, Claire Cardie, Nathan Gilbert, Ellen Riloff, David Buttler, and David Hysom. 2010. Coreference resolution with reconcile. In Proceedings of the ACL 2010 Conference Short Papers, pages 156–161. find
  38. Veselin Stoyanov, Uday Babbar, Pracheer Gupta, and Claire Cardie. 2011. Reconciling ontonotes: Unrestricted coreference resolution in ontonotes with reconcile. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 122–126. find
  39. Olga Uryupina, Sriparna Saha, Asif Ekbal, and Massimo Poesio. 2011. Multi-metric optimization for coreference: The unitn / iitp / essex submission to the 2011 conll shared task. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 61–65. find
  40. Yannick Versley, Simone Paolo Ponzetto, Massimo Poesio, Vladimir Eidelman, Alan Jern, Jason Smith, Xiaofeng Yang, and Alessandro Moschitti. 2008. Bart: a modular toolkit for coreference resolution. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Demo Session, pages 9–12. find
  41. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the Sixth Message Uunderstanding Conference, pages 45–52. find our card
  42. Hao Xiong, Linfeng Song, Fandong Meng, Yang Liu, Qun Liu, and Yajuan Lv. 2011. Ets: An error tolerable system for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 76–80. find
  43. Yaqin Yang, Nianwen Xue, and Peter Anick. 2011. A machine learning-based coreference detection system for ontonotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 117–121. find
  44. Bo Yuan, Qingcai Chen, Yang Xiang, Xiaolong Wang, Liping Ge, Zengjian Liu, Meng Liao, and Xianbo Si. 2012. A mixed deterministic model for coreference resolution. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 76–82. find
  45. Desislava Zhekova and Sandra Kübler. 2011. Ubiu: A robust system for resolving unrestricted coreference. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 112–116. find
  46. Desislava Zhekova, Sandra Kübler, Joshua Bonner, Marwa Ragheb, and Yu-Yin Hsu. 2012. Ubiu for multilingual coreference resolution in ontonotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 88–94. find
  47. Huiwei Zhou, Yao Li, Degen Huang, Yan Zhang, Chunlong Wu, and Yuansheng Yang. 2011. Combining syntactic and semantic features by svm for unrestricted coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 66–70. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kummerfeld-2013-error-driven-analysis-challenges-coreference/