Recall Error Analysis for Coreference Resolution

What it is about

The paper proposes a link-based framework for analysing coreference errors and uses it for a recall error analysis of four English systems, two deterministic and two learning-based. Recall errors are missed links, such as the link between "the factory" and "the Qinghai Petroleum Bureau's Ge'ermu oil refinery", which needs the hypernymy that "factory" is a hypernym of "refinery"; no string match gives it. The framework counts, for each missed link, which mention pairs a system could have linked and why it did not. The main finding, from the abstract, is that most recall errors for nouns and names are shared by all four systems, and the authors describe this shared error set by lexical and semantic properties of the mentions. A brief precision analysis is also given. A toolkit implementing the framework is released. I read the abstract, the introduction and the framework's opening; I did not read the result tables.

Ideas

• I1. Link-based error log for coref/src/eval.rs: for every gold link the sieve misses, store the pair, the rule that rejected it and the reason, so the recall errors can be counted by rule and by lexical class.
• I2. Shared-error set: run the sieve and a second configuration (for example, with and without the head-match rule), and report the missed links that both miss. The paper suggests such shared errors are the ones that need a new knowledge source.
• I3. Hypernym source for en/src/dict.rs: for nominal pairs with no string overlap, check whether the head of one mention is a hypernym of the other in the lexicon, and count how many shared recall errors that would fix before adding it to the sieve.

For Mova

Code: coref/src/eval.rs (the link-based log), coref/src/sieve.rs (rule that rejects a pair), en/src/dict.rs and world/src/dictionary.rs (hypernyms for I3). Chapter: 35. Reference and coreference, 19. The lexicon: vocabulary, multiword expressions and register. Measure: the count of missed noun and name links by rule, with the shared set between two configurations, run twice to confirm the counts are identical.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/v1/d14-1221

@inproceedings{martschat2014recall,
  title = {{Recall Error Analysis for Coreference Resolution}},
  author = {Martschat, Sebastian and Strube, Michael},
  booktitle = {Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2014},
  doi = {10.3115/v1/d14-1221},
  url = {https://doi.org/10.3115/v1/d14-1221}
}

References

33 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mira Ariel. 1988. Referring and accessibility. Journal of Linguistics, 24(1):65–87. find
2. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain, 28–30 May 1998, pages 563–566. find our card
3. Mohit Bansal and Dan Klein. 2012. Coreference semantics from web features. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Jeju Island, Korea, 8–14 July 2012, pages 389–398. find
4. Eric Bengtson and Dan Roth. 2008. Understanding the value of features for coreference resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Waikiki, Honolulu, Hawaii, 25–27 October 2008, pages 294– 303. find
5. Anders Björkelund and Richárd Farkas. 2012. Data-driven multilingual coreference resolution using resolver stacking. In Proceedings of the Shared Task of the 16th Conference on Computational Natural Language Learning, Jeju Island, Korea, 12–14 July 2012, pages 49–55. find
6. Jie Cai and Michael Strube. 2010. End-to-end coreference resolution via hypergraph partitioning. In Proceedings of the 23rd International Conference on Computational Linguistics, Beijing, China, 23– 27 August 2010, pages 143–151. find
7. Chen Chen and Vincent Ng. 2012. Combining the best of two worlds: A hybrid approach to multilingual coreference resolution. In Proceedings of the Shared Task of the 16th Conference on Computational Natural Language Learning, Jeju Island, Korea, 12–14 July 2012, pages 56–63. find
8. Chen Chen and Vincent Ng. 2013. Linguistically aware coreference evaluation metrics. In Proceedings of the 6th International Joint Conference on Natural Language Processing, Nagoya, Japan, 14– 18 October 2013, pages 1366–1374. find
9. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Wash., 18–21 October 2013, pages 1971–1982. find our card
10. Eraldo Fernandes, Cı́cero dos Santos, and Ruy Milidiú. 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In Proceedings of the Shared Task of the 16th Conference on Computational Natural Language Learning, Jeju Island, Korea, 12–14 July 2012, pages 41–48. find
11. Jonathan K. Kummerfeld and Dan Klein. 2013. Error-driven analysis of challenges in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Wash., 18–21 October 2013, pages 265– 277. find our card
12. Shalom Lappin and Herbert J. Leass. 1994. An algorithm for pronominal anaphora resolution. Computational Linguistics, 20(4):535–561. find our card
13. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. find our card
14. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of the Human Language Technology Conference and the 2005 Conference on Empirical Methods in Natural Language Processing, Vancouver, B.C., Canada, 6–8 October 2005, pages 25–32. find our card
15. Sebastian Martschat, Jie Cai, Samuel Broscheit, Éva Mújdricza-Maydt, and Michael Strube. 2012. A multigraph model for coreference resolution. In Proceedings of the Shared Task of the 16th Conference on Computational Natural Language Learning, Jeju Island, Korea, 12–14 July 2012, pages 100–106. find
16. Sebastian Martschat. 2013. Multigraph clustering for unsupervised coreference resolution. In 51st Annual Meeting of the Association for Computational Linguistics: Proceedings of the Student Research Workshop, Sofia, Bulgaria, 5–7 August 2013, pages 81– 88. find
17. Alexis Mitchell, Stephanie Strassel, Shudong Huang, and Ramez Zakhary. 2004. ACE 2004 multilingual training corpus. LDC2005T09, Philadelphia, Penn.: Linguistic Data Consortium. find
18. Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Penn., 7–12 July 2002, pages 104–111. find
19. Vincent Ng. 2010. Supervised noun phrase coreference research: The first fifteen years. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, Uppsala, Sweden, 11–16 July 2010, pages 1396–1411. find our card
20. Simone Paolo Ponzetto and Michael Strube. 2006. Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, New York, N.Y., 4–9 June 2006, pages 192–199. find
21. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 Shared Task: Modeling unrestricted coreference in OntoNotes. In Proceedings of the Shared Task of the 15th Conference on Computational Natural Language Learning, Portland, Oreg., 23–24 June 2011, pages 1–27. find our card
22. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 Shared Task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Shared Task of the 16th Conference on Computational Natural Language Learning, Jeju Island, Korea, 12–14 July 2012, pages 1–40. find our card
23. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Baltimore, Md., 22–27 June 2014, pages 30– 35. find our card
24. Karthik Raghunathan, Heeyoung Lee, Sudarshan Rangarajan, Nathanael Chambers, Mihai Surdeanu, Dan Jurafsky, and Christopher Manning. 2010. A multi-pass sieve for coreference resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, Cambridge, Mass., 9–11 October 2010, pages 492–501. find our card
25. Altaf Rahman and Vincent Ng. 2011. Coreference resolution with world knowledge. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Portland, Oreg., 19–24 June 2011, pages 814–824. find
26. Lev Ratinov and Dan Roth. 2012. Learning-based multi-sieve co-reference resolution with knowledge. In Proceedings of the 2012 Conference on Empirical Methods in Natural Language Processing and Natural Language Learning, Jeju Island, Korea, 12–14 July 2012, pages 1234–1244. find
27. Don Tuggener. 2014. Coreference resolution evaluation for higher level applications. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, Gothenburg, Sweden, 26–30 April 2014, pages 231–235. find
28. Olga Uryupina, Massimo Poesio, Claudio Giuliano, and Kateryna Tymoshenko. 2011. Disambiguation and filtering methods in using web knowledge for coreference resolution. In Proceedings of the 24th International Florida Artificial Intelligence Research Society Conference, Palm Beach, USA, 18– 20 May 2011, pages 317–322. find
29. Olga Uryupina. 2007. Knowledge acquisition for coreference resolution. Ph.D. thesis, Saarland University, Saarbrücken, Germany. find
30. Olga Uryupina. 2008. Error analysis for learning-based coreference resolution. In Proceedings of the 6th International Conference on Language Resources and Evaluation, Marrakech, Morocco, 26 May – 1 June 2008, pages 1914–1919. find
31. Renata Vieira and Massimo Poesio. 2000. An empirically-based system for processing definite descriptions. Computational Linguistics, 26(4):539– 593. find
32. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference (MUC-6), pages 45–52, San Mateo, Cal. Morgan Kaufmann. find our card
33. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. OntoNotes release 5.0. LDC2013T19, Philadelphia, Penn.: Linguistic Data Consortium. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/martschat-2014-recall-error-analysis-coreference-resolution/