Using Automatically Extracted Minimum Spans to Disentangle Coreference Evaluation from Boundary Detection

What it is about

Scoring coreference by maximum spans mixes coreference errors with boundary errors such as PP attachment: «an extensive presence» and «an extensive presence, of course in this country» count as different mentions, and the system is penalised as if it had linked the wrong thing. MUC, ACE and ARRAU annotate minimum spans by hand, CoNLL-2012 does not. MINA extracts a minimum span automatically from one constituency tree: walk the mention's subtree breadth-first through NP-internal nodes and keep the shallowest terminal nodes that contain a word other than a determiner or conjunction (so a name keeps all its words, unlike a single head word, which makes «Mary Green» and «John Green» both «Green»). Gold and system mentions are reduced on the same tree and compared by their minimum spans. MINA spans agree with expert minimum spans better than head words, and minimum-span scores reduce the effect of parser differences on system rankings.

Ideas

• I1. Add a minimum-span matching mode to coref/src/score.rs next to the head mode and the exact-span mode, defined on UD: the head plus its flat, fixed and name-internal compound dependents (and for a coordination, each conjunct's minimum span). This keeps head-based robustness to attachment errors but separates «George W. Bush» from «George H. W. Bush», which head matching merges. Report CoNLL F1 on GUM dev under all three modes; the difference to head mode measures how much name-internal confusion the head scorer hides.
• I2. Use the same minimum span inside the proper_head and exact/strict_* sieves (coref/src/sieve.rs): two proper-name mentions link only if their minimum spans are compatible (one contains the other's full name words) — a direct attack on the «[George W.] Bush ~ [George H. W.] Bush» error listed in coref — SLM coreference: a sieve of rules with a reason for every link. Measure proper_head and strict_a precision on GUM dev.
• I3. Use minimum-span scoring when comparing gold-tree and SLM-tree runs: part of the 6-point loss on SLM trees is boundary detection (e.g. reference numbers parsed as appositions), and minimum spans separate that from real linking loss.

For Mova

35. Reference and coreference §7 (annotating coreference), §9.3. Code: coref/src/score.rs (matching mode), coref/src/sieve.rs, coref/src/mention.rs. Measures: CoNLL F1 on GUM dev in three matching modes, gold vs SLM trees; precision of name sieves.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1408 · arXiv 1906.06703

@inproceedings{moosavi2019using,
  title = {{Using Automatically Extracted Minimum Spans to Disentangle Coreference Evaluation from Boundary Detection}},
  author = {Moosavi, Nafise Sadat and Born, Leo and Poesio, Massimo and Strube, Michael},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1408},
  eprint = {1906.06703},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p19-1408}
}

References

18 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain, 28–30 May 1998, pages 563–566. find our card
2. Kevin Clark and Christopher D. Manning. 2016a. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643–653, Berlin, Germany. Association for Computational Linguistics. find
3. Kevin Clark and Christopher D. Manning. 2016b. Deep reinforcement learning for mention-ranking coreference models. pages 2256–2262, Austin, Texas. Association for Computational Linguistics. find
4. Michael Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania, Philadelphia, Penn. find our card
5. Abbas Ghaddar and Philippe Langlais. 2016. Wiki-Coref: An English coreference-annotated corpus of Wikipedia articles. In Proceedings of the 10th International Conference on Language Resources and Evaluation, Portorož, Slovenia, 23–28 May 2016. find our card
6. Lynette 1997. Hirschman and Nancy Chinchor. MUC-7 coreference task definition, http://www.muc.saic.com/proceedings/ . ↗
7. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. ↗ our card
8. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692. Association for Computational Linguistics. find
9. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver, British Columbia, Canada. Association for Computational Linguistics. find our card
10. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics, 3:405–418. find
11. Alexis Mitchell, Stephanie Strassel, Mark Przybocki, JK Davis, George Doddington, Ralph Grishman, Adam Meyers, Ada Brunstain, Lisa Ferro, and Beth Sundheim. 2002. ACE-2 Version 1.0. LDC2003T11, Philadelphia, Penn.: Linguistic Data Consortium. find
12. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632–642, Berlin, Germany. Association for Computational Linguistics. ↗ our card
13. Nafise Sadat Moosavi and Michael Strube. 2017. Lexical features in coreference resolution: To be used with caution. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 14–19, Vancouver, Canada. Association for Computational Linguistics. find our card
14. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 423–430, Sapporo, Japan. Association for Computational Linguistics. find our card
15. Nafise Sadat Moosavi and Michael Strube. 2018. Using linguistic features to improve the generalization capability of neural coreference resolvers. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 193– 203, Brussels, Belgium. Association for Computational Linguistics. find our card
16. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. find our card
17. Haoruo Peng, Kai-Wei Chang, and Dan Roth. 2015. A joint framework for coreference resolution and mention head detection. In Proceedings of the 19th Conference on Computational Natural Language Learning, Beijing, China, 30–31 July 2015, pages 12–21. find
18. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 30–35, Baltimore, Maryland. Association for Computational Linguistics. ↗ our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/moosavi-2019-automatically-extracted-minimum-spans-disentangle/