Lexical Features in Coreference Resolution: To be Used With Caution

What it is about

Learned coreference resolvers owe much of their gain over rule systems to lexical features (surface words or embeddings of heads, first/last words, context). The paper shows the cost. On WikiCoref, annotated with the same guidelines as CoNLL-2012, the Stanford rule-based sieve system (CoNLL F1 about 51.0) is on par with or ahead of berkeley, cort and deep-coref (51.0 down to 49.9, 52.7 for the best deep-coref variant), although on the CoNLL test set the learned systems are 6–10 points ahead. The reason is overlap: a large share of the coreferent mention-head pairs in the CoNLL test set already occur, as coreferent pairs, in training (38–51% by genre, 76–77% for the pivot/Bible genre), so lexical features memorise rather than generalise. The authors call for out-of-domain evaluation and features that capture linguistic properties rather than words.

Ideas

• I1. Keep this as the argument for Mova's rule-first coreference, with numbers to defend it: report CoNLL F1 per GUM genre (bio, academic, conversation, fiction, …) on dev, not only the total, since coref was tuned on dev overall. A rule that helps one genre and hurts another shows up there.
• I2. Compute the paper's overlap diagnostic for Mova's own tuning: the share of coreferent head pairs in GUM test that also occur as coreferent pairs in GUM dev. If it is high, dev-tuned string sieves (exact, strict_*) are flattered on test, and the frozen-code test result in coref — SLM coreference: a sieve of rules with a reason for every link should be read with that number beside it.
• I3. When a learned ranker is added (see lee-2017-end-end-neural-coreference-resolution I2), start from non-lexical features (agreement, syntactic role, distance, Hobbs rank, speaker, mention type) and add head words only if a cross-genre check (train on some genres, test on others, or test on tales) shows no loss.

For Mova

35. Reference and coreference §8–9. Code: coref/src/eval.rs (per-genre and overlap reports), coref/src/sieve.rs. Measures: CoNLL F1 per GUM genre on dev; head-pair overlap between dev and test.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-2003 · arXiv 1704.06779

@inproceedings{moosavi2017lexical,
  title = {{Lexical Features in Coreference Resolution: To be Used With Caution}},
  author = {Moosavi, Nafise Sadat and Strube, Michael},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2003},
  eprint = {1704.06779},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p17-2003}
}

References

15 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Conference on Empirical Methods in Natural Language Processing, Vancouver, B.C., Canada, 6–8 October 2005. pages 25–32. find
2. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain, 28–30 May 1998. pages 563–566. find our card
3. Sebastian Martschat and Michael Strube. 2014. Recall error analysis for coreference resolution. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 25–29 October 2014. pages 2070–2081. http://www.aclweb.org/anthology/D14-1221.pdf. ↗ our card
4. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Tex., 1–5 November 2016. pages 2256–2262. https://www.aclweb.org/anthology/D16-1245.pdf. ↗
5. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics 3:405–418. http://www.aclweb.org/anthology/Q15-1029.pdf. ↗
6. Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, 7–12 August 2016. http://www.aclweb.org/anthology/P16-1061.pdf. ↗
7. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? A proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, 7–12 August 2016. pages 632–642. http://www.aclweb.org/anthology/P16-1060.pdf. ↗ our card
8. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Wash., 18–21 October 2013. pages 1971–1982. http://www.aclweb.org/anthology/D13-1203.pdf. ↗ our card
9. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Baltimore, Md., 22–27 June 2014. pages 30–35. http://www.aclweb.org/anthology/P14-2006.pdf. ↗ our card
10. Abbas Ghaddar and Philippe Langlais. 2016a. Coreference in Wikipedia: Main concept resolution. In Proceedings of the 20th Conference on Computational Natural Language Learning, Berlin, Germany, 11–12 August 2016. pages 229–238. https://www.aclweb.org/anthology/K16-1023.pdf. ↗
11. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference (MUC-6). Morgan Kaufmann, San Mateo, Cal., pages 45–52. find our card
12. Abbas Ghaddar and Philippe Langlais. 2016b. WikiCoref: An English coreference-annotated corpus of Wikipedia articles. In Proceedings of the 10th International Conference on Language Resources and Evaluation, Portorož, Slovenia, 23–28 May 2016. http://www.lrec-conf.org/proceedings/lrec2016/pdf/192 Paper.pdf. ↗ our card
13. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics 39(4):885–916. https://www.aclweb.org/anthology/J13-4004.pdf. ↗ our card
14. Omer Levy, Steffen Remus, Chris Biemann, and Ido Dagan. 2015. Do supervised distributional methods really learn lexical inference relations? In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, Col., 31 May – 5 June 2015. pages 970–976. http://www.aclweb.org/anthology/N15-1098. ↗
15. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of the Human Language Technology Conference and the 2005 find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/moosavi-2017-lexical-features-coreference-resolution-be/