Do UD Trees Match Mention Spans in Coreference Annotations?

What it is about

Coreference mention spans in most corpora are plain token intervals, drawn without reference to syntax. The paper asks how often those spans match the subtrees of UD trees. It hand-checks mismatches in nine CorefUD datasets for eight languages. In eight of the nine, the trees come from UDPipe; English-GUM is the only one with hand-annotated trees. Most mentions (86% to 98%, depending on the dataset) are catenae, that is, connected parts of the tree. Among the remaining mismatches, more than half come from wrong trees, most often a case-marked nominal attached to the clause predicate instead of to the noun it modifies. The rest come from coordination, flat multiword names and span errors. The authors suggest that the mismatches can be used to find errors on either side.

Ideas

• I1. prag / world: measure the share of entity mentions that are catenae in the trees Mova builds, on GUM test (test only) and on silver dev; the paper's range (86% to 98%) is the reference.
• I2. tools/treeagree: add a gate for the "generalized PP attachment" error, a case-marked nominal attached to a verb when a nominal head is available, and report violations per 1k sentences from treeagree check on silver dev.
• I3. en/src/ctx.rs: add a feature for the syntactic head of a mention, not the whole span, for pronoun-antecedent agreement, and measure agreement errors on a small hand-marked dev set.
• I4. annot: when a coreference span from corpus/ is not a catena, send the sentence with the tree and the span to the teacher, and let the Rust gate decide; count how often the tree witnesses in tools/parsecmp side with the tree.

More ideas (a second reading)

• I5. Use gold coreference spans as a free check of Mova's parser: on GUM dev, compute for every gold mention whether it is a catena in the SLM tree (en annotate output via --trees). Non-catena mentions are mostly parser errors, typically PP attachment; list them by error type and feed the list to the parser's repair work (the absurdity-matrix repairs in world). Report the non-catena rate for gold vs SLM trees as one number per parser version.
• I6. The same check on Mova's own coreference output: a chain whose mentions are all catenae in the SLM tree but whose antecedent mention crosses an attachment the parser got wrong (e.g. reference numbers as appos, the third error type in coref — SLM coreference: a sieve of rules with a reason for every link) can be flagged in coref explain, so that link errors caused by trees are separated from sieve errors in the error analysis.

For Mova

The ideas touch the mention output of prag and world, the context features of en/src/ctx.rs and the gates of tools/treeagree. The relevant chapters are 22. Phrases: noun, adjective, adverb and prepositional phrases (what a phrase covers) and 12-pronouns-determiners-numerals.md (the head that carries agreement). The first measure is I1 on GUM test and silver dev, compared with the paper's catena rates.

Topics: Coreference resolution · Universal Dependencies

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.findings-emnlp.303

@inproceedings{popel2021do,
  title = {{Do UD Trees Match Mention Spans in Coreference Annotations?}},
  author = {Popel, Martin and Žabokrtský, Zdeněk and Nedoluzhko, Anna and Novák, Michal and Zeman, Daniel},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2021},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.303},
  url = {https://doi.org/10.18653/v1/2021.findings-emnlp.303}
}

References

41 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Shane Bergsma and Dekang Lin. 2006. Bootstrapping path-based pronoun resolution. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 33–40, Sydney, Australia. Association for Computational Linguistics. find
2. Peter Bourgonje and Manfred Stede. 2020. The Potsdam commentary corpus 2.2: Extending annotations for shallow discourse parsing. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1061–1066, Marseille, France. European Language Resources Association. find
3. David Maclean Carter. 1986. A shallow processing approach to anaphor resolution. Technical Report UCAM-CL-TR-88, University of Cambridge, Computer Laboratory, 15 JJ Thomson Avenue, Cambridge CB3 0FD, United Kingdom, phone +44 1223 763500. find
4. Noam Chomsky. 1993. Lectures on Government and Binding. Studies in Generative Grammar (SGG). De Gruyter Mouton, Berlin, Germany. find
5. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1405–1415, Beijing, China. Association for Computational Linguistics. find
6. Dóra Csendes, János Csirik, Tibor Gyimóthy, and András Kocsor. 2005. The Szeged treebank. In Proceedings of the 8th International Conference on Text, Speech and Dialogue, TSD’05, page 123–131, Berlin, Heidelberg. Springer-Verlag. find
7. Adèle Désoyer, Frédéric Landragin, Isabelle Tellier, Anaïs Lefeuvre, Jean-Yves Antoine, and Marco Dinarelli. 2016. Coreference Resolution for French Oral Data: Machine Learning Experiments with AN-COR. In 17th International Conference on Intelligent Text Processing and Computational Linguistics (CICLing’2016), Konya, Turkey. find
8. Yuan Ding and Martha Palmer. 2004. Automatic learning of parallel dependency treelet pairs. In International Conference on Natural Language Processing, pages 233–243. Springer. find
9. Jenny Rose Finkel and Christopher D. Manning. 2009. Joint parsing and named entity recognition. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 326–334. Association for Computational Linguistics. find
10. Liane Guillou, Christian Hardmeier, Aaron Smith, Jörg Tiedemann, and Bonnie Webber. 2014. Parcor 1.0: A parallel pronoun-coreference corpus to support statistical mt. In Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC-2014), Reykjavik, Iceland. European Language Resources Association (ELRA). find
11. Jan Hajič, Eduard Bejček, Jaroslava Hlaváčová, Marie Mikulová, Milan Straka, Jan Štěpánek, and Barbora Štěpánková. 2020. Prague Dependency Treebank -Consolidated 1.0. In Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC 2020), pages 5208–5218, Marseille, France. European Language Resources Association. find
12. Iris Hendrickx, Gosse Bouma, Frederik Coppens, Walter Daelemans, Veronique Hoste, Geert Kloosterman, Anne-Marie Mineur, Joeri Van Der Vloet, and Jean-Luc Verschelde. 2008. A coreference corpus and resolution system for Dutch. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA). find
13. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. find
14. Erhard W. Hinrichs, Sandra Kübler, and Karin Naumann. 2005. A unified representation for morphological, syntactic, semantic, and referential annotations. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 13– 20, Ann Arbor, Michigan. Association for Computational Linguistics. find
15. J.R. Hobbs. 1978. Resolving pronoun references. Lingua 44, pages 311–338. find our card
16. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. find
17. Frédéric Landragin. 2016. Description, modélisation et détection automatique des chaînes de référence (DEMOCRAT). Bulletin de l’Association Française pour l’Intelligence Artificielle, 92:11–15. find
18. Shalom Lappin and Herbert J. Leass. 1994. An algorithm for pronominal anaphora resolution. Computational Linguistics, 20(4):535–561. find our card
19. Ekaterina Lapshinova-Koltunski, Christian Hardmeier, and Pauline Krielke. 2018. ParCorFull: a Parallel Corpus Annotated with Full Coreference. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
20. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics. find
21. Tomasz Limisiewicz, David Mareček, and Rudolf Rosa. 2020. Universal Dependencies According to BERT: Both More Specific and More General. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2710–2722, Online. Association for Computational Linguistics. find
22. Igor A. Mel’čuk. 1981. Meaning-text models: A recent trend in soviet linguistics. Annual Review of Anthropology, 10:27–62. find
23. Marie Mikulová, Jiří Mírovský, Anna Nedoluzhko, Petr Pajas, Jan Štěpánek, and Jan Hajič. 2017. PDTSC 2.0 - spoken corpus with rich multi-layer structural annotation. In Text, Speech, and Dialogue. TSD 2017, volume 10415 of Lecture Notes in Computer Science, Cham, Switzerland. Springer. find
24. Anna Nedoluzhko, Michal Novák, Silvie Cinková, Marie Mikulová, and Jiří Mírovský. 2016. Coreference in Prague Czech-English Dependency Treebank. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 169–176, Portorož, Slovenia. European Language Resources Association (ELRA). find
25. Anna Nedoluzhko, Michal Novák, Martin Popel, Zdeněk Žabokrtský, and Daniel Zeman. 2021. Coreference meets Universal Dependencies – a pilot experiment on harmonizing coreference datasets for 11 languages. Technical Report 66, ÚFAL MFF UK, Praha, Czechia. find our card
26. Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 104– 111, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. find
27. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Jan Hajič, Christopher Manning, Sampo Pyysalo, Sebastian Schuster, Francis Tyers, and Daniel Zeman. 2020. Universal Dependencies v2: An evergrowing multilingual treebank collection. In Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC 2020), pages 4034–4043, Marseille, France. European Language Resources Association. find our card
28. Michal Novák. 2017. Coreference Resolution System Not Only for Czech. In Proceedings of the 17th conference ITAT 2017: Slovenskočeský NLP workshop (SloNLP 2017), pages 193–200, Praha, Czechia. CreateSpace Independent Publishing Platform. find
29. Maciej Ogrodniczuk, Katarzyna Glowińska, Mateusz Kopeć, Agata Savary, and Magdalena Zawisławska. 2013. Polish coreference corpus. In Human Language Technology. Challenges for Computer Science and Linguistics - 6th Language and Technology Conference, LTC 2013, Poznań, Poland, December 7-9, 2013. Revised Selected Papers, volume 9561 of Lecture Notes in Computer Science, pages 215–226. Springer. find
30. Timothy Osborne, Michael Putnam, and Thomas Groß. 2012. Catenae: Introducing a novel unit of syntactic analysis. Syntax, 15(4):354–396. find
31. Martin Popel, David Mareček, Jan Štěpánek, Daniel Zeman, and Zdeněk Žabokrtský. 2013. Coordination structures in dependency treebanks. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 517–527, Sofia, Bulgaria. Association for Computational Linguistics. broad range of anaphoric phenomena, in a variety of genres: the ARRAU Corpus. Natural Language Engineering, 26(1):95–128. find our card
32. Veronika Vincze, Klára Hegedűs, Alex Sliz-Nagy, and Richárd Farkas. 2018. SzegedKoref: A Hungarian coreference corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
33. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. Ontonotes: A large training corpus for enhanced processing. In Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation, pages 54–63, New York. Springer-Verlag. find
34. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963, Online. Association for Computational Linguistics. find
35. Amir Zeldes. 2017. The GUM Corpus: Creating Multilayer Resources in the Classroom. Language Resources and Evaluation, 51(3):581–612. ↗
36. Voldemaras Žitkus and Rita Butkienė. 2018. Coreference annotation scheme and corpus for Lithuanian language. In Fifth International Conference on Social Networks Analysis, Management and Security, SNAMS 2018, Valencia, Spain, October 15-18, 2018, pages 243–250. IEEE. ↗
37. Marta Recasens and M. Antònia Martí. 2010. AnCora-CO: Coreferentially Annotated Corpora for Spanish and Catalan. Lang. Resour. Eval., 44(4):315–345. find
38. Petr Sgall, Eva Hajičová, and Jarmila Panevová. 1986. The Meaning of the Sentence and Its Semantic and Pragmatic Aspects. Academia/Reidel Publishing Company, Prague/Dordrecht. find
39. Milan Straka and Jana Straková. 2017. Tokenizing, POS Tagging, Lemmatizing and Parsing UD 2.0 with UDPipe. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 88–99, Vancouver, Canada. Association for Computational Linguistics. find our card
40. S. Toldova, A. Roytberg, A. A. Ladygina, M. D. Vasilyeva, I. L. Azerkovich, M. Kurzukov, G. Sim, D. V. Gorshkov, A. Ivanova, A. Nedoluzhko, and Y. Grishina. 2014. Evaluating anaphora and coreference resolution for russian. In Komp’juternaja lingvistika i intellektual’nye tehnologii. Po materialam ezhegodnoj Mezhdunarodnoj konferencii Dialog, pages 681– 695. find
41. Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa J. Rodriguez, and Massimo Poesio. 2020. Annotating a find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/coref-2021-popel-do-ud-trees-match-mention/