Mova0.5 pre-release⌕
☰
home / docs / papers / Findings of the Fourth Shared Task on Multilingual Coreference Resolution: Can LLMs Dethrone Traditional Approaches?

Findings of the Fourth Shared Task on Multilingual Coreference Resolution: Can LLMs Dethrone Traditional Approaches?

Summary

This overview reports the fourth edition of the multilingual coreference shared task at CODI-CRAC 2025. Participants had to detect mentions, including zero mentions in pro-drop languages, and cluster them by identity coreference. The data came from CorefUD 1.3, with 22 datasets in 17 languages used in the task; new additions included spoken French, Hindi and Korean. The main novelty was a dedicated LLM track with a simplified plaintext encoding of coreference written inline in the text, easier for generative models than CoNLL-U, alongside an unconstrained track. Nine systems took part, four of them LLM-based. A traditional encoder-based system (CorPipe) again ranked first, but LLM approaches matched or beat it on some datasets. The official primary metric is CoNLL F1 computed with head-based mention matching.

How Mova uses it

Effectiveness in Mova

This paper provides the evaluation protocol and reference numbers rather than a technique, so it is not measured separately. Indicative comparison on GUM (CoNLL F1, head match, without singletons): CRAC 2025 systems range from 76.1 (CorPipeEnsemble) to 61.7 (mBERT end-to-end baseline); Mova's deterministic resolver scores 60.3 on gold trees and 54.0 on its own predicted trees. The settings differ (CorefUD 1.3 and a reduced mini-test there, the full UD 2.18 test here; trained neural systems there, untrained rules here), so the comparison is only approximate.

👨‍🔬💥

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.crac-1.9

@inproceedings{novak2025findings,
  title = {{Findings of the Fourth Shared Task on Multilingual Coreference Resolution: Can LLMs Dethrone Traditional Approaches?}},
  author = {Novák, Michal and Konopik, Miloslav and Nedoluzhko, Anna and Popel, Martin and Prazak, Ondrej and Sido, Jakub and Straka, Milan and Žabokrtský, Zdeněk and Zeman, Daniel},
  booktitle = {Proceedings of the Eighth Workshop on Computational Models of Reference, Anaphora and Coreference},
  year = {2025},
  doi = {10.18653/v1/2025.crac-1.9},
  url = {https://doi.org/10.18653/v1/2025.crac-1.9}
}

References

53 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for Scoring Coreference Chains. In Proceedings of The First International Conference on Language Resources and Evaluation Workshop on Linguistics Coreference, pages 563–566. find our card
  2. David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An Annotated Dataset of Coreference in English Literature. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 44–54, Marseille, France. European Language Resources Association. find our card
  3. Peter Bourgonje and Manfred Stede. 2020. The Potsdam Commentary Corpus 2.2: Extending Annotations for Shallow Discourse Parsing. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1061–1066, Marseille, France. European Language Resources Association. find
  4. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440– 8451, Online. Association for Computational Linguistics. ↗
  5. Pascal Denis and Jason Baldridge. 2009. Global joint models for coreference resolution and named entity classification. Procesamiento del lenguaje natural, 42. find
  6. Vladimir Dobrovolskii. 2021. Word-Level Coreference Resolution. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7670–7675, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  7. Karel D’Oosterlinck, Semere Kiros Bitew, Brandon Pa-pineau, Christopher Potts, Thomas Demeester, and Chris Develder. 2023. CAW-coref: Conjunction-Aware Word-level Coreference Resolution. In Proceedings of the Sixth Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC 2023), pages 8–14, Singapore. Association for Computational Linguistics. find
  8. Ali Emami, Paul Trichelair, Adam Trischler, Kaheer Suleman, Hannes Schulz, and Jackie Chi Kit Cheung. 2019. The KnowRef Coreference Corpus: Removing Gender and Number Cues for Difficult Pronominal Anaphora Resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3952–3961, Florence, Italy. Association for Computational Linguistics. find our card
  9. Yujian Gan, Massimo Poesio, and Juntao Yu. 2024. Assessing the Capabilities of Large Language Models in Coreference: An Evaluation. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1645–1665, Torino, Italia. ELRA and ICCL. find
  10. Ine Gevers, Victor De Marez, Luna De Bruyne, and Walter Daelemans. 2025. WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorization. In Proceedings of the 29th Conference on Computational Natural Language Learning, pages 68–80, Vienna, Austria. Association for Computational Linguistics. find
  11. Jan Hajič, Eduard Bejček, Jaroslava Hlaváčová, Marie Mikulová, Milan Straka, Jan Štěpánek, and Barbora Štěpánková. 2020. Prague Dependency Treebank -Consolidated 1.0. In Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC 2020), pages 5208–5218, Marseille, France. European Language Resources Association. find
  12. Dag Trygve Truslew Haug and Marius L. Jøhndal. 2008. Creating a Parallel Treebank of the Old Indo-European Bible Translations. In Proceedings of the second workshop on language technology for cultural heritage data (LaTeCH 2008). find
  13. Rebecca Hicke and David Mimno. 2024. [Lions: 1] and [Tigers: 2] and [Bears: 3], Oh My! Literary Coreference Annotation with LLMs. In Proceedings of the 8th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2024), pages 270–277, St. Julians, Malta. Association for Computational Linguistics. find
  14. Frédéric Landragin. 2021. Le corpus Democrat et son exploitation. Présentation. Langages, 224:11–24. ↗
  15. Ekaterina Lapshinova-Koltunski, Christian Hardmeier, and Pauline Krielke. 2018. ParCorFull: a Parallel Corpus Annotated with Full Coreference. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association. ↗
  16. Nghia T. Le and Alan Ritter. 2023. Are Large Language Models Robust Coreference Resolvers? ArXiv:2305.14489 [cs]. ↗
  17. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end Neural Coreference Resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. ↗ our card
  18. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, pages 552–561, Rome, Italy. AAAI Press. find
  19. Houjun Liu, John Bauer, Karel D’Oosterlinck, Christopher Potts, and Christopher D. Manning. 2024. MSCAW-coref: Multilingual, Singleton and Conjunction-Aware Word-Level Coreference Resolution. In Proceedings of the Seventh Workshop on Computational Models of Reference, Anaphora and Coreference, pages 33–40, Miami. Association for Computational Linguistics. ↗
  20. Xiaoqiang Luo. 2005. On Coreference Resolution Performance Metrics. In Proceedings of the Conference on Human Language Technology and Empirical Methods in Natural Language Processing, HLT 2005, pages 25–32. Association for Computational Linguistics. ↗ our card
  21. Petter Mæhlum, Dag Haug, Tollef Jørgensen, Andre Kåsen, Anders Nøklestad, Egil Rønningstad, Per Erik Solberg, Erik Velldal, and Lilja Øvrelid. 2022. NARC–Norwegian Anaphora Resolution Corpus. In Proceedings of the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 48–60, Gyeongju, Korea. Association for Computational Linguistics. find
  22. Nafise Sadat Moosavi and Michael Strube. 2016. Which Coreference Evaluation Metric Do You Trust? A Proposal for a Link-based Entity Aware Metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632–642, Berlin, Germany. Association for Computational Linguistics. ↗ our card
  23. Vandan Mujadia, Palash Gupta, and Dipti Misra Sharma. 2016. Coreference Annotation Scheme and Relation Types for Hindi. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 161–168, Portorož, Slovenia. European Language Resources Association (ELRA). ↗
  24. Judith Muzerelle, Anaïs Lefeuvre, Emmanuel Schang, Jean-Yves Antoine, Aurore Pelletier, Denis Maurel, Iris Eshkol, and Jeanne Villaneau. 2014. AN-COR_Centre, a large free spoken French coreference corpus: description of the resource and reliability measures. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 843–847, Reykjavik, Iceland. European Language Resources Association (ELRA). ↗
  25. Sangha Nam, Minho Lee, Donghwan Kim, Kijong Han, Kuntae Kim, Sooji Yoon, Eun-kyung Kim, and Key-Sun Choi. 2020. Effective Crowdsourcing of Multiple Tasks for Comprehensive Knowledge Extraction. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 212–219, Marseille, France. European Language Resources Association. find
  26. Anna Nedoluzhko, Michal Novák, Silvie Cinková, Marie Mikulová, and Jiří Mírovský. 2016. Coreference in Prague Czech-English Dependency Treebank. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 169–176, Portorož, Slovenia. European Language Resources Association. ↗
  27. Anna Nedoluzhko, Michal Novák, Martin Popel, Zdeněk Žabokrtský, Amir Zeldes, and Daniel Zeman. 2022. CorefUD 1.0: Coreference Meets Universal Dependencies. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4859–4872, Marseille, France. European Language Resources Association. ↗ our card
  28. Michal Novák, Barbora Dohnalová, Miloslav Konopik, Anna Nedoluzhko, Martin Popel, Ondrej Prazak, Jakub Sido, Milan Straka, Zdeněk Žabokrtský, and Daniel Zeman. 2024. Findings of the Third Shared Task on Multilingual Coreference Resolution. In Proceedings of the Seventh Workshop on Computational Models of Reference, Anaphora and Coreference, pages 78–96, Miami. Association for Computational Linguistics. ↗
  29. Michal Novák, Martin Popel, Daniel Zeman, Zdeněk Žabokrtský, Anna Nedoluzhko, Kutay Acar, David Bamman, Peter Bourgonje, Silvie Cinková, Hanne Eckhoff, Gülşen Cebiroğlu Eryiğit, Jan Hajič, Christian Hardmeier, Dag Haug, Tollef Jørgensen, Andre Kåsen, Pauline Krielke, Frédéric Landragin, Ekaterina Lapshinova-Koltunski, Petter Mæhlum, M. Antònia Martí, Marie Mikulová, Kirill Milintsevich, Vandan Mujadia, Judith Muzerelle, Sangha Nam, Anders Nøklestad, Maciej Ogrodniczuk, Lilja Øvrelid, Tuğba Pamay Arslan, Ian Porada, Marta Recasens, Per Erik Solberg, Manfred Stede, Milan Straka, Daniel Swanson, Svetlana Toldova, Noémi Vadász, Erik Velldal, Veronika Vincze, Amir Zeldes, and Voldemaras Žitkus. 2025. Coreference in Universal Dependencies 1.3 (CorefUD 1.3). LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  30. Maciej Ogrodniczuk, Katarzyna Glowińska, Mateusz Kopeć, Agata Savary, and Magdalena Zawisławska. 2013. Polish Coreference Corpus. In Human Language Technology. Challenges for Computer Science and Linguistics — 6th Language and Technology Conference (LTC 2013), Revised Selected Papers, volume 9561 of Lecture Notes in Computer Science, pages 215–226. Springer. find
  31. Maciej Ogrodniczuk, Katarzyna Głowińska, Mateusz Kopeć, Agata Savary, and Magdalena Zawisławska. 2015. Coreference in Polish: Annotation, Resolution and Evaluation. Walter De Gruyter. find
  32. Martha Palmer, Rajesh Bhatt, Bhuvana Narasimhan, Owen Rambow, Dipti Misra Sharma, and Fei Xia. 2009. Hindi syntax: Annotating dependency, lexical predicate-argument structure, and phrase structure. In The 7th International Conference on Natural Language Processing, pages 14–17. find
  33. Tuğba Pamay and Gülşen Eryiğit. 2018. Turkish Coreference Resolution. In 2018 Innovations in Intelligent Systems and Applications (INISTA), pages 1–7. ↗
  34. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring Coreference Partitions of Predicted Mentions: A Reference Implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 30–35, Baltimore, Maryland. Association for Computational Linguistics. ↗ our card
  35. Svetlana Toldova, Anna Roytberg, Alina Ladygina, Maria Vasilyeva, Ilya Azerkovich, Matvei Kurzukov, G. Sim, D.V. Gorshkov, A. Ivanova, Anna Nedoluzhko, and Yulia Grishina. 2014. Evaluating Anaphora and Coreference Resolution for Russian. In Komp’juternaja lingvistika i intellektual’nye tehnologii. Po materialam ezhegodnoj Mezhdunarodnoj konferencii Dialog, pages 681–695. find
  36. Ondřej Pražák, Miloslav Konopík, and Jakub Sido. 2021. Multilingual Coreference Resolution with Harmonized Annotations. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1119– 1123, Held Online. INCOMA Ltd. find
  37. Noémi Vadász. 2022. Building a Manually Annotated Hungarian Coreference Corpus: Workflow and Tools. In Proceedings of the Fifth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 38–47, Gyeongju, Korea. Association for Computational Linguistics. find
  38. Marta Recasens, Eduard Hovy, and M. Antònia Martí. 2010. A Typology of Near-Identity Relations for Coreference (NIDENT). In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC 2010), Valletta, Malta. European Language Resources Association. ↗
  39. Noémi Vadász. 2023. Resolving Hungarian Anaphora with ChatGPT. In Text, Speech, and Dialogue: 26th International Conference, TSD 2023, Pilsen, Czech Republic, September 4–6, 2023, Proceedings, page 45–57, Berlin, Heidelberg. Springer-Verlag. find
  40. Marta Recasens and Eduard H. Hovy. 2011. BLANC: Implementing the Rand index for coreference evaluation. Natural Language Engineering, 17(4):485–510. ↗
  41. Marta Recasens and M. Antònia Martí. 2010. AnCora-CO: Coreferentially Annotated Corpora for Spanish and Catalan. Language Resources and Evaluation, 44(4):315–345. ↗
  42. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A Model-Theoretic Coreference Scoring Scheme. In Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995. ↗ our card
  43. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Commun. ACM, 64(9):99–106. find
  44. Veronika Vincze, Klára Hegedűs, Alex Sliz-Nagy, and Richárd Farkas. 2018. SzegedKoref: A Hungarian Coreference Corpus. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association. ↗
  45. Karol Saputa, Angelika Peljak-Łapińska, and Maciej Ogrodniczuk. 2024. Polish Coreference Corpus as an LLM Testbed: Evaluating Coreference Resolution within Instruction-Following Language Models by Instruction–Answer Alignment. In Proceedings of The Seventh Workshop on Computational Models of Reference, Anaphora and Coreference, pages 23–32, Miami. Association for Computational Linguistics. find
  46. Juntao Yu, Michal Novák, Abdulrahman Aloraini, Nafise Sadat Moosavi, Silviu Paun, Sameer Pradhan, and Massimo Poesio. 2023. The Universal Anaphora Scorer 2.0. In Proceedings of the 15th International Conference on Computational Semantics, pages 183– 194, Nancy, France. Association for Computational Linguistics. find
  47. Milan Straka. 2024. CorPipe at CRAC 2024: Predicting Zero Mentions from Raw Text. In Proceedings of the Seventh Workshop on Computational Models of Reference, Anaphora and Coreference, pages 97–106, Miami. Association for Computational Linguistics. find
  48. Daniel G. Swanson, Bryce D. Bussert, and Francis Tyers. 2024. Towards Named-Entity and Coreference Annotation of the Hebrew Bible. In Proceedings of the Third Workshop on Language Technologies for Historical and Ancient Languages (LT4HALA) @ LREC-COLING-2024, pages 36–40, Torino, Italia. ELRA and ICCL. ↗ our card
  49. Mariona Taulé, M. Antònia Martí, and Marta Recasens. 2008. AnCora: Multilevel Annotated Corpora for Catalan and Spanish. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC 2008), Marrakech, Morocco. European Language Resources Association. ↗
  50. Zdeněk Žabokrtský, Miloslav Konopík, Anna Nedoluzhko, Michal Novák, Maciej Ogrodniczuk, Martin Popel, Ondřej Pražák, Jakub Sido, and Daniel Zeman. 2023. Findings of the Second Shared Task on Multilingual Coreference Resolution. In Proceedings of the CRAC 2023 Shared Task on Multilingual Coreference Resolution, pages 1–18, Singapore. Association for Computational Linguistics. ↗
  51. Zdeněk Žabokrtský, Miloslav Konopík, Anna Nedoluzhko, Michal Novák, Maciej Ogrodniczuk, Martin Popel, Ondřej Pražák, Jakub Sido, Daniel Zeman, and Yilun Zhu. 2022. Findings of the Shared Task on Multilingual Coreference Resolution. In Proceedings of the CRAC 2022 Shared Task on Multilingual Coreference Resolution, pages 1–17, Gyeongju, Korea. Association for Computational Linguistics. ↗
  52. Amir Zeldes. 2017. The GUM Corpus: Creating Multilayer Resources in the Classroom. Language Resources and Evaluation, 51(3):581–612. ↗
  53. Voldemaras Žitkus and Rita Butkienė. 2018. Coreference Annotation Scheme and Corpus for Lithuanian Language. In Fifth International Conference on Social Networks Analysis, Management and Security, SNAMS 2018, Valencia, Spain, October 15-18, 2018, pages 243–250. IEEE. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/novak-2025-findings-fourth-shared-task-multilingual/