First Shared Task on UMR Parsing

What it is about

The paper sets up the first shared task on UMR parsing. A system gets plain text and must return sentence-level graphs, the alignment of each node to tokens, and document-level relations such as coreference, temporal and modal links. Seven languages from four families take part: Indo-European, Sino-Tibetan, Na-Dene and Algic. Six have some training data, and one is a zero-shot case. Training data and gold test sets are released. Two graph matchers score the systems: a traditional Smatch-style one, and an alignment-sensitive one built for UMR. Two systems are compared. The results show that alignment prediction and document-level semantics are the hardest parts, and that scores vary widely across languages and annotation conditions.

Ideas

• I1. Score alignment on its own. Report the translator's node alignment, or its token links, as a separate score next to the meaning score. The shared task shows that alignment is where parsers fail, so the two must not be merged.
• I2. Split by sub-task. Report sentence structure, alignment and document relations as three separate residual classes in tools/translate/src/measure.rs. Keep each class as a count in an ordered map.
• I3. A zero-shot control. A target with no delta, only the standard, is the Mova analogue of the zero-shot language. Measure how far its output falls from a target that has a delta, to show what the delta is worth.

For Mova

The shared task gives a split of evaluation into parts that Mova can copy. It bears on open problem 7 (better alignment for the Hebrew reader, by Strong's numbers and an HMM) and on the zero-shot case, which is how the standard behaves when no delta exists for an author. The alignment-sensitive matcher is in zeman-2026-word-alignment-based-evaluation-uniform.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/3taoes4u6cmk

@inproceedings{stepanek2026first,
  title = {{First Shared Task on UMR Parsing}},
  author = {Štěpánek, Jan and Zeman, Daniel and Lopatkova, Marketa and Gamba, Federica and Hledíková, Hana and Xue, Nianwen},
  booktitle = {Proceedings of The Seventh International Workshop on Designing Meaning Representations (DMR 2026) @ LREC 2026},
  year = {2026},
  doi = {10.63317/3taoes4u6cmk},
  url = {https://doi.org/10.63317/3taoes4u6cmk}
}

References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics. find our card
2. Julia Bonn, Matthew J. Buchholz, Jayeol Chun, Andrew Cowell, William Croft, Lukas Denk, Sijia Ge, Jan Hajič, Kenneth Lai, James H. Martin, Skatje Myers, Alexis Palmer, Martha Palmer, Claire Benet Post, James Pustejovsky, Kristine Stenzel, Haibo Sun, Zdeňka Urešová, Rosa Vallejos, Jens E. L. Van Gysel, Meagan Vigus, Nianwen Xue, and Jin Zhao. 2024. Building a broad infrastructure for uniform meaning representations. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2537– 2547, Torino, Italia. ELRA and ICCL. find
3. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. find
4. Jayeol Chun and Nianwen Xue. 2024. Uniform Meaning Representation Parsing as a Pipelined Approach. In Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing, pages 40–52, Bangkok, Thailand. Association for Computational Linguistics. find
5. Rémi de Vergnette and Maxime Amblard. 2026. Sema system for the DMR 2026 shared task: Multistage UMR parsing with Qwen3-4B. In Proceedings of the Seventh International Workshop on Designing Meaning Representations, Palma, Spain. ELRA. find
6. Federica Gamba, Alexis Palmer, and Daniel Zeman. 2025. Bootstrapping UMRs from Universal Dependencies for Scalable Multilingual Annotation. In Proceedings of the 19th Linguistic Annotation Workshop ((LAW-XIX-2025), pages 126–136, Wien, Austria. Association for Computational Linguistics. find our card
7. Johannes Heinecke and Munshi Asadullah. 2026. Orange @ UMR parsing shared task. In Proceedings of the Seventh International Workshop on Designing Meaning Representations, Palma, Spain. ELRA. find
8. Emma Markle, Javier Gutierrez Bach, and Shira Wein. 2026. SETUP: Sentence-level English-To-Uniform Meaning Representation Parser. In Proceedings of the 2026 International Conference on Language Resources and Evaluation (LREC 2026), Palma, Spain. ELRA. find
9. Haibo Sun, Nianwen Xue, Jin Zhao, Liulu Yue, Yao Sun, Keer Xu, and Jiawei Wu. 2024. Chinese UMR annotation: Can LLMs help? In Proceedings of the Fifth International Workshop on Designing Meaning Representations @ LREC-COLING 2024, pages 131–139, Torino, Italia. ELRA and ICCL. find
10. Jens E. L. Van Gysel, Meagan Vigus, Jayeol Chun, Kenneth Lai, Sarah Moeller, Jiarui Yao, Tim O’Gorman, Andrew Cowell, William Croft, Chu-Ren Huang, Jan Hajič, James H. Martin, Stephan Oepen, Martha Palmer, James Pustejovsky, Rosa Vallejos, and Nianwen Xue. 2021. Designing a Uniform Meaning Representation for Natural Language Processing. Künstliche Intelligenz, 35(3):343–360. find
11. Daniel Zeman and Federica Gamba. 2026. Word alignment-based evaluation of Uniform Meaning Representations. In arXiv:2603.26401 [cs.CL]. arXiv.org. ↗ our card
12. Daniel Zeman, Martin Popel, Milan Straka, Jan Hajič, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinková, Jan Hajič jr., Jaroslava Hlaváčová, Václava Kettnerová, Zdeňka Urešová, Jenna Kanerva, Stina Ojala, Anna Missilä, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria de Paiva, Kira Droganova, Héctor Martínez Alonso, Çağrı Çöltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadová, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonça, Tatiana Lando, Rattima Nitisaroj, and Josie Li. 2017. CoNLL 2017 shared task: Multilingual parsing from raw text to Universal Dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1–19, Vancouver, Canada. Association for Computational Linguistics. 8. Language Resource References Julia Bonn and Claire Bonial and Matt Buchholz and Hsiao-Jung Cheng and Alvin Chen and Ching-wen Chen and Andrew Cowell and William Croft and Lukas Denk and Ahmed Elsayed and Eva Fučíková and Federica Gamba and Carlos Gomez and Jan Hajič and Eva Hajičová and Jiří Havelka and Loden Havenmeier and Ath Kilgore and Veronika Kolářová and Lucie Kučová and Kenneth Lai and Bin Li and Jingyi Li and Markéta Lopatková and Marie Mac-Gregor and Marie Mikulová and Jiří Mírovský and Anna Nedoluzhko and Skatje Myers and Michal Novák and Tim O’Gorman and Petr Pajas and Alexis Palmer and Nartha Palmer and Jarmila Panevová and Benét Post and James Pustejovsky and Petr Sgall and Jialin Song and Li Song and Magda Ševčíková and Jan Štěpánek and Zdeňka Urešová and Haibo Sun and Yao Sun and Rosa Vallejos Yopán and Jens Van Gysel and Meagan Vigus and Kristin Wright‑Bettner and Jiawei Wu and Nianwen Xue and Dan Xing and Keer Xu and Zhixing Xu and Liulu Yue and Daniel Zeman and Jin Zhao and Šárka Zikánová and Zdeněk Žabokrtský. 2025. Uniform Meaning Representation 2.0. LINDAT/CLARIAH-CZ Digital Library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find our card
13. Julia Bonn and Claire Bonial and Matt Buchholz and Hsiao-Jung Cheng and Alvin Chen and Ching-wen Chen and Andrew Cowell and William Croft and Lukas Denk and Ahmed Elsayed and Eva Fučíková and Federica Gamba and Carlos Gomez and Jan Hajič and Eva Hajičová and Jiří Havelka and Loden Havenmeier and Hana Hledíková and Ath Kilgore and Veronika Kolářová and Lucie Kučová and Kenneth Lai and Bin Li and Jingyi Li and Markéta Lopatková and Marie MacGregor and Marie Mikulová and Jiří Mírovský and Anna Nedoluzhko and Skatje Myers and Michal Novák and Tim O’Gorman and Petr Pajas and Alexis Palmer and Martha Palmer and Jarmila Panevová and Benét Post and James Pustejovsky and Petr Sgall and Jialin Song and Li Song and Magda Ševčíková and Jan Štěpánek and Zdeňka Urešová and Haibo Sun and Yao Sun and Rosa Vallejos Yopán and Jens Van Gysel and Meagan Vigus and Kristin Wright‑Bettner and Jiawei Wu and Nianwen Xue and Dan Xing and Keer Xu and Zhixing Xu and Liulu Yue and Daniel Zeman and Jin Zhao and Šárka Zikánová and Zdeněk Žabokrtský. 2026. Uniform Meaning Representation 2.2. LINDAT/CLARIAH-CZ Digital Library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
14. Jan Štěpánek and Markéta Lopatková and Daniel Zeman and Federica Gamba and Hana Hledíková and Eva Fučíková and Michal Novák and Šárka Zikánová and Eva Hajičová and Jiří Havelka and Veronika Kolářová and Lucie Kučová and Marie Mikulová and Jiří Mírovský and Anna Nedoluzhko and Petr Pajas and Jarmila Panevová and Petr Sgall and Magda Ševčíková and Zdeňka Urešová and Zdeněk Uniform Žabokrtský and Jan Hajič. 2025. Meaning Representation 2.1 (Czech and Latin). LINDAT/CLARIAH-CZ Digital Library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/tepanek-2026-first-shared-task-umr-parsing/