Word Alignment-Based Evaluation of Uniform Meaning Representations

What it is about

Graph comparison, such as Smatch, forces every node of one graph onto a node of the other, even when the two nodes are unrelated, so the mismatch scores say little. The paper proposes juːmæʧ, a node matcher for UMR graphs. It first pairs nodes that are aligned to the same word, using the alignment that UMR already stores. Nodes without an alignment are paired only when their concepts are identical. The matching is symmetric and one-to-one. Nodes that find no partner stay unmapped. Partial scores for concepts, relations and attributes are reported. In a small test on 209 English sentences, a parser output converted from UD was compared with gold UMR. Whole-graph scores are lower than Smatch's, because nodes are left unmapped, and on the mapped triples they approach Smatch's. A manual check found that 166 of 203 unmapped nodes (82 percent) were rightly left unmapped. The method is open source.

Ideas

• I1. Source-word matching for Mova. The meta tokens in tools/translate keep their source word links. The same first-by-word, then-by-concept mapping can compare two translations of a verse, without the NP-hard search. Run it next to exact match.
• I2. Unmapped as its own class. Count unmatched meta tokens as a class of their own, not as errors or as forced matches. The 82 percent figure suggests that forced matches hide real differences in the residual.
• I3. Longer words first. The paper prefers alignment to longer words, which avoids leaning on function words. Test the same rule in the Hebrew reader's choice between candidate words.
• I4. One-to-one symmetrisation. The Hebrew alignment from IBM Model 1 gives many-to-many links. Apply the paper's symmetric one-to-one resolution, keeping the most similar candidate, and measure the change in the Hebrew residual.

For Mova

The matcher is a comparison method, so it belongs in tools/translate/src/measure.rs, not in the translator. It bears on open problem 1 (two translations of one verse parse into different trees) and on the alignment work for the Hebrew reader in hebrew.rs (open problem 7). The method's own limit, that UMR has no alignment guidelines, applies to any Mova alignment too.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2603.26401

@misc{zeman2026word,
  title = {{Word Alignment-Based Evaluation of Uniform Meaning Representations}},
  author = {Zeman, Daniel and Gamba, Federica},
  year = {2026},
  eprint = {2603.26401},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.26401}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omri Abend and Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228–238, Sofia, Bulgaria. Association for Computational Linguistics. find our card
2. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics. find our card
3. Julia Bonn, Matthew J. Buchholz, Jayeol Chun, Andrew Cowell, William Croft, Lukas Denk, Sijia Ge, Jan Hajič, Kenneth Lai, James H. Martin, Skatje Myers, Alexis Palmer, Martha Palmer, Claire Benet Post, James Pustejovsky, Kristine Stenzel, Haibo Sun, Zdeňka Urešová, Rosa Vallejos, Jens E. L. Van Gysel, Meagan Vigus, Nianwen Xue, and Jin Zhao. 2024. Building a broad infrastructure for uniform meaning representations. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2537– 2547, Torino, Italia. ELRA and ICCL. find
4. Julia Bonn, Skatje Myers, Jens E. L. Van Gysel, Lukas Denk, Meagan Vigus, Jin Zhao, Andrew Cowell, William Croft, Jan Hajič, James H. Martin, Alexis Palmer, Martha Palmer, James Pustejovsky, Zdenka Urešová, Rosa Vallejos, and Nianwen Xue. 2023. Mapping AMR to UMR: Resources for adapting existing corpora for cross-lingual compatibility. In Proceedings of the 21st International Workshop on Treebanks and Linguistic Theories (TLT, GURT/SyntaxFest 2023), pages 74–95, Washington, D.C. Association for Computational Linguistics. find our card
5. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. find
6. Federica Gamba, Alexis Palmer, and Daniel Zeman. 2025. Bootstrapping UMRs from Universal Dependencies for scalable multilingual annotation. In Proceedings of the 19th Linguistic Annotation Workshop (LAW-XIX-2025), pages 126– 136, Vienna, Austria. Association for Computational Linguistics. ↗ our card
7. Angelina Ivanova, Stephan Oepen, Lilja Øvrelid, and Dan Flickinger. 2012. Who did what to whom? a contrastive study of syntacto-semantic dependencies. In Proceedings of the Sixth Linguistic Annotation Workshop, pages 2–11, Jeju, Republic of Korea. Association for Computational Linguistics. find
8. James J. McGregor. 1982. Backtrack search algorithms and the maximal common subgraph problem. Software: Practice and Experience, 12. find
9. Stephan Oepen, Omri Abend, Lasha Abzianidze, Johan Bos, Jan Hajič, Daniel Hershcovich, Bin Li, Tim O’Gorman, Nianwen Xue, and Daniel Zeman. 2020. MRP 2020: The second shared task on cross-framework and cross-lingual meaning representation parsing. In Proceedings of the CoNLL 2020 Shared Task: Cross-Framework Meaning Representation Parsing, pages 1–22, Online. Association for Computational Linguistics. find
10. Petr Sgall, Eva Hajičová, and Jarmila Panevová. 1986. The Meaning of the Sentence in its Semantic and Pragmatic Aspects. Springer Science & Business Media. find
11. Jan Štěpánek, Daniel Zeman, Markéta Lopatková, Federica Gamba, and Hana Hledíková. 2025. Comparing manual and automatic UMRs for Czech and Latin. In Proceedings of the Sixth International Workshop on Designing Meaning Representations, pages 1–12, Prague, Czechia. Association for Computational Lingustics. find
12. Milan Straka. 2018. UDPipe 2.0 prototype at CoNLL 2018 UD shared task. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 197–207, Brussels, Belgium. Association for Computational Linguistics. find our card
13. Haibo Sun, Jayeol Chun, and Nianwen Xue. 2025. AnCast++: Document-level evaluation of graph-based meaning representations. In Findings of the Association for Computational Linguistics: ACL 2025, pages 19642–19654, Vienna, Austria. Association for Computational Linguistics. find
14. Haibo Sun and Nianwen Xue. 2024. Anchor and broadcast: An efficient concept alignment approach for evaluation of semantic graphs. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1052–1062, Torino, Italia. ELRA and ICCL. find
15. Jens van Gysel, Meagan Vigus, Jayeol Chun, Kenneth Lai, Sarah Moeller, Jiarui Yao, Tim O’Gorman, James Cowell, William Croft, Chu-Ren Huang, Jan Hajič, James Martin, Stephan Oepen, Martha Palmer, James Pustejovsky, and Rosa Vallejos. 2021. Designing a uniform meaning representation for natural language processing. KI - Künstliche Intelligenz, 35(2):343–360. find
16. Zdeněk Žabokrtský, Daniel Zeman, and Magda Ševčíková. 2020. Sentence meaning representations across languages: What can we learn from existing frameworks? Computational Linguistics, 46(3):605–665. find
17. Aleksandr K. Žolkovskij and Igor A. Mel’čuk. 1967. O semantičeskom sinteze (on semantic synthesis). Problemy kybernetiki, 19:177–238. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/zeman-2026-word-alignment-based-evaluation-uniform/