Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation

What it is about

Sentence-level judgments of cross-lingual divergence miss small meaning differences. The paper parses both sides of a parallel sentence pair into AMR, aligns their words with fast_align, translates the concept and role labels into English, and runs Smatch. A score of 1 means the two graphs are the same, and any other score marks a divergence. On 100 gold English-French pairs from the REFreSD dataset, the AMR judgment is never looser than the sentence-level one. Of the pairs that humans called equivalent, 26 had diverging AMRs. The same pipeline works with automatic parses on 1,033 items, and it is more precise than multilingual BERTScore at spotting strictly equivalent sentence pairs. The authors also release the first French AMR set.

Ideas

• I1. A strict gate for the translator. Count a verse and its translation as equivalent only when their meta-worlds coincide after the same reader. Report that strict rate next to chrF, as a second number, not a replacement.
• I2. Word-aligned node mapping. The paper maps nodes through word alignment before comparing. Mova's meta tokens keep their source word links, so the same mapping can be done with ordered BTreeMap structures and no search. Compare it with exact match on the Bible pairs, and run it twice to check that it is deterministic.
• I3. Lexicon candidates from divergence. Pairs that the reader treats as identical but whose graphs differ point to concept merges that drop meaning. List them as candidates for the residual-driven lexicon.

For Mova

The paper is an evaluation method and does not give a translation result. Its main use for Mova is as a stricter check for tools/translate, in measure.rs. It bears on open problem 2, the residual-driven lexicon, because a merge such as be with have should show up as a strict divergence. The matcher that pairs nodes by word is in zeman-2026-word-alignment-based-evaluation-uniform.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{wein2023measuring,
  title = {{Measuring Fine-Grained Semantic Equivalence with Abstract Meaning Representation}},
  author = {Wein, Shira and Wang, Zhuxin and Schneider, Nathan},
  booktitle = {Proceedings of the 15th International Conference on Computational Semantics},
  year = {2023},
  url = {https://aclanthology.org/2023.iwcs-1.16/}
}

References

41 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics. find our card
2. Susan Bassnett. 2013. Translation studies. Routledge. find
3. Rexhina Blloshmi, Rocco Tripodi, and Roberto Navigli. 2020. XL-AMR: Enabling cross-lingual AMR parsing with transfer learning techniques. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2487–2500, Online. Association for Computational Linguistics. find
4. Eleftheria Briakou and Marine Carpuat. 2020. Detecting Fine-Grained Cross-Lingual Semantic Divergences without Supervision by Learning to Rank. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1563–1580, Online. Association for Computational Linguistics. find
5. Eleftheria Briakou and Marine Carpuat. 2021. Beyond noise: Mitigating the impact of fine-grained semantic divergences on neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 7236–7249, Online. Association for Computational Linguistics. find
6. Shu Cai and Kevin Knight. 2013. Smatch: an evaluation metric for semantic feature structures. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 748–752, Sofia, Bulgaria. Association for Computational Linguistics. find
7. Marine Carpuat, Yogarshi Vyas, and Xing Niu. 2017. Detecting cross-lingual semantic divergence for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 69– 79, Vancouver. Association for Computational Linguistics. find
8. Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Evaluating the efficacy of summarization evaluation across languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 801–812, Online. Association for Computational Linguistics. find
9. Marco Damonte. 2019. Understanding and Generating Language with Abstract Meaning Representation. Ph.D. thesis, University of Edinburgh. find
10. Wai Ching Leung, Shira Wein, and Nathan Schneider. 2022. Semantic similarity as a window into vector-and graph-based metrics. In Proc. of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 106–115, Abu Dhabi, United Arab Emirates (Hybrid). find
11. Marco Damonte and Shay B. Cohen. 2018. Cross-lingual Abstract Meaning Representation parsing. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1146–1155, New Orleans, Louisiana. Association for Computational Linguistics. find
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
13. Bonnie Dorr. 1990. Solving thematic divergences in machine translation. In Proceedings of the 28th Annual Meeting on Association for Computational Linguistics, ACL ’90, page 127–134, USA. Association for Computational Linguistics. find our card
14. Bonnie J. Dorr. 1994. Machine translation divergences: A formal description and proposed solution. Computational Linguistics, 20(4):597–633. find our card
15. Pascale Fung and Percy Cheung. 2004. Multi-level bootstrapping for extracting parallel sentences from a quasi-comparable corpus. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 1051–1057, Geneva, Switzerland. COLING. find
16. Valdis Girgzdis, Maija Kale, Martins Vaicekauskis, Ieva Zarina, and Inguna Skadin , a. 2014. Tracing mistakes and finding gaps in automatic word alignments for Latvian-English translation. In Andrius Utka, Gintarė Grigonytė, Jurgita Kapočiūtė-Dzikienė, and Jurgita Vaičenonienė, editors, Human Language Technologies – The Baltic Perspective, volume 268 of Frontiers in Artificial Intelligence and Applications, pages 87–94. IOS Press. find
17. Spence Green, Jeffrey Heer, and Christopher D. Manning. 2013. The efficacy of human post-editing for language translation. In Proc. of CHI, pages 439– 448, New York, NY, USA. find
18. Paul Kingsbury and Martha Palmer. 2002. From Tree-Bank to PropBank. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC’02), Las Palmas, Canary Islands - Spain. European Language Resources Association (ELRA). find
19. Ha Linh and Huyen Nguyen. 2019. A case study on meaning representation for Vietnamese. In Proceedings of the First International Workshop on Designing Meaning Representations, pages 148–153, Florence, Italy. Association for Computational Linguistics. find
20. Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 881–893, Valencia, Spain. Association for Computational Linguistics. find
21. Noelia Migueles Abraira. 2017. A study towards Spanish Abstract Meaning Representation. Master’s thesis, University of the Basque Country, Donostia-San Sebastián, Spain, June. find
22. Noelia Migueles-Abraira, Rodrigo Agerri, and Arantza Diaz de Ilarraza. 2018. Annotating Abstract Meaning Representations for Spanish. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
23. Juri Opitz, Angel Daza, and Anette Frank. 2021. Weisfeiler-leman in the bamboo: Novel AMR graph metrics and a benchmark for AMR graph similarity. Transactions of the Association for Computational Linguistics, 9:1425–1441. find our card
24. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. ↗ our card
25. Martin Potthast, Alberto Barrón-Cedeño, Benno Stein, and Paolo Rosso. 2011. Cross-language plagiarism detection. Language Resources and Evaluation, 45(1):45–62. find
26. Luigi Procopio, Rocco Tripodi, and Roberto Navigli. 2021. SGL: Speaking the graph languages of semantic parsing via multilingual translation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 325–337, Online. Association for Computational Linguistics. find
27. Michael Roth and Talita Anthonio. 2021. UnImplicit shared task report: Detecting clarification requirements in instructional text. In Proceedings of the 1st Workshop on Understanding Implicit and Underspecified Language, pages 28–32, Online. Association for Computational Linguistics. find
28. Naomi Saphra and Adam Lopez. 2015. AMRICA: an AMR inspector for cross-language alignments. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 36–40, Denver, Colorado. Association for Computational Linguistics. find
29. Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021. Wiki-Matrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1351–1361, Online. Association for Computational Linguistics. find
30. Jason R. Smith, Chris Quirk, and Kristina Toutanova. 2010. Extracting parallel sentences from comparable corpora using document level alignment. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 403–411, Los Angeles, California. Association for Computational Linguistics. find
31. Marco Antonio Sobrevilla Cabezudo and Thiago Pardo. 2019. Towards a general Abstract Meaning Representation corpus for Brazilian Portuguese. In Proceedings of the 13th Linguistic Annotation Workshop, pages 236–244, Florence, Italy. Association for Computational Linguistics. find
32. Naoko Taguchi. 2005. Comprehending implied meaning in English as a foreign language. The Modern Language Journal, 89(4):543–562. find
33. Sarah Uhrig, Yoalli Garcia, Juri Opitz, and Anette Frank. 2021. Translate, then parse! a strong baseline for cross-lingual AMR parsing. In Proceedings of the 17th International Conference on Parsing Technologies and the IWPT 2021 Shared Task on Parsing into Enhanced Universal Dependencies (IWPT 2021), pages 58–64, Online. Association for Computational Linguistics. find
34. Yogarshi Vyas, Xing Niu, and Marine Carpuat. 2018. Identifying semantic divergences in parallel text without annotations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1503–1515, New Orleans, Louisiana. Association for Computational Linguistics. find
35. Shira Wein and Nathan Schneider. 2021. Classifying divergences in cross-lingual AMR pairs. In Proceedings of The Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 56–65, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
36. Shira Wein and Nathan Schneider. 2022. Accounting for language effect in the evaluation of cross-lingual AMR parsers. In Proceedings of the 29th International Conference on Computational Linguistics, pages 3824–3834, Gyeongju, Republic of Korea. International Committee on Computational Linguistics. find
37. Nianwen Xue, Ondřej Bojar, Jan Hajič, Martha Palmer, Zdeňka Urešová, and Xiuhong Zhang. 2014. Not an interlingua, but close: Comparison of English AMRs to Chinese and Czech. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 1765– 1772, Reykjavik, Iceland. European Language Resources Association (ELRA). find our card
38. Laura Zeidler, Juri Opitz, and Anette Frank. 2022. A dynamic, interpreted CheckList for meaning-oriented NLG metric evaluation – through the lens of semantic similarity rating. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 157–172, Seattle, Washington. Association for Computational Linguistics. find
39. Yuming Zhai, Gabriel Illouz, and Anne Vilnat. 2020. Detecting non-literal translations by fine-tuning cross-lingual pre-trained language models. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5944–5956, Barcelona, Spain (Online). International Committee on Computational Linguistics. find
40. Yuming Zhai, Aurélien Max, and Anne Vilnat. 2018. Construction of a multilingual corpus annotated with translation relations. In Proceedings of the First Workshop on Linguistic Resources for Natural Language Processing, pages 102–111, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
41. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Proc. of ICLR, Online. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/wein-2023-measuring-fine-grained-semantic-equivalence/