Incorporate Semantic Structures into Machine Translation Evaluation via UCCA
What it is about
The paper defines semantic core words: words that appear in every good translation of one source sentence. They are found from UCCA units, mainly the main relations of scenes and their participants. The proposed measure, Semantically Weighted Sentence Similarity (SWSS), computes precision and recall over the overlap of core words after stem matching, takes their F1, and then penalises differences between the two UCCA structures and in length. Used as a wrapper around BLEU and METEOR, it improves their correlation with human judgments consistently across the tested settings. The approach borrows the copying idea from neural generation, but the measure itself is count-based.
Ideas
- I1. Core weights from agreement across authors. A concept that every Bible translation of a verse keeps (the divine name, the verb of the main event) gets a high weight in scoring; a concept only one translator keeps gets a low weight. No UCCA is needed: the agreement among KJV, WEB, ASV and JPS is the signal.
- I2. Use the core weight as a priority in the lexicon work (open problem 2): a merge that breaks the agreement on a core concept counts as a larger residual than one that breaks a function word.
- I3. Penalise the structural difference between two readings of one verse, in the way SWSS penalises UCCA differences, using the number of frame roles that differ in
prag.
For Mova
The places are tools/translate (meta.rs for the lexicon, the author delta) and prag. The UCCA parser is not part of Mova. The cross-translation vote, which the paper uses for the core words, is the part that fits, because the Bible pairs are already in the repo's data flow.
Cite
@inproceedings{xu2020incorporate,
title = {{Incorporate Semantic Structures into Machine Translation Evaluation via UCCA}},
author = {Xu, Jin and Guo, Yinuo and Hu, Junfeng},
booktitle = {Proceedings of the Fifth Conference on Machine Translation},
year = {2020},
doi = {10.18653/v1/2020.wmt-1.104},
url = {https://doi.org/10.18653/v1/2020.wmt-1.104}
}References
- Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (ucca). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 228–238. 0.645 0.512 0.543 0.627 0.564 0.627 0.586 0.649 0.503 0.537 0.626 0.574 0.609 0.583 find our card
- Eleftherios Avramidis, Maja Popovic, David Vilar, and Aljoscha Burchardt. 2011. Evaluate with confidence estimation: Machine ranking of translation outputs using grammatical features. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 65–70. Association for Computational Linguistics. *Table 3: Results of ablation experiments. ”+UCCA” is the complete SWSS model combined with Meteor, ”-repr” means the penalties based on UCCA representation (P S , P N , P E ) are removed, ”-len” means the length penalty is removed, and ”None” contains only Meteor without SWSS.* Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 178–186. words is clearly a good and large-scale representation of copy words, according to the results. find our card
- Alexandra Birch, Omri Abend, Ondřej Bojar, and Barry Haddow. 2016. HUME: Human UCCA-based evaluation of machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1264–1274, Austin, Texas. Association for Computational Linguistics. find our card
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/xu-2020-incorporate-semantic-structures-into-machine/