Syntactic Features for Evaluation of Machine Translation

What it is about

The paper adds syntax to n-gram evaluation. The headword-chain metric (HWCM) matches the head-modifier paths of the reference against the hypothesis. The subtree metric (STM) matches constituent subtrees, and a dependency subtree variant (DSTM) matches head subtrees. All of them need a parser on the hypothesis. At sentence level, most syntactic metrics predict fluency better than BLEU, and HWCM predicts overall sentence judgments better too. At corpus level the kernel-based subtree metric reaches a correlation of about 0.88 with human overall judgments on the 2005 test systems, against about 0.78 for BLEU at its best n-gram length. The authors note that the kernel metric is weak at sentence level, so the two levels can disagree.

Ideas

• I1. Headword-chain matching on meta trees: take the head-modifier chains of the reference's trees, read by the same reader, and check that each chain appears in the output's trees in the same order. This measures open problem 3, whether the encoder writes trees.
• I2. Report sentence-level and verse-level-aggregated scores separately, as the paper shows they can disagree. A delta that raises the corpus score but lowers the per-verse agreement is flagged.
• I3. A syntactic fluency check on the encoder output, computed as chain matches, to catch order errors that bag-of-concept scores miss.

For Mova

The places are tools/roundtrip (UD trees) and en/src/lin.rs (order). Mova does not need a parser on the output: the paper's cost is the parsing of noisy MT, and Mova can compare meta trees built from the references instead.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{liu2005syntactic,
  title = {{Syntactic Features for Evaluation of Machine Translation}},
  author = {Liu, Ding and Gildea, Daniel},
  booktitle = {Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization},
  year = {2005},
  url = {https://aclanthology.org/W05-0904/}
}

References

12 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Hiyan Alshawi, Srinivas Bangalore, and Shona Douglas. 2000. Learning dependency translation models as collections of finite state head transducers. Computational Linguistics, 26(1):45–60. find
2. John Blatz, Erin Fitzgerald, George Foster, Simona Gandrabur, Cyril Goutte, Alex Kulesza, Alberto Sanchis, and Nicola Ueffing. 2003. Confidence estimation for machine translation. Technical report, Center for Language and Speech Processing, Johns Hopkins University, Baltimore. Summer Workshop Final Report. find
3. Eugene Charniak, Kevin Knight, and Kenji Yamada. 2003. Syntax-based language models for machine translation. In Proc. MT Summit IX. find
4. Michael Collins and Nigel Duffy. 2001. Convolution kernels for natural language. In Advances in Neural Information Processing Systems. find
5. Michael John Collins. 1999. Head-driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania, Philadelphia. find our card
6. G. Doddington. 2002. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In In HLT 2002, Human Language Technology Conference, San Diego, CA. find
7. Heidi J. Fox. 2002. Phrasal cohesion and statistical machine translation. In In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pages 304–311. find
8. Alex Kulesza and Stuart M. Shieber. 2004. A learning approach to improving sentence-level MT evaluation. In Proceedings of the 10th International Conference on Theoretical and Methodological Issues in Machine Translation (TMI), Baltimore, MD, October. find
9. Dekang Lin. 2004. A path-based transfer model for machine translation. In Proceedings of the 20th International Conference on Computational Linguistics (COLING-04), pages 625–630, Geneva, Switzerland. find
10. Franz Josef Och, Daniel Gildea, Sanjeev Khudanpur, Anoop Sarkar, Kenji Yamada, Alex Fraser, Shankar Kumar, Libin Shen, David Smith, Katherine Eng, Viren Jain, Zhen Jin, and Dragomir Radev. 2004. A smorgasbord of features for statistical machine translation. In Proceedings of the 2004 Meeting of the North American chapter of the Association for Computational Linguistics (NAACL-04), Boston. find
11. Franz Josef Och. 2003. Minimum error rate training for statistical machine translation. In Proceedings of the 41th Annual Conference of the Association for Computational Linguistics (ACL-03). find
12. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Conference of the Association for Computational Linguistics (ACL-02), Philadelphia, PA. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/liu-2005-syntactic-features-evaluation-machine-translation/