METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments

What it is about

The paper proposes METEOR, a sentence-level metric built on explicit word-to-word alignment between the system output and each reference. The alignment is made in stages: exact surface match first, then stem match, then synonym match, each stage using only words not yet aligned, with ties broken by the fewest crossing links. From the alignment the metric takes unigram precision and recall, combines them into a recall-weighted harmonic mean, and multiplies by a penalty for fragmentation: matched words are grouped into chunks, and the penalty rises with the number of chunks, up to half the score. The authors argue that BLEU's geometric mean fails at sentence level and that its brevity penalty does not compensate for lost recall. On the LDC TIDES 2003 Arabic and Chinese data with human adequacy and fluency scores, segment-level Pearson correlation was about 0.35 for Arabic and 0.33 for Chinese, higher than unigram precision, recall or their F1, and recall alone was already better than BLEU at system level.

Ideas

• I1. Make the alignment staged in tools/translate/src/measure.rs: exact match, then stem, then synonym from a BTreeMap concept table. The translator's meta-world already merges spellings and synonyms (thou, you, ye), so its concept table is the natural synonym source. Count matches per stage, so each report says where a match came from.
• I2. Use the fragmentation penalty as an order measure. Count the chunks of matched words in the output; a linearizer change in en/src/lin.rs that increases the chunk count will show in this number without touching the exact-match score.
• I3. Compute the recall-weighted form as a dev check. The paper finds recall the most important part, which bears on the translator's open question of dropped function words (problem 4).
• I4. Count crossing alignments, the paper's measure of reordering, as a probe for the Hebrew verb-subject order (open problem 7).

For Mova

The code is tools/translate/src/measure.rs. The staged alignment is the cleanest way to bring the meta-world's concepts into a scorer without changing exact mode, and the stage counts make the scorer's output inspectable, which matches the translator's goal of saying why. Run it on the dev split twice before any claim, as the metric has tunable parameters that the paper itself did not train.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{banerjee2005meteor,
  title = {{METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments}},
  author = {Banerjee, Satanjeev and Lavie, Alon},
  booktitle = {Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization},
  year = {2005},
  url = {https://aclanthology.org/W05-0909/}
}

References

7 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. George Doddington. 2002. Automatic Evaluation of Machine Translation Quality using N-gram Co-occurrence Statistics. In Proceedings of 2 nd Human Language Technologies Conference (HLT-02). San Diego, CA. pp. 128-132. find
2. Margaret King, Andrei Popescu-Belis and Eduard Hovy. 2003. FEMTI: Creating and Using a Framework for MT Evaluation. In Proceedings of MT Summit IX, New Orleans, LA. find
3. Alon Lavie, Kenji Sagae and Shyamsundar Jayaraman, 2004. The Significance of Recall in Automatic Metrics for MT Evaluation. In Proceedings of AMTA-2004, Washington DC. September 2004. find
4. Bo Pang, Kevin Knight and Daniel Marcu. 2003. Syntax-based Alignment of Multiple Translations: Extracting Paraphrases and Generating New Sentences. In Proceedings of HLT-NAACL 2003. Edmonton, Canada. May 2003. find our card
5. Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu. 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40 th Annual Meeting of the Association for Computational Linguistics (ACL-02). Philadelphia, PA. July 2002. pp. 311-318. find our card
6. Joseph P. Turian, Luke Shen and I. Dan Melamed. 2003. Evaluation of Machine Translation and its Evaluation. In Proceedings of MT Summit IX, New Orleans, LA. find
7. C. van Rijsbergen. 1979. Information Retrieval. Butterworths. London, England. 2 nd Edition. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/banerjee-2005-meteor-automatic-metric-mt-evaluation/