A Human Evaluation of AMR-to-English Generation Systems

What it is about

Most AMR-to-English systems had been scored only with automatic metrics such as BLEU. The paper runs a new human evaluation of five systems on 100 sentences per system, with 100 human references for comparison. Annotators gave fluency scores (without the input) and adequacy scores (with the AMR), each on a 0–100 slider, and ticked error boxes for incomprehensible text, missing information and added information. Nine annotators took part, two per sentence. The references score 87.56 on fluency and 93.68 on adequacy; the best system (Konstas et al.) scores 78.14 and 81.46. The rule-based system (Manning, 2019) scores lowest on both (36.89 and 54.10) but adds the least information, and it often drops function words (for example thank you read). Automatic metrics rank the systems about as humans do, and BERTScore correlates best with adequacy. Agreement on the error boxes is low: kappa is 0.28 for added information.

Ideas

• I1. Error labels from frames: label each output as missing or added information by comparing its frames with the input frames. The paper's error types then become automatic labels that can be checked on a small human sample.
• I2. Function-word check: the rule-based failure (dropped for or thanks for before a gerund) is a complementation problem. Test the realiser in en/src/lin.rs against the complement patterns of 23. Complementation.
• I3. Before annotating added information, write a decision test: for instance, tense is never added if the input has no tense. The paper's kappa of 0.28 is what such a guideline would have to fix.
• I4. Run Mova's realisers on the same inputs and compare the system ranking with the paper's; agreement on fluency is the more useful check.

For Mova

Chapters: 20. Syntax: the Universal Dependencies frame, 23. Complementation, 13. Verbs. Code: en/src/lin.rs (realisation) and tools/roundtrip (frame comparison for the missing and added labels). Measure: counts of missing and added information per generator under the automatic labels, with two identical runs.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.coling-main.420

@inproceedings{manning2020human,
  title = {{A Human Evaluation of AMR-to-English Generation Systems}},
  author = {Manning, Emma and Wein, Shira and Schneider, Nathan},
  booktitle = {Proceedings of the 28th International Conference on Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.420},
  url = {https://doi.org/10.18653/v1/2020.coling-main.420}
}

References

31 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria. Association for Computational Linguistics. find our card
2. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
3. Loïc Barrault, Onďrej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 Conference on Machine Translation (WMT19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, Florence, Italy. Association for Computational Linguistics. find
4. Steven Bird, Edward Loper, and Ewan Klein. 2009. Natural Language Processing with Python. O’Reilly Media Inc. find
5. Onďrej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pages 131–198, Berlin, Germany. Association for Computational Linguistics. find
6. Kris Cao and Stephen Clark. 2019. Factorising AMR generation through syntax. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2157–2163, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
7. Marco Damonte and Shay B. Cohen. 2019. Structural neural encoders for AMR-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3649–3658, Minneapolis, Minnesota. Association for Computational Linguistics. https://github.com/mjpost/sacreBLEU https://www.cs.cmu.edu/~alavie/METEOR/http://www.cs.umd.edu/~snover/tercom/https://github.com/m-popovic/chrF https://github.com/Tiiiger/bert_score ↗
8. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 376–380, Baltimore, Maryland, USA. Association for Computational Linguistics. find
9. Jeffrey Flanigan, Chris Dyer, Noah A. Smith, and Jaime Carbonell. 2016. Generation from Abstract Meaning Representation using tree transducers. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 731–739, San Diego, California. Association for Computational Linguistics. find our card
10. Albert Gatt and Anja Belz. 2010. Introducing Shared Tasks to NLG: The TUNA Shared Task Evaluation Challenges. In Emiel Krahmer and Mariët Theune, editors, Empirical Methods in Natural Language Generation: Data-oriented Methods and Empirical Evaluation, pages 264–293. Springer, Berlin, Heidelberg, Berlin, Heidelberg. find
11. Dimitra Gkatzia and Saad Mahamood. 2015. A snapshot of NLG evaluation practices 2005 - 2014. In Proceedings of the 15th European Workshop on Natural Language Generation (ENLG), pages 57–60, Brighton, UK. Association for Computational Linguistics. find
12. Zhijiang Guo, Yan Zhang, Zhiyang Teng, and Wei Lu. 2019. Densely connected graph convolutional networks for graph-to-sequence learning. Transactions of the Association for Computational Linguistics, 7:297–312. find
13. Kevin Knight, Bianca Badarau, Laura Banarescu, Claire Bonial, Madalina Bardocz, Kira Griffitt, Ulf Hermjakob, Daniel Marcu, Martha Palmer, Tim O’Gorman, and Nathan Schneider. 2017. Abstract Meaning Representation (AMR) Annotation Release 2.0. Technical Report LDC2017T10, Linguistic Data Consortium, Philadelphia, PA. find
14. Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-sequence models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 146–157, Vancouver, Canada. Association for Computational Linguistics. find our card
15. Guy Lapalme. 2019. Verbalizing AMR Structures. http://rali.iro.umontreal.ca/rali/sites/ default/files/publis/GoPhi.pdf . ↗
16. Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Abstract Meaning Representation for multi-document summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1178–1190, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
17. Manuel Mager, Ramón Fernandez Astudillo, Tahira Naseem, Md Arafat Sultan, Young-Suk Lee, Radu Florian, and Salim Roukos. 2020. GPT-too: A language-model-first approach for AMR-to-text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1846–1852, Online. Association for Computational Linguistics. find
18. Emma Manning. 2019. A partially rule-based approach to AMR generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 61–70, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
19. Jonathan May and Jay Priyadarshi. 2017. SemEval-2017 task 9: Abstract Meaning Representation parsing and generation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 536–545, Vancouver, Canada. Association for Computational Linguistics. find
20. Jekaterina Novikova, Onďrej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics. find
21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. find our card
22. Maja Popovíc. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics. find our card
23. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186–191, Brussels, Belgium. Association for Computational Linguistics. find our card
24. Nima Pourdamghani, Kevin Knight, and Ulf Hermjakob. 2016. Generating English from Abstract Meaning Representations. In Proceedings of the 9th International Natural Language Generation conference, pages 21–25, Edinburgh, UK. Association for Computational Linguistics. find our card
25. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393– 401. find our card
26. Leonardo F. R. Ribeiro, Claire Gardent, and Iryna Gurevych. 2019. Enhancing AMR-to-text generation with dual graph representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3183–3194, Hong Kong, China. Association for Computational Linguistics. find
27. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas (AMTA 2006), pages 223–231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas. find our card
28. Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019. Semantic neural machine translation using AMR. Transactions of the Association for Computational Linguistics, 7:19–31. find our card
29. Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for AMR-to-text generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1616–1626, Melbourne, Australia. Association for Computational Linguistics. find
30. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In International Conference on Learning Representations (ICLR 2020), Online. find
31. Jie Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Min Zhang, and Guodong Zhou. 2019. Modeling graph structure in transformer for better AMR-to-text generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5459–5468, Hong Kong, China. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/manning-2020-human-evaluation-amr-english-generation/