Linguistic realisation as machine translation: Comparing different MT models for AMR-to-text generation

What it is about

The paper frames AMR-to-text as translation and compares phrase-based and neural MT on the same data. Three preprocessing steps are studied, and every combination is tried (2^3 = 8 strategies): delexicalisation (constants replaced by tags), compression (a conditional random field decides which nodes and edges to keep, using their alignment to words in context), and linearisation (depth-first order, optionally reordered by a two-step maximum-entropy preorderer that places each subtree before or after its head). On the SemEval 2017 data of about 36,000 pairs, preordering helps both models. The best phrase-based setting (delexicalised, compressed, preordered) reaches 26.8 BLEU, comparable to the best published phrase-based result. The neural models are weaker at this data size, which the authors attribute to the small corpus.

Ideas

• I1. Run the full 2x2x2 grid as a standing measurement for the translator's reader (keep tags, drop nodes, reorder), reporting each setting on dev, twice. The paper shows that the best option depends on the decoder family, so the count-based encoder should be measured the same way.
• I2. Compression by counts: for each concept and relation pair, count how often its node is aligned to any target word in the Bible pairs, and drop it when the rate is under a threshold tuned on dev. Measure the precision of the drop.
• I3. Preordering by counts: decide whether each subtree goes before or after its head with a table keyed by (head frame, subtree frame, side). The side tables in en/src/lin.rs already approximate this; a keyed table makes the decision visible and countable.
• I4. Data-size caution: the neural decoders lose at about 36,000 pairs. Any learned component of the translator should first be compared with a count baseline on the same pairs before a claim is made.

For Mova

The grid fits tools/translate as a measurement, the drop rule fits tools/translate/meta.rs, and the preorder table fits en/src/lin.rs. The paper's neural models are not for Mova; its finding that preordering matters most is the part to test first.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-3501

@inproceedings{ferreira2017linguistic,
  title = {{Linguistic realisation as machine translation: Comparing different MT models for AMR-to-text generation}},
  author = {Ferreira, Thiago Castro and Calixto, Iacer and Wubben, Sander and Krahmer, Emiel},
  booktitle = {Proceedings of the 10th International Conference on Natural Language Generation},
  year = {2017},
  doi = {10.18653/v1/w17-3501},
  url = {https://doi.org/10.18653/v1/w17-3501}
}

References

35 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Yoav Artzi, Kenton Lee, and Luke Zettlemoyer. 2015. Broad-coverage ccg semantic parsing with amr. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1699–1710, Lisbon, Portugal, September. Association for Computational Linguistics. find
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In International Conference on Learning Representations, ICLR 2015, San Diego, California. find
3. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186, Sofia, Bulgaria, August. Association for Computational Linguistics. find our card
4. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Carolina Scarton, Lucia Specia, and Marco Turchi. 2015. Findings of the 2015 Workshop on Statistical Machine Translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 1–46, Lisbon, Portugal, September. Association for Computational Linguistics. find
5. Johan Bos. 2016. Expressive power of abstract meaning representations. Comput. Linguist., 42(3):527–535, September. find
6. Colin Cherry and George Foster. 2012. Batch tuning strategies for statistical machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT ’12, pages 427–436, Stroudsburg, PA, USA. Association for Computational Linguistics. find
7. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 103–111, Doha, Qatar, October. find
8. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 363–370, Ann Arbor, Michigan, June. Association for Computational Linguistics. find
9. Jeffrey Flanigan, Sam Thomson, Jaime Carbonell, Chris Dyer, and Noah A. Smith. 2014. A discriminative graph-based parser for the abstract meaning representation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1426–1436, Baltimore, Maryland, June. Association for Computational Linguistics. find
10. Jeffrey Flanigan, Chris Dyer, Noah A. Smith, and Jaime Carbonell. 2016. Generation from abstract meaning representation using tree transducers. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 731–739, San Diego, California, June. Association for Computational Linguistics. find our card
11. Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In Advances in Neural Information Processing Systems, NIPS, pages 1019–1027, Barcelona, Spain. find
12. Michel Galley and Christopher D. Manning. 2008. A simple and effective hierarchical phrase reordering model. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 848–856, Honolulu, Hawaii, October. Association for Computational Linguistics. find
13. A. Gatt and E. Krahmer. 2017. Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation. ArXiv e-prints, March. find
14. Dimitra Gkatzia, Helen Hastie, and Oliver Lemon. 2014. Comparing multi-label classification with reinforcement learning for summarisation of time-series data. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1231–1240, Baltimore, Maryland, June. Association for Computational Linguistics. find
15. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified kneser-ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 690–696, Sofia, Bulgaria, August. Association for Computational Linguistics. find
16. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 48–54, Stroudsburg, PA, USA. Association for Computational Linguistics. find
17. Philipp Koehn, Amittai Axelrod, Alexandra Birch, Chris Callison-Burch, Miles Osborne, and David Talbot. 2005. Edinburgh System Description for the 2005 IWSLT Speech Translation Evaluation. In International Workshop on Spoken Language Translation. find
18. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, ACL ’07, pages 177–180, Stroudsburg, PA, USA. Association for Computational Linguistics. find
19. I. Konstas, S. Iyer, M. Yatskar, Y. Choi, and L. Zettlemoyer. 2017. Neural AMR: Sequence-to-Sequence Models for Parsing and Generation. ArXiv e-prints, April. find our card
20. Alon Lavie and Abhaya Agarwal. 2007. Meteor: An Automatic Metric for MT Evaluation with High Levels of Correlation with Human Judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT ’07, pages 228–231, Prague, Czech Republic. find
21. Uri Lerner and Slav Petrov. 2013. Source-side classifier preordering for machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 513–523, Seattle, Washington, USA, October. Association for Computational Linguistics. find
22. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160–167, Sapporo, Japan, July. Association for Computational Linguistics. find
23. Naoaki Okazaki. 2007. Crfsuite: a fast implementation of conditional random fields (crfs). find
24. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July. Association for Computational Linguistics. find our card
25. Nima Pourdamghani, Yang Gao, Ulf Hermjakob, and Kevin Knight. 2014. Aligning english strings with abstract meaning representation graphs. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 425–429, Doha, Qatar, October. Association for Computational Linguistics. find our card
26. Nima Pourdamghani, Kevin Knight, and Ulf Hermjakob. 2016. Generating english from abstract meaning representations. In Proceedings of the 9th International Natural Language Generation conference, pages 21– 25, Edinburgh, UK, September 5-8. Association for Computational Linguistics. find our card
27. Ehud Reiter and Robert Dale. 2000. Building natural language generation systems. Cambridge University Press, New York, NY, USA. find
28. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of Association for Machine Translation in the Americas, AMTA, pages 223–231, Cambridge, MA, USA. find our card
29. Linfeng Song, Yue Zhang, Xiaochang Peng, Zhiguo Wang, and Daniel Gildea. 2016. Amr-to-text generation as a traveling salesman problem. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2084–2089, Austin, Texas, November. Association for Computational Linguistics. find
30. L. Song, X. Peng, Y. Zhang, Z. Wang, and D. Gildea. 2017. AMR-to-text Generation with Synchronous Node Replacement Grammar. ArXiv e-prints, February. find our card
31. M. Theune, E. Klabbers, J. R. De Pijper, E. Krahmer, and J. Odijk. 2001. From data to speech: A general approach. Nat. Lang. Eng., 7(1):47–86, March. find
32. Sanja Štajner, Iacer Calixto, and Horacio Saggion. 2015. Automatic text simplification for spanish: Comparative evaluation of various simplification strategies. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 618–626, Hissar, Bulgaria, September. find
33. Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers - Volume 1, ACL ’12, pages 1015–1024, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
34. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In David Blei and Francis Bach, editors, Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 2048–2057. JMLR Workshop and Conference Proceedings. find
35. Matthew D. Zeiler. 2012. ADADELTA: An Adaptive Learning Rate Method. CoRR, abs/1212.5701. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ferreira-2017-linguistic-realisation-as-machine-translation/