Factorising AMR generation through syntax

What it is about

The paper splits AMR generation into two steps. A syntax model first produces a delexicalised constituency tree (only the part-of-speech shape of the sentence) from the AMR; a lexicalisation model then fills in the words given the tree and the AMR. Both are sequence-to-sequence models that share the AMR encoder, and the lexicalisation model has a copy mechanism for rare concept words. The authors argue that the AMR does not fix the syntax, so the choice of shape is a separate decision. Trained on the two AMR releases with silver parses from the Stanford parser, the model gives 26.8 BLEU on test (23.5 for the LDC2015E86 version) and beats the single-step baselines without extra unlabelled data. Sampling three syntactic shapes and realising each one gives 1.52 acceptable paraphrases out of three on average, against 1.19 for a single-step sampler, as judged by crowd annotators.

Ideas

• I1. Two-stage encoder: first choose a delexicalised shape for the meta-world (heads and relations placed in a part-of-speech frame), then fill the words from the chunk table. This is open problem 3, the encoder that writes trees; the shape stage picks structure and the fill stage uses the counts already in profile.rs.
• I2. Shape choice from counts: for each concept and relation set, count the shapes seen in the Bible pairs, take the three most frequent (ties broken by key order), and realise each one deterministically. Measure how many variants a reader round trip accepts against a single-stage baseline.
• I3. Keep the shape choice and the word choice as separate residual layers. The bits spent on each would show whether a loss comes from structure or from wording, which exact mode already needs.
• I4. Paraphrase control: for a verse with several Bible translations, check whether the encoder gives the same meta-world for each rendering. This is a meaning-preservation test that does not depend on BLEU.

For Mova

The ideas fit tools/translate/profile.rs (encoder), tools/translate/measure.rs (residual layers) and en/src/lin.rs (order given a shape). The neural syntax model is not for Mova; the count-based shape table, with a dev-only sampling check, is what carries over.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1223

@inproceedings{cao2019factorising,
  title = {{Factorising AMR generation through syntax}},
  author = {Cao, Kris and Clark, Stephen},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1223},
  url = {https://doi.org/10.18653/v1/n19-1223}
}

References

28 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450. ↗
2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. ICLR. find
3. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 178–186. Association for Computational Linguistics. find our card
4. Daniel Beck, Gholamreza Haffari, and Trevor Cohn. 2018. Graph-to-sequence learning using gated graph neural networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 273–283. Association for Computational Linguistics. find
5. Anna Currey and Kenneth Heafield. 2018. Multi-source syntactic neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2961– 2966, Brussels, Belgium. Association for Computational Linguistics. find
6. Ondřej Dušek and Filip Jurcicek. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 45–51, Berlin, Germany. Association for Computational Linguistics. find
7. Jeffrey Flanigan, Chris Dyer, Noah A. Smith, and Jaime Carbonell. 2016. Generation from abstract meaning representation using tree transducers. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Z. Ghahramani, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 3111–3119. Curran Associates, Inc. pages 731–739, San Diego, California. Association for Computational Linguistics. find our card
8. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems 29 (NIPS). find
9. Claire Gardent and Laura Perez-Beltrachini. 2017. A statistical, grammar-based approach to microplanning. Computational Linguistics, 43(1):1–30. find
10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Compututation, 9(8):1735–1780. find
11. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875–1885, New Orleans, Louisiana. Association for Computational Linguistics. find
12. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. ICLR. find
13. Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural amr: Sequence-to-sequence models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 146–157, Vancouver, Canada. Association for Computational Linguistics. find our card
14. Jindřich Libovický and Jindřich Helcl. 2017. Attention strategies for multi-source sequence-to-sequence learning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 196–202, Vancouver, Canada. Association for Computational Linguistics. find
15. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal. Association for Computational Linguistics. find
16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. find our card
17. Nima Pourdamghani, Kevin Knight, and Ulf Hermjakob. 2016. Generating english from abstract meaning representations. In Proceedings of the 9th International Natural Language Generation conference, pages 21–25, Edinburgh, UK. Association for Computational Linguistics. find our card
18. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073– 1083. Association for Computational Linguistics. find
19. Linfeng Song, Xiaochang Peng, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2017. Amr-to-text generation with synchronous node replacement grammar. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 7–13. Association for Computational Linguistics. find our card
20. Linfeng Song, Yue Zhang, Xiaochang Peng, Zhiguo Wang, and Daniel Gildea. 2016. Amr-to-text generation as a traveling salesman problem. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2084–2089. Association for Computational Linguistics. find
21. Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for amr-to-text generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1616– 1626. Association for Computational Linguistics. find
22. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
23. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. find
24. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 3104–3112. Curran Associates, Inc. find
25. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In C. J. C. Burges, L. Bottou, M. Welling, Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015a. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692–2700. Curran Associates, Inc. find
26. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015b. Grammar as a foreign language. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2773–2781. Curran Associates, Inc. find
27. Marilyn A. Walker, Owen Rambow, and Monica Rogati. 2001. Spot: A trainable sentence planner. In Second Meeting of the North American Chapter of the Association for Computational Linguistics. find
28. Pengcheng Yin, Chunting Zhou, Junxian He, and Graham Neubig. 2018. Structvae: Tree-structured latent variable models for semi-supervised semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 754–765, Melbourne, Australia. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/cao-2019-factorising-amr-generation-through-syntax/