Designing a Symbolic Intermediate Representation for Neural Surface Realization
What it is about
Neural generators often hallucinate, repeat or contradict the input. The paper proposes a symbolic intermediate representation between the content and the text. First a model turns each reference sentence of the E2E restaurant data into this representation, and then a second neural model realises the text from it. The representation is exact enough that the realiser reaches a BLEU score of 82.47, and 83.38 when the training data are extended with TripAdvisor sentences. A manual review found that the meaning was lost in only about 5 percent of outputs. The full pipeline beat the E2E challenge winner on automatic scores. The paper argues that the symbolic layer lets non-neural content selection and planning feed a surface realiser that can be trained on more text.
Ideas
- I1. Treat the meta-world as the symbolic layer and count recovery. For each verse, check that every concept in the meta-world is found in the output of the target stack. Report the lost-concept count per author. This is the paper's meaning-recovery check, done with counts.
- I2. Draw a stratified sample of residual failures, about five percent as the paper does, and score each one by hand against the reference. Keep the sample and the scores in the worklog so later changes can be compared.
- I3. Add target-language text to the standard. The paper shows that extra unannotated sentences help the realiser. In the translator, add more target text to the count tables of
profile.rs, and test on a held-out set whether the most specific contexts become more reliable (open problem 6).
For Mova
The paper validates the idea of a symbolic meta-level as the place where meaning is checked. It fits tools/translate (meta.rs as the layer, profile.rs as the standard) and the residual reports. The neural realiser is not for Mova, and the BLEU numbers are for a different task. The counting checks and the stratified hand review are what to borrow.
Cite
@inproceedings{elder2019designing,
title = {{Designing a Symbolic Intermediate Representation for Neural Surface Realization}},
author = {Elder, Henry and Foster, Jennifer and Barry, James and O'Connor, Alexander},
booktitle = {Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation},
year = {2019},
doi = {10.18653/v1/w19-2308},
eprint = {1905.10486},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/w19-2308}
}References
- Anja Belz, Michael White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The First Surface Realisation Shared Task: Overview and Evaluation Results. In Proceedings of the European Workshop on Natural Language Generation, December, pages 217–226. find our card
- Ondej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, and Christof Monz. 2018. Findings of the 2018 Conference on Machine Translation (WMT18). In Proceedings of the Third Conference on Machine Translation, Volume 2: Shared Task Papers, pages 272–307, Belgium, Brussels. Association for Computational Linguistics. find
- Chun-Yen Chen, Dian Yu, Weiming Wen, Yi Mang Yang, Jiaping Zhang, Mingyang Zhou, Kevin Jesse, Austin Chau, Antara Bhowmick, Shreenath Iyer, Giritheja Sreenivasulu, Runxiang Cheng, Ashwin Bhandare, and Zhou Yu. 2018. Gunrock: Building A Human-Like Social Bot By Leveraging Large Scale Real User Data. find
- Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, and Stephan Vogel. 2017. Understanding and Improving Morphological Learning in the Neural Machine Translation Decoder. IJCNLP, pages 142–151. find
- Andrea F Daniele, Matthew R Walter, Mohit Bansal, and Matthew R Walter. 2017. Navigational Instruction Generation as Inverse Reinforcement Learning with Neural Machine Translation. In Proceedings of HRI. find
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/elder-2019-designing-symbolic-intermediate-representation-neural/