The WebNLG Challenge: Generating Text from RDF Data

What it is about

The WebNLG challenge maps sets of RDF triples taken from DBpedia to text, and is built to test microplanning: referring expressions, aggregation, lexicalisation, surface realisation and sentence segmentation chosen together. The dataset has 25,298 data-text pairs over 9,674 distinct triple sets, drawn from 15 DBpedia categories and 373 properties. Inputs were chosen for variety in shape (sibling triples, chains) and in the number of distinct property patterns, so that the same content can be said in different syntax. The authors compare it with an older dataset (RNNLG): WebNLG has about twice the distinct input patterns and ten times the input shapes, and a richer lexicalisation. Eight systems from six teams were scored with BLEU, METEOR and TER on seen and unseen categories. Pipeline systems built on templates or grammar were weakest on seen categories, while learned systems did better there. The results are automatic only, since human judging was planned for later. A larger follow-up dataset is also released.

Ideas

• I1. A micro-benchmark for the translator. Build a small test set of meta-world inputs whose shapes (siblings, chains) and number of distinct patterns follow the paper's recipe. Count the shapes in the current verse set first, and see which shapes Mova's rules never meet.
• I2. Seen versus unseen concepts. Report exact match separately for concepts that occur in the training Bible pairs and for concepts that do not, as the paper splits seen and unseen categories. This shows directly how much the bare form from morph costs, and it bears on open problem 2.
• I3. Delexicalise names. The UMelbourne entry replaces entity names by typed identifiers before generation and fills them back after. In the translator, proper names in genealogies may be a large error source; check this with a count, then try typed name slots in the encoder and measure on Genesis.
• I4. Keep an explicit rule-based baseline in every report, as the paper does for its own baseline. A simple per-relation template gives the floor the encoder must beat, and the report should show it.

For Mova

The relevant code is tools/translate/src/profile.rs (encoder), tools/translate/src/measure.rs (reports) and morph for unseen forms. The dataset is not Bible text and the paper's numbers do not transfer. Its method of diversity-controlled inputs and seen-versus-unseen reporting does.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-3518

@inproceedings{gardent2017webnlg,
  title = {{The WebNLG Challenge: Generating Text from RDF Data}},
  author = {Gardent, Claire and Shimorina, Anastasia and Narayan, Shashi and Perez-Beltrachini, Laura},
  booktitle = {Proceedings of the 10th International Conference on Natural Language Generation},
  year = {2017},
  doi = {10.18653/v1/w17-3518},
  url = {https://doi.org/10.18653/v1/w17-3518}
}

References

26 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR-2015 (abs/1409.0473). ↗
2. Eva Banik, Claire Gardent, and Eric Kow. 2013. The kbgen challenge. In the 14th European Workshop on Natural Language Generation (ENLG), pages 94–97. find
3. Anja Belz and Albert Gatt. 2007. The attribute selection for gre challenge: Overview and evaluation results. Proceedings of UCNLG+ MT: Language Generation and Machine Translation, pages 75–83. find
4. Anja Belz, Eric Kow, Jette Viethen, and Albert Gatt. 2008. The grec challenge: Overview and evaluation results. find
5. Anja Belz, Eric Kow, and Jette Viethen. 2009. The grec named entity generation challenge 2009: overview and evaluation results. In Proceedings of the 2009 Workshop on Language Generation and Summarisation, pages 88–98. Association for Computational Linguistics. find
6. Anja Belz, Eric Kow, Jette Viethen, and Albert Gatt. 2010. Generating referring expressions in context: The grec task evaluation challenges. In Empirical methods in natural language generation, pages 294– 327. Springer. find
7. Anja Belz, Michael White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The first surface realisation shared task: Overview and evaluation results. In Proceedings of the 13th European Workshop on Natural Language Generation, ENLG ’11, pages 217–226, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
8. Nadjet Bouayad-Agha, Gerard Casamayor, Leo Wanner, and Chris Mellish. 2013. Overview of the first content selection challenge from open semantic web data. In ENLG, pages 98–102. find
9. J. B. Carroll. 1964. Language and thought. NJ: Prentice-Hall. Englewood Cliffs. find
10. Thiago Castro Ferreira, Emiel Krahmer, and Sander Wubben. 2016. Towards more variation in text generation: Developing and evaluating variation models for choice of referential form. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 568–577. find
11. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the EACL 2014 Workshop on Statistical Machine Translation. find
12. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for nlg micro-planners. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 179–188. find
13. Albert Gatt, Anja Belz, and Eric Kow. 2008. The tuna challenge 2008: Overview and evaluation results. In Proceedings of the Fifth International Natural Language Generation Conference, pages 198–206. Association for Computational Linguistics. find
14. Albert Gatt, Anja Belz, and Eric Kow. 2009. The tuna-reg challenge 2009: Overview and evaluation results. In Proceedings of the 12th European Workshop on Natural Language Generation, pages 174–182. Association for Computational Linguistics. find
15. G. Klein, Y. Kim, Y. Deng, J. Senellart, and A. M. Rush. 2017. OpenNMT: Open-Source Toolkit for Neural Machine Translation. ArXiv e-prints. find
16. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between european languages. In Proceedings of the Workshop on Statistical Machine Translation, StatMT ’06, pages 102–121. find
17. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL on Interactive Poster and Demonstration Sessions, pages 177– 180. find
18. Xiaofei Lu. 2008. Automatic measurement of syntactic complexity using the revised developmental level scale. In FLAIRS Conference, pages 153–158. find
19. Simon Mille, Roberto Carlini, Alicia Burga, and Leo Wanner. 2017. FORGe at SemEval-2017 task 9: Deep sentence generation based on a sequence of graph transducers. In Proceedings of SemEval-2017, pages 917–920. find
20. Shashi Narayan, Claire Gardent, Shay B. Cohen, and Anastasia Shimorina. 2017. Split and rephrase. In Proceedings of EMNLP. ↗
21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
22. Laura Perez-Beltrachini and Claire Gardent. 2017. Analysing data-to-text generation benchmarks. In Proceedings of the tenth International Natural Language Generation Conference, INLG. find
23. Laura Perez-Beltrachini, Rania Sayed, and Claire Gardent. 2016. Building rdf content for data-to-text generation. In COLING, pages 1493–1502. find
24. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch-Mayne, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel Laubli, Antonio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: A toolkit for neural machine translation. In Proceedings of the EACL 2017 Software Demonstrations, pages 65–68, 4. ↗
25. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of association for machine translation in the Americas, volume 200. find our card
26. Tsung-Hsien Wen, Milica Gašić, Nikola Mrkšić, Lina M. Rojas-Barahona, Pei-Hao Su, David Vandyke, and Steve Young. 2016. Multi-domain neural network language generation for spoken dialogue systems. In Proceedings of NAACL-HLT. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/gardent-2017-webnlg-challenge-generating-text-rdf/