The First Multilingual Surface Realisation Shared Task (SR'18): Overview and Evaluation Results

What it is about

The report presents the first multilingual shared task on surface realisation, the step that turns an unordered dependency tree into a sentence. Inputs come from Universal Dependencies treebanks with word order removed and tokens lemmatised. The shallow track covers ten languages. The deep track goes further and also removes function words and surface morphology, and it covers three languages. Systems were scored with automatic intrinsic metrics (BLEU-type scores) and by human judges on readability and meaning similarity. The organisers report high participation for a first run and give per-language tables of both kinds of score. The paper is an evaluation report: the systems themselves are described in other papers of the same volume.

Ideas

• I1. Split the linearizer's test into two levels, as the shallow and deep tracks do. Feed en/src/lin.rs the same trees with and without function words and score each separately. A large drop in the second level shows which information the realiser depends on.
• I2. Score outputs with more than one measure, as the shared task does with automatic scores and human judgements. For the translator, keep exact match and chrF, and add a small hand-judged sample for meaning similarity. Check whether the measures agree on dev before any claim.
• I3. Use the deep-track idea as a test for open problem 4: give the encoder meta tokens with function words removed and measure how many of them it restores. The gap is the bit cost of the function words the meta-world does not yet carry.

For Mova

The shared task is a benchmark, not a method, so the value is in its test design. It fits tools/translate (measure.rs), en/src/lin.rs and tools/roundtrip. The shallow and deep inputs map onto the translator's residual classes (word order, function words, morphology). Use the track split as a measurement design, and keep the test set held out until the once-per-release run.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/w18-3601

@inproceedings{mille2018first,
  title = {{The First Multilingual Surface Realisation Shared Task (SR'18): Overview and Evaluation Results}},
  author = {Mille, Simon and Belz, Anja and Bohnet, Bernd and Graham, Yvette and Pitler, Emily and Wanner, Leo},
  booktitle = {Proceedings of the First Workshop on Multilingual Surface Realisation},
  year = {2018},
  doi = {10.18653/v1/w18-3601},
  url = {https://doi.org/10.18653/v1/w18-3601}
}

References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Anja Belz and Eric Kow. 2011. Discrete vs. continuous rating scales for language evaluation in NLP. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL-HLT’11). find
2. Anja Belz, Michael White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The first surface realisation shared task: Overview and evaluation results. In Proceedings of the 13th European Workshop on Natural Language Generation, ENLG ’11, pages 217–226, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017a. Findings of the 2017 conference on machine translation (WMT’17). In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pages 169– 214, Copenhagen, Denmark. Association for Computational Linguistics. find
4. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. Creating training corpora for micro-planners. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada. Association for Computational Linguistics. find
5. Yvette Graham, George Awad, and Alan Smeaton. 2017. Evaluation of Automatic Video Captioning Using Direct Assessment. ArXiv e-prints. find
6. Yvette Graham and Timothy Baldwin. 2014. Testing for significance of increased correlation with human judgment. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 172–176, Doha, Qatar. Association for Computational Linguistics. find
7. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, FirstView:1–28. find
8. Jonathan May and Jay Priyadarshi. 2017. Semeval-2017 task 9: Abstract meaning representation parsing and generation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 534–543, Vancouver, Canada. Association for Computational Linguistics. find
9. Adam Meyers, R. Reeves, C. Macleod, R. Szekely, V. Zielinska, B. Young, and R. Grishman. 2004. The NomBank project: An interim report. In HLT-NAACL 2004 Workshop: Frontiers in Corpus Annotation, Boston, MA, May 2004, pages 24–31. find
10. Simon Mille, Roberto Carlini, Ivan Latorre, and Leo Wanner. 2017. Upf at epe 2017: Transduction-based deep analysis. In Shared Task on Extrinsic Parser Evaluation (EPE 2017), pages 80–88, Pisa, Italy. find
11. Joakim Nivre and Marie-Catherine de Marneffe et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of LREC, Portorož, Slovenia. find our card
12. Jekaterina Novikova, Ondrej Dušek, and Verena Rieser. 2017. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Saarbrücken, Germany. ArXiv:1706.09254. ↗
13. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–105. find our card
14. K. Papineni, S. Roukos, T. Ward, and W. j. Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proc. 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/mille-2018-first-multilingual-surface-realisation-shared/