Mova0.5 pre-release⌕
☰
home / docs / papers / Machine translation / The Third Multilingual Surface Realisation Shared Task (SR'20): Overview and Evaluation Results

The Third Multilingual Surface Realisation Shared Task (SR'20): Overview and Evaluation Results

What it is about

The third edition of the surface realisation task keeps the two tracks of the earlier editions: a shallow track with unordered lemmatised UD trees, and a deep track that also removes function words and morphology. Each track now has two subtracks. In the restricted subtrack only the data approved for the track may be used for training. In the open subtrack any resource is allowed. The shallow track covers eleven languages and the deep track three. A new test set was built from Wikipedia articles, which tests how well systems generalise to a new domain. Systems were judged by automatic metrics and by direct assessment of readability and meaning similarity against the reference sentences. The paper reports the results for each subtrack and gives short summaries of the participating systems.

Ideas

For Mova

The restricted/open split is a clean way to report the translator's standard against extra data. It fits tools/translate (profile.rs, measure.rs) and the corpus layout in the project data. The Wikipedia-derived test shows how to get a domain-shift check without touching the once-per-release test. The paper itself is a benchmark report, so the idea to take is the protocol, not any system.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{mille2020third,
  title = {{The Third Multilingual Surface Realisation Shared Task (SR'20): Overview and Evaluation Results}},
  author = {Mille, Simon and Belz, Anya and Bohnet, Bernd and Ferreira, Thiago Castro and Graham, Yvette and Wanner, Leo},
  booktitle = {Proceedings of the Third Workshop on Multilingual Surface Realisation},
  year = {2020},
  url = {https://aclanthology.org/2020.msr-1.1/}
}

References

39 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual NLP. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 183–192, Sofia, Bulgaria, August. Association for Computational Linguistics. find
  2. G. Awad, A. Butt, K. Curtis, Y. Lee, J. Fiscus, A. Godil, A. Delgado, J. Zhang, E. Godard, L. Diduch, A. F. Smeaton, Y. Graham, and W. Kraaij. 2019. Trecvid 2019: An evaluation campaign to benchmark video activity 13 https://sites.google.com/site/genchalrepository/surface-realisation/ sr-20-multilingual detection, video captioning and matching, and video search & retrieval. In Proceedings of TRECVID, volume 2019. ↗
  3. Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019. Findings of the 2019 conference on machine translation (wmt19). In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 1–61, Florence, Italy, August. Association for Computational Linguistics. find
  4. Loïc Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Alexander Fraser, Yvette Graham, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, and Matteo Negri, editors. 2020. Proceedings of the Fifth Conference on Machine Translation. Association for Computational Linguistics, Online, November. find
  5. Anya Belz and Eric Kow. 2011. Discrete vs. continuous rating scales for language evaluation in NLP. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL-HLT’11). find
  6. Anya Belz, Michael White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The first surface realisation shared task: Overview and evaluation results. In Proceedings of the 13th European Workshop on Natural Language Generation, ENLG ’11, pages 217–226, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  7. Wanxiang Che, Jiang Guo, Yuxuan Wang, Bo Zheng, Huaipeng Zhao, Yang Liu, Dechuan Teng, and Ting Liu. 2017. The hit-scir system for end-to-end parsing of universal dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 52–62. find our card
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. ↗
  9. Ondřej Dušek, David M Howcroft, and Verena Rieser. 2019. Semantic noise matters for neural natural language generation. arXiv preprint arXiv:1911.03905. ↗
  10. Thiago Castro Ferreira and Emiel Krahmer. 2019. find
  11. Surface Realization Shared Task 2019 (SR’19): The Tilburg University Approach. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation, Hong Kong, China. find
  12. Thiago Castro Ferreira, Sander Wubben, and Emiel Krahmer. 2018. Surface realization shared task 2018 (sr18): The tilburg university approach. In Proceedings of the First Workshop on Multilingual Surface Realisation, pages 35–38. find
  13. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2016. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, FirstView:1–28, 1. find
  14. Yvette Graham, George Awad, and Alan Smeaton. 2017. Evaluation of Automatic Video Captioning Using Direct Assessment. ArXiv e-prints, October. find
  15. Yvette Graham, George Awad, and Alan Smeaton. 2018. Evaluation of automatic video captioning using direct assessment. PLOS ONE, 13(9):1–20, 09. find
  16. Yvette Graham, Barry Haddow, and Philipp Koehn. 2019. Translationese in machine translation evaluation. CoRR, abs/1906.09833. ↗
  17. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693–1701. find
  18. Xinyu Hua and Lu Wang. 2019. Sentence-level content planning and style specification for neural text generation. find
  19. Adám Kovács, Evelin Ács, Judit Ács, András Kornai, and Gábor Recski. 2019. BME-UW at SR’19: Surface Realization with Interpreted Regular Tree Grammars. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation, Hong Kong, China. find
  20. Guy Lapalme. 2019. Realizing Universal Dependencies Structures Using a Symbolic Approach. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation, Hong Kong, China. find
  21. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871–7880, Online, July. Association for Computational Linguistics. find
  22. KyungTae Lim, Cheoneum Park, Changki Lee, and Thierry Poibeau. 2018. Sex bist: A multi-source trainable parser with deep contextualized lexical representations. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 143–152. find our card
  23. Qingsong Ma, Yvette Graham, Timothy Baldwin, and Qun Liu. 2017. Further investigation into reference bias in monolingual evaluation of machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2466–2475, Copenhagen, Denmark, September. Association for Computational Linguistics. find
  24. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843. ↗
  25. Simon Mille, Bernd Bohnet, Leo Wanner, and Anja Belz. 2017. Shared task proposal: Multilingual surface realization using universal dependency trees. In Proceedings of the 10th International Conference on Natural Language Generation, pages 120–123. find
  26. Simon Mille, Anya Belz, Bernd Bohnet, Yvette Graham, Emily Pitler, and Leo Wanner. 2018. The First Multilingual Surface Realisation Shared Task (SR’18): Overview and Evaluation Results. In Proceedings of the 1st Workshop on Multilingual Surface Realisation (MSR), 56th Annual Meeting of the Association for Computational Linguistics (ACL), pages 1–12, Melbourne, Australia. find our card
  27. Simon Mille, Anya Belz, Bernd Bohnet, Yvette Graham, and Leo Wanner. 2019. The second multilingual surface realisation shared task (sr’19): Overview and evaluation results. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019), pages 1–17. find
  28. Paul Molins and Guy Lapalme. 2015. Jsrealb: A bilingual text realizer for web programming. In Proceedings of the 15th European Workshop on Natural Language Generation (ENLG), pages 109–111. find
  29. K. Papineni, S. Roukos, T. Ward, and W. j. Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proc. 40th Annual Meeting on Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, July 7–12,. find our card
  30. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. ↗
  31. Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D Manning. 2019. Universal dependency parsing from scratch. arXiv preprint arXiv:1901.10457. ↗
  32. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. ↗ our card
  33. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8). find
  34. Aaron Smith, Bernd Bohnet, Miryam de Lhoneux, Joakim Nivre, Yan Shao, and Sara Stymne. 2018. 82 treebanks, 34 models: Universal dependency parsing with cross-treebank models. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Asso. for Comp. Linguistics. find our card
  35. Xiang Yu, Agnieszka Falenska, Marina Haid, Ngoc Thang Vu, and Jonas Kuhn. 2019. IMSurReal: IMS at the Surface Realization Shared Task 2019. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation, Hong Kong, China. find
  36. Xiang Yu, Simon Tannert, Ngoc Thang Vu, and Jonas Kuhn. 2020. Fast and accurate non-projective dependency tree linearization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1451–1462, Online, July. Association for Computational Linguistics. find our card
  37. Fangzhou Zhai, Vera Demberg, Pavel Shkadzko, Wei Shi, and Asad Sayeed. 2019. A hybrid model for globally coherent story generation. In Proceedings of the Second Workshop on Storytelling, pages 34–45. find
  38. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. find
  39. Liang Zhao, Jingjing Xu, Junyang Lin, Yichang Zhang, Hongxia Yang, and Xu Sun. 2020. Graph-based multi-hop reasoning for long text generation. arXiv preprint arXiv:2009.13282. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/mille-2020-third-multilingual-surface-realisation-shared/