The First Surface Realisation Shared Task: Overview and Evaluation Results

What it is about

The paper reports the first surface realisation shared task. Teams received a common input (a shallow track and a deep track, both derived from Penn Treebank parses) and produced English sentences. Five teams submitted six systems, and the corpus sentences were scored as a human topline. Systems were judged automatically with BLEU and related metrics, and by human raters on clarity, readability and meaning similarity. The statistical dependency realisers performed best. The two systems that used a hand-written grammar or a reversible grammar did poorly, and the authors trace much of this to converting the common input into the form their realisers expected, which the report treats as a large obstacle. The paper also notes that BLEU scores from earlier, non-shared work could not be compared, because each used a different input.

Ideas

• I1. Report input conversion coverage separately from realiser quality. The report shows that a weak result can come from the conversion of the input rather than from the realiser. The translator's residual per layer already separates these; keep that split in every table.
• I2. Use one common input for all encoder variants, so that comparisons between author deltas run on the same meta-world and cannot differ by conversion.
• I3. Add the reference text as a row in every table, as the human topline does, so the gap to the ceiling is always visible.
• I4. Split human-style judgements into clarity, readability and meaning similarity. For Mova's normal mode, report exact match, chrF and a meaning-only score on the same rows, so that a fluent but wrong output shows as a meaning drop.

For Mova

The ideas fit tools/translate/measure.rs and the layout of its README tables. The shared-task design, a common-ground input for comparing systems, is the same argument behind the translator's meta-world, and the paper's warning about conversion cost is a reason to measure the reader separately.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{belz2011first,
  title = {{The First Surface Realisation Shared Task: Overview and Evaluation Results}},
  author = {Belz, Anja and White, Michael and Espinosa, Dominic and Kow, Eric and Hogan, Deirdre and Stent, Amanda},
  booktitle = {Proceedings of the 13th European Workshop on Natural Language Generation},
  year = {2011},
  url = {https://aclanthology.org/W11-2832/}
}

References

17 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proc. of the ACL-05 Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. find our card
2. Anja Belz and Eric Kow. 2011. Discrete vs. continuous rating scales for language evaluation in nlp. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL-HLT’11). find
3. A. Belz, M. White, J. van Genabith, D. Hogan, and A. Stent. 2010. Finding common ground: Towards a Clarity Mean Readability Mean Homogeneous subsets Corpus 88.06 A STUMABA - D 64.91 B OSU 22.83 C Homogeneous subsets A B C Corpus 83.28 60.30 22.71 STUMABA - D OSU Table 9: SR-Task, Deep Track: Results for Clarity, Readability and Meaning Similarity evaluations, in terms of means and homogeneous subsets determined by post-hoc Tukey’s HSD (sig. < 0.05). Surface Realisation shared task. In Proc. of INLG’10, pages 267–271. find
4. Bernd Bohnet, Leo Wanner, Simon Mille, and Alicia Burga. 2010. Broad coverage multilingual deep sentence generation with a stochastic multi-level realizer. In Proceedings of the 23rd International Conference on Computational Linguistics, Beijing, China. find our card
5. A. Cahill and J. van Genabith. 2006. Robust PCFG-based generation using automatically acquired LFG approximations. In Proc. ACL’06, pages 1033–44. find
6. Charles Callaway. 2003. Evaluating coverage for large symbolic NLG grammars. In Proceedings of the 18th International Joint Conference on Artificial Intelligence (IJCAI 2003), pages 811–817. find
7. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic Metric for Reliable Optimization and Evaluation of Machine Translation Systems. In Proceedings of the EMNLP 2011 Workshop on Statistical Machine Translation. find
8. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to-dependency conversion for english. In Joakim Nivre, Heiki-Jaan Kaalep, Kadri Muischnek, and Mare Koit, editors, Proceedings of NODALIDA 2007, pages 105–112, Tartu, Estonia. find our card
9. I. Langkilde. 2002. An empirical verification of coverage and correctness for a general-purpose sentence generator. In Proc. 2nd International Natural Language Generation Conference (INLG ’02). find
10. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1995. The PENN treebank: Annotating predicate argument structure. Distributed on the PENN Treebank Release 2 CD-ROM, Linguistic Data Consortium. find our card
11. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The nombank project: An interim report. In NAACL/HLT Workshop Frontiers in Corpus Annotation. find
12. Hiroko Nakanishi, Yusuke Miyao, and Jun’ichi Tsujii. 2005. Probabilistic models for disambiguation of an hpsg-based chart generator. In Proceedings of the 9th International Workshop on Parsing Technology (Parsing’05), pages 93–102. Association for Computational Linguistics. Meaning Similarity Mean Homogeneous subsets Corpus 99.86 A STUMABA - D 72.54 B OSU 32.44 C Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: A corpus annotated with semantic roles. In Computational Linguistics Journal, pages 71–105. find
13. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA, July. Association for Computational Linguistics. find our card
14. Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proc. of the Association for Machine Translation in the Americas (AMTA-06). find our card
15. Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluı́s Màrquez, and Joakim Nivre. 2008. The conll-2008 shared task on joint parsing of syntactic and semantic dependencies. In Proceedings of the Twelfth Conference on Computational Natural Language Learning, Manchester, UK. find our card
16. Michael White and Rajakrishnan Rajkumar. 2009. Perceptron reranking for CCG realisation. In Proceedings of the 2009 Conference on Empririal Methods in Natural Language Processing (EMNLP’09), pages 410– 419. find
17. H. Zhong and A. Stent. 2005. Building surface realizers automatically from corpora. In A. Belz and S. Varges, editors, Proceedings of UCNLG’05, pages 49–54. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/belz-2011-first-surface-realisation-shared-task/