Mova0.5 pre-release⌕
☰
home / docs / papers / Machine translation / An Error Analysis Framework for Shallow Surface Realization

An Error Analysis Framework for Shallow Surface Realization

What it is about

Scores such as BLEU say how good a system is but not which input features make it fail. The paper offers an error analysis for shallow surface realisation with two parts. The first part correlates syntactic complexity metrics of the input trees (tree shape, projectivity, entropy of dependency relations) with the performance scores, using Spearman correlation with a multiple-testing correction. The second part is tree error mining, adapted from grammar-based generation. It gives each input subtree a suspicion score based on how often that subtree co-occurs with failure and rarely with success. The framework was applied to 174 system runs from the SR'18 and SR'19 shared tasks. The results show which structures, such as non-projective ones, go with low scores, and that edge accuracy tracks the automatic metrics.

Ideas

For Mova

The paper fits tools/translate (measure.rs) and tools/roundtrip (prag frames). The error-mining score is the most useful part: it finds the construction that the translator fails on, which is the question open problem 2 asks about lexicon merges. The shuffle control is cheap and should be part of the gate.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00376

@article{shimorina2021error,
  title = {{An Error Analysis Framework for Shallow Surface Realization}},
  author = {Shimorina, Anastasia and Parmentier, Yannick and Gardent, Claire},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.1162/tacl\_a\_00376},
  url = {https://doi.org/10.1162/tacl_a_00376}
}

References

54 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. for sentence-level BLEU. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 362–367. DOI: https://doi.org/10.3115/v1/W14 -3346 ↗
  2. Ryan Cotterell, Christo Kirov, John Sylak-Glassman, Géraldine Walther, Ekaterina Vylomova, Patrick Xia, Manaal Faruqui, Sandra Kübler, David Yarowsky, Jason Eisner, and Mans Hulden. 2017. CoNLL-SIGMORPHON 2017 shared task: Universal morphological reinflection in 52 languages. In Proceedings of the CoNLL SIGMORPHON 2017 Shared Task: Universal Morphological Reinflection, pages 1–30, Vancouver. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/K17 -2001 ↗
  3. Hoa Trang Dang and Karolina Owczarzak. 2008. Overview of the TAC 2008 update summarization task. In Proceedings of the First Text Analysis Conference, TAC 2008, Gaithersburg, Maryland, USA, November 17-19, 2008. NIST. find
  4. Anja Belz, Mike White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The first surface realisation shared task: Overview and evaluation results. In Proceedings of the 13th European Workshop on Natural Language Generation, pages 217–226. Association for Computational Linguistics. find our card
  5. Ondřej Dušek and Filip Jurčı́ček. 2016. Sequence-to-sequence generation for spoken dialogue via deep syntax trees and strings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 45–51, Berlin, Germany. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/P16 -2008 ↗
  6. Aoife Cahill. 2009. Correlating human and automatic evaluation of a German surface realiser. In Proceedings of the ACL-IJCNLP 2009 Conference Short Papers, pages 97–100, Suntec, Singapore. Association for Computational Linguistics. DOI: https://doi.org/10.3115/1667583 .1667615, PMID: 19468038 ↗
  7. William Dyer. 2019. Weighted posets: Learning surface order from dependency trees. In Proceedings of the 18th International Workshop on Treebanks and Linguistic Theories (TLT, SyntaxFest 2019), pages 61–73, Paris, France. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/W19-7807 ↗
  8. Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019. Neural data-to-text generation: A comparison between pipeline and end-to-end architectures. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 552–562, Hong Kong, China. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/D19 -1052 ↗
  9. Henry Elder, Jennifer Foster, James Barry, and Alexander O’Connor. 2019. Designing a symbolic intermediate representation for neural surface realization. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 65–73, Minneapolis, Minnesota. Association for Computational Linguistics. DOI: Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques inflects but OK: Making sense of morphological generation errors. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 140–151, Hong Kong, China. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/K19 -1014 https://doi.org/10.18653/v1/W19 -2308 PMCID: PMC6981808 ↗ our card
  10. Henry Elder and Chris Hokamp. 2018. Generating high-quality surface realizations using data augmentation and factored sequence models. In Proceedings of the First Workshop on Multilingual Surface Realisation, pages 49–53, Melbourne, Australia. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/W18 -3606 ↗
  11. Kristina Gulordava and Paola Merlo. 2016. Multi-lingual dependency parsing evaluation: A large-scale analysis of word order properties using artificial data. Transactions of the Association for Computational Linguistics, 4:343–356. DOI: https://doi.org/10.1162/tacl a 00103 ↗
  12. Katja Filippova and Michael Strube. 2009. Tree linearization in English: Improving language model based approaches. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers, pages 225–228, Boulder, Colorado. Association for Computational Linguistics. DOI: https://doi.org/10.3115/1620853 .1620915 ↗ our card
  13. Eduard Hovy, Chin-Yew Lin, and Liang Zhou. 2005. find
  14. Evaluating DUC 2005 using basic elements. In Proceedings of the 5th Document Understanding Conference (DUC). find
  15. Sylvain Kahane, Chunxiao Yan, and Marie-Amélie Botalla. 2017. What are the limitations on the flux of syntactic dependencies? Evidence from UD treebanks. In Proceedings of the Fourth International Conference on Dependency Linguistics (Depling 2017), pages 73–82, Pisa, Italy. Linköping University Electronic Press. find
  16. Richard Futrell, Kyle Mahowald, and Edward Gibson. 2015. Quantifying word order freedom in dependency corpora. In Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), pages 91–100, Uppsala, Sweden. Uppsala University, Uppsala, Sweden. find
  17. Rahul Katragadda. 2009. On alternative automated content evaluation measures. In Proceedings of the Second Text Analysis Conference, Gaithersburg, Maryland, USA. find
  18. Claire Gardent and Shashi Narayan. 2012. Error mining on dependency trees. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 592–600, Jeju Island, Korea. Association for Computational Linguistics. find
  19. David King and Michael White. 2018. The OSU realizer for SRST ‘18: Neural sequence-to-sequence inflection and incremental locality-based linearization. In Proceedings of the First Workshop on Multilingual Surface Realisation, pages 39–48, Melbourne, Australia. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/W18-3605 ↗
  20. Jesús Giménez and Lluı́s Màrquez. 2009. On the robustness of syntactic and semantic features for automatic MT evaluation. In Proceedings of the Fourth Workshop on Statistical Machine Translation, pages 250–258, Athens, Greece. Association for Computational Linguistics. DOI: https://doi.org/10.3115/1626431 .1626479 ↗
  21. Ioannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. 2017. Neural AMR: Sequence-to-sequence models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 146–157, Vancouver, Kyle Gorman, Arya D. McCarthy, Ryan Cotterell, Ekaterina Vylomova, Miikka Silfverberg, and Magdalena Markowska. 2019. Weird Canada. Association for Computational Linguistics. DOI: https://doi.org/10 .18653/v1/P17-1014 In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 85–92, Hong Kong. Association for Computational Linguistics. DOI: https://doi.org/10.3115/1075218 .1075230 ↗ our card
  22. Ádám Kovács, Evelin Ács, Judit Ács, Andras Kornai, and Gábor Recski. 2019. BME-UW at SRST-2019: Surface realization with interpreted regular tree grammars. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019), pages 35–40, Hong Kong, China. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/D19 -6304 PMID: 30739462 PMCID: PMC7044605 ↗
  23. Dennis N. Mehay and Chris Brew. 2007. Bleuâtre: Flattening syntactic dependencies for MT evaluation. In Proceedings of the 11th International Conference on Theoretical and Methodological Issues in Machine Translation, pages 122–131. find
  24. Simon Mille, Anja Belz, Bernd Bohnet, Yvette Graham, Emily Pitler, and Leo Wanner. 2018. The first multilingual surface realisation shared task (SR’18): Overview and evaluation results. In Proceedings of the First Workshop on Multilingual Surface Realisation, pages 1–12. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/W18 -3601 ↗ our card
  25. Wei Li. 2015. Abstractive multi-document summarization with semantic information extraction. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1908–1913, Lisbon, Portugal. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/D15-1219 PMCID: PMC4665338 ↗
  26. Ding Liu and Daniel Gildea. 2005. Syntactic features for evaluation of machine translation. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 25–32, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
  27. Simon Mille, Anja Belz, Bernd Bohnet, Yvette Graham, and Leo Wanner. 2019. The second multilingual surface realisation shared task (SR’19): Overview and evaluation results. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019), pages 1–17, Hong Kong, China. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/D19 -6301 ↗
  28. Haitao Liu. 2008. Dependency distance as a metric of language comprehension difficulty. Journal of Cognitive Science, 9(2):159–191. find
  29. George A. Miller. 1956. The magical number seven plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63(2):81–97. DOI: https://doi.org/10.1037/h0043158 ↗
  30. Haitao Liu. 2010. Dependency direction as a means of word-order typology: A method based on dependency treebanks. Lingua, 120(6):1567–1578. DOI: https://doi.org/10.17791/jcs.2008.9.2.159 ↗
  31. Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019. Step-by-step: Separating planning from realization in neural data-to-text generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2267–2277, Minneapolis, Minnesota. Association for Computational Linguistics. find
  32. Chi-kiu Lo, Anand Karthik Tumuluru, and Dekai Wu. 2012. Fully automatic semantic MT evaluation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, pages 243–252, Montréal, Canada. Association for Computational Linguistics. find
  33. Robert Malouf. 2000. The order of prenominal adjectives in natural language generation. find
  34. Shashi Narayan and Claire Gardent. 2012. Error mining with suspicion trees: Seeing the forest for the trees. In Proceedings of COLING 2012, pages 2011–2026, Mumbai, India. find
  35. The COLING 2012 Organizing Committee. Joakim Nivre, Željko Agić, Lars Ahrenberg, Maria Jesus Aranzabe, Masayuki Asahara, Aitziber Atutxa, Miguel Ballesteros, John Bauer, Kepa Bengoetxea, Riyaz Ahmad Bhat, Eckhard Bick, Cristina Bosco, Gosse Bouma, Sam Bowman, Marie Candito, Gülşen Cebiroğlu Eryiğit, Giuseppe G. A. Celano, Fabricio Chalub, Jinho Choi, Çağrı Çöltekin, Miriam Connor, Elizabeth Davidson, Marie-Catherine de Marneffe, Valeria de Paiva, Arantza Diaz de Ilarraza, Kaja Dobrovoljc, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Marhaba Eli, Tomaž Erjavec, Richárd Farkas, Jennifer Foster, Cláudia Freitas, Katarı́na Gajdošová, Daniel Galbraith, Marcos Garcia, Filip Ginter, Iakes Goenaga, Koldo Gojenola, Memduh Gökırmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta Gonzáles Saavedra, Matias Grioni, Normunds Grūzı̄tis, Bruno Guillaume, Nizar Habash, Jan Hajič, Linh Hà Mỹ, Dag Haug, Barbora Hladká, Petter Hohle, Radu Ion, Elena Irimia, Anders Johannsen, Fredrik Jørgensen, Hüner Kaşikara, Hiroshi Kanayama, Jenna Kanerva, Natalia Kotsyba, Simon Krek, Veronika Laippala, Phuong Lê Hô` ng, Alessandro Lenci, Nikola Ljubešić, Olga Lyashevskaya, Teresa Lynn, Aibek Makazhanov, Christopher Manning, C̆atălina Mărănduc, David Mareček, Héctor Martı́nez Alonso, André Martins, Jan Mašek, Yuji Matsumoto, Ryan McDonald, Anna Missilä, Verginica Mititelu, Yusuke Miyao, Simonetta Montemagni, Amir More, Shunsuke Mori, Bohdan Moskalevskyi, Kadri Muischnek, Nina Mustafina, Kaili Müürisep, Luong Nguyê˜ n Thi., Huyê` n Nguyê˜ n Thi. Minh, Vitaly Nikolaev, Hanna Nurmi, Stina Ojala, Petya Osenova, Lilja Øvrelid, Elena Pascual, Marco Passarotti, Cenel-Augusto Perez, Guy Perrier, Slav Petrov, Jussi Piitulainen, Barbara Plank, Martin Popel, Lauma Pretkalnin , a, Prokopis Prokopidis, Tiina Puolakainen, Sampo Pyysalo, Alexandre Rademaker, Loganathan Ramasamy, Livy Real, Laura Rituma, Rudolf Rosa, Shadi Saleh, Manuela Sanguinetti, Baiba Saulı̄te, Sebastian Schuster, Djamé Seddah, Wolfgang Seeker, Mojgan Seraji, Lena Shakurova, Mo Shen, Dmitry Sichinava, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simkó, Mária Šimková, Kiril Simov, Aaron Smith, Alane Suhr, Umut Sulubacak, Zsolt Szántó, Dima Taji, Takaaki Tanaka, Reut Tsarfaty, Francis Tyers, Sumire Uematsu, Larraitz Uria, Gertjan van Noord, Viktor Varga, Veronika Vincze, Jonathan North Washington, Zdeněk Žabokrtský, Amir Zeldes, Daniel Zeman, and Hanzhi Zhu. 2017. Universal dependencies 2.0. LINDAT/CLARIN digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  36. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/D17 -1238 ↗
  37. Karolina Owczarzak. 2009. DEPEVAL(summ): Dependency-based evaluation for automatic summaries. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 190–198, Suntec, Singapore. Association for Computational Linguistics. DOI: https://doi.org/10.3115/1687878 .1687907 ↗
  38. Karolina Owczarzak, Josef van Genabith, and Andy Way. 2007. Dependency-based automatic evaluation for machine translation. In Proceedings of SSST, NAACL-HLT 2007 / AMTA Workshop on Syntax and Structure in Statistical Translation, pages 80–87, Rochester, New York. Association for Computational Linguistics. DOI: https://doi.org/10.3115/1626281 .1626292 ↗
  39. Ratish Puduppully, Yue Zhang, and Manish Shrivastava. 2016. Transition-based syntactic linearization with lookahead features. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 488–493, San Diego, California. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/N16 -1058 https://doi.org/10.18653/v1/D19 -1305 ↗ our card
  40. Yevgeniy Puzikov, Claire Gardent, Ido Dagan, and Iryna Gurevych. 2019. Revisiting the binary linearization technique for surface realization. In Proceedings of the 12th International Conference on Natural Language Generation, pages 268–278, Tokyo, Japan. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/W19 -8635 ↗
  41. Peng Qi, Timothy Dozat, Yuhao Zhang, and Christopher D. Manning. 2018. Universal Dependency parsing from scratch. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 160–170, Brussels, Belgium. Association for Computational Linguistics. find our card
  42. Linfeng Song, Yue Zhang, and Daniel Gildea. 2018. Neural transition-based syntactic linearization. In Proceedings of the 11th International Conference on Natural Language Generation, pages 431–440, Tilburg University, The Netherlands. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/W18-6553, PMCID: PMC6219880 ↗ our card
  43. Amanda Stent, Matthew Marge, and Mohit Singhai. 2005. Evaluating evaluation methods for generation in the presence of variation. In Computational Linguistics and Intelligent Text Processing, 6th International Conference, CICLing 2005, Mexico City, Mexico, February 13-19, 2005, Proceedings, pages 341–351. DOI: https://doi.org/10.1007/978 -3-540-30586-6 38 ↗
  44. Stephen Tratz and Eduard H. Hovy. 2009. BEwT-E for TAC 2009’s AESOP task. In Proceedings of the Second Text Analysis Conference. Gaithersburg, Maryland, USA. find
  45. J. R. Quinlan. 1986. Induction of decision trees. Machine Learning, 1(1):81–106. find
  46. Michael White and Rajakrishnan Rajkumar. 2012. Minimal dependency length in realization ranking. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 244–255, Jeju Island, Korea. Association for Computational Linguistics. find
  47. Ehud Reiter. 2018. A structured review of the validity of BLEU. Computational Linguistics, 44(3):393–401. DOI: https://doi.org/10.1162/coli a 00322 ↗ our card
  48. Allen Schmaltz, Alexander M. Rush, and Stuart Shieber. 2016. Word ordering without syntax. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2319–2324, Austin, Texas. Association for Computational Linguistics. DOI: https://doi.org/10.18653/v1/D16 -1255 ↗
  49. Hui Yu, Xiaofeng Wu, Jun Xie, Wenbin Jiang, Qun Liu, and Shouxun Lin. 2014. RED: A reference dependency based MT evaluation metric. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 2042–2051, Dublin, Ireland. Dublin City University and Association for Computational Linguistics. find our card
  50. Anastasia Shimorina and Claire Gardent. 2019. Surface realisation using full delex-icalisation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3086–3096, Hong Kong, China. Association for Computational Linguistics. DOI: Xiang Yu, Agnieszka Falenska, Marina Haid, Ngoc Thang Vu, and Jonas Kuhn. 2019. IMSurReal: IMS at the surface realization shared task 2019. In Proceedings of the 2nd Workshop on Multilingual Surface Realisation (MSR 2019), pages 50–58, Hong Kong, China. Association for Computational Linguistics. find
  51. Xiang Yu, Simon Tannert, Ngoc Thang Vu, and Jonas Kuhn. 2020. Fast and accurate non-projective dependency tree linearization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1451–1462, Online. Association for Computational Linguistics. https://doi.org/10.1515/pralin -2016-0007 ↗ our card
  52. Daniel Zeman. 2016. Universal annotation of Slavic verb forms. The Prague Bulletin of Mathematical Linguistics, 105:143–193. find
  53. Yue Zhang, Graeme Blackwood, and Stephen Clark. 2012. Syntax-based word ordering incorporating a large-scale language model. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 736–746, Avignon, France. Association for Computational Linguistics. find
  54. Yue Zhang. 2013. Partial-tree linearization: Generalized word ordering for text synthesis. In Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, pages 2232–2238. AAAI Press. DOI: Yue Zhang and Stephen Clark. 2015. Discriminative syntax-based word ordering for text generation. Computational Linguistics, 41(3):503–538. DOI: https://doi.org/10.1162/COLI a 00229 ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/shimorina-2021-error-analysis-framework-shallow-surface/