Semantic Structural Evaluation for Text Simplification

What it is about

The paper asks how to score a simplified sentence when reference simplifications are scarce and not unique. It proposes SAMSA, a measure built on UCCA semantic structure: the source is split into its scenes (predicate-argument structures), and an output scores well when each scene gets its own sentence and no scene's relations are broken. The score needs only the source side annotated, by a parser or by hand, and the output is matched to it by words, so no reference is used. Five annotators rated six simplification systems for grammaticality, meaning preservation and structural simplicity. The SAMSA system ranking agrees with the average human score at a Spearman of about 0.58 (semi-automatic version), while the reference-based SARI and BLEU correlate weakly or negatively. On the QATS shared task data, SAMSA reaches a sentence-level Pearson of 0.32.

Ideas

• I1. A count-based structure check for Mova's own output: take the frames from prag for each input sentence and count, for each produced sentence, how many frames it keeps whole. No parser is needed on the output side, which is the paper's own argument for source-side annotation.
• I2. Report a structure score next to the lexical ones that the translator already reports in normal mode (exact match, chrF). Measure on the same verses whether the two disagree; the paper's main finding is that lexical measures can be blind to structure.
• I3. Use the scene-splitting idea as a diagnostic for the reader: one verse read from two translations should keep the same number of frames. A mismatch marks a candidate attachment error, the first open problem of the translator.

For Mova

The SAMSA scene is the sentence-level analogue of a frame-preservation test for tools/translate, with the frames of prag and tools/roundtrip in the role of UCCA scenes. The numbers are for English simplification and do not transfer; only the count and the source-side design carry over. Any such gate needs a negative control: shuffling frames inside a sentence should drop the score, and if it does not, the gate is not measuring structure.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1063

@inproceedings{sulem2018semantic,
  title = {{Semantic Structural Evaluation for Text Simplification}},
  author = {Sulem, Elior and Abend, Omri and Rappoport, Ari},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1063},
  url = {https://doi.org/10.18653/v1/n18-1063}
}

References

52 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 384. http://aclweb.org/anthology/P/ P06/P06-1048.pdf. ↗
2. Omri Abend and Ari Rappoport. 2013. Universal Conceptual Cognitive Annotation (UCCA). In Proc. of ACL-13. pages 228–238. http://aclweb.org/anthology/P/P13/P13-1023.pdf. ↗ our card
3. Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit. Psychological bulletin 70(4):213. find
4. Omri Abend and Ari Rappoport. 2017. The state of the art in semantic representation. In Proc. of ACL’17. pages 77–89. http://aclweb.org/ anthology/P/P17/P17-1008.pdf. ↗
5. William Coster and David Kauchak. 2011. Simple English Wikipedia: A new text simplification task. In Proc. of ACL’11. pages 665– 669. http://aclweb.org/anthology/P/ P11/P11-2117.pdf. ↗
6. Omri Abend, Shai Yerushalmi, and Ari Rappoport. 2017. UCCAApp: Web-application for syntactic and semantic phrase-based annotation. In Proc. of ACL’17, System Demonstrations. pages 109–114. http://www.aclweb.org/ anthology/P17-4019. ↗
7. Michael Denkowski and Alon Lavie. 2011. Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems. In Proc. of WMT’11. pages 85–91. http://aclweb.org/anthology/W/W11/W11-2107.pdf. ↗
8. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract Meaning Rrepresen-tation for sembanking. Proc. of Linguistic Annotation Workshop and Interoperability with Discourse pages 178–186. http://aclweb.org/ anthology/W/W13/W13-2322.pdf. ↗ our card
9. Robert M.W. Dixon. 2010a. Basic Linguistic Theory: Grammatical Topics, volume 2. Oxford University Press. find
10. Robert M.W. Dixon. 2010b. Basic Linguistic Theory: Methodology, volume 1. Oxford University Press. find
11. Robert M.W. Dixon. 2012. Basic Linguistic Theory: Further Grammatical Topics, volume 3. Oxford University Press. find
12. Regina Barzilay and Noemie Elhadad. 2003. Sentence alignment for monolingual comparable corpora. In Proc. of EMNLP’03. pages 25– 32. http://www.aclweb.org/anthology/ W/W03/W03-1004.pdf. ↗
13. Goran Glavas and Sanja Štajner. 2013. Event-centered simplification of news stories. In Proc. of the Student Research Workshop associated with RANLP 2013. pages 71–78. http://aclweb.org/ anthology/R13-2011. ↗
14. Or Biran, Samuel Brody, and Noémie Elhadad. 2011. Putting it simply: a context-aware approach to lexical simplification. In Proc. of ACL’11. pages 465– 501. http://aclweb.org/anthology/P/ P11/P11-2087.pdf. ↗
15. Goran Glavaš and Sanja Štajner. 2015. Simplifying lexical simplification: Do we need simplified corpora? In Proc. of ACL’15 (Short papers). pages 63– 68. http://www.aclweb.org/anthology/ P15-2011. ↗
16. Alexandra Birch, Omri Abend, Ondřej Bojar, and Barry Haddow. 2016. HUME: Human UCCA-based evaluation of machine translation. In Proc. of EMNLP’16. pages 1264–1274. http://aclweb.org/anthology/D/D16/ D16-1134.pdf. ↗ our card
17. Daniel Hershcovich, Omri Abend, and Ari Rappoport. 2017. A transition-based directed acyclic graph parser for UCCA. In Proc. of ACL’17. pages 1127– 1138. http://aclweb.org/anthology/P/ P17/P17-1104.pdf. ↗
18. Yvonne Margaret Canning. 2002. Syntactic simplification of text. Ph.D. thesis, University of Sunderland, UK. find
19. Colby Horn, Cathryn Manduca, and David Kauchak. 2014. Learning a lexical simplifier using Wikipedia. In Proc. of ACL’14 (Short papers). pages 458– 463. http://aclweb.org/anthology/P/ P14/P14-2075.pdf. ↗
20. Raman Chandrasekar, Christine Doran, and Bangalore Srinivas. 1996. Motivations and methods for sentence simplification. In Proc. of COL-ING’96. pages 1041–1044. http://aclweb.org/anthology/C/C96/C96-2183.pdf. ↗
21. William Hwang, Hannaneh Hajishirzi, Mari Ostendorf, and Wei Wu. 2015. Aligning sentences from Standard Wikipedia to Simple Wikipedia. In Proc. of NAACL’15. pages 211–217. http://aclweb.org/anthology/N/N15/N15-1022.pdf. ↗
22. Leshem Choshen and Omri Abend. 2018. Reference-less measure of faithfulness for grammatical error correction. In Proc. of NAACL’18 (Short papers). To appear. find
23. Hans Kamp. 1981. A theory of truth and semantic representation. In J.A.G. Groenendijk, T.M.V. Jassen, B.J. Stokhof, and M.J.B/ Stokhof, editors, Formal methods in the study of language. Mathematisch Centrum. Number pt.1 in Mathematical Centre tracts. find
24. James Clarke and Mirella Lapata. 2006. Models for sentence compression: A comparison across domains, training requirements and evaluation measures. In Proc. of ACL-COLING’06. pages 377– Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. of ACL’02. pages 311–318. http://aclweb.org/anthology/P/P02/P02-1040.pdf. ↗ our card
25. J. Peter Kincaid, Robert P. Fishburne Jr., Richard L. Rogers, and Brad S. Chissom. 1975. Derivation of new readability formulas (automated readability index, fog count and Flesch reading ease formula) for Navy enlisted personnel. Technical report, DTIC Document. find
26. Luz Rello, Ricardo Baeza-Yates, Stefan Bott, and Horacio Saggion. 2013. Simplify or help?: text simplification strategies for people with dyslexia. In Proc. of the 10th International Cross-Disciplinary Conference on Web Accesibility. pages 15:1 – 15:10. find
27. Ronald W. Langacker. 2008. Cognitive Grammar: A Basic Introduction. Oxford University Press, USA. find
28. Chi-kiu Lo, Meriem Beloucif, Markus Saers, and Dekai Wu. 2014. XMEANT: Better semantic mt evaluation without reference translations. In Proc. of ACL’14 (Short Papers). pages 765– 771. http://aclweb.org/anthology/P/ P14/P14-2124.pdf. ↗
29. Miguel Rios, Wilker Aziz, and Lucia Specia. 2011. TINE: A metric to assess MT adequacy. In Proc. of WMT’11. pages 116–122. http://aclweb.org/anthology/W/W11/W11-2112.pdf. ↗
30. Advaith Siddharthan. 2002. An architecture for a text simplification system. In Proc. of LEC. pages 64– 71. find
31. Chi-kiu Lo, Anand Karthik Tumuluru, and Dekai Wu. 2012. Fully automatic semantic MT evaluation. In Proc. of WMT’12. pages 243– 252. http://aclweb.org/anthology/W/ W12/W12-3129.pdf. ↗
32. Advaith Siddharthan and M. A. Angrosh. 2014. Hybrid text simplification using synchronous dependency grammars with hand-written and automatically harvested rules. In Proc. of EACL’14. pages 722– 731. http://aclweb.org/anthology/E/ E14/E14-1076.pdf. ↗ our card
33. Edward Loper and Steven Bird. 2002. NLTK: the natural language toolkit. In Proc. of EMNLP’02. pages 63–70. http://www.aclweb.org/ anthology/W/W02/W02-0109.pdf. ↗
34. Matthew Snover, Nitin Madnani, Bonnie J. Dorr, and Richard Schwartz. 2009. Fluency, adequacy, or HTER? Exploring different human judgments with a tunable MT metric. In Proc. of WMT’09. pages 85– 91. http://www.aclweb.org/anthology/ W09-0441. ↗
35. Kshitij Mishra, Ankush Soni, Rahul Sharma, and Dipti Misra Sharma. 2014. Exploring the effects of sentence simplification on Hindi to English Machine Translation systems. In Proc. of the Workshop on Automatic Text Simplification: Methods and Applications in the Multilingual Society. pages 21– 29. http://www.aclweb.org/anthology/ W14-5603. ↗
36. Elior Sulem, Omri Abend, and Ari Rappoport. 2015. Conceptual annotations preserve structure across translations. In Proc. of 1st Workshop on Semantics-Driven Statistical Machine Translation (S2Mt 2015). pages 11–22. http://aclweb.org/anthology/W/W15/W15-3502.pdf. ↗
37. Shashi Narayan and Claire Gardent. 2014. Hybrid simplification using deep semantics and machine translation. In Proc. of ACL14. pages 435– 445. http://aclweb.org/anthology/P/ P14/P14-1041.pdf. ↗
38. Md Arafat Sultan, Steven Bethard, and Tamara Sumner. 2014. Back to basics for monolingual alignment: Exploiting word similarity and contextual evidence. TACL 2:219–230. http://aclweb.org/anthology/Q/Q14/Q14-1018.pdf. ↗
39. Shashi Narayan and Claire Gardent. 2016. Unsupervised sentence simplification using deep semantics. In Proc. of INLG’16. pages 111– 120. http://aclweb.org/anthology/W/ W16/W16-6620.pdf. ↗ our card
40. Hong Sun and Ming Zhou. 2012. Joint learning of a dual SMT system for paraphrase generation. In Proc. of ACL’12. pages 38– 42. http://aclweb.org/anthology/P/ P12/P12-2008.pdf. ↗
41. Shashi Narayan, Claire Gardent, Shay B. Cohen, and Anastasia Shimorina. 2017. Split and rephrase. In Proc. of EMNLP’17. pages 617– 627. http://aclweb.org/anthology/D/ D17/D17-1065.pdf. ↗
42. Kristina Toutanova, Chris Brockett, Ke M. Tran, and Saleema Amershi. 2016. A dataset and evaluation metrics for abstractive compression of sentences and short paragraphs. In Proc. of EMNLP’16. pages 340–350. http://aclweb.org/anthology/D/D16/D16-1033.pdf. ↗
43. Christina Niklaus, Bernahard Bermeitinger, Siegfried Handschuh, and André Freitas. 2016. A sentence simplification system for improving relation extraction. In Proc. of COLING’16. http://aclweb.org/anthology/C/C16/C16-2036.pdf. ↗
44. Sanja Štajner, Hannah Bechara, and Horacio Saggion. 2015. A deeper exploration of the standard PB-SMT approach to text simplification and its evaluation. In Proc. of ACL’15, Short papers. pages 823– 828. http://aclweb.org/anthology/P/ P15/P15-2135.pdf. ↗
45. Sergiu Nisioi and Fabrice Nauze. 2016. An ensemble method for quality assessment of text simplification. In Workshop on Quality Assessment for Text Simplification (QATS). find
46. Sanja Štajner, Ruslan Mitkov, and Horacio Saggion. 2014. One step closer to automatic evaluation of text simplification systems. Proc. of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations pages 1–10. http:// www.aclweb.org/anthology/W14-1201. ↗
47. Sanja Štajner, Maja Popović, Horacio Saggion, Lucia Specia, and Mark Fishel. 2016. Shared task on quality assessment for text simplification. In Workshop on Quality Assessment for Text Simplification (QATS). find
48. Kristian Woodsend and Mirella Lapata. 2011. Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proc. of EMNLP’11. pages 409–420. http://aclweb.org/anthology/D/D11/D11-1038.pdf. ↗ our card
49. Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2012. Sentence simplification by monolingual machine translation. In Proc. of ACL’12. pages 1015–1024. http://aclweb.org/ anthology/P/P12/P12-1107.pdf. ↗ our card
50. Wei Xu, Chris Callison-Burch, and Courtney Napoles. 2015. Problems in current text simplification research: new data can help. TACL 3:283– 297. http://aclweb.org/anthology/Q/ Q15/Q15-1021.pdf. ↗
51. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. TACL 4:401–415. http://aclweb.org/ anthology/Q/Q16/Q16-1029.pdf. ↗ our card
52. Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010. A monolingual tree-based translation model for sentence simplification. In Proc. of COL-ING’10. pages 1353–1361. http://aclweb.org/anthology/C/C10/C10-1152.pdf. ↗ our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/sulem-2018-semantic-structural-evaluation-text-simplification/