Optimizing Statistical Machine Translation for Text Simplification

What it is about

The paper adapts a syntax-based statistical translation system to simplify sentences. Its paraphrase rules come from large bilingual paraphrase tables (PPDB), which are easier to get than manually simplified text, and they are ranked with simplicity-specific features. The main contribution is the metric SARI. It compares the output with the input and with several references, and rewards words added, kept and deleted correctly. Keep is scored by precision and recall, addition by precision and recall, and deletion by precision only, because over-deleting harms readability more. Eight references were collected by crowdsourcing. The system tuned on SARI beats a phrase-based baseline and the BLEU-tuned variants in human judgement of simplicity, while BLEU tracks meaning and grammar. SARI ranks the three human references and five systems in the same order as human raters. Among the paper's findings, existing readability formulas do not correlate with human simplicity ratings.

Ideas

I1. SARI for the translator. Score each normal-mode output against the other verse translations as references, and against the source meta-world tokens, using add, keep and delete F-measures over content-token n-grams. This rewards real change and penalises copying, which exact match cannot do.
I2. Tune deltas on the metric. The paper tunes its system on the metric. Mova can accept a learned author override only if the SARI-like score rises on dev, never on test; the SVAMP test rule is untouched.
I3. Diagnostic on rule pairs. List the most frequent source-to-target pairs in the meta-lexicon, and mark each as a real paraphrase or a merge error. This gives a direct view of noisy merges such as "be" with "have", which is open problem 2.
I4. Asymmetric penalty. Penalise dropped meta-tokens more than added ones in the exact-mode residual, as the paper does for deletions.

For Mova

SARI is the most transferable metric in the set. It belongs in tools/translate/measure.rs next to exact match and chrF, and the Bible translations supply the multiple references the paper had to collect by hand. The SMT and PPDB machinery is not for Mova. Relevant to open problems 2 and 8.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00107

@article{xu2016optimizing,
  title = {{Optimizing Statistical Machine Translation for Text Simplification}},
  author = {Xu, Wei and Napoles, Courtney and Pavlick, Ellie and Chen, Quanze and Callison-Burch, Chris},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2016},
  doi = {10.1162/tacl\_a\_00107},
  url = {https://doi.org/10.1162/tacl_a_00107}
}

References

69 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Allen, D. (2009). A study of the role of relative clauses in the simplification of news texts for learners of English. System, 37(4):585–599. find
Amancio, M. A. and Specia, L. (2014). An analysis of crowdsourced text simplifications. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR). find
Angrosh, M., Nomoto, T., and Siddharthan, A. (2014). Lexico-syntactic text simplification and compression with typed dependencies. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics (EACL). find
Bannard, C. and Callison-Burch, C. (2005). Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL). find our card
Biran, O., Brody, S., and Elhadad, N. (2011). Putting it simply: A context-aware approach to lexical simplification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). find
Carroll, J., Minnen, G., Pearce, D., Canning, Y., Devlin, S., and Tait, J. (1999). Simplifying text for language-impaired readers. In Proceedings of the 14th Conference of the 9th European Conference for Computational Linguistics (EACL). find
Chan, T. P., Callison-Burch, C., and Van Durme, B. (2011). Reranking bilingually extracted paraphrases using monolingual distributional similarity. In Proceedings of the Workshop on Geometrical Models of Natural Language Semantics (MTTG). find
Chandrasekar, R., Doran, C., and Srinivas, B. (1996). Motivations and methods for text simplification. In Proceedings of the 16th Conference on Computational linguistics (COLING). find
Chen, B., Kuhn, R., and Larkin, S. (2012a). PORT: A precision-order-recall MT evaluation metric for tuning. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). find
Chen, D. L. and Dolan, W. B. (2011). Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics (ACL). find
Chen, H.-B., Huang, H.-H., Chen, H.-H., and Tan, C.-T. (2012b). A simplification-translation-restoration framework for cross-domain SMT applications. In Proceedings of the 24th International Conference on Computational Linguistics (COLING). find
Chiang, D. (2007). Hierarchical phrase-based translation. Computational Linguistics, 33(2):201– 228. find our card
Clarke, J. and Lapata, M. (2006). Models for sentence compression: A comparison across domains, training requirements and evaluation measures. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics (ACL-COLING). find
Coster, W. and Kauchak, D. (2011a). Learning to simplify sentences using Wikipedia. In Proceedings of the Workshop on Monolingual Text-To-Text Generation. find
Coster, W. and Kauchak, D. (2011b). Simple English Wikipedia: A new text simplification task. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). find
Dahlmeier, D. and Ng, H. T. (2012). Better evaluation for grammatical error correction. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). find
De Belder, J. and Moens, M.-F. (2010). Text simplification for children. In Proceedings of the SIGIR Workshop on Accessible Search Systems. find
Devlin, S., Tail, J., Canning, Y., Carroll, J., Minnen, G., and Pearce, D. (1999). The application of assistive technology in facilitating the comprehension of newspaper text by aphasic people. Assistive Technology on the Threshold of the New Millennium, page 160. find
Evans, R., Orasan, C., and Dornescu, I. (2014). An evaluation of syntactic simplification rules for people with autism. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR). find
Felice, M. and Briscoe, T. (2015). Towards a standard evaluation method for grammatical error detection and correction. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
Feng, L. (2008). Text simplification: A survey. The City University of New York, Technical Report. find
Filippova, K., Alfonseca, E., Colmenares, C. A., Kaiser, L., and Vinyals, O. (2015). Sentence compression by deletion with LSTMs. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
Filippova, K. and Strube, M. (2008). Dependency tree based sentence compression. In Proceedings of the Fifth International Natural Language Generation Conference (INLG). find
Galley, M., Brockett, C., Sordoni, A., Ji, Y., Auli, M., Quirk, C., Mitchell, M., Gao, J., and Dolan, B. (2015). deltaBLEU: A discriminative metric for generation tasks with intrinsically diverse targets. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL). find
Ganitkevitch, J., Cao, Y., Weese, J., Post, M., and Callison-Burch, C. (2012a). Joshua 4.0: Packing, PRO, and paraphrases. In Proceedings of the Seventh Workshop on Statistical Machine Translation (WMT). find
Ganitkevitch, J., Van Durme, B., and Callison-Burch, C. (2012b). Monolingual distributional similarity for text-to-text generation. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM). find
Ganitkevitch, J., Van Durme, B., and Callison-Burch, C. (2013). PPDB: The paraphrase database. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find our card
Gao, M., Xu, W., and Callison-Burch, C. (2015). Cost optimization in crowdsourcing translation. find
In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
Hopkins, M. and May, J. (2011). Tuning as ranking. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
Horn, C., Manduca, C., and Kauchak, D. (2014). Learning a lexical simplifier using Wikipedia. In Proceedings of the 52th Annual Meeting of the As-sociatioin for Computational Linguistics (ACL). find
Hwang, W., Hajishirzi, H., Ostendorf, M., and Wu, W. (2015). Aligning sentences from Standard Wikipedia to Simple Wikipedia. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
Inui, K., Fujita, A., Takahashi, T., Iida, R., and Iwakura, T. (2003). Text simplification for reading assistance: A project note. In Proceedings of the 2nd International Workshop on Paraphrasing (IWP). find
Kaji, N., Kawahara, D., Kurohash, S., and Sato, S. (2002). Verb paraphrase based on case frame alignment. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics (ACL). find
Kauchak, D. (2013). Improving text simplification language modeling using unsimplified text data. In Proceedings of the 2013 Conference of the Association for Computational Linguistics (ACL). find
Kincaid, J. P., Fishburne Jr, R. P., Rogers, R. L., and Chissom, B. S. (1975). Derivation of new readability formulas (automated readability index, fog count and Flesch reading ease formula) for Navy enlisted personnel. Technical report, Defence Technical Information Center (DTIC) Document. find
Knight, K. and Marcu, D. (2002). Summarization beyond sentence extraction: A probabilistic approach to sentence compression. Artificial Intelligence. find
Liu, C., Dahlmeier, D., and Ng, H. T. (2010). TESLA: Translation evaluation of sentences with linear-programming-based analysis. In Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and Metrics (MATR). find
Miwa, M., Saetre, R., Miyao, Y., and Tsujii, J. (2010). Entity-focused sentence simplification for relation extraction. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING). find
Napoles, C., Sakaguchi, K., Post, M., and Tetreault, J. (2015). Ground truth for grammatical error correction metrics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL). find
Narayan, S. and Gardent, C. (2014). Hybrid simplification using deep semantics and machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL). find
Och, F. J. (2003). Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics (ACL). find
Pavlick, E., Bos, J., Nissim, M., Beller, C., Durme, B. V., and Callison-Burch, C. (2015). Adding semantics to data-driven paraphrasing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL). find
Pavlick, E. and Callison-Burch, C. (2016). Simple PPDB: A paraphrase database for simplification. In The 54th Annual Meeting of the Association for Computational Linguistics (ACL). find
Pellow, D. and Eskenazi, M. (2014a). An open corpus of everyday documents for simplification tasks. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR). find
Pellow, D. and Eskenazi, M. (2014b). Tracking human process using crowd collaboration to enrich data. In Proceedings of Second AAAI Conference on Human Computation and Crowdsourcing (HCOMP). find
Petersen, S. E. and Ostendorf, M. (2007). Text simplification for language learners: A corpus analysis. In Proceedings of Workshop on Speech and Language Technology for (SLaTE). find
Post, M., Ganitkevitch, J., Orland, L., Weese, J., Cao, Y., and Callison-Burch, C. (2013). Joshua 5.0: Sparser, better, faster, server. In Proceedings of the Eighth Workshop on Statistical Machine Translation (WMT). find
Rello, L., Baeza-Yates, R. A., and Saggion, H. (2013). The impact of lexical simplification by verbal paraphrases for people with and without dyslexia. In Proceedings of the 14th International Conference on Intelligent Text Processing and Computational Linguistics (CICLing). find
Rush, A. M., Chopra, S., and Weston, J. (2015). A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
Siddharthan, A. (2006). Syntactic simplification and text cohesion. Research on Language and Computation, 4(1):77–109. find
Siddharthan, A. (2014). A survey of research on text simplification. Special issue of International Journal of Applied Linguistics, 165(2). find
Siddharthan, A. and Angrosh, M. (2014). Hybrid text simplification using synchronous dependency grammars with hand-written and automatically harvested rules. In Proceedings of the 25th International Conference on Computational Linguistics (COLING). find our card
Siddharthan, A. and Katsos, N. (2010). Reformulating discourse connectives for non-expert readers. In Proceedings of the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). find
Siddharthan, A., Nenkova, A., and McKeown, K. (2004). Syntactic simplification for improving content selection in multi-document summarization. In Proceedings of the 20th International Conference on Computational Linguistics (COL-ING). find
Specia, L., Jauhar, S. K., and Mihalcea, R. (2012). SemEval-2012 task 1: English lexical simplification. In Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval). find
Štajner, S., Béchara, H., and Saggion, H. (2015). A deeper exploration of the standard PB-SMT approach to text simplification and its evaluation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL). find
Štajner, S., Mitkov, R., and Saggion, H. (2014). One step closer to automatic evaluation of text simplification systems. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR). find
Unsupervised extraction of lexical simplifications from Wikipedia. In Proceedings of the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). find
Sun, H. and Zhou, M. (2012). Joint learning of a dual SMT system for paraphrase generation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). find
Zhu, Z., Bernhard, D., and Gurevych, I. (2010). A monolingual tree-based translation model for sentence simplification. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING). find our card
Vickrey, D. and Koller, D. (2008). Sentence simplication for semantic role labeling. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT). find
Watanabe, W. M., Junior, A. C., Uzêda, V. R., Fortes, R. P. d. M., Pardo, T. A. S., and Aluı́sio, S. M. (2009). Facilita: Reading assistance for low-literacy readers. In Proceedings of the 27th ACM International Conference on Design of Communication (SIGDOC). find
Weese, J., Ganitkevitch, J., Callison-Burch, C., Post, M., and Lopez, A. (2011). Joshua 3.0: Syntax-based machine translation with the thrax grammar extractor. In Proceedings of the Sixth Workshop on Statistical Machine Translation (WMT). find
Woodsend, K. and Lapata, M. (2011). Learning to simplify sentences with quasi-synchronous grammar and integer programming. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
Wubben, S., van den Bosch, A., and Krahmer, E. (2012). Sentence simplification by monolingual machine translation. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL). find our card
Xu, W., Callison-Burch, C., and Napoles, C. (2015). Problems in current text simplification research: New data can help. Transactions of the Association for Computational Linguistics (TACL), 3:283–297. find
Xu, W., Ritter, A., Dolan, B., Grishman, R., and Cherry, C. (2012). Paraphrasing for style. In Proceedings of the 24th International Conference on Computational Linguistics (COLING). find our card
Yatskar, M., Pang, B., Danescu-Niculescu-Mizil, C., and Lee, L. (2010). For the sake of simplicity: Zollmann, A. and Venugopal, A. (2006). Syntax augmented machine translation via chart parsing. In Proceedings of the Workshop on Statistical Machine Translation (WMT). find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/xu-2016-optimizing-statistical-machine-translation-text/