Shape of Synth to Come: Why We Should Use Synthetic Data for English Surface Realization

What it is about

Surface realization turns a UD-like tree of lemmas back into a sentence with word order and inflection. The system is a bidirectional LSTM encoder-decoder trained on the English EWT data of the 2018 shared task (12,375 training sentences, 1,978 dev, 2,062 test). Three changes (several random linearisations per tree, scoping brackets, a restricted beam) raise dev BLEU-4 from 57.3 to 72.2. The main claim concerns synthetic data: unlabelled sentences from Wikitext and CNN are parsed, their word order shuffled and forms removed, and about 4.5 million such sentences are added. Test BLEU-4 rises from 72.3 to 80.1, above the earlier best of 72.7; the gain is largest for long sentences. The authors argue that the 2019 ban on synthetic data should be lifted.

Ideas

• I1. en/src/lin.rs: make training pairs from unlabelled text, as the paper does. Parse plain sentences with en/src/parse.rs, drop the word order and the surface forms, keep the lemmas, and count the word-order tables on the result. Use the same 5 to 50 token and 80% vocabulary filters. Measure the exact-match rate on the EWT dev set, run twice.
• I2. Several random orders per tree: for each training tree, count the orders produced by a fixed set of seeds, rather than one order, so that the count tables see the variety of child orders. Compare the dev exact-match rate with one order against three and five, and keep the seed list in the repository.
• I3. Error categories from the paper's analysis: exact match, punctuation-only, inflection-only, non-breaking (valid but different order) and breaking (wrong meaning). Add these categories to tools/roundtrip, so that each run reports the counts, not just one score.
• I4. Report accuracy by sentence length bucket (5–10, 10–20, 20+ tokens) on en parse-eval and on the linearizer, since the paper finds the gain concentrated in long sentences.

For Mova

Word order from a tree is en/src/lin.rs, and the tree source is en/src/parse.rs. The chapters that describe the order rules are 20. Syntax: the Universal Dependencies frame, 21. The simple clause and 29. Ellipsis, word order and information structure. The error breakdown belongs in tools/roundtrip. The corpora used in the paper are licensed in ways Mova cannot carry (Wikitext is CC BY-SA), so only open text Mova may use should feed I1. Measures: dev exact match, the error breakdown and the length buckets, all run twice for determinism.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.665 · arXiv 2005.02693

@inproceedings{elder2020shape,
  title = {{Shape of Synth to Come: Why We Should Use Synthetic Data for English Surface Realization}},
  author = {Elder, Henry and Burke, Robert and O'Connor, Alexander and Foster, Jennifer},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.665},
  eprint = {2005.02693},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.665}
}

References

1 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. ## 1. Non-breaking Anja Belz, Michael White, Dominic Espinosa, Eric Kow, Deirdre Hogan, and Amanda Stent. 2011. The First Surface Realisation Shared Task: Overview and Evaluation Results. In Proceedings of the European Workshop on Natural Language Generation, December, pages 217–226. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/elder-2020-shape-synth-come-why-we/