The Types and Distributions of Errors in a Wide Coverage Surface Realizer Evaluation

What it is about

A study of why a wide-coverage realiser makes errors when it is scored on a treebank. The realiser is FUF/SURGE, scored on Penn Treebank section 23 after a transformer turns trees into its input. The newer version covers 98.5% of sentences with 69.3% exact matches, against 82.8% and 57.5% for the statistical HALogen, and has a higher simple string accuracy. The paper then asks where the remaining errors come from. The author generated 4,240 sentences from sections 20 to 22 and hand-checked 629 that had large accuracy penalties, classifying each error twice: first its source (corpus annotation, transformer, or grammar), then the grammar rule behind it. Most failing sentences trace back to the transformer, not the grammar; about 29% trace to syntax. Among syntactic constructions, mixed noun-phrase modifiers are the most frequent, while topicalisation has the highest average penalty per instance (about 73 edit-distance points). Rare constructions such as it-clefts are found with regular expressions, and the corpus tags for them are often wrong, so a treebank is a noisy test of rare phenomena.

Ideas

• I1. Separate error sources in Mova's roundtrip report: corpus or annotation error, reader and transformer error, and encoder error, as the paper does. Estimate the corpus floor first by hand-checking a sample, so the topline is known before the rules are blamed.
• I2. Find rare constructions by pattern, and keep them as a probe set. The paper finds it-clefts and wh-clefts with regular expressions over text. In KJV and WEB, run the same kind of pattern as a held-out probe, not a training source, and count the correct and incorrect outputs.
• I3. Report the average penalty per error class next to the counts. Topicalisation was rare but costly; the translator's report should say which class costs most per verse, not only how many verses fail.
• I4. Coverage beside exact match. The paper's gain came mostly from coverage. Give the translator's normal-mode report the share of verses translated without a fallback next to the exact-match rate.

For Mova

The measures belong in tools/roundtrip (divergence classes) and tools/translate/src/measure.rs. The ideas touch the residual-by-class measure (open problem 2) and the reader's noise (open problem 1). The paper's counts are for English newspaper text and do not transfer; its method does.

Topics: Treebanks · Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{callaway2005types,
  title = {{The Types and Distributions of Errors in a Wide Coverage Surface Realizer Evaluation}},
  author = {Callaway, Charles},
  booktitle = {Proceedings of the Tenth European Workshop on Natural Language Generation (ENLG-05)},
  year = {2005},
  url = {https://aclanthology.org/W05-1619/}
}

References

18 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. S. Bangalore and O. Rambow. Exploiting a probabilistic hierarchical model for generation. In COLING–2000: Proceedings of the 18th International Conference on Computational Linguistics, Saarbruecken, Germany, 2000. find
2. John A. Bateman. KPML: The KOMET-penman (multilingual) development environment. Technical Report Release 0.8, Institut für Integrierte Publikations- und Informationssysteme (IPSI), GMD, Darmstadt, 1995. find
3. Charles B. Callaway. Evaluating coverage for large symbolic NLG grammars. In Proceedings of the Eighteenth International Joint Conference on Artificial Intelligence, pages 811–817, Acapulco, Mexico, August 2003. find
4. Charles Callaway. Wide coverage symbolic surface realization. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 125–128, Barcelona, Spain, July 2004. find
5. M. Dickinson and D. Meurers. Detecting errors in part-of-speech annotation. In Proceedings of the 10th Conference of the European Chapter of the ACL, Budapest, Hungary, April 2003. find
6. George Doddington. Automatic evaluation of machine translation quality using n-gram co-occurrence statistics. In Proceedings of the 2002 Conference on Human Language Technology, San Diego, CA, March 2002. find
7. Christine Doran. Incorporating Punctuation into the Sentence Grammar: A Lexicalized Tree Adjoining Grammar Perspective. PhD thesis, University of Pennsylvania, Philadelphia, PA, 1998. find
8. Michael Elhadad. FUF: The universal unifier user manual version 5.0. Technical Report CUCS-038-91, Dept. of Computer Science, Columbia University, 1991. Total SSA Penalty 601 8.99% 47 0.70% 1080 16.15% 1197 17.90% 560 8.37% 307 4.59% 542 8.11% 1231 18.41% 317 4.74% 805 12.04% 6687 100.0% Avg. Penalty 50.08 15.67 41.54 18.70 56.00 34.11 41.69 42.45 19.81 73.18 34.65 find
9. Georgia Green. Pragmatic motivation and exploitation of syntactic rules. Technical report, University of Illinois, http://www.linguistics.uiuc.edu/g-green/441/prmoex/index.html, 2001. ↗
10. Michael Halliday. System and Function in Language. Oxford University Press, Oxford, 1976. find
11. Irene Langkilde and Kevin Knight. Generation that exploits corpus-based statistical knowledge. In COLING-ACL-98: Proceedings of the Joint 36th Meeting of the ACL andthe 17th International COL-ING, pages 704–710, Montréal, Canada, August 1998. find
12. Irene Langkilde-Geary. An empirical verification of coverage and correctness for a general-purpose sentence generator. In Second International Natural Language Generation Conference, Harriman, NY, July 2002. find
13. Benoit Lavoie and Owen Rambow. A fast and portable realizer for text generation systems. In Proceedings of the 5th Conference on Applied Natural Language Processing, 1997. find our card
14. M. Marcus, B. Santorini, and M. Marcinkiewicz. Building a large annotated corpus of English: The PennTreeBank. Computational Linguistics, 26(2), 1993. find
15. Igor A. Mel’cuk. Dependency Syntax: Theory and Practice. SUNY Publications, 1988. find
16. K. Papineni, S. Roukos, T. Ward, and W. J. Zhu. BLEU: A method for automatic evaluation of MT. Technical Report RC22176, IBM Research, New York, September 2001. find
17. Adwait Ratnaparkhi. Trainable methods for surface natural language generation. In Proceedings of the First North American Conference of the ACL, Seattle, WA, May 2000. find
18. Michael White and Ted Caldwell. EXEMPLARS: A practical, extensible framework for dynamic text generation. In Proceedings of the Ninth International Workshop on NLG, pages 266–275, Niagara-on-the-Lake, Ontario, August 1998. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/callaway-2005-types-distributions-errors-wide-coverage/