Mova0.5 pre-release⌕
☰
home / docs / papers / English / Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output

Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output

What it is about

A single F-score on WSJ section 23 says nothing about which errors a constituency parser makes. The authors repair each wrong parse into the gold tree with a few tree transformations (subtree movement, node creation, node deletion), and name each repair by its linguistic type: PP attachment, clause attachment, modifier attachment, coordination, label, and others. They analyse thirteen parsers. PP attachment is the largest group (846 error groups, 1,455 node errors on section 23), and coordination is rarer but costs more nodes per error (209 groups, 557 node errors). Rerankers gain mainly on some types, and out-of-domain text adds a disproportionate number of coordination and clause-attachment errors. The method works on constituency trees, but the idea of typed errors carries over to dependency arcs.

Ideas

For Mova

Touches 20. Syntax: the Universal Dependencies frame, 21. The simple clause and 22. Phrases: noun, adjective, adverb and prepositional phrases. This gives Mova a way to say why a parser fails, which fits the explainable goal. Measure I1 first; it needs only the existing en parse-eval output.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{kummerfeld2012parser,
  title = {{Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output}},
  author = {Kummerfeld, Jonathan K. and Hall, David and Curran, James R. and Klein, Dan},
  booktitle = {Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning},
  year = {2012},
  url = {https://aclanthology.org/D12-1096/}
}

References

42 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Abney, S. Flickenger, C. Gdaniec, C. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, M. Marcus, S. Roukos, B. Santorini, and T. Strzalkowski. 1991. Procedure for quantitatively comparing the syntactic coverage of english grammars. In Proceedings of the workshop on Speech and Natural Language, pages 306–311, Pacific Grove, California, USA, February. find
  2. Michael Auli and Adam Lopez. 2011. A comparison of loopy belief propagation and dual decomposition for integrated ccg supertagging and parsing. In Proceedings of ACL, pages 470–480, Portland, Oregon, USA, June. find
  3. Emily M. Bender, Dan Flickinger, Stephan Oepen, and Yi Zhang. 2011. Parser evaluation over local and non-local deep dependencies in a large corpus. In Proceedings of EMNLP, pages 397–408, Edinburgh, United Kingdom, July. find our card
  4. Daniel M. Bikel. 2004. Intricacies of collins’ parsing model. Computational Linguistics, 30(4):479–511. find
  5. Nathan Bodenstab, Aaron Dunlop, Keith Hall, and Brian Roark. 2011. Beam-width prediction for efficient context-free parsing. In Proceedings of ACL, pages 440–449, Portland, Oregon, USA, June. find
  6. Ted Briscoe and John Carroll. 2006. Evaluating the accuracy of an unlexicalized statistical parser on the PARC DepBank. In Proceedings of ACL, pages 41– 48, Sydney, Australia, July. find
  7. Ted Briscoe, John Carroll, Jonathan Graham, and Ann Copestake, 2002. Relational Evaluation Schemes, pages 4–8. Las Palmas, Canary Islands, Spain, May. find
  8. John Carroll, Ted Briscoe, and Antonio Sanfilippo. 1998. Parser evaluation: a survey and a new proposal. In Proceedings of LREC, pages 447–454, Granada, Spain, May. find
  9. Daniel Cer, Marie-Catherine de Marneffe, Daniel Jurafsky, and Christopher D. Manning. 2010. Parsing to stanford dependencies: Trade-offs between speed and accuracy. In Proceedings of LREC, Valletta, Malta, May. find our card
  10. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of ACL, pages 173–180, Ann Arbor, Michigan, USA, June. find our card
  11. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of NAACL, pages 132–139, Seattle, Washington, USA, April. find
  12. Stephen Clark and Julia Hockenmaier. 2002. Evaluating a wide-coverage ccg parser. In Proceedings of the LREC Beyond Parseval Workshop, Las Palmas, Canary Islands, Spain, May. find
  13. Andrew B. Clegg and Adrian J. Shepherd. 2005. Evaluating and integrating treebank parsers on a biomedical corpus. In Proceedings of the ACL Workshop on Software, pages 14–33, Ann Arbor, Michigan, USA, June. find
  14. Michael Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proceedings of ACL, pages 16–23, Madrid, Spain, July. find
  15. Michael Collins. 2000. Discriminative reranking for natural language parsing. In Proceedings of ICML, pages 175–182, Palo Alto, California, USA, June. find our card
  16. Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational Linguistics, 29(4):589–637. find our card
  17. Rebecca Dridan and Stephan Oepen. 2011. Parser evaluation using elementary dependency matching. In Proceedings of IWPT, pages 225–230, Dublin, Ireland, October. find
  18. Aaron Dunlop, Nathan Bodenstab, and Brian Roark. 2011. Efficient matrix-encoded grammars and low latency parallelization strategies for cyk. In Proceedings of IWPT, pages 163–174, Dublin, Ireland, October. find
  19. Jennifer Foster and Josef van Genabith. 2008. Parser evaluation and the bnc: Evaluating 4 constituency parsers with 3 metrics. In Proceedings of LREC, Marrakech, Morocco, May. find our card
  20. Daniel Gildea. 2001. Corpus variation and parser performance. In Proceedings of EMNLP, pages 167–202, Pittsburgh, Pennsylvania, USA, June. find our card
  21. Tadayoshi Hara, Yusuke Miyao, and Jun’ichi Tsujii. 2007. Evaluating impact of re-training a lexical disambiguation model on domain adaptation of an hpsg parser. In Proceedings of IWPT, pages 11–22, Prague, Czech Republic, June. find
  22. Tadayoshi Hara, Yusuke Miyao, and Jun’ichi Tsujii. 2009. Descriptive and empirical approaches to capturing underlying dependencies among parsing errors. In Proceedings of EMNLP, pages 1162–1171, Singapore, August. find our card
  23. James Henderson. 2003. Inducing history representations for broad coverage statistical parsing. In Proceedings of NAACL, pages 24–31, Edmonton, Canada, May. find
  24. James Henderson. 2004. Discriminative training of a neural network statistical parser. In Proceedings of ACL, pages 95–102, Barcelona, Spain, July. find
  25. Liang Huang. 2008. Forest reranking: Discriminative parsing with non-local features. In Proceedings of ACL, pages 586–594, Columbus, Ohio, USA, June. find
  26. Tracy H. King, Richard Crouch, Stefan Riezler, Mary Dalrymple, and Ronald M. Kaplan. 2003. The PARC 700 dependency bank. In Proceedings of the 4th International Workshop on Linguistically Interpreted Corpora at EACL, Budapest, Hungary, April. find
  27. Dan Klein and Christopher D. Manning. 2003a. Accurate unlexicalized parsing. In Proceedings of ACL, pages 423–430, Sapporo, Japan, July. find our card
  28. Dan Klein and Christopher D. Manning. 2003b. Fast exact inference with a factored model for natural language parsing. In Proceedings of NIPS, pages 3–10, Vancouver, British Columbia, Canada, December. find
  29. Dekang Lin. 1998. A dependency-based method for evaluating broad-coverage parsers. Natural Language Engineering, 4(2):97–114. find
  30. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: the penn treebank. Computational Linguistics, 19(2):313–330. find our card
  31. David McClosky, Eugene Charniak, and Mark Johnson. 2006. Effective self-training for parsing. In Proceedings of NAACL, pages 152–159, New York, New York, USA, June. find our card
  32. Ryan McDonald and Joakim Nivre. 2007. Characterizing the errors of data-driven dependency parsing models. In Proceedings of EMNLP, pages 122–131, Prague, Czech Republic, June. find our card
  33. Yusuke Miyao, Rune Sætre, Kenji Sagae, Takuya Matsuzaki, and Jun’ichi Tsujii. 2008. Task-oriented evaluation of syntactic parsers and their representations. In Proceedings of ACL, pages 46–54, Columbus, Ohio, USA, June. find our card
  34. Dominick Ng and James R. Curran. 2012. N-best CCG parsing and reranking. In Proceedings of ACL, Jeju, South Korea, July. find
  35. Dominick Ng, Matthew Honnibal, and James R. Curran. 2010. Reranking a wide-coverage ccg parser. In Proceedings of ALTA, pages 90–98, Melbourne, Australia, December. find
  36. Joakim Nivre, Laura Rimell, Ryan McDonald, and Carlos Gómez-Rodrı́guez. 2010. Evaluation of dependency parsers on unbounded dependencies. In Proceedings of Coling, pages 833–841, Beijing, China, August. find our card
  37. Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In Proceedings of NAACL, pages 404–411, Rochester, New York, USA, April. find
  38. Slav Petrov and Ryan McDonald. 2012. SANCL Shared Task. LDC2012E43. Linguistic Data Consortium. Philadelphia, Philadelphia, USA. find
  39. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proceedings of ACL, pages 433–440, Sydney, Australia, July. find our card
  40. Chris Quirk and Simon Corston-Oliver. 2006. The impact of parse quality on syntactically-informed statistical machine translation. In Proceedings of EMNLP, pages 62–69, Sydney, Australia, July. find
  41. Kun Yu, Yusuke Miyao, Takuya Matsuzaki, Xiangli Wang, and Junichi Tsujii. 2011. Analysis of the difficulties in chinese deep parsing. In Proceedings of IWPT, pages 48–57, Dublin, Ireland, October. find
  42. Deniz Yuret, Aydin Han, and Zehra Turgut. 2010. Semeval-2010 task 12: Parser evaluation using textual entailments. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 51–56, Uppsala, Sweden, July. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2012-kummerfeld-parser-showdown-wall-street-corral/