Parser Evaluation Using Derivation Trees: A Complement to evalb

What it is about

The standard metric for constituency parsers, evalb, scores matching brackets, so it cannot say which construction went wrong. The paper decomposes each tree into small tree-adjoining-style units, and records how the units attach. A set of 49 regular expressions names the construction of each unit, such as right modification (NP-modr), coordination or a non-recursive NP. Each word is then scored on whether it has the right head for its construction (F-h) and whether the span is right too (F-s), and the attachment of each word is scored separately. The Berkeley parser is analysed on OntoNotes WSJ sections 2–21 (where it is trained, so this is test-on-train), on section 22, and on the English Web Treebank "Answers" section. On 2–21 the non-recursive NP covers 30.7% of brackets with F-s 97.6, and the final F-s of 92.5 is lower than the evalb score of 93.8. On Answers the right edge of VP-t has a span score of only 85.4, against 95.8 on 2–21.

Ideas

• I1. tools/parsecmp: group the UD arcs of the English parser into a small set of construction types, using rule patterns over the tree, and report head and attachment accuracy per type next to LAS and UAS from en parse-eval. Run it for en/src/parse.rs and for the tools/treeagree output.
• I2. Report PP attachment and SBAR attachment on their own, as the paper does, because a single attachment score hides which errors dominate.
• I3. Negative control: a test-on-train split. If the rate on training sentences is near perfect for a construction type, the parser is memorising that type and the type needs a separate test.

For Mova

The chapters are 22. Phrases: noun, adjective, adverb and prepositional phrases and 24. Coordination and subordination, with 25. Relative clauses for the relative-clause types. The code is tools/parsecmp, en/src/parse.rs and en parse-eval. The measures: per-type head and attachment accuracy on UD EWT dev, and the train-versus-dev gap per type. The decomposition itself is tied to the TAG treebank; the idea to take is the per-construction score.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-2109

@inproceedings{kulick2014parser,
  title = {{Parser Evaluation Using Derivation Trees: A Complement to evalb}},
  author = {Kulick, Seth and Bies, Ann and Mott, Justin and Kroch, Anthony and Santorini, Beatrice and Liberman, Mark},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-2109},
  url = {https://doi.org/10.3115/v1/p14-2109}
}

References

16 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Srinivas Bangalore and Aravind K. Joshi, editors. 2010. Supertagging: Using Complex Lexical Descriptions in Natural Language Processing. MIT Press. find
2. Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English Web Treebank. LDC2012T13. Linguistic Data Consortium. find
3. Michael Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, Department of Computer and Information Sciences, University of Pennsylvania. find our card
4. A.K. Joshi and Y. Schabes. 1997. Tree-adjoining grammars. In G. Rozenberg and A. Salomaa, editors, Handbook of Formal Languages, Volume 3: Beyond Words, pages 69–124. Springer, New York. find
5. Seth Kulick, Ann Bies, and Justin Mott. 2011. Using derivation trees for treebank error detection. Association for Computational Linguistics. find
6. Seth Kulick, Ann Bies, and Justin Mott. 2012. Using supertags and encoded annotation principles for improved dependency to phrase structure conversion. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 305–314, Montréal, Canada, June. Association for Computational Linguistics. find
7. Seth Kulick, Ann Bies, Justin Mott, Mohamed Maamouri, Beatrice Santorini, and Anthony Kroch. 2013. Using derivation trees for informative treebank inter-annotator agreement evaluation. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 550–555, Atlanta, Georgia, June. Association for Computational Linguistics. find
8. Jonathan K. Kummerfeld, David Hall, James R. Curran, and Dan Klein. 2012. Parser showdown at the wall street corral: An empirical investigation of error types in parser output. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1048–1059, Jeju Island, Korea, July. Association for Computational Linguistics. find our card
9. Mitchell P. Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. 1999. Treebank-3. LDC99T42, Linguistic Data Consortium, Philadelphia. find
10. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. In Proceedings of the First Workshop on Syntactic Analysis of Non-Canonical Language. find
11. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2008. The Berkeley Parser. https://code.google.com/p/berkeleyparser/. ↗
12. Owen Rambow and Aravind Joshi. 1997. A formal look at dependency grammars and phrase-structure grammars, with special consideration of word-order phenomena. In L. Wanner, editor, Recent Trends in Meaning-Text Theory, pages 167–190. John Benjamins, Amsterdam and Philadelphia. find
13. Satoshi Sekine and Michael Collins. 2008. Evalb. http://nlp.cs.nyu.edu/evalb/. ↗
14. Libin Shen and Aravind Joshi. 2008. LTAG dependency parsing with bidirectional incremental construction. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 495–504, Honolulu, Hawaii, October. Association for Computational Linguistics. find
15. Libin Shen, Lucas Champollion, and Aravind Joshi. 2008. LTAG-spinal and the Treebank: A new resource for incremental, dependency and semantic parsing. Language Resources and Evaluation, 42(1):1–19. find
16. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2011. OntoNotes 4.0. Linguistic Data Consortium LDC2011T03. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/kulick-2014-parser-evaluation-derivation-trees-complement/