Descriptive and Empirical Approaches to Capturing Underlying Dependencies among Parsing Errors

What it is about

Parser errors are usually counted one by one, but one wrong decision often causes several errors. The authors look for these underlying dependencies in two ways. The descriptive approach defines combinations of error patterns, such as argument inconsistencies around a verb, and extracts them from the errors. The empirical approach re-parses a sentence with one error corrected and sees which other errors disappear. On section 0 of the Penn Treebank the Enju HPSG parser makes 4,709 errors, of which 3,085 are correctable; these form 1,978 co-occurring error groups and 501 inductive relations. Fourteen error patterns cover more than 40% of the errors, and prepositional attachments stand out among the error types.

Ideas

• I1. Co-occurring error groups in en ud-eval: on EWT dev, group the wrong arcs that share a wrong head or a wrong subtree into co-occurring groups and report how many errors fall in groups of two or more, against the paper's 1,978 groups from 3,085 correctable errors.
• I2. Re-parse with one arc fixed: for each wrong arc of the en parse of an EWT dev sentence, force the gold arc and re-parse, and count the other wrong arcs that disappear; en explain can show why the other arcs changed.
• I3. Prepositional attachment as a pattern: measure the share of wrong arcs that are obl or nmod attachments of prepositions in EWT dev, and try one attachment feature in en/src/parse.rs for the preposition and its candidate heads, with dev LAS as the measure.

For Mova

The error analysis tooling is en ud-eval and en explain in en/src/explain.rs, the parser is en/src/parse.rs. The chapters that the ideas touch are 22. Phrases: noun, adjective, adverb and prepositional phrases and 23. Complementation. Measure first: the share of EWT dev errors that sit in co-occurring groups, since it decides whether pattern-level fixes are worth more than arc-level ones.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1699648.1699662

@inproceedings{hara2009descriptive,
  title = {{Descriptive and Empirical Approaches to Capturing Underlying Dependencies among Parsing Errors}},
  author = {Hara, Tadayoshi and Miyao, Yusuke and Tsujii, Jun'ichi},
  booktitle = {Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing},
  year = {2009},
  doi = {10.3115/1699648.1699662},
  url = {https://doi.org/10.3115/1699648.1699662}
}

References

9 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mark Dredze, John Blitzer, Partha Pratim Talukdar, Kuzman Ganchev, João V. Graça, and Fernando Pereira. 2007. Frustratingly hard domain adaptation for dependency parsing. In Proceedings of the CoNLL Shared Task Session of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 1051–1055. find our card
2. Jesús Giménez and Lluı́s Màrquez. 2008. Towards heterogeneous automatic MT error analysis. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), pages 1894–1901. find our card
3. Ronald M. Kaplan and Joan Bresnan. 1995. Lexical-functional grammar: A formal system for grammatical representation. Formal Issues in Lexical-Functional Grammar, pages 29–130. find
4. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert Macintyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: Annotating predicate argument structure. In Proceedings of ARPA Human Language Technology Workshop. find our card
5. Ryan McDonald and Joakim Nivre. 2007. Characterizing the errors of data-driven dependency parsing models. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 122–131. find our card
6. Yusuke Miyao and Jun’ichi Tsujii. 2005. Probabilistic disambiguation models for wide-coverage HPSG parsing. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL), pages 83–90. find our card
7. Takashi Ninomiya, Takuya Matsuzaki, Yoshimasa Tsuruoka, Yusuke Miyao, and Jun’ichi Tsujii. 2006. Extremely lexicalized models for accurate and fast HPSG parsing. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 155–163. find
8. Carl J. Pollard and Ivan A. Sag. 1994. Head-Driven Phrase Structure Grammar. University of Chicago Press. find
9. Mark Steedman. 2000. The Syntactic Process. THE MIT Press. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2009-hara-descriptive-empirical-approaches-capturing-underlying/