Effective Analysis of Causes and Inter-dependencies of Parsing Errors

What it is about

F-scores hide where a parser fails and how its errors depend on each other. The authors analyse errors of the Enju HPSG parser on predicate-argument triplets in two ways. First, 14 hand-written pattern rules map sets of mismatched triplets to cause categories such as PP-attachment, coordination, relative-clause attachment and the infinitive as adjunct or complement. Second, each erroneous decision is forcibly corrected and the sentence is re-parsed; errors that disappear with it are treated as dependent on it, giving groups of inter-dependent errors and a propagation network. On Penn Treebank Section 0 the parser makes 4,709 triplet errors (F-score 90.69). The rules cover 2,078 of them (over 40%), and 3,085 are correctable. The re-parse analysis finds 1,978 groups and 501 propagation links. In a domain adaptation study on GENIA, the adapted model lowers the error count from 3,120 to 2,229 and the F-score rises from 87.41 to 90.93.

Ideas

• I1. Pattern rules over dependency arcs: write the paper's cause categories (PP-attachment, relative clause, coordination scope, infinitive as complement) as rules over en/src/parse.rs output, and count the hits on the school texts. Each rule should say why it fired, which fits Mova's explainable style.
• I2. Forced-correction re-parse as a test: for each wrong arc in a tree, fix it and re-run the parser; record which other arcs change. Build the propagation graph in a BTreeMap and use it to order the fixes.
• I3. Coverage report by category: like the paper's Table 3, report how many errors each rule explains and how many remain unclassified, so each new rule is measured on held-out lessons.
• I4. Negative control: apply the same rules to a shuffled or corrupted parse. The rule hit rate must drop sharply, or the rules are not measuring real errors.
• I5. Domain check: run en parse-eval separately on the school questions and the narrative texts, and report the category distribution per domain, as the paper does for PTB and GENIA.

For Mova

The cause categories map to 25. Relative clauses, 23. Complementation (infinitive and gerund), and 24. Coordination and subordination. Code: en/src/parse.rs for the arcs, a new error-analysis step in tools/parsecmp for the categories and the re-parse, and en parse-eval as the base measure. Measure: the share of parser errors covered by rules, the number of propagation links, and the held-out UAS on UD EWT dev, run twice to confirm determinism.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{hara2009effective,
  title = {{Effective Analysis of Causes and Inter-dependencies of Parsing Errors}},
  author = {Hara, Tadayoshi and Miyao, Yusuke and Tsujii, Jun'ichi},
  booktitle = {Proceedings of the 11th International Conference on Parsing Technologies (IWPT'09)},
  year = {2009},
  url = {https://aclanthology.org/W09-3828/}
}

References

11 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mark Dredze, John Blitzer, Partha Pratim Talukdar, Kuzman Ganchev, João V. Graça, and Fernando Pereira. 2007. Frustratingly hard domain adaptation for dependency parsing. In Proceedings of the CoNLL Shared Task Session of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 1051–1055. find our card
2. Jesús Giménez and Lluı́s Màrquez. 2008. Towards heterogeneous automatic mt error analysis. In Proceedings of the Sixth International Language Resources and Evaluation (LREC’08), pages 1894–1901. find our card
3. Tadayoshi Hara, Yusuke Miyao, and Jun’ichi Tsujii. 2007. Evaluating impact of re-training a lexical disambiguation model on domain adaptation of an hpsg parser. In Proceedings of 10th International Conference on Parsing Technologies (IWPT 2007), pages 11–22. find
4. Ronald M. Kaplan and Joan Bresnan. 1995. Lexical-functional grammar: A formal system for grammatical representation. Formal Issues in Lexical-Functional Grammar, pages 29–130. find
5. Jin-Dong Kim, Tomoko Ohta, Yuka Teteisi, and Jun’ichi Tsujii. 2003. GENIA corpus - a semantically annotated corpus for bio-textmining. Bioinformatics, 19(suppl. 1):i180–i182. find
6. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert Macintyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: Annotating predicate argument structure. In Proceedings of ARPA Human Language Technology Workshop. find our card
7. Ryan McDonald and Joakim Nivre. 2007. Characterizing the errors of data-driven dependency parsing models. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 122–131. find our card
8. Yusuke Miyao and Jun’ichi Tsujii. 2005. Probabilistic disambiguation models for wide-coverage HPSG parsing. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL), pages 83–90. find our card
9. Takashi Ninomiya, Takuya Matsuzaki, Yoshimasa Tsuruoka, Yusuke Miyao, and Jun’ichi Tsujii. 2006. Extremely lexicalized models for accurate and fast HPSG parsing. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 155–163. find
10. Carl J. Pollard and Ivan A. Sag. 1994. Head-Driven Phrase Structure Grammar. University of Chicago Press. find
11. Mark Steedman. 2000. The Syntactic Process. THE MIT Press. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/hara-2009-effective-analysis-causes-inter-dependencies/