Characterizing the Errors of Data-Driven Dependency Parsing Models

What it is about

The paper compares the two top systems of the CoNLL-X shared task. MSTParser is a global, exhaustive graph-based model with maximum spanning tree decoding and arc-pair features; MaltParser is a local, greedy transition-based model with an SVM classifier. Both reach almost the same labeled accuracy over 13 languages (about 80.8 and 80.8 LAS when the output files are pooled), but they make different kinds of errors. The authors break the errors down by arc length, distance to the root, number of siblings, degree of non-projectivity, part of speech and dependency type. MSTParser is more precise for arcs far from the root and for long arcs; MaltParser is better on short arcs and on nouns and pronouns. A simple oracle over the two outputs reaches 84.5 to 86.9% against about 81% for each system.

Ideas

• I1. Error profile by distance to the root in en parse-eval and in gparse (Eisner decoding, gparse/src/eisner.rs): precision and recall of arcs at distance 1, 2, 3–4 and 5 or more, on EWT test; the question is whether the arc-standard parser fragments the tree near the root as the paper suggests for greedy parsers.
• I2. Root-modifier check: count predicted root arcs that are not verbs or auxiliaries in en explain output over EWT test; the paper reads low root-modifier precision with stable recall as a sign of fragmented parses.
• I3. Oracle diagnostic (test only, not shipped): take the en and gparse outputs on EWT test and measure the oracle LAS and the share of sentences where one of them is fully correct; if the gap is large, a combiner through tools/parsecmp is worth a later decision.
• I4. Non-projectivity split: report LAS separately for sentences with a gold arc of degree 1 or more; the arc-standard parser in en/src/parse.rs cannot build those arcs, so the number is its ceiling on EWT and GUM test (GUM is test only).

For Mova

The relevant chapters are 20. Syntax: the Universal Dependencies frame (the main clause and root) and 21-clause-structure.md. The code is en/src/parse.rs (arc-standard, beam 8), gparse/src/eisner.rs, and the en parse-eval and tools/parsecmp tools. Measure first: EWT test LAS by distance to the root for both parsers, against the current baseline LAS 80.24. The paper's lesson is that an error analysis by structure says more than one accuracy number.

Topics: Dependency parsing

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{mcdonald2007characterizing,
  title = {{Characterizing the Errors of Data-Driven Dependency Parsing Models}},
  author = {McDonald, Ryan and Nivre, Joakim},
  booktitle = {Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)},
  year = {2007},
  url = {https://aclanthology.org/D07-1013/}
}

References

17 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Böhmová, J. Hajič, E. Hajičová, and B. Hladká. 2003. The PDT: A 3-level annotation scenario. In A. Abeillé, editor, Treebanks: Building and Using Parsed Corpora, chapter 7. Kluwer Academic Publishers. find
2. S. Buchholz and E. Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proc. CoNLL. find our card
3. Hal Daumé III, John Langford, and Daniel Marcu. 2006. Search-based structured prediction. In Submission. find
4. Y. Ding and M. Palmer. 2004. Synchronous dependency insertion grammars: A grammar formalism for syntax based statistical MT. In Workshop on Recent Advances in Dependency Grammars (COLING). find
5. R. Hudson. 1984. Word Grammar. Blackwell. find
6. H. Maruyama. 1990. Structural disambiguation with constraint propagation. In Proc. ACL. find
7. R. McDonald, K. Crammer, and F. Pereira. 2005a. Online large-margin training of dependency parsers. In Proc. ACL. find our card
8. R. McDonald, F. Pereira, K. Ribarov, and J. Hajič. 2005b. Non-projective dependency parsing using spanning tree algorithms. In Proc. HLT/EMNLP. find our card
9. R. McDonald, K. Lerman, and F. Pereira. 2006. Multilingual dependency analysis with a two-stage discriminative parser. In Proc. CoNLL. find
10. I.A. Mel’čuk. 1988. Dependency Syntax: Theory and Practice. State University of New York Press. find
11. J. Nivre and J. Nilsson. 2005. Pseudo-projective dependency parsing. In Proc. ACL. find
12. J. Nivre, J. Hall, J. Nilsson, G. Eryigit, and S. Marinov. 2006. Labeled pseudo-projective dependency parsing with support vector machines. In Proc. CoNLL. find
13. J. Nivre. 2003. An efficient algorithm for projective dependency parsing. In Proc. IWPT. find our card
14. J. Nivre. 2006. Constraints on non-projective dependency parsing. In Proc. EACL. find
15. K. Sagae and A. Lavie. 2006. Parser combination by reparsing. In Proc. HLT/NAACL. find
16. P. Sgall, E. Hajičová, and J. Panevová. 1986. The Meaning of the Sentence in Its Pragmatic Aspects. Reidel. find
17. R. Snow, D. Jurafsky, and A. Y. Ng. 2004. Learning syntactic patterns for automatic hypernym discovery. In Proc. NIPS. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/missing4-2007-mcdonald-characterizing-errors-data-driven-dependency/