Evaluation of Dependency Parsers on Unbounded Dependencies

What it is about

Unbounded dependencies, such as an object that is extracted from a relative clause, are a standard test of grammar-based parsers, but had not been applied to dependency parsers. The authors train MSTParser and MaltParser on the Penn Treebank converted to Stanford dependencies, and add a simple post-processor that extracts long-range arcs from the basic tree. Their test follows Rimell et al. (2009). With the post-processor, the two parsers recover unbounded dependencies about 50% of the time, slightly below two grammar-based parsers designed for this. Recall varies by construction, from about 65% down to 1%.

Ideas

• I1. en/src/parse.rs plus a post-processor in prag: add long-range arcs for object relatives and reduced relatives on the en tree, and measure the recall of these arcs on a 100-sentence dev set of relatives and questions, before and after, two runs each.
• I2. gparse: the biaffine and Eisner model may recover non-local arcs that the arc-standard parser cannot; run the same per-construction recall for en and gparse on the same dev set and report both.
• I3. en parse-eval: report recall per construction of unbounded arcs next to LAS, so that an overall gain which hides a loss on relatives is visible.

For Mova

Touches 25. Relative clauses, 23. Complementation and 26. Questions. The prag and world layers need these arcs to link a relative clause to its antecedent. Measure I3 first, because it shows where the existing en output already fails.

Topics: Dependency parsing

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{nivre2010evaluation,
  title = {{Evaluation of Dependency Parsers on Unbounded Dependencies}},
  author = {Nivre, Joakim and Rimell, Laura and McDonald, Ryan and Gómez-Rodríguez, Carlos},
  booktitle = {Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)},
  year = {2010},
  url = {https://aclanthology.org/C10-1094/}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Black, E., S. Abney, D. Flickinger, C. Gdaniec, R. Grishman, P. Harrison, D. Hindle, R. Ingria, F. Jelinek, J. Klavans, M. Liberman, S. Roukos, B. Santorini, and T. Strzalkowski. 1991. A procedure for quantitatively comparing the syntactic coverage of English grammars. In Proceedings of 4th DARPA Workshop, 306–311. find
2. Briscoe, T., J. Carroll, and R. Watson. 2006. The second release of the RASP system. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, 77–80. find
3. Buchholz, S. and E. Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of CoNLL, 149–164. find our card
4. Cahill, A., M. Burke, R. O’Donovan, J. Van Genabith, and A. Way. 2004. Long-distance dependency resolution in automatically acquired wide-coverage PCFG-based LFG approximations. In Proceedings of ACL, 320–327. find
5. Carreras, X., M. Collins, and T. Koo. 2008. TAG, dynamic programming, and the perceptron for efficient, feature-rich parsing. In Proceedings of CoNLL, 9–16. find
6. Charniak, E. and M. Johnson. 2005. Coarse-to-fine nbest parsing and MaxEnt discriminative reranking. In Proceedings of ACL, 173–180. find
7. Clark, S. and J. R. Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. Computational Linguistics, 33:493–552. find our card
8. de Marneffe, M.-C., B. MacCartney, and C. D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of LREC. find our card
9. Giménez, J. and L. Màrquez. 2004. SVMTool: A general POS tagger generator based on support vector machines. In Proceedings of LREC. find
10. Hall, J., J. Nivre, and J. Nilsson. 2006. Discriminative classifiers for deterministic dependency parsing. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, 316–323. find
11. Hall, J., J. Nilsson, J. Nivre, G. Eryiğit, B. Megyesi, M. Nilsson, and M. Saers. 2007. Single malt or blended? A study in multilingual parser optimization. In Proceedings of the CoNLL Shared Task. find
12. Huang, L. 2008. Forest reranking: Discriminative parsing with non-local features. In Proceedings of ACL, 586–594. find
13. Judge, J., A. Cahill, and J. van Genabith. 2006. QuestionBank: Creating a corpus of parse-annotated questions. In Proceedings of COLING-ACL, 497– 504. find
14. Klein, D. and C. D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of ACL, 423–430. find our card
15. Kübler, S., R. McDonald, and J. Nivre. 2008. Dependency Parsing. Morgan and Claypool. find
16. McDonald, R. and J. Nivre. 2007. Characterizing the errors of data-driven dependency parsing models. In Proceedings of EMNLP-CoNLL, 122–131. find our card
17. McDonald, R. and F. Pereira. 2006. Online learning of approximate dependency parsing algorithms. In Proceedings of EACL, 81–88. find our card
18. McDonald, R. and G. Satta. 2007. On the complexity of non-projective data-driven dependency parsing. In Proceedings of IWPT, 122–131. find
19. McDonald, R., K. Crammer, and F. Pereira. 2005. Online large-margin training of dependency parsers. In Proceedings of ACL, 91–98. find our card
20. McDonald, R., K. Lerman, and F. Pereira. 2006. Multilingual dependency analysis with a two-stage discriminative parser. In Proceedings of CoNLL, 216– 220. find
21. McDonald, R.. 2006. Discriminative Learning and Spanning Tree Algorithms for Dependency Parsing. Ph.D. thesis, University of Pennsylvania. find
22. Miyao, Y. and J. Tsujii. 2005. Probabilistic disambiguation models for wide-coverage HPSG parsing. In Proceedings of ACL, 83–90. find our card
23. Nivre, J., J. Hall, and J. Nilsson. 2006a. MaltParser: A data-driven parser-generator for dependency parsing. In Proceedings of LREC, 2216–2219. find
24. Nivre, J., J. Hall, J. Nilsson, G. Eryiğit, and S. Marinov. 2006b. Labeled pseudo-projective dependency parsing with support vector machines. In Proceedings of CoNLL, 221–225. find
25. Nivre, J., J. Hall, J. Nilsson, A. Chanev, G. Eryiǧit, S. Kübler, S. Marinov, and E. Marsi. 2007. Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13:95–135. find
26. Nivre, J. 2009. Non-projective dependency parsing in expected linear time. In Proceedings of ACL-IJCNLP, 351–359. find
27. Rimell, L., S. Clark, and M. Steedman. 2009. Unbounded dependency recovery for parser evaluation. In Proceedings EMNLP, 813–821. find our card
28. Sagae, K. and A. Lavie. 2006. Parser combination by reparsing. In Proceedings of NAACL HLT: Short Papers, 129–132. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2010-nivre-evaluation-dependency-parsers-unbounded-dependencies/