Mova0.5 pre-release⌕
☰
home / docs / papers / English / It Depends: Dependency Parser Comparison Using A Web-based Evaluation Tool

It Depends: Dependency Parser Comparison Using A Web-based Evaluation Tool

What it is about

The authors compare ten statistical dependency parsers on the English OntoNotes 5 corpus, which has seven genres and about 2.1 million training tokens, all trained with the same automatic POS tags and dependencies from the ClearNLP converter. They report LAS, UAS, label accuracy and exact match, speed in sentences and tokens per second, accuracy by sentence length, dependency distance, projectivity, relation and POS, and genre. Greedy ClearNLP, Mate, RBG, Redshift, Turbo and Yara are close at the top, with Mate best overall (LAS 90.03 with punctuation); spaCy is the fastest greedy parser (about 755 sentences per second) but a few LAS points behind the top group. Non-projective trees cost about five LAS points for most parsers, and the authors list the parsers' typical failures: punctuation attachment, fragments and backchannels in conversation, and wrong roots. A web tool, DEPENDABLE, shows side-by-side outputs and is the paper's second contribution.

Ideas

For Mova

Touches languages/en/20-syntax-overview, 21-clause-structure and 37-register-and-variation, and the en crate (parse.rs, ud.rs), en beam-curve and tools/parsecmp. OntoNotes is licensed for research: test only. This paper is also the model for Mova's own parser comparison table. Measure first the speed and LAS point of the Mova parser at beam 8, twice, since the paper's speed numbers are on other hardware.

Topics: Dependency parsing

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1038

@inproceedings{choi2015it,
  title = {{It Depends: Dependency Parser Comparison Using A Web-based Evaluation Tool}},
  author = {Choi, Jinho D. and Tetreault, Joel and Stent, Amanda},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1038},
  url = {https://doi.org/10.3115/v1/p15-1038}
}

References

42 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily M. Bender, Dan Flickinger, Stephan Oepen, and Yi Zhang. 2011. Parser evaluation over local and non-local deep dependencies in a large corpus. In Proceedings of EMNLP. find our card
  2. Anders Björkelund et al. 2014. Introducing the IMS-Wrocław-Szeged-CIS entry at the SPMRL 2014 shared task: Reranking and morpho-syntax meet unlabeled data. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages and Syntactic Analysis of Non-Canonical Languages. find
  3. Bernd Bohnet. 2010. Very high accuracy and fast dependency parsing is not a contradiction. In Proceedings of COLING. find
  4. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of CoNLL. find our card
  5. Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of EMNLP. find our card
  6. Jinho D. Choi and Andrew McCallum. 2013. Transition-based Dependency Parsing with Selectional Branching. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, ACL’13, pages 1052–1062. find
  7. Jinho D. Choi and Martha Palmer. 2012a. Fast and Robust Part-of-Speech Tagging Using Dynamic Model Selection. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, ACL’12, pages 363–367. find
  8. Jinho D. Choi and Martha Palmer. 2012b. Guidelines for the Clear Style Constituent to Dependency Conversion. Technical Report 01-12, Institute of Cognitive Science, University of Colorado Boulder, Boulder, CO, USA. find
  9. Ronan Collobert et al. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. find
  10. Marie-Catherine de Marneffe and Christopher D. Manning. 2008. The Stanford typed dependencies representation. In Proceedings of the COLING workshop on Cross-Framework and Cross-Domain Parser Evaluation. find our card
  11. Yoav Goldberg and Joakim Nivre. 2013. Training deterministic parser with non-deterministic oracles. In Proceedings of TACL. find
  12. Gholamreza Haffari, Marzieh Razavi, and Anoop Sarkar. 2011. An ensemble model that combines syntactic and semantic clustering for discriminative dependency parsing. In Proceedings of ACL-HLT. find
  13. Jan Hajič et al. 2009. The CoNLL-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of CoNLL. find our card
  14. Matthew Honnibal, Yoav Goldberg, and Mark Johnson. 2013. A non-monotonic arc-eager transition system for dependency parsing. In Proceedings of CoNLL. find our card
  15. Liang Huang, Suphan Fayong, and Yang Guo. 2012. Structured perceptron with inexact search. In Proceedings of the NAACL. find our card
  16. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to-dependency conversion for English. In Proceedings of NODALIDA. find our card
  17. Jonathan K. Kummerfeld et al. 2012. Parser showdown at the wall street corral: An empirical investigation of error types in parser output. In Proceedings of EMNLP. find our card
  18. Joseph Le Roux et al. 2012. DCU-Paris13 systems for the SANCL 2012 shared task. In Proceedings of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL). find
  19. Joseph Le Roux, Antoine Rozenknop, and Jennifer Foster. 2013. Combining PCFG-LA models with dual decomposition: A case study with function labels and binarization. In Proceedings of EMNLP. find
  20. Tao Lei, Yu Xin, Yuan Zhang, Regina Barzilay, and Tommi Jaakkola. 2014. Low-rank tensors for scoring dependency structures. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1381–1391, Baltimore, Maryland, June. Association for Computational Linguistics. find
  21. Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: the Penn treebank. Computational Linguistics, 19(2):313–330. find our card
  22. André F. T. Martins, Miguel B. Almeida, and Noah A. Smith. 2013. Turning on the turbo: Fast third-order non-projective turbo parsers. In Proceedings of the ACL. find our card
  23. Ryan McDonald and Joakim Nivre. 2007. Characterizing the errors of data-driven dependency parsing models. In Proceedings of EMNLP-CoNLL. find our card
  24. Ryan McDonald and Joakim Nivre. 2011. Analyzing and integrating dependency parsers. Computational Linguistics, 37(1):197–230. find
  25. Makoto Miwa et al. 2010. A comparative study of syntactic parsers for event extraction. In Proceedings of BioNLP. find
  26. Jens Nilsson and Joakim Nivre. 2008. MaltEval: An evaluation and visualization tool for dependency parsing. In Proceedings of LREC. find
  27. Joakim Nivre et al. 2007. find
  28. The CoNLL 2007 shared task on dependency parsing. In Proceedings of CoNLL. find
  29. Joakim Nivre, Laura Rimell, Ryan McDonald, and Carlos Gómez Rodrı́guez. 2010. Evaluation of dependency parsers on unbounded dependencies. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 833–841, Beijing, China, August. Coling 2010 Organizing Committee. find our card
  30. Stephan Oepen et al. 2014. SemEval 2014 Task 8: Broad-Coverage Semantic Dependency Parsing. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 63–72. find
  31. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on parsing the web. In Proceedings of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL) Shared Task. find
  32. Slav Petrov, Pi-Chuan Chang, Michael Ringgaard, and Hiyan Alshawi. 2010. Uptraining for accurate deterministic question parsing. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 705–713, Cambridge, MA, October. Association for Computational Linguistics. find
  33. Sameer Pradhan et al. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of CoNLL. find
  34. Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. Yara parser: A fast and accurate dependency parser. CoRR, abs/1503.06733. ↗
  35. Kenji Sagae and Alon Lavie. 2006. Parser combination by reparsing. In Proceedings HLT-NAACL. find
  36. Kenji Sagae and Jun’ichi Tsujii. 2010. Dependency parsing and domain adaptation with data-driven LR models and parser ensembles. In Trends in Parsing Technology: Dependency Parsing, Domain Adaptation, and Deep Parsing, pages 57–68. Springer. find
  37. Djamé Seddah et al. 2013. Overview of the SPMRL 2013 shared task: A cross-framework evaluation of parsing morphologically rich languages. In Proceedings of the 4th Workshop on Statistical Parsing of Morphologically Rich Languages. find
  38. Pontus Stenetorp et al. 2012. BRAT: A web-based tool for NLP-assisted text annotation. In Proceedings of the EACL. find
  39. Mihai Surdeanu et al. 2008. The CoNLL-2008 shared task on joint parsing of syntactic and semantic dependencies. In Proceedings of CoNLL. find our card
  40. Reut Tsarfaty, Joakim Nivre, and Evelina Andersson. 2011. Evaluating dependency parsing: Robust and heuristics-free cross-annotation evaluation. In Proceedings of EMNLP. find our card
  41. Ralph Weischedel et al. 2011. OntoNotes: A large training corpus for enhanced processing. In Joseph Olive, Caitlin Christianson, and John McCary, editors, Handbook of Natural Language Processing and Machine Translation. Springer. find
  42. Deniz Yuret, Laura Rimell, and Aydin Han. 2013. Parser evaluation using textual entailments. Language Resources and Evaluation, 47(3). find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2015-choi-it-depends-dependency-parser-comparison/