Identifying Cascading Errors using Constraints in Dependency Parsing

What it is about

Parser evaluation counts wrong arcs, but one wrong arc can force several more, so the count hides which error is the root cause. The authors force each parser (MSTParser, graph-based, and ZPar, transition-based with a beam) to take the gold head and label for every word of one error class (NP, PP, coordination, clause, modifier, root, punctuation, and others), while letting the rest of the parse be built freely. They measure the direct gain on the constrained words and the cascaded gain on the other words. On WSJ section 22 with Stanford dependencies, the parsers score UAS 91.3 and 91.7. NP attachments and punctuation cause the largest cascades; punctuation constraints have the largest cascaded effect for both parsers, and wrong root arcs are displaced by about nine words on average. Forcing punctuation also lowers ZPar's coverage to 93.2%, which shows how a beam can lose the constrained state.

Ideas

• I1. en/src/parse.rs constraint mode: force the gold head and label for one error class at a time during beam decoding, and report direct and cascaded gains per class, as the paper does; measure on EWT dev (test only for the final numbers).
• I2. en/src/ud.rs and the parser's features: test punctuation as context features rather than as arcs, as the paper's cascade result suggests and as Ma et al. (2014) propose; measure LAS on EWT test with punctuation arcs excluded from training, and the change in attachment of non-punctuation words.
• I3. tools/treeagree root gate: check that the root is the predicate of the main clause (a verb or copula, and not a fragment head) and count violations per 1k silver trees; the paper says root errors are rare but displace the tree most.
• I4. en beam-curve coverage: report, for each beam width, the share of sentences where the constrained decoder finds no tree; measure it on EWT dev with the class constraints from I1.

For Mova

Touches languages/en/07-punctuation-capitals-numbers, 21-clause-structure, 22-phrases and 24-coordination-subordination, and the en crate (parse.rs, ud.rs), tools/treeagree and en beam-curve. OntoNotes and WSJ material is research-licensed: test only. Measure first the punctuation share of cascaded errors on EWT dev, since that number decides whether I2 is worth building.

Topics: Dependency parsing

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1111

@inproceedings{ng2015identifying,
  title = {{Identifying Cascading Errors using Constraints in Dependency Parsing}},
  author = {Ng, Dominick and Curran, James R.},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1111},
  url = {https://doi.org/10.3115/v1/p15-1111}
}

References

25 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2014. Tailoring continuous word representations for dependency parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL-14), pages 809–815. Baltimore, Maryland, USA. find
2. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X Shared Task on Multilingual Dependency Parsing. In Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-06), pages 149–164. find our card
3. Daniel Cer, Marie-Catherine de Marneffe, Dan Jurafsky, and Chris Manning. 2010. Parsing to Stanford Dependencies: Trade-offs between Speed and Accuracy. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC-10). find our card
4. Eugene Charniak and Mark Johnson. 2005. Coarse-to-Fine n-Best Parsing and MaxEnt Discriminative Reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL-05), pages 173–180. find our card
5. Stephen Clark and James R. Curran. 2007. Formalism-Independent Parser Evaluation with CCG and DepBank. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics (ACL-07), pages 248–255. Prague, Czech Republic. find our card
6. Marie-Catherine de Marneffe and Christopher D. Manning. 2008. Stanford dependencies manual. Technical report, Stanford University. find
7. Jason Eisner. 1996. Three New Probabilistic Models for Dependency Parsing: An Exploration. In Proceedings of the 16th International Conference on Computational Linguistics (COLING-96), pages 340–345. find our card
8. Jason Eisner and A. Noah Smith. 2005. Parsing with Soft and Hard Constraints on Dependency Length. In Proceedings of the Ninth International Workshop on Parsing Technology (IWPT-05), pages 30–41. find
9. Richard Johansson and Pierre Nugues. 2007. Extended Constituent-to-dependency Conversion for English. In Proceedings of the 16th Nordic Conference of Computational Linguistics (NODALIDA-07), pages 105–112. Tartu, Estonia. find our card
10. Jonathan K. Kummerfeld, David Hall, James R. Curran, and Dan Klein. 2012. Parser Showdown at the Wall Street Corral: An Empirical Investigation of Error Types in Parser Output. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL-12), pages 1048– 1059. find our card
11. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a Large Annotated Corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. ↗ our card
12. Ryan McDonald and Joakim Nivre. 2011. Analyzing and Integrating Dependency Parsers. Computational Linguistics, 37(1):197–230. find
13. Ryan McDonald and Fernando Pereira. 2006. Online Learning of Approximate Dependency Parsing Algorithms. In Proceedings of the 11th Conference of the European Chapter of the Association for Computational Linguistics (EACL-06), pages 81–88. find our card
14. Dominick Ng and James R. Curran. 2012. Dependency Hashing for n-best CCG Parsing. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL-12), pages 497–505. find
15. Joakim Nivre. 2009. Non-Projective Dependency Parsing in Expected Linear Time. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP (ACL-09), pages 351– 359. find
16. Joakim Nivre, Yoav Goldberg, and Ryan McDonald. 2014. Constrained arc-eager dependency parsing. Computational Linguistics, 40(2):249– 257. find
17. Joakim Nivre, Johan Hall, and Jens Nilsson. 2004. Memory-Based Dependency Parsing. In HLT-NAACL 2004 Workshop: Eighth Conference on Computational Natural Language Learning (CoNLL-04), pages 49–56. find
18. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 Shared Task on Parsing the Web. In Notes of the First Workshop on the Syntactic Analysis of Non-Canonical Language (SANCL-12). find
19. Adwait Ratnaparkhi. 1996. A Maximum Entropy Model for Part-of-Speech Tagging. In Proceedings of the 1996 Conference on Empirical Methods in Natural Language Processing (EMNLP-96), pages 133–142. find
20. Alexander Rush and Slav Petrov. 2012. Vine Pruning for Efficient Multi-Pass Dependency Parsing. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-12), pages 498– 507. find
21. Anders Søgaard. 2013. An Empirical Study of Differences between Conversion Schemes and Annotation Guidelines. In Proceedings of the 2nd International Conference on Dependency Linguistics, pages 298–307. find our card
22. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A Large Training Corpus for Enhanced Processing. In Joseph Olive, Caitlin Christianson, and John McCary, editors, Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation, pages 54–63. Springer. find
23. Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical Dependency Analysis with Support Vector Machines. In Proceedings of the 8th International Workshop of Parsing Technologies (IWPT-03), pages 196–206. find our card
24. Yue Zhang, Byung-Gyu Ahn, Stephen Clark, Curt Van Wyk, James R. Curran, and Laura Rimell. 2010. Chart Pruning for Fast Lexicalised-Grammar Parsing. In Proceedings of the 23rd International Conference on Computational Linguistics (COLING-10), pages 1471– 1479. find
25. Yue Zhang and Stephen Clark. 2011. Syntactic Processing Using the Generalized Perceptron and Beam Search. Computational Linguistics, 37(1):105–151. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2015-ng-identifying-cascading-errors-constraints-dependency/