Mova0.5 pre-release⌕
☰
home / docs / papers / English / Evaluating Dependency Parsing: Robust and Heuristics-Free Cross-Annotation Evaluation

Evaluating Dependency Parsing: Robust and Heuristics-Free Cross-Annotation Evaluation

What it is about

Attachment scores compare parsers only within one treebank scheme. Two parsers trained on different conversions of the Penn Treebank can differ mainly because of annotation choices, such as the head of a prepositional phrase or of a copula. The authors propose TEDEVAL: each dependency tree is mapped by deterministic unification steps into a common functional tree, the mapped trees are generalized to remove scheme-specific choices, and a tree edit distance scores the hypothesis against one or several gold standards. For different conversions of the Penn Treebank, the ranking of parsers seen with LAS changes or disappears once the outputs are normalized. The paper's conclusion is that a score difference across experiments can be a difference in annotation, not in parsing.

Ideas

For Mova

Touches 08. National varieties and 20. Syntax: the Universal Dependencies frame. The paper gives a principled answer to the question why Mova's cross-treebank numbers (ATIS 63.0, GENTLE 67.9) are not directly comparable with other systems. Measure I1 first on the existing outputs.

Topics: Dependency parsing · Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{tsarfaty2011evaluating,
  title = {{Evaluating Dependency Parsing: Robust and Heuristics-Free Cross-Annotation Evaluation}},
  author = {Tsarfaty, Reut and Nivre, Joakim and Andersson, Evelina},
  booktitle = {Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing},
  year = {2011},
  url = {https://aclanthology.org/D11-1036/}
}

References

36 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Philip Bille. 2005. A survey on tree edit distance and related. problems. Theoretical Computer Science, 337:217–239. find
  2. Ezra Black, Steven P. Abney, D. Flickenger, Claudia Gdaniec, Ralph Grishman, P. Harrison, Donald Hindle, Robert Ingria, Frederick Jelinek, Judith L. Klavans, Mark Liberman, Mitchell P. Marcus, Salim Roukos, Beatrice Santorini, and Tomek Strzalkowski. 1991. Procedure for quantitatively comparing the syntactic coverage of English grammars. In E. Black, editor, Proceedings of the workshop on Speech and Natural Language, HLT, pages 306–311. Association for Computational Linguistics. find
  3. Ted Briscoe, John Carroll, Jonathan Graham, and Ann Copestake. 2002. Relational evaluation schemes. In Proceedings of LREC Workshop“Beyond Parseval – Towards improved evaluation measures for parsing systems”. find
  4. Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of CoNLL-X, pages 149–164. find our card
  5. Ekaterina Buyko and Udo Hahn. 2010. Evaluating the impact of alternative dependency graph encodings on solving event extraction tasks. In Proceedings of EMNLP, pages 982–992. find
  6. John Carroll, Ted Briscoe, and Antonio Sanfilippo. 1998. Parser evaluation: a survey and a new proposal. In Proceedings of LREC, pages 447–454. find
  7. Daniel Cer, Marie-Catherine de Marneffe, Daniel Jurafsky, and Christopher D. Manning. 2010. Parsing to stanford dependencies: Trade-offs between speed and accuracy. In Proceedings of LREC. find our card
  8. Jinho D. Choi and Martha Palmer. 2010. Robust constituent-to-dependency conversion for English. In Proceedings of TLT. find
  9. Paul Cohen. 1995. Empirical Methods for Artificial Intelligence. The MIT Press. find
  10. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of LREC, pages 449–454. find our card
  11. Martin Emms. 2008. Tree-distance and some other variants of evalb. In Proceedings of LREC. find our card
  12. Erhard Hinrichs, Sandra Kübler, Karin Naumann, Heike Telljohan, and Julia Trushkina. 2004. Recent development in linguistic annotations of the TüBa-D/Z Treebank. In Proceedings of TLT. find
  13. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to-dependency conversion for English. In Proceedings of NODALIDA. find our card
  14. Sandra Kübler and Heike Telljohann. 2002. Towards a dependency-oriented evaluation for partial parsing. In Proceedings of LREC Workshop“Beyond Parseval – Towards improved evaluation measures for parsing systems”. find
  15. Sandra Kübler, Ryan McDonald, and Joakim Nivre. 2009. Dependency Parsing. Number 2 in Synthesis Lectures on Human Language Technologies. Morgan & Claypool Publishers. find
  16. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19:313–330. find our card
  17. Ryan McDonald and Fernando Pereira. 2006. Online learning of approximate dependency parsing algorithms. In Proceedings of EACL, pages 81–88. find our card
  18. Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Online large-margin training of dependency parsers. In Proceedings of ACL, pages 91–98. find our card
  19. Igor Mel’čuk. 1988. Dependency Syntax: Theory and Practice. State University of New York Press. find
  20. Yusuke Miyao, Rune Sætre, Kenji Sagae, Takuya Matsuzaki, and Jun’ichi Tsujii. 2008. Task-oriented evaluation of syntactic parsers and their representations. In Proceedings of ACL, pages 46–54. find our card
  21. Joakim Nivre and Beata Megyesi. 2007. Bootstrapping a Swedish Treebank using cross-corpus harmonization and annotation projection. In Proceedings of TLT. find
  22. Joakim Nivre and Jens Nilsson. 2005. Pseudo projective dependency parsing. In Proceeding of ACL, pages 99– 106. find
  23. Joakim Nivre and Mario Scholz. 2004. Deterministic dependency parsing of English text. In Proceedings of COLING, pages 64–70. find our card
  24. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan McDonald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007a. find
  25. The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915–932. find
  26. Joakim Nivre, Jens Nilsson, Johan Hall, Atanas Chanev, Gülsen Eryigit, Sandra Kübler, Svetoslav Marinov, and Erwin Marsi. 2007b. Maltparser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(1):1–41. find
  27. Joakim Nivre, Laura Rimell, Ryan McDonald, and Carlos Gómez-Rodrı́guez. 2010. Evaluation of dependency parsers on unbounded dependencies. pages 813–821. find our card
  28. Owen Rambow. 2010. The Simple Truth about Dependency and Phrase Structure Representations: An Opinion Piece. In Proceedings of HLT-ACL, pages 337– 340. find
  29. Ines Rehbein and Josef van Genabith. 2007. Why is it so difficult to compare treebanks? Tiger and TüBa-D/Z revisited. In Proceedings of TLT, pages 115–126. find
  30. Brian Roark. 2002. Evaluating parser accuracy using edit distance. In Proceedings of LREC Workshop“Beyond Parseval – Towards improved evaluation measures for parsing systems”. find
  31. Roy Schwartz, Omri Abend, Roi Reichart, and Ari Rappoport. 2011. Neutralizing linguistically problematic annotations in unsupervised dependency parsing evaluation. In Proceedings of ACL, pages 663–672. find
  32. Stuart M. Shieber. 1986. An Introduction to Unification-Based Grammars. Center for the Study of Language and Information. find
  33. Wojciech Skut, Brigitte Krenn, Thorsten Brants, and Hans Uszkoreit. 1997. An annotation scheme for free word-order languages. In Proceedings of the fifth conference on Applied natural language processing, pages 88–95. find
  34. Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proceeding of IWPT, pages 195–206. find our card
  35. Kaizhong Zhang and Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. In SIAM Journal of Computing, volume 18, pages 1245–1262. find
  36. Arnold M. Zwicky. 1993. Heads, bases, and functors. In G.G. Corbett, N. Fraser, and S. McGlashan, editors, Heads in Grammatical Theory, pages 292–315. Cambridge University Press. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2011-tsarfaty-evaluating-dependency-parsing-robust-heuristics/