An Empirical Study of Differences between Conversion Schemes and Annotation Guidelines

What it is about

Many dependency treebanks are converted from constituency treebanks, and different conversion schemes make different choices about heads and relation labels. The author measures how far such schemes differ with a generalized tree-edit distance, and compares them with learnability and with downstream performance. Experiments convert the Penn Treebank with four schemes (Yamada, CoNLL, Stanford and LTH). Parser accuracy on section 23 differs clearly for the Stanford scheme (LAS 81.36) against 87.52 to 88.99 for the others, but the parser's own bias makes many differences invisible to the parser. Learnability of a scheme does not predict its downstream score, but the choice of scheme does change the results of downstream applications.

Ideas

• I1. Scheme noise floor for en: build two of the mova-dialect conversions of EWT that differ only in head choice, and report the LAS spread between them, so that every claimed en gain is compared with this spread rather than with zero.
• I2. Tree-edit agreement in treeagree: compute a generalized tree-edit distance between the silver trees from annot and the Stanza or spaCy trees per sentence, and use it as one more agreement gate beside R1 to R8 (measure how many sentences the new gate changes on EWT dev).
• I3. Do not use learnability as a selection rule: when choosing among conversions or among gparse checkpoints, select on downstream LAS (the prag/world output on ESLSpok) and not on training loss, as the paper finds loss does not track downstream results.

For Mova

The conversions are in en/src/convert.rs and the dialect in The Mova dialect: a UD metadialect, treebanks, and converters, the silver checks in tools/treeagree and the witnesses in tools/parsecmp. The chapter that the ideas touch is 20. Syntax: the Universal Dependencies frame. Measure first: the LAS spread between two conversions of the same EWT trees.

Topics: Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{sogaard2013empirical,
  title = {{An Empirical Study of Differences between Conversion Schemes and Annotation Guidelines}},
  author = {Søgaard, Anders},
  booktitle = {Proceedings of the Second International Conference on Dependency Linguistics (DepLing 2013)},
  year = {2013},
  url = {https://aclanthology.org/W13-3733/}
}

References

34 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bernd Bohnet. 2010. Top accuracy and fast dependency parsing is not a contradiction. In COLING. find
2. Jonathan H. Clark, Chris Dyer, Alon Lavie, and Noah A. Smith. 2011. Better hypothesis testing for statistical machine translation: controlling for optimizer instability. In ACL. find
3. Mike Collins. 1999. Head-driven statistical models for natural language parsing. Ph.D. thesis, University of Pennsylvania. find
4. Michael Collins. 2002. Discriminative training methods for Hidden Markov Models. In EMNLP. find
5. Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning, 20(3):273– 297. find
6. Koby Crammer and Yoram Singer. 2003. Ultraconservative algorithms for multiclass problems. In JMLR. find
7. Jakob Elming and Martin Haulrich. 2011. Reordering by parsing. In Proceedings of International Workshop on Using Linguistic Information for Hybrid Machine Translation (LIHMT-2011). find
8. Jakob Elming, Anders Johannsen, Sigrid Klerke, Emanuele Lapponi, Hector Martinez Alonso, and Anders Søgaard. 2013. Down-stream effects of tree-to-dependency conversions. In NAACL. find
9. Michel Galley and Christopher Manning. 2009. Quadratic-time dependency parsing for machine translation. In ACL. find
10. Angelina Ivanova, Stephan Oepen, Lilja Øvrelid, and Dan Flickinger. 2012. Who did what to whom? a contrastive study of syntactico-semantic dependencies. In LAW. find
11. Richard Johansson and Alessandro Moschitti. 2010. Syntactic and semantic structure for opinion expression detection. In CoNLL. find
12. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to-dependency conversion for English. In NODALIDA. find our card
13. Richard Johansson and Pierre Nugues. Dependency-based syntactic-semantic analysis with propbank and nombank. In CoNLL. find
14. Richard Johansson. 2013. Training parsers on incompatible treebanks. In NAACL. find
15. Mahesh Joshi and Carolyn Penstein-Rose. 2009. Generalizing dependency features for opinion mining. In ACL. find
16. Kevin Knight and Daniel Marcu. 2002. Summarization beyond sentence extraction: a probabilistic approach to sentence compression. Artificial Intelligence, 139:91–107. find
17. Emanuele Lapponi, Erik Velldal, Lilja Øvrelid, and Jonathon Read. 2012. UiO2: Sequence-labeling negation using dependency features. In *SEM. find
18. Alon Lavie and Abhaya Agarwal. 2007. Meteor: an automatic metric for mt evaluation with high levels of correlation with human judgments. In WMT. find
19. Wei-Hao Lin and Alexander Hauptmann. 2006. Are these documents written from different perspectives? In COLING-ACL. find
20. Mitchell Marcus, Mary Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
21. Ryan McDonald and Joakim Nivre. 2007. Characterizing the errors of data-driven dependency parsers. In EMNLP-CoNLL. find
22. Ryan McDonald. 2006. Discriminative sentence compression with soft syntactic evidence. In EACL. find
23. Roser Morante and Eduardo Blanco. 2012. *sem 2012 shared task: Resolving the scope and focus of negation. In *SEM. find
24. Roser Morante and Caroline Sporleder. 2012. Modality and negation: An introduction to the special issue. Computational linguistics, 38(2):223–260. find
25. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan Mc-Donald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. find
26. The CoNLL 2007 Shared Task on Dependency Parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915–932, Prague, Czech Republic. find
27. Kishore Papineni, Salim Roukus, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania. find our card
28. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 Shared Task on Parsing the Web. In Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL). find
29. Roy Schwartz, Omri Abend, and Ari Rappoport. 2012. Learnability-based syntactic annotation design. In 2008. COLING. find
30. Anders Søgaard and Martin Haulrich. 2010. On the derivation perplexity of treebanks. In TLT. find
31. Reut Tsarfaty, Joakim Nivre, and Evelina Andersson. 2012. Cross-framework evaluation for statistical parsing. In EACL. find
32. Erik Velldal, Lilja Øvrelid, Jonathon Read, and Stephan Oepen. 2012. Speculation and negation: Rules, rankers, and the role of synta. Computational linguistics, 38(2):369–410. find
33. Peng Xu, Jaeho Kang, Michael Ringgaard, and Franz Och. 2009. Using a dependency parser to improve SMT for subject-object-verb languages. In Proceedings of the North American Chapter of the Association for Computational Linguistics - Human Language Technologies (NAACL-HLT) 2009, pages 245–253, Boulde… find
34. Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proceedings of the 8th International Workshop on Parsing Technologies, pages 195–206, Nancy, France. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/bank-2013-sogaard-empirical-study-differences-between-conversion/