Mova0.5 pre-release⌕
☰
home / docs / papers / English / The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency Parsers

The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency Parsers

What it is about

The Second Extrinsic Parser Evaluation (EPE 2018) runs the basic UD trees of the English parsers of seventeen participating teams through three downstream systems: biological event extraction (TEES), negation resolution (Sherlock) and opinion analysis. The question is how far intrinsic scores predict downstream utility. The best 2018 system (UDPipe-Future) reaches downstream F1 of 49.66, 58.45 and 56.19, below the 2017 best system, which used enhanced UD on about 1.7 million tokens of training text, so the two campaigns are not strictly comparable. The correlation analysis is the main result: the average downstream rank correlates most with intrinsic UPOS and MLAS (0.88), event extraction with UPOS (0.71), negation with XPOS (0.75), and opinion analysis with MLAS (0.83). Parsers that predict no fine-grained tags (Uppsala-18) do well on events and opinion but poorly on negation.

Ideas

For Mova

The chapters are 20. Syntax: the Universal Dependencies frame and 37-register-and-variation.md for the claim that intrinsic scores are not enough. The code is en/src/ud.rs (the UPOS and feature tables), en/src/ptag.rs, and the prag/world layer as the natural downstream check. Measure UPOS and MLAS first, because they are the cheap predictors.

Topics: Dependency parsing

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/k18-2002

@inproceedings{fares20182018,
  title = {{The 2018 Shared Task on Extrinsic Parser Evaluation: On the Downstream Utility of English Universal Dependency Parsers}},
  author = {Fares, Murhaf and Oepen, Stephan and Øvrelid, Lilja and Björne, Jari and Johansson, Richard},
  booktitle = {Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies},
  year = {2018},
  doi = {10.18653/v1/k18-2002},
  url = {https://doi.org/10.18653/v1/k18-2002}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Miguel Ballesteros, Bernd Bohnet, Simon Mille, and Leo Wanner. 2015. Data-driven deep-syntactic dependency parsing. Natural Language Engineering 22:1 – 36. find
  2. Thorsten Joachims. 1999. Making large-scale SVM learning practical. In Bernhard Schölkopf, Christopher J. C. Burges, and Alexander J. Smola, editors, Advances in Kernel Methods. Support Vector Learning, MIT Press, Cambridge, MA, USA, page 41 – 56. find
  3. Richard Johansson. 2017. EPE 2017: The Trento– Gothenburg opinion extraction system. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 31 – 39. find
  4. Richard Johansson and Alessandro Moschitti. 2013. Relational features in fine-grained opinion analysis. Computational Linguistics 39(3):473 – 509. find
  5. Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshi-nobu Kano, and Jun’ichi Tsujii. 2009. Overview of BioNLP’09 Shared Task on event extraction. In Proceedings of the Workshop on Current Trends in Biomedical Natural Language Processing. Shared Task. Boulder, CO, USA, page 1 – 9. find
  6. Marco Kuhlmann and Stephan Oepen. 2016. Towards a catalogue of linguistic graph banks. Computational Linguistics 42(4):819 – 827. find
  7. Emanuele Lapponi, Stephan Oepen, and Lilja Øvrelid. 2017. EPE 2017: The Sherlock negation resolution downstream application. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 25 – 30. find
  8. Jari Björne. 2014. Biomedical Event Extraction with Machine Learning. Ph.D. thesis, University of Turku, Turku, Finland. find
  9. Yufei Chen, Junjie Cao, Weiwei Sun, and Xiaojun Wan. 2017. Peking at EPE 2017: A comparison of tree approximation, transition-based, and maximum subgraph models for semantic dependency analysis. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 60 – 64. find
  10. Emanuele Lapponi, Erik Velldal, Lilja Øvrelid, and Jonathon Read. 2012. UiO2. Sequence-labeling negation using dependency features. In Proceedings of the 1st Joint Conference on Lexical and Computational Semantics. Montréal, Canada, page 319 – 327. find
  11. Thomas Lavergne, Olivier Cappé, and François Yvon. 2010. Practical very large scale CRFs. In Proceedings of the 48th Meeting of the Association for Computational Linguistics. Uppsala, Sweden, page 504 – 513. find
  12. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th International Conference on Language Resources and Evaluation. Genoa, Italy, page 449 – 454. find our card
  13. Angelina Ivanova, Stephan Oepen, Lilja Øvrelid, and Dan Flickinger. 2012. Who did what to whom? A contrastive study of syntacto-semantic dependencies. In Proceedings of the 6th Linguistic Annotation Workshop. Jeju, Republic of Korea, page 2 – 11. find
  14. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, and Oscar Täckström. 2013. Universal dependency annotation for multilingual parsing. In Proceedings of the 51th Meeting of the Association for Computational Linguistics. Sofia, Bulgaria, page 92 – 97. find
  15. Tao Ji, Yuekun Yao, Qi Zheng, Yuanbin Wu, and Man Lan. 2017. ECNU at EPE 2017: Universal dependencies representations parser. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Simon Mille, Roberto Carlini, Ivan Latorre, and Leo Wanner. 2017. UPF at EPE 2017: Transduction-based deep analysis. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Milan Straka, Jana Straková, and Jan Hajič. 2017. Prague at EPE 2017: The UDPipe system. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 65 – 74. Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 76 – 84. find
  16. Yusuke Miyao. 2006. From Linguistic Theory to Syntactic Analysis. Corpus-Oriented Grammar Development and Feature Forest Model. Doctoral dissertation, University of Tokyo, Tokyo, Japan. find
  17. Roser Morante and Eduardo Blanco. 2012. *SEM 2012 Shared Task. Resolving the scope and focus of negation. In Proceedings of the 1st Joint Conference on Lexical and Computational Semantics. Montréal, Canada, page 265 – 274. find
  18. Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluís Màrquez, and Joakim Nivre. 2008. find
  19. The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies. In Proceedings of the 12th Conference on Natural Language Learning. Manchester, UK, page 159 – 177. find our card
  20. Roser Morante and Walter Daelemans. 2012. ConanDoyle-neg. Annotation of negation in Conan Doyle stories. In Proceedings of the 8th International Conference on Language Resources and Evaluation. Istanbul, Turkey, page 1563 – 1568. find
  21. Zsolt Szántó and Richárd Farkas. 2017. Szeged at EPE 2017: First experiments in a generalized syntactic parsing framework. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 75 – 79. find
  22. Joakim Nivre, Marie-Catherine de Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajič, Christopher D. Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo, Natalia Silveira, Reut Tsarfaty, and Daniel Zeman. 2016. Universal Dependencies v1. A multilingual treebank collection. In Proceedings of the 10th International Conference on Language Resources and Evaluation. Portorož, Slovenia, page 1659 – 1666. find our card
  23. Yuka Tateisi, Akane Yakushiji, Tomoko Ohta, and Jun’ichi Tsujii. 2005. Syntax annotation for the GE-NIA corpus. In Proceedings of the 2nd International Joint Conference on Natural Language Processing. Companion Volume. Jeju, Korea. find
  24. Stephan Oepen, Lilja Øvrelid, Jari Björne, Richard Johansson, Emanuele Lapponi, Filip Ginter, and Erik Velldal. 2017. The 2017 Shared Task on Extrinsic Parser Evaluation. Towards a reusable community infrastructure. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 1 – 16. find
  25. Erik Velldal, Lilja Øvrelid, Jonathon Read, and Stephan Oepen. 2012. Speculation and negation: Rules, rankers, and the role of syntax. Computational Linguistics 38(2):369 – 410. find
  26. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation 39(2-3):165 – 210. find
  27. Hao Peng, Sam Thomson, and Noah A. Smith. 2017. Deep multitask learning for semantic dependency parsing. In Proceedings of the 55th Meeting of the Association for Computational Linguistics. Vancouver, Canada, page 2037 – 2048. find
  28. Daniel Zeman, Jan Hajič, Martin Popel, Martin Potthast, Milan Straka, Filip Ginter, Joakim Nivre, and Slav Petrov. 2018. CoNLL 2018 Shared Task. Multilingual Parsing from Raw Text to Universal Dependencies. In Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies. Brussels, Belgium, page 1 – 20. find our card
  29. Martin Potthast, Tim Gollub, Francisco Rangel, Paolo Rosso, Efstathios Stamatatos, and Benno Stein. 2014. Improving the reproducibility of PAN’s shared tasks. Plagiarism detection, author identification, and author profiling. In Evangelos Kanoulas, Mihai Lupu, Paul Clough, Mark Sanderson, Mark Hall, Allan Hanbury, and Elaine Toms, editors, Information Access Evaluation meets Multilinguality, Multimodality, and Visualization. 5th International Conference of the CLEF Initiative. Springer, Berlin, Germany, page 268 – 299. find
  30. Sebastian Schuster, Eric De La Clergerie, Marie Candito, Benoît Sagot, Christopher D. Manning, and Djamé Seddah. 2017. Paris and Stanford at EPE 2017: Downstream evaluation of graph-based dependency representations. In Proceedings of the 2017 Shared Task on Extrinsic Parser Evaluation at the Fourth International Conference on Dependency Linguistics and the 15th International Conference on Parsing Technologies. Pisa, Italy, page 47 – 59. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/ud-2018-fares-2018-shared-task-extrinsic-parser/