The CoNLL-2009 Shared Task: Syntactic and Semantic Dependencies in Multiple Languages

What it is about

This is the overview of the 2009 CoNLL shared task on joint syntactic dependency parsing and semantic role labelling. It extends the 2008 English task to six more languages (Catalan, Chinese, Czech, German, Japanese, Spanish), with one unified dependency format. Participants chose a joint task (parse and label arguments) or an SRL-only task (parses supplied). Each task had a closed challenge (only the given training data) and an open one (any resources). For three languages (Czech, English, German) out-of-domain test data were added. Twenty systems took part in the closed challenge: 13 in the joint task and 7 in SRL-only. The best macro-averaged joint F1 in the closed challenge was 82.64 (across languages), and on English it was 87.69. The paper's own contributions are the data conversion for the new languages, the evaluation rules, and a summary of the approaches; only four systems tried a true joint learning formulation.

Ideas

• I1. Report every Mova measurement under the paper's two-challenge labels: closed (only the school and grammar data in the repo) and open (any external tool). The parse is then never mixed with a resource that the closed run does not have. Keep the label in the result file of tools/jobs so a gain can always be traced to its resources.
• I2. Split the error source as the joint and SRL-only tasks do: run the frames reader (prag/src/frames.rs) on gold trees and on Mova's own parse, and report both. The gap between the two numbers is the cost of the parser, the rest is the frames rule.
• I3. Use an out-of-domain split from the start: the Early Modern texts in languages/en-x-emode are a natural out-of-domain test for the modern English rules. Report the in-domain and out-of-domain scores side by side, as the paper does for Czech, English and German.
• I4. Keep the paper's rule that a system is scored only on the words it was asked to label (the predicates were given). For Mova's roles, score only the verbs the lesson marks as events, so a missing predicate is not counted as a role error. Run it twice to check the numbers are identical.

For Mova

The chapters are 20. Syntax: the Universal Dependencies frame (the dependency view), 13. Verbs (predicates) and 28. Voice and the passive (role mapping under voice). The code is en parse-eval (en/src/parse.rs, LAS/UAS on UD EWT), prag/src/roles.rs for the role labels, tools/parsecmp for the parser comparison, and languages/en-x-emode with tools/gutenberg for the out-of-domain set. The measures are LAS with the gold-tree and parsed-tree split, and the in-domain and out-of-domain figures, reported with the closed or open label.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{hajic2009conll,
  title = {{The CoNLL-2009 Shared Task: Syntactic and Semantic Dependencies in Multiple Languages}},
  author = {Hajič, Jan and Ciaramita, Massimiliano and Johansson, Richard and Kawahara, Daisuke and Martí, Maria Antònia and Màrquez, Lluís and Meyers, Adam and Nivre, Joakim and Padó, Sebastian and Štěpánek, Jan and Straňák, Pavel and Surdeanu, Mihai and Xue, Nianwen and Zhang, Yi},
  booktitle = {Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL 2009): Shared Task},
  year = {2009},
  url = {https://aclanthology.org/W09-1201/}
}

References

32 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Sabine Brants, Stefanie Dipper, Silvia Hansen, Wolfgang Lezius, and George Smith. 2002. The TIGER treebank. In Proceedings of the Workshop on Treebanks and Linguistic Theories, Sozopol. find
2. Aljoscha Burchardt, Katrin Erk, Anette Frank, Andrea Kowalski, Sebastian Padó, and Manfred Pinkal. 2006. The SALSA corpus: a German corpus resource for lexical semantics. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC-2006), Genoa, Italy. find
3. Xavier Carreras. 2007. Experiments with a higher-order projective dependency parser. In Proceedings of EMNLP-CoNLL 2007, pages 957–961, June. Prague, Czech Republic. find
4. Montserrat Civit, M. Antònia Martı́, and Núria Bufı́. 2006. Cat3LB and Cast3LB: from constituents to dependencies. In Proceedings of the 5th International Conference on Natural Language Processing, FinTAL, pages 141–153, Turku, Finland. Springer Verlag, LNAI 4139. find
5. Qifeng Dai, Enhong Chen, and Liu Shi. 2009. An iterative approach for joint dependency parsing and semantic role labeling. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL-2009), June 4-5, Boulder, Colorado, USA. June 4-5. find
6. Jason Eisner. 2000. Bilexical grammars and their cubic-time parsing algorithms. In Harry Bunt and Anton Nijholt, editors, Advances in Probabilistic and Other Parsing Tehcnologies, pages 29–62. Kluwer Academic Publishers. find
7. Christiane Fellbaum, editor. 1998. WordNet: An Electronic Lexical Database. The MIT Press, Cambridge. find
8. Andrea Gesmundo, James Henderson, Paola Merlo, and Ivan Titov. 2009. A latent variable model of synchronous syntactic-semantic parsing for multiple languages. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL-2009), June 4-5, Boulder, Colorado, USA. June 4-5. find
9. Jan Hajič, Jarmila Panevová, Zdeňka Urešová, Alevtina Bémová, Veronika Kolářová-Řeznı́čková‘, and Petr Pajas. 2003. PDT-VALLEX: Creating a Large-coverage Valency Lexicon for Treebank Annotation. In J. Nivre and E. Hinrichs, editors, Proceedings of The Second Workshop on Treebanks and Linguistic Theories, pages 57–68, Vaxjo, Sweden. Vaxjo University Press. find
10. Jan Hajič, Jarmila Panevová, Eva Hajičová, Petr Sgall, Petr Pajas, Jan Štěpánek, Jiřı́ Havelka, Marie Mikulová, and Zdeněk Žabokrtský. 2006. Prague Dependency Treebank 2.0. find
11. Daisuke Kawahara, Sadao Kurohashi, and Kôiti Hasida. 2002. Construction of a Japanese relevance-tagged corpus. In Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC-2002), pages 2008–2013, Las Palmas, Canary Islands. find
12. Xavier Lluı́s, Stefan Bott, and Lluı́s Màrquez. 2009. A second-order joint eisner model for syntactic and semantic dependency parsing. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL-2009), June 4-5, Boulder, Colorado, USA. June 4-5. find
13. M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz. 1994. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2):313–330. find our card
14. Lluı́s Màrquez, Luis Villarejo, M. Antònia Martı́, and Mariona Taulé. 2007. SemEval-2007 Task 09: Multilevel semantic annotation of catalan and spanish. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 42–47, Prague, Czech Republic. find
15. M. Antònia Martı́, Mariona Taulé, Lluı́s Màrquez, and Manu Bertran. 2007. Anotación semiautomática con papeles temáticos de los corpus CESS-ECE. Procesamiento del Lenguaje Natural, SEPLN Journal, 38:67–76. find
16. Ryan McDonald, Fernando Pereira, Jan Hajič, and Kiril Ribarov. 2005. Non-projective dependency parsing using spanning tree algortihms. In Proceedings of NAACL-HLT’05, Vancouver, Canada, pages 523–530. find
17. A. Meyers, R. Reeves, C. Macleod, R. Szekely, V. Zielinska, B. Young, and R. Grishman. 2004. The Nom-Bank Project: An Interim Report. In NAACL/HLT 2004 Workshop Frontiers in Corpus Annotation, Boston. find
18. Roser Morante, Vincent Van Asch, and Antal van den Bosch. 2009. A simple generative pipeline approach to dependency parsing and semantic role labeling. In Proceedings of the 13th Conference on Computational Natural Language Learning (CoNLL-2009), Boulder, Colorado, USA. June 4-5. find
19. Joakim Nivre, Johann Hall, Sandra Kübler, Ryan Mc-Donald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. The conll 2007 shared task on dependency parsing. In Proceedings of the EMNLP-CoNLL 2007 Conference, pages 915–932, Prague, Czech Republic. find
20. Sebastian Pado and Mirella Lapata. 2005. Cross-lingual projection of role-semantic information. In Proceedings of the Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP-2005), pages 859– 866, Vancouver, BC. find
21. Petr Pajas and Jan Štěpánek. 2008. Recent advances in a feature-rich framework for treebank annotation. In The 22nd International Conference on Computational Linguistics - Proceedings of the Conference (COL-ING’08), pages 673–680, Manchester. find
22. M. Palmer, D. Gildea, and P. Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. find our card
23. Helmut Schmid. 1994. Probabilistic part-of-speech tagging using decision trees. In Proceedings of International Conference on New Methods in Language Processing. find
24. Drahomı́ra ”Johanka” Spoustová, Jan Hajič, Jan Raab, and Miroslav Spousta. 2009. Semi-supervised training for the averaged perceptron POS tagger. In Proceedings of the European ACL Cenference EACL’09, Athens, Greece. find
25. Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluı́s Màrquez, and Joakim Nivre. 2008. The CoNLL-2008 shared task on joint parsing of syntactic and semantic dependencies. In Proceedings of the 12th Conference on Computational Natural Language Learning (CoNLL-2008), pages 159–177. find our card
26. Mariona Taulé, Maria Antònia Martı́, and Marta Recasens. 2008. AnCora: Multilevel Annotated Corpora for Catalan and Spanish. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC-2008), Marrakesh, Morroco. find
27. Martin Čmejrek, Jan Cuřı́n, Jan Hajič, Jiřı́ Havelka, and Vladislav Kuboň. 2004. Prague Czech-English Dependency Treebank: Syntactically Anntoated Resources for Machine Translation. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC-2004), pages 1597– 1600, Lisbon, Portugal. find
28. W. N. Francis and H. Kucera. 1964. Brown Corpus Manual of Information to accompany A Standard Corpus of Present-Day Edited American English, for use with Digital Computers. Revised 1971, Revised and Amplified 1979, available at www.clarinet/brown. find
29. R. Weischedel and A. Brunstein. 2005. BBN pronoun coreference and entity type corpus. Technical report, Lin- guistic Data Consortium. find
30. Nianwen Xue and Martha Palmer. 2009. Adding semantic roles to the Chinese Treebank. Natural Language Engineering, 15(1):143–172. find
31. Nianwen Xue, Fei Xia, Fu Dong Chiou, and Martha Palmer. 2005. The Penn Chinese TreeBank: Phrase Structure Annotation of a Large Corpus. Natural Language Engineering, 11(2):207–238. find
32. Andrea Zielinski and Christian Simon. 2008. Morphisto: An open-source morphological analyzer for german. In Proceedings of the Conference on Finite State Methods in Natural Language Processing. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/hajic-2009-conll-2009-shared-task-syntactic/