CoNLL-X Shared Task on Multilingual Dependency Parsing

What it is about

The shared task asked for dependency parsers for 13 languages, trained and tested on the same data converted into one ten-column format (the parser predicts HEAD and DEPREL). The test set in each language was made to hold about 5,000 scoring tokens, and the score was labelled attachment, with punctuation tokens left out. Nineteen groups submitted parsed test data, and LAS ranged from 37.8% on Turkish to 91.7% on Japanese. The paper describes the conversion of each treebank, the evaluation and the approaches used, and draws conclusions about which languages, treebanks and annotation choices are hard for any dependency parser, including the role of morphology, data size and non-projectivity.

Ideas

• I1. en ud-eval: check that the evaluation excludes exactly the tokens the official script excludes (FORM made only of Unicode punctuation, with a test file that includes the characters underscore and percent sign), and compare en ud-eval with the official script on EWT dev; the scores must match.
• I2. en ud-eval: when comparing English treebanks, cut each test set to the same number of scoring tokens (about 5,000, as in this task) so that differences in LAS are not driven by test size.
• I3. Use the per-language spread of the shared task as a stress set: report en LAS on the least and most morphologically rich languages only as context, and never as a claim about English.

For Mova

The ideas touch the scorer behind en ud-eval and how Mova reports results across treebanks. The chapter that gives the reporting context is languages/en/08-national-varieties, since treebanks differ in their annotation choices. Measure first I1, a scorer check that costs nothing and makes every later LAS number comparable.

Topics: Dependency parsing · Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1596276.1596305

@inproceedings{buchholz2006conll,
  title = {{CoNLL-X Shared Task on Multilingual Dependency Parsing}},
  author = {Buchholz, Sabine and Marsi, Erwin},
  booktitle = {Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X)},
  year = {2006},
  doi = {10.3115/1596276.1596305},
  url = {https://doi.org/10.3115/1596276.1596305}
}

References

60 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. A. Arun and F. Keller. 2005. Lexicalization in crosslinguistic probabilistic parsing: The case of French. In Proc. of the 43rd Annual Meeting of the ACL. find
2. D. Bikel. 2002. Design of a multi-lingual, parallel-processing statistical parsing engine. In Proc. of the Human Language Technology Conf. (HLT). find
3. E. Black, S. Abney, D. Flickenger, et al. 1991. A procedure for quantitatively comparing the syntactic coverage of English grammars. In Speech and Natural Language: Proceedings of a Workshop Held at Pacific Grove, California. find
4. E. Black, F. Jelinek, J. Lafferty, D. Magerman, R. Mercer, and S. Roukos. 1993. Towards history-based grammars: Using richer models for probabilistic parsing. In Proc. of the 31rd Annual Meeting of the ACL. find
5. S. Buchholz and D. Green. 2006. Quality control of treebanks: documenting, converting, patching. In LREC 2006 workshop on Quality assurance and quality measurement for language and speech resources. 34 For the Swedish Talbanken05 corpus, that work has been done after the shared task (see the treeban… find
6. A. Chanev, K. Simov, P. Osenova, and S. Marinov. 2006. Dependency conversion and parsing of the BulTreeBank. In Proc. of the LREC-Workshop Merging and Layering Linguistic Information. find
7. Y. Cheng, M. Asahara, and Y. Matsumoto. 2005. Chinese deterministic dependency analyzer: Examining effects of global features and root node finder. In Proc. of SIGHAN-2005. find
8. Y.J. Chu and T.H. Liu. 1965. On the shortest arborescence of a directed graph. Science Sinica, 14:1396–1400. find
9. M. Collins, J. Hajic, L. Ramshaw, and C. Tillmann. 1999. A statistical parser for Czech. In Proc. of the 37th Annual Meeting of the ACL. find
10. M. Collins. 1996. A new statistical parser based on bigram lexical dependencies. In Proc. of the 34th Annual Meeting of the ACL. find
11. M. Collins. 1997. Three generative, lexicalised models for statistical parsing. In Proc. of the 35th Annual Meeting of the ACL. find
12. M. Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania. find
13. B. Cowan and M. Collins. 2005. Morphology and reranking for the statistical parsing of Spanish. In Proc. of the Joint Conf. on Human Language Technology and Empirical Methods in Natural Language Processing (HLT/EMNLP). find
14. D. Csendes, J. Csirik, and T. Gyimóthy. 2004. The Szeged corpus: a POS tagged and syntactically annotated Hungarian natural language corpus. In Proc. of the 5th Intern. Workshop on Linguistically Interpreteted Corpora (LINC). find
15. W. Daelemans, J. Zavrel, P. Berck, and S. Gillis. 1996. MBT: A memory-based part of speech tagger-generator. In Proc. of the 4th Workshop on Very Large Corpora (VLC). find
16. A. Dubey and F. Keller. 2003. Probabilistic parsing for German using sister-head dependencies. In Proc. of the 41st Annual Meeting of the ACL. find
17. J. Edmonds. 1967. Optimum branchings. Journal of Research of the National Bureau of Standards, 71B:233–240. find
18. J. Einarsson. 1976. Talbankens skriftspråkskonkordans. find
19. J. Eisner. 1996. Three new probabilistic models for dependency parsing: An exploration. In Proc. of the 16th Intern. Conf. on Computational Linguistics (COLING), pages 340– 345. find our card
20. D. Elworthy. 2000. A finite-state parser with dependency structure output. In Proc. of the 6th Intern. Workshop on Parsing Technologies (IWPT). find
21. H. Gaifman. 1965. Dependency systems and phrase-structure systems. Information and Control, 8:304–337. find
22. D. Hays. 1964. Dependency theory: A formalism and some observations. Language, 40:511–525. find
23. R. Hudson. 1984. Word Grammar. Blackwell. find
24. T. Kudo and Y. Matsumoto. 2000. Japanese dependency structure analysis based on support vector machines. In Proc. of the Joint Conf. on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC). find
25. S. Kurohashi and M. Nagao. 1994. KN parser: Japanese dependency/case structure analyzer. In Proceedings of the Workshop on Sharable Natural Language, pages 48–55. find
26. S. Kurohashi and M. Nagao. 1997. Kyoto University text corpus project. In Proc. of the 5th Conf. on Applied Natural Language Processing (ANLP), pages 115–118. find
27. Y. Lepage, S. Ando, S. Akamine, and H. Iida. 1998. An annotated corpus in Japanese using Tesnière’s structural syntax. In ACL-COLING Workshop on Processing of Dependency-Based Grammars, pages 109–115. find
28. D. Lin. 1995. A dependency-based method for evaluating broad-coverage parsers. In Proc. of the International Joint Conference on Artificial Intelligence (IJCAI). find
29. D. Magerman. 1995. Statistical decision-tree models for parsing. In Proc. of the 33rd Annual Meeting of the ACL, pages 276–283. find
30. M. Marcus, G. Kim, M. Marcinkiewicz, R. Mac-Intyre, A. Bies, M. Ferguson, K. Katz, and B. Schasberger. 1994. The Penn treebank: Annotating predicate argument structure. In Proc. of the Workshop on Human Language Technology (HLT). find our card
31. S. Marinov and J. Nivre. 2005. A data-driven dependency parser for Bulgarian. In Proc. of the 4th Workshop on Treebanks and Linguistic Theories (TLT), pages 89–100. find
32. R. McDonald and F. Pereira. 2006. Online learning of approximate dependency parsing algorithms. In Proc. of the 11th Conf. of the European Chapter of the ACL (EACL). find our card
33. R. McDonald, K. Crammer, and F. Pereira. 2005a. Online large-margin training of dependency parsers. In Proc. of the 43rd Annual Meeting of the ACL. find our card
34. R. McDonald, F. Pereira, K. Ribarov, and J. Hajic. 2005b. Non-projective dependency parsing using spanning tree algorithms. In Proc. of the Joint Conf. on Human Language Technology and Empirical Methods in Natural Language Processing (HLT/EMNLP). find our card
35. I. Mel’čuk. 1988. Dependency Syntax: Theory and Practice. The SUNY Press, Albany, N.Y. find
36. J. Nivre and J. Nilsson. 2005. Pseudo-projective dependency parsing. In Proc. of the 43rd Annual Meeting of the ACL, pages 99–106. find
37. J. Nivre and M. Scholz. 2004. Deterministic dependency parsing of English text. In Proc. of the 20th Intern. Conf. on Computational Linguistics (COLING), pages 64–70. find our card
38. J. Nivre, J. Hall, and J. Nilsson. 2004. Memory-based dependency parsing. In Proc. of the 8th Conf. on Computational Natural Language Learning (CoNLL), pages 49–56. find
39. K. Oflazer. 1999. Dependency parsing with an extended finite state approach. In Proc. of the 37th Annual Meeting of the ACL, pages 254–260. find
40. G. Sampson. 1995. English for the Computer: The SUSANNE Corpus and analytic scheme. Clarendon Press. find
41. K. Sima’an, A. Itai, Y. Winter, A. Altman, and N. Nativ. 2001. Building a tree-bank of modern Hebrew text. In Journal Traitement Automatique des Langues (t.a.l.) — Special Issue on Natural Language Processing and Corpus Linguistics. find
42. D. Sleator and D. Temperley. 1993. Parsing English with a link grammar. In Proc. of the 3rd Intern. Workshop on Parsing Technologies (IWPT). find our card
43. P. Tapanainen and T. Järvinen. 1997. A non-projective dependency parser. In Proc. of the 5th Conf. on Applied Natural Language Processing (ANLP). find
44. U. Teleman, 1974. Manual för grammatisk beskrivning av talad och skriven svenska (MAMBA). find
45. L. Tesnière. 1959. Elément de syntaxe structurale. Klincksieck, Paris. find
46. H. Yamada and Y. Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proc. of the 8th Intern. Workshop on Parsing Technologies (IWPT), pages 195– 206. find our card
47. Papers by participants of CoNLL-X (this volume) find
48. G. Attardi. 2006. Experiments with a multilanguage non-projective dependency parser. find
49. E. Bick. 2006. LingPars, a linguistically inspired, language-independent machine learner for dependency treebanks. find
50. S. Canisius, T. Bogers, A. van den Bosch, J. Geertzen, and E. Tjong Kim Sang. 2006. Dependency parsing by inference over high-recall dependency predictions. find
51. M. Chang, Q. Do, and D. Roth. 2006. A pipeline model for bottom-up dependency parsing. find
52. S. Corston-Oliver and A. Aue. 2006. Dependency parsing with reference to Slovene, Spanish and Swedish. find
53. M. Dreyer, D. Smith, and N. Smith. 2006. Vine parsing and minimum risk reranking for speed and precision. find
54. R. Johansson and P. Nugues. 2006. Investigating multilingual dependency parsing. find
55. R. McDonald, K. Lerman, and F. Pereira. 2006. Multilingual dependency analysis with a two-stage discriminative parser. find
56. J. Nivre, J. Hall, J. Nilsson, G. Eryiǧit, and S. Marinov. 2006. Labeled pseudo-projective dependency parsing with support vector machines. find
57. S. Riedel, R. Çakıcı, and I. Meza-Ruiz. 2006. Multi-lingual dependency parsing with incremental integer linear programming. find
58. M. Schiehlen and K. Spranger. 2006. Language independent probabilistic context-free parsing bolstered by machine learning. find
59. N. Shimizu. 2006. Maximum spanning tree algorithm for non-projective labeled dependency parsing. find
60. D. Yuret. 2006. Dependency parsing as a classication problem. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/parsers-2006-buchholz-conll-x-shared-task-multilingual/