The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies

What it is about

The 2008 CoNLL task joined syntactic dependency parsing and semantic role labelling in one dependency formalism. The English data convert the Penn Treebank constituents into dependencies, and the semantic layer covers both verbal predicates (PropBank) and nominal predicates (NomBank). Each system had to produce a syntactic dependency tree and the arguments of each predicate. The paper defines the subtasks, the closed and open challenges, and the evaluation measures: labelled attachment score, labelled F1 for the semantic dependencies, exact match of the whole sentence and perfect-proposition F1, which are much lower than the token scores. The main analysis is that nominal predicates are harder than verbal ones: the average gap in labelled F1 is 7.84 points on WSJ and Brown together, and 12.36 points on Brown alone, so nominal roles are more sensitive to domain change. Dividing the semantic F1 by the LAS gives a normalised score that isolates the role labeller from the parser. The paper also finds that wh-movement and split-clause links, which are non-projective, are picked out by simple patterns (a wh-word, a pair of commas).

Ideas

• I1. Separate models for verbal and nominal predicates: a verb frame reader and a noun frame reader in prag/src/roles.rs, with their own feature lists. The paper's two best systems did this. Measure the gap between the two readers on the school texts, in the same way the paper measures it on WSJ and Brown.
• I2. Non-projective arcs by pattern: a wh-word at the start of a clause, or a comma pair around a clause, is a candidate for a non-projective link in en/src/parse.rs. Measure the precision and recall of that rule on UD EWT dev, with the baseline of no non-projective arcs, and run it twice.
• I3. A normalised role score for Mova: divide the labelled role F1 by the attachment score, as the paper does, so that a frame error is not blamed on a parse error. Report both numbers in the same table as en parse-eval.
• I4. Report exact match over whole sentences and perfect propositions (all roles of a predicate right at once) next to the token scores. The paper shows that these figures are much lower, and a lesson-level claim should be judged on them.

For Mova

The chapters are 11. Nouns (nominal predicates, relational nouns), 13. Verbs (verbal predicates and argument structure), 25. Relative clauses and 26. Questions (wh-movement and the non-projective links). The code is en/src/parse.rs for the attachment and the non-projective rule, prag/src/roles.rs and prag/src/frames.rs for the role readers, and en parse-eval for the scores. The measures are the verb-noun gap on the school texts and the Early Modern texts, the non-projective precision and recall on UD EWT dev, and the normalised role score.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{surdeanu2008conll,
  title = {{The CoNLL 2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies}},
  author = {Surdeanu, Mihai and Johansson, Richard and Meyers, Adam and Màrquez, Lluís and Nivre, Joakim},
  booktitle = {CoNLL 2008: Proceedings of the Twelfth Conference on Computational Natural Language Learning},
  year = {2008},
  url = {https://aclanthology.org/W08-2121/}
}

References

40 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. X. Carreras. 2007. Experiments with a Higher-Order Projective Dependency Parser. In Proc. of CoNLL-2007 Shared Task. http://code.google.com/p/whatswrong/ ↗
2. X. Carreras and L. Màrquez. 2005. Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling. In Proc. of CoNLL-2005. find our card
3. X. Carreras and L. Màrquez. 2004. Introduction to the CoNLL-2004 Shared Task: Semantic Role Labeling. In Proc. of CoNLL-2004. find
4. W. Che, Z. Li, Y. Hu, Y. Li, B. Qin, T. Liu and S. Li. 2008. A Cascaded Syntactic and Semantic Dependency Parsing System. In Proc. of CoNLL-2008 Shared Task. find
5. E. Chen, L. Shi and D. Hu. 2008. Probabilistic Model for Syntactic and Semantic Dependency Parsing. In Proc. of CoNLL-2008 Shared Task. find
6. Chinchor, N. and P. Robinson. 1998. MUC-7 Named Entity Task Definition. In Proc. of Seventh Message Understanding Conference (MUC-7). http://www.itl.nist.gov/iaui/894.02/related projects/ /muc/proceedings/muc 7 toc.html. ↗
7. Chomsky, Noam. 1981. Lectures on Government and Binding. Foris Publications, Dordrecht. find
8. Y.J. Chu and T.H. Liu. 1965. On the Shortest Arborescence of a Directed Graph. In Science Sinica, 14:1396-1400. find
9. M. Ciaramita, G. Attardi, F. Dell’Orletta, and M. Surdeanu. 2008. DeSRL: A Linear-Time Semantic Role Labeling System. In Proc. of CoNLL-2008 Shared Task. find
10. M. Ciaramita and Y. Altun. 2006. Broad Coverage Sense Disambiguation and Information Extraction with a Supersense Sequence Tagger. In Proc. of EMNLP. find
11. M. Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania. find our card
12. J. Edmonds. 1967. Optimum Branchings. In Journal of Research of the National Bureau of Standards, 71B:233–240. find
13. J. Eisner. 2000. Bilexical Grammars and Their Cubic-Time Parsing Algorithms. New Developments in Parsing Algorithms, Kluwer Academic Publishers. find
14. W. N. Francis and H. Kuĉera. 1964. Brown Corpus. Manual of Information to accompany A Standard Corpus of Present-Day Edited American English, for use with Digital Computers. Revised 1971, Revised and Amplified 1979. find
15. C. Fellbaum, editor. 1998. WordNet: An electronic lexical database. MIT Press. find
16. J. Giménez and L. Màrquez. 2004. SVMTool: A general POS tagger generator based on Support Vector Machines. In Proc. of LREC. find
17. K. Hacioglu. 2004. Semantic Role Labeling Using Dependency Trees. In Proc. of COLING-2004. find
18. J. Henderson, P. Merlo, G. Musillo and I. Titov. 2008. A Latent Variable Model of Synchronous Parsing for Syntactic and Semantic Dependencies. In Proc. of CoNLL-2008 Shared Task. find
19. R. Johansson and P. Nugues. 2008. Dependency-based Syntactic–Semantic Analysis with PropBank and NomBank. In Proc. of CoNLL-2008 Shared Task. find
20. R. Johansson and P. Nugues. 2007. Extended Constituent-to-Dependency Conversion for English. In Proc. of NODALIDA. find our card
21. X. Lluı́s and L. Màrquez. 2008. A Joint Model for Parsing Syntactic and Semantic Dependencies. In Proc. of CoNLL-2008 Shared Task. find
22. D. Magerman. 1994. Natural Language Parsing as Statistical Pattern Recognition. Ph.D. thesis, Stanford University. find
23. M.P. Marcus, B. Santorini, and M.A. Marcinkiewicz. 1993. Building a Large Annotated Corpus of English: the Penn Treebank. Computational Linguistics, 19. find our card
24. R. McDonald, F. Pereira, K. Ribarov and J. Hajic. 2005. Non-Projective Dependency Parsing using Spanning Tree Algorithms In Proc. of HLT-EMNLP. find our card
25. A. Meyers, R. Grishman, M. Kosaka, and S. Zhao. 2001. Covering Treebanks with GLARF. In Proc. of the ACL/EACL 2001 Workshop on Sharing Tools and Resources for Research and Education. find
26. Meyers, A., R. Reeves, C. Macleod, R. Szekely, V. Zielinska, B. Young, and R. Grishman. 2004. The NomBank Project: An Interim Report. In NAACL/HLT 2004 Workshop Frontiers in Corpus Annotation, Boston. find
27. J. Nivre, J. Hall, J. Nilsson and G. Eryigit. 2006. Labeled Pseudo-Projective Dependency Parsing with Support Vector Machines. In Proc. of CoNLL-X Shared Task. find
28. J. Nivre, J. Hall, S. Kübler, R. McDonald, J. Nilsson, S. Riedel, D. Yuret. 2007. find
29. The CoNLL 2007 Shared Task on Dependency Parsing. In Proc. of CoNLL-2007. find
30. J. Nivre, J. Hall, J. Nilsson, A. Chanev, G. Eryiğit, S. Kübler, S. Marinov, and E. Marsi. 2007b. Malt-Parser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(2):95–135. find
31. M. Palmer, D. Gildea, and P. Kingsbury. 2005. The Proposition Bank: An Annotated Corpus of Semantic Roles. Computational Linguistics, 31(1). find our card
32. S. Riedel and I. Meza-Ruiz. 2008. Collective Semantic Role Labelling with Markov Logic. In Proc. of CoNLL-2008 Shared Task. find
33. Y. Samuelsson, O. Täckström, S. Velupillai, J. Eklund, M. Fishel and M. Saers. 2008. Mixing and Blending Syntactic and Semantic Dependencies. In Proc. of CoNLL-2008 Shared Task. find
34. W. Sun, H. Li and Z. Sui. 2008. The Integration of Dependency Relation Classification and Semantic Role Labeling Using Bilayer Maximum Entropy Markov Models. In Proc. of CoNLL-2008 Shared Task. find
35. E. F. Tjong Kim San and F. De Meulder. 2003. Introduction to the CoNLL-2003 Shared Task: Language-Independent Named Entity Recognition. In Proc. of CoNLL-2003. find
36. D. Vickrey and D. Koller. 2008. Applying Sentence Simplification to the CoNLL-2008 Shared Task. In Proc. of CoNLL-2008 Shared Task. find
37. R. Weischedel and A. Brunstein. 2005. BBN pronoun coreference and entity type corpus. Technical report, Linguistic Data Consortium. find
38. H. Yamada and Y. Matsumoto. 2003. Statistical Dependency Analysis with Support Vector Machines. In Proc. of IWPT. find our card
39. Y. Zhang, R. Wang and H. Uszkoreit. 2008. Hybrid Learning of Dependency Structures from Heterogeneous Linguistic Resources. In Proc. of CoNLL-2008 Shared Task. find
40. Zhao, S. 2005. Information Extraction from Multiple Syntactic Sources. Ph.D. thesis, NYU. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/surdeanu-2008-conll-2008-shared-task-joint/