Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling

What it is about

The overview of the 2005 shared task on SRL over PropBank. Systems label the arguments of given target verbs with roles. The training data is the Wall Street Journal sections 02 to 21 with full syntactic parses from two parsers, and the test is section 23 plus three Brown sections (ck01 to 03) to test robustness across corpora. Nineteen systems took part, all in the closed setting. The best system reached F1 79.44 on the WSJ test. Every system lost about 10 F1 points on the Brown set, which the authors attribute to parser error propagation in the pipeline. The top four systems were combinations of several systems working on different parses, and the best single system reached 75.04 on the combined test. A seven-rule baseline (for example, the noun phrase before the verb is A0, the one after is A1) is far below all systems. The argument is strict: a span is right only if its boundaries and its label are both right, and the verb itself is not scored.

Ideas

• I1. Cross-corpus gate: evaluate Mova's rule-based role labeller on a second genre, such as the school books against other text, and report the drop. The paper's 10-point drop for every system is a useful expectation, and a claim of generality should show it.
• I2. Implement the seven-rule baseline in prag/src/roles.rs as Mova's first-tier fallback and as the floor of every role measure. It is a deterministic rule set, so it is easy to audit.
• I3. Parse combination: take the agreed trees from tools/treeagree and count role votes over several parsers, as the top systems did. Measure the gain over the best single parser on school sentences.
• I4. Adopt the strict scoring (span and label both correct, verb excluded) in the role evaluation of prag, so that Mova's numbers are comparable to published ones.

For Mova

The chapters are 21. The simple clause and 13. Verbs. The code is prag/src/roles.rs and tools/treeagree. The measure is span-and-label F1 on school sentences, run twice to check determinism, and the cross-genre gate in I1.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1706543.1706571

@inproceedings{carreras2005introduction,
  title = {{Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling}},
  author = {Carreras, Xavier and Màrquez, Lluís},
  booktitle = {Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005)},
  year = {2005},
  doi = {10.3115/1706543.1706571},
  url = {https://doi.org/10.3115/1706543.1706571}
}

References

36 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Xavier Carreras and Lluı́s Màrquez. 2003. Phrase recognition by filtering and ranking with perceptrons. In Proceedings of RANLP-2003, Borovets, Bulgaria. find
2. Xavier Carreras and Lluı́s Màrquez. 2004. Introduction to the CoNLL-2004 Shared Task: Semantic Role Labeling. In Proceedings of CoNLL-2004. find
3. Eugene Charniak. 2000. A maximum-entropy inspired parser. In Proceedings of NAACL-2000. find
4. Wanxiang Che, Ting Liu, Sheng Li, Yuxuan Hu, and Huaijun Liu. 2005. Semantic role labeling system using maximum entropy classifier. In Proceedings of CoNLL-2005. find
5. Hai Leong Chieu and Hwee Tou Ng. 2003. Named entity recognition with a maximum entropy approach. In Proceedings of CoNLL-2003, Edmonton, Canada. find
6. Trevor Cohn and Philip Blunsom. 2005. Semantic role labelling with tree conditional random fields. In Proceedings of CoNLL-2005. find
7. Michael Collins. 1999. Head-driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania. find our card
8. Charles J. Fillmore, Charles Wooters, and Collin F. Baker. 2001. Building a large lexical databank which provides deep semantics. In Proceedings of the Pacific Asian Conference on Language, Informa tion and Computation, Hong Kong, China. find
9. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. find our card
10. Jesús Giménez and Lluı́s Màrquez. 2003. Fast and accurate part-of-speech tagging: The svm approach revisited. In Proceedings of RANLP-2003, Borovets, Bulgaria. find
11. Aria Haghighi, Kristina Toutanova, and Christopher Manning. 2005. A joint model for semantic role labeling. In Proceedings of CoNLL-2005. find
12. Richard Johansson and Pierre Nugues. 2005. Sparse bayesian classification of predicate arguments. In Proceedings of CoNLL-2005. find
13. Chi-San Lin and Tony C. Smith. 2005. Semantic role labeling via consensus in pattern-matching. In Proceedings of CoNLL-2005. find
14. Ken Litkowski. 2004. Senseval-3 task: Automatic labeling of semantic roles. In Proceedings of the Senseval-3 ACL-SIGLEX Workshop. find our card
15. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19. find our card
16. Lluı́s Màrquez, Pere Comas, Jesús Giménez, and Neus Català. 2005. Semantic role labeling as sequential tagging. In Proceedings of CoNLL-2005. find
17. Tomohiro Mitsumori, Masaki Murata, Yasushi Fukuda, Kouichi Doi, and Hirohumi Doi. 2005. Semantic role labeling using support vector machines. In Proceedings of CoNLL-2005. find
18. Alessandro Moschitti, Ana-Maria Giuglea, Bonaventura Coppola, and Roberto Basili. 2005. Hierarchical semantic role labeling. In Proceedings of CoNLL-2005. find
19. Alessandro Moschitti. 2004. A study on convolution kernel for shallow semantic parsing. In Proceedings of the 42nd Annual Conference of the Association for Computational Linguistics (ACL-2004). find
20. Necati Ercan Ozgencil and Nancy McCracken. 2005. Semantic role labeling using libSVM. In Proceedings of CoNLL-2005. find
21. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1). find our card
22. Kyung-Mi Park and Hae-Chang Rim. 2005. Maximum entropy based semantic role labeling. In Proceedings of CoNLL-2005. find
23. Simone Paolo Ponzetto and Michael Strube. 2005. Semantic role labeling using lexical statistical information. In Proceedings of CoNLL-2005. find
24. Sameer Pradhan, Kadri Hacioglu, Valerie Krugler, Wayne Ward, James Martin, and Daniel Jurafsky. 2005a. Support vector learning for semantic argument classification. Machine Learning. Special issue on Speech and Natural Language Processing. To appear. find
25. Sameer Pradhan, Kadri Hacioglu, Wayne Ward, James H. Martin, and Daniel Jurafsky. 2005b. Semantic role chunking combining complementary syntactic views. In Proceedings of CoNLL-2005. find
26. Vasin Punyakanok, Dan Roth, Wen-Tau Yih, and Dav Zimak. 2004. Semantic role labeling via integer linear programming inference. In Proceedings of the International Conference on Computational Linguistics (COLING). find
27. Vasin Punyakanok, Peter Koomen, Dan Roth, and Wen tau Yih. 2005. Generalized inference with multiple semantic role labeling systems. In Proceedings of CoNLL-2005. find
28. Mihai Surdeanu and Jordi Turmo. 2005. Semantic role labeling using complete syntactic analysis. In Proceedings of CoNLL-2005. find
29. Mihai Surdeanu, Sanda Harabagiu, John Williams, and Paul Aarseth. 2003. Using predicate-argument structures for information extraction. In Proceedings of ACL 2003, Sapporo, Japan. find
30. Charles Sutton and Andrew McCallum. 2005. Joint parsing and semantic role labeling. In Proceedings of CoNLL-2005. find
31. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL-2003. find
32. Erik Tjong Kim Sang, Sander Canisius, Antal van den Bosch, and Toine Bogers. 2005. Applying spelling error correction techniques for improving semantic role labelling. In Proceedings of CoNLL-2005. find
33. Tzong-Han Tsai, Chia-Wei Wu, Yu-Chun Lin, and Wen-Lian Hsu. 2005. Exploiting full parsing information to label semantic roles using an ensemble of me and svm via integer linear programming. In Proceedings of CoNLL-2005. find
34. Sriram Venkatapathy, Akshar Bharati, and Prashanth Reddy. 2005. Inferring semantic roles using subcategorization frames and maximum entropy model. In Proceedings of CoNLL-2005. find
35. Nianwen Xue and Martha Palmer. 2004. Calibrating features for semantic role labeling. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find
36. Szu-ting Yi and Martha Palmer. 2005. The integration of syntactic parsing and semantic role labeling. In Proceedings of CoNLL-2005. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/carreras-2005-introduction-conll-2005-shared-task/