Abstraction and Generalisation in Semantic Role Labels: PropBank, VerbNet or both?

What it is about

The paper compares two role inventories, PropBank (abstract labels A0 to A5, AA, and adjunct tags) and VerbNet (23 thematic roles, grouped into 7 classes for the comparison). It does not train a parser or a labeller. It uses information-theoretic counts over the SemLink mapping of the Wall Street Journal training and development sections (106,459 role pairs; 68,268 dependencies for the syntax analysis). PropBank labels are easier to learn (entropy 1.37 bits, against 2.06 for the VerbNet groups). VerbNet labels carry more information and depend more on the verb: mutual information with the verb is 1.03 bits for VerbNet and 0.26 bits for PropBank. PropBank role sets correlate more with dependency labels (symmetric uncertainty 0.49, against 0.39). An animacy effect on the subject pronouns is significant for VerbNet only. The authors conclude that both inventories are useful: VerbNet when the verb is known, PropBank when it is not.

Ideas

• I1. A two-tier role lookup in the frame reader: (verb, sense) gives a verb-specific thematic role; when the verb is not in the lexicon, fall back to a coarse abstract label (A0 or A1 style). Implement the order in prag/src/roles.rs, and measure accuracy at each step on school sentences.
• I2. Report the entropy, mutual information and symmetric uncertainty of Mova's own role labels against the verb and against the dependency relation, as a diagnostic table. It shows how much the rules depend on the verb, which the paper argues is the key property.
• I3. Test the animacy effect with a fixed pronoun list (I, you, he, she, it and so on) as a deterministic feature for agent assignment. The paper's test is a count-based ANOVA, which Mova can run with plain counts.
• I4. Use the role-set counts (PropBank: 62 role sets; VerbNet: 116 combinations) as a closed-world check: a proposition whose role set was never seen in the lessons is flagged rather than silently accepted. Measure the flag rate on held-out lessons.

For Mova

The chapters are 13. Verbs and 28. Voice and the passive. The code is prag/src/roles.rs, with the verb lexicon in en/src/dict.rs. Measure with the school test sentences and the two-fold cv split. Run twice and compare the outputs, since the counts must be ordered by BTreeMap.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{merlo2009abstraction,
  title = {{Abstraction and Generalisation in Semantic Role Labels: PropBank, VerbNet or both?}},
  author = {Merlo, Paola and Plas, Lonneke van der},
  booktitle = {Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP},
  year = {2009},
  url = {https://aclanthology.org/P09-1033/}
}

References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Judith Aissen. 2003. Differential object marking: Iconicity vs. economy. Natural Language and Linguistic Theory, 21:435–483. find
2. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of the Thirty-Sixth Annual Meeting of the Association for Computational Linguistics and Seventeenth International Conference on Computational Linguistics (ACL-COLING’98), pages 86–90, Montreal, Canada. find our card
3. David Dowty. 1991. Thematic proto-roles and argument selection. Language, 67(3):547–619. find
4. Charles Fillmore. 1968. The case for case. In Emmon Bach and Harms, editors, Universals in Linguistic Theory, pages 1–88. Holt, Rinehart, and Winston. find
5. Jane Grimshaw. 1990. Argument Structure. MIT Press. find
6. Jeffrey Gruber. 1965. Studies in Lexical Relation. MIT Press, Cambridge, MA. find
7. Ray Jackendoff. 1972. Semantic Interpretation in Generative Grammar. MIT Press, Cambridge, MA. find
8. Karin Kipper. 2005. VerbNet: A broad-coverage, comprehensive verb lexicon. Ph.D. thesis, University of Pennsylvania. find
9. Beth Levin and Malka Rappaport Hovav. 2005. Argument Realization. Cambridge University Press, Cambridge, UK. find
10. Edward Loper, Szu ting Yi, and Martha Palmer. 2007. Combining lexical resources: Mapping between PropBank and VerbNet. In Proceedings of the IWCS. find
11. Christopher Manning and Hinrich Schuetze. 1999. Foundations of Statistical Natural Language Processing. MIT Press. find
12. Mitch Marcus, Beatrice Santorini, and M.A. Marcinkiewicz. 1993. Building a large annotated corpus of English: the Penn Treebank. Computational Linguistics, 19:313–330. find our card
13. Paola Merlo and Gabriele Musillo. 2008. Semantic parsing for high-precision semantic role labelling. In Proceedings of the Twelfth Conference on Computational Natural Language Learning (CONLL-08), pages 1–8, Manchester, UK. find
14. Paola Merlo and Suzanne Stevenson. 2001. Automatic verb classification based on statistical distributions of argument structure. Computational Linguistics, 27(3):373–408. find
15. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31:71–105. find our card
16. Josef Ruppenhofer, Michael Ellsworth, Miriam Petruck, Christopher Johnson, and Jan Scheffczyk. 2006. Framenet ii: Theory and practice. Technical report, Berkeley,CA. find
17. Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluı́s Màrquez, and Joakim Nivre. 2008. The CoNLL-2008 shared task on joint parsing of syntactic and semantic dependencies. In Proceedings of the 12th Conference on Computational Natural Language Learning (CoNLL-2008), pages 159–177. find our card
18. Kristina Toutanova, Aria Haghighi, and Christopher D. Manning. 2008. A global joint model for semantic role labeling. Computational Linguistics, 34(2). find
19. Ian Witten and Eibe Frank. 2005. Data Mining. Elsevier. find
20. Szu-ting Yi, Edward Loper, and Martha Palmer. 2007. Can semantic roles generalize across genres? In Proceedings of the Human Language Technologies 2007 (NAACL-HLT’07), pages 548–555, Rochester, New York, April. find
21. Beñat Zapirain, Eneko Agirre, and Lluı́s Màrquez. 2008. Robustness and generalization of role sets: PropBank vs. VerbNet. In Proceedings of ACL-08: HLT, pages 550–558, Columbus, Ohio, June. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/merlo-2009-abstraction-generalisation-semantic-role-labels/