Automatic Labeling of Semantic Roles

What it is about

The first statistical system for labeling semantic roles with FrameNet frame elements (Agent, Patient, Speaker, Topic and others). Given a sentence, a target word and its frame, the system classifies the constituents. It is trained on about 50,000 hand-annotated FrameNet sentences (36,995 of them for training in the experiments). Each constituent gets features from a syntactic parse: phrase type, grammatical function, position, head word, and the path in the tree from the target to the constituent. The probabilities are estimated with backoff over these features. Given the boundaries of constituents, the system reaches 82% accuracy on the role; segmenting and labeling together gives 65% precision and 61% recall. Clustering of head nouns, WordNet classes and bootstrapping from unlabeled text each extend the statistics to head words never seen in training, with similar accuracy where they apply and different coverage. Performance drops for predicates unseen in training and further for unseen frames.

Ideas

• I1. Add a dependency-path feature to the frame reader: the sequence of relation labels and up or down moves from the predicate to the constituent. The UD trees from en/src/parse.rs give the same kind of path. Measure role accuracy with and without it on school sentences, and run a negative control in which role labels are shuffled.
• I2. Use the paper's backoff lattice as a count table in BTreeMap: P(role | head, phrase type, target) backs off to P(role | phrase type, target), then to P(role | target). Report the coverage at each level.
• I3. The paper compares three ways of generalising over head words (clusters, WordNet, bootstrapping) and finds them about equally accurate where each applies. Mova's dictionary relations (world/src/dictionary.rs) play the WordNet role; measure how many unseen head words they cover before adding anything learned.
• I4. The subcategorization feature gave no gain once frame-element priors were used (81.6%). Before adding a feature to prag/src/roles.rs, measure it alone and combined with the prior, and keep only what helps.

For Mova

The chapters are 13. Verbs and 11. Nouns for the head-word and verb-frame side. The code is prag/src/roles.rs and en/src/parse.rs. Measure with the school test sentences; the held-out check is the two-fold cv split, not the release test.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1162/089120102760275983

@article{gildea2002automatic,
  title = {{Automatic Labeling of Semantic Roles}},
  author = {Gildea, Daniel and Jurafsky, Daniel},
  journal = {Computational Linguistics},
  year = {2002},
  doi = {10.1162/089120102760275983},
  url = {https://doi.org/10.1162/089120102760275983}
}

References

47 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Baayen, R. H., R. Piepenbrock, and L. Gulikers. 1995. The CELEX Lexical Database (Release 2) [CD-ROM]. Linguistic Data Consortium, University of Pennsylvania [Distributor], Philadelphia, PA. find
2. Baker, Collin F., Charles J. Fillmore, and John B. Lowe. 1998. “The Berkeley FrameNet project.” In Proceedings of COLING/ACL, pages 86–90, Montreal, Canada. find our card
3. Blaheta, Don and Eugene Charniak. 2000. “Assigning function tags to parsed text.” In Proceedings of the First Annual Meeting of the North American Chapter of the ACL (NAACL), pages 234–240, Seattle, Washington. find
4. Carroll, Glenn and Mats Rooth. 1998. “Valence induction with a head-lexicalized PCFG.” In Proceedings of the Third Conference on Empirical Methods in Natural Language Processing (EMNLP 3), Granada, Spain. find
5. Charniak, Eugene. 1997. “Statistical parsing with a context-free grammar and word statistics.” In AAAI-97, pages 598–603, Menlo Park, August. AAAI Press, Menlo Park, California. find
6. Collins, Michael. 1997. “Three generative, lexicalised models for statistical parsing.” In Proceedings of the 35th Annual Meeting of the ACL, pages 16–23, Madrid, Spain. find
7. Collins, Michael. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. dissertation, University of Pennsylvania, Philadelphia. find our card
8. Dahiya, Yajan Veer. 1995. Panini as a Linguist: Ideas and Patterns. Eastern Book Linkers, Delhi, India. find
9. Defense Advanced Research Projects Agency, editor. 1998. Proceedings of the Seventh Message Understanding Conference. find
10. Dowty, David R. 1991. Thematic proto-roles and argument selection. Language 67(3):547–619. find
11. Fellbaum, Christiane, editor. 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, Massachusetts. find
12. Fillmore, Charles J. 1968. “The case for case.” In Emmon W. Bach and Robert T. Harms, editors, Universals in Linguistic Theory. Holt, Rinehart & Winston, New York, pages 1–88. find
13. Fillmore, Charles J. 1971. “Some problems for case grammar.” In R. J. O’Brien, editor, 22nd Annual Round Table. Linguistics: Developments of the Sixties—Viewpoints of the Seventies. Volume 24 of Monograph Series on Language and Linguistics. Georgetown University Press, Washington, D.C., pages 35–56. find
14. Fillmore, Charles J. 1976. “Frame semantics and the nature of language.” In Annals of the New York Academy of Sciences: Conference on the Origin and Development of Language and Speech, Volume 280, pages 20–32. New York Academy of Sciences, New York. find
15. Fillmore, Charles J. 1986. “Pragmatically controlled zero anaphora.” In Proceedings of Berkeley Linguistics Society, pages 95–107, Berkeley, California. find
16. Fillmore, Charles J. and Collin F. Baker. 2000. “FrameNet: Frame semantics meets the corpus.” Poster presentation, 74th Annual Meeting of the Linguistics Society of America. find
17. Gildea, Daniel and Thomas Hofmann. 1999. “Probabilistic topic analysis for language modeling.” In Eurospeech-99, pages 2167–2170, Budapest. find
18. Hearst, Marti. 1999. “Untangling text data mining.” In Proceedings of the 37th Annual Meeting of the ACL, pages 3–10, College Park, Maryland. find
19. Hobbs, Jerry R., Douglas Appelt, John Bear, David Israel, Megumi Kameyama, Mark E. Stickel, and Mabry Tyson. 1997. “FASTUS: A cascaded finite-state transducer for extracting information from natural-language text.” In Emmanuel Roche and Yves Schabes, editors, Finite-State Language Processing. MIT Press, Cambridge, Massachusetts, pages 383–406. find
20. Hofmann, Thomas and Jan Puzicha. 1998. “Statistical models for co-occurrence data.” Memorandum, Massachussetts Institute of Technology Artificial Intelligence Laboratory, Cambridge, Massachusetts. find
21. Jackendoff, Ray. 1972. Semantic Interpretation in Generative Grammar. MIT Press, Cambridge, Massachusetts. find
22. Jelinek, Frederick and Robert L. Mercer. 1980. “Interpolated estimation of Markov source parameters from sparse data.” In Proceedings: Workshop on Pattern Recognition in Practice, pages 381–397. Amsterdam. North Holland. find
23. Johnson, Christopher R., Charles J. Fillmore, Esther J. Wood, Josef Ruppenhofer, Margaret Urban, Miriam R. L. Petruk, and Collin F. Baker. 2001. The FrameNet project: Tools for lexicon building. Version 0.7. Available at http://www.icsi. berkeley.edu/˜framenet/book.html. ↗
24. Kipper, Karin, Hoa Trang Dang, William Schuler, and Martha Palmer. 2000. “Building a class-based verb lexicon using TAGs.” In TAG+5 Fifth International Workshop on Tree Adjoining Grammars and Related Formalisms, Paris, May. find
25. Lapata, Maria and Chris Brew. 1999. “Using subcategorization to resolve verb class ambiguity.” In Joint SIGDAT Conference on Empirical Methods in NLP and Very Large Corpora, pages 266–274, College Park, Maryland. find
26. Levin, Beth. 1993. English Verb Classes and Alternations: A Preliminary Investigation. University of Chicago Press, Chicago. find
27. Levin, Beth and Malka Rappaport Hovav. 1996. “From lexical semantics to argument realization.” Unpublished manuscript. find
28. Marcus, Mitchell P., Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. “The Penn Treebank: Annotating predicate argument structure.” In ARPA Human Language Technology Workshop, pages 114–119, Plainsboro, New Jersey. Morgan Kaufmann, San Francisco. find our card
29. Marcus, Mitchell P., Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn treebank. Computational Linguistics 19(2):313–330. find our card
30. McCarthy, Diana. 2000. “Using semantic preferences to identify verbal participation in role switching alternations.” In Proceedings of the First Annual Meeting of the North American Chapter of the ACL (NAACL), pages 256–263, Seattle, Washington. find
31. Miller, Scott, Heidi Fox, Lance Ramshaw, and Ralph Weischedel. 2000. “A novel use of statistical parsing to extract information from text.” In Proceedings of the First Annual Meeting of the North American Chapter of the ACL (NAACL), pages 226–233, Seattle, Washington. find
32. Miller, Scott, David Stallard, Robert Bobrow, and Richard Schwartz. 1996. “A fully statistical approach to natural language interfaces.” In Proceedings of the 34th Annual Meeting of the ACL, pages 55–61, Santa Cruz, California. find
33. Misra, Vidya Niwas. 1966. The Descriptive Technique of Panini. Mouton, The Hague. find
34. Pereira, Fernando, Naftali Tishby, and Lillian Lee. 1993. “Distributional clustering of English words.” In Proceedings of the 31st Annual Meeting of the ACL, pages 183–190, Columbus, Ohio. find
35. Pietra, Stephen Della, Vincent Della Pietra, and John Lafferty. 1997. Inducing features of random fields. IEEE Transactions on Pattern Analysis and Machine Intelligence 19(4):380–393. find
36. Pollard, Carl and Ivan A. Sag. 1994. Head-Driven Phrase Structure Grammar. University of Chicago Press, Chicago. find
37. Riloff, Ellen. 1993. Automatically constructing a dictionary for information extraction tasks. In Proceedings of the 11th National Conference on Artificial Intelligence (AAAI), pages 811–816, Washington, D.C. find
38. Riloff, Ellen and Mark Schmelzenbach. 1998. “An empirical approach to conceptual case frame acquisition.” In Proceedings of the Sixth Workshop on Very Large Corpora, pages 49–56, Montreal, Canada. find
39. Rocher, Rosane. 1964. “Agent” et “Objet” chez Panini. Journal of the American Oriental Society 84:44–54. find
40. Rooth, Mats. 1995. “Two-dimensional clusters in grammatical relations.” In AAAI Symposium on Representation and Acquisition of Lexical Knowledge, Stanford, California. find
41. Rooth, Mats, Stefan Riezler, Detlef Prescher, Glenn Carroll, and Franz Beil. 1999. “Inducing a semantically annotated lexicon via EM-based clustering.” In Proceedings of the 37th Annual Meeting of the ACL, pages 104–111, College Park, Maryland. find
42. Schank, Roger C. 1972. Conceptual dependency: A theory of natural language understanding. Cognitive Psychology 3:552–631. find
43. Siegel, Sidney and N. John Castellan, Jr. 1988. Nonparametric Statistics for the Behavioral Sciences. 2nd ed. McGraw-Hill, New York. find
44. Somers, Harold L. 1987. Valency and Case in Computational Linguistics. Edinburgh University Press, Edinburgh, Scotland. find
45. Stallard, David. 2000. “Talk’n’travel: A conversational system for air travel planning.” In Proceedings of the Sixth Applied Natural Language Processing Conference (ANLP’00), pages 68–75. find
46. Van Valin, Robert D. 1993. A synopsis of role and reference grammar. In Robert D. Van Valin, editor, Advances in Role and Reference Grammar. John Benjamins Publishing Company, Amsterdam, pages 1–166. find
47. Winograd, Terry. 1972. Understanding natural language. Cognitive Psychology, 3(1). Reprinted as a book by Academic Press, 1972. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/gildea-2002-automatic-labeling-semantic-roles/