PropBank Comes of Age—Larger, Smarter, and more Diverse

What it is about

A twenty-year review of PropBank. Rolesets are coarse senses with numbered arguments, and every role now also carries a function tag (PAG proto-agent, PPT proto-patient, GOL goal, LOC, TMP…) after Dowty. Since the 2017 unification, one roleset covers etymologically related verbs, nouns, adjectives, light-verb constructions, multiword expressions and even prepositions and adverbs (afraid shares fear's roles), to serve AMR. Release 3.4.0 adds links from rolesets and roles to VerbNet 3.4, FrameNet and OntoNotes senses inside the frame files, usage tags per project, tokenised examples with start/end indices, and a validation script (indices match the text, arguments do not overlap, no numbered argument twice). The annotated data now spans OntoNotes genres, the English Web Treebank, BOLT chat and forums, an enlarged Brown fiction set, LORELEI, Flickr captions and earth-science and clinical texts, offered as new out-of-domain benchmarks.

Ideas

• I1. A larger out-of-domain yardstick for the frame reader: PropBank's annotation of the English Web Treebank and of the Brown fiction subset (closest to tales), mapped through function tags (PAG → agent, PPT → patient, GOL → recipient or place, LOC → place, TMP → when:, CAU/PRP → cause). It runs outside the repo as a test, like the PropBank tables already converted in prag lexicon convert, and gives per-role recall on thousands of predicates where the frame bench has 240 lines. Mova's parser is trained on EWT, so use EWT's test split only.
• I2. Adjective and noun predicates with arguments, as PropBank's unified rolesets treat them: «the hare was afraid of the fox», «proud of his speed», «fond of honey» become frames of the related verb (fear, pride, like) with the subject as experiencer and the of-phrase as patient. Build the link from WordNet derivations (WordNet already feeds prag/data/nominals.tsv), not from PropBank. Measure with world knowledge diagnose on questions «What was the hare afraid of?» and course test counts.
• I3. Copy PropBank's validation checks for the frame bench gold (runs/prag-bench/gold.tsv): each filler is a word of its line, no frame has two agents, link targets exist. A gold error found this way is cheaper than a reader change chasing it. The check needs a negative control (a deliberately broken gold line).

For Mova

15. Adjectives and adverbs (adjective complements), 23-complementation.md, 30-pragmatics-overview.md §5. Code: prag/src/vnconv.rs (test-only conversion), prag/src/frames.rs (adjective predicates), prag/data/nominals.tsv, prag/src/fbench.rs (gold validation). Measures: per-role recall on PropBank EWT/Brown (tests only), diagnosis on adjective questions, frame bench.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.starsem-1.24

@inproceedings{pradhan2022propbank,
  title = {{PropBank Comes of Age—Larger, Smarter, and more Diverse}},
  author = {Pradhan, Sameer and Bonn, Julia and Myers, Skatje and Conger, Kathryn and O'gorman, Tim and Gung, James and Wright-bettner, Kristin and Palmer, Martha},
  booktitle = {Proceedings of the 11th Joint Conference on Lexical and Computational Semantics},
  year = {2022},
  doi = {10.18653/v1/2022.starsem-1.24},
  url = {https://doi.org/10.18653/v1/2022.starsem-1.24}
}

References

47 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alan Akbik, Laura Chiticariu, Marina Danilevsky, Yunyao Li, Shivakumar Vaithyanathan, and Huaiyu Zhu. 2015. Generating high quality Proposition Banks for multilingual semantic role labeling. In Proceedings of the 53rd Annual Meeting of the ACL and the 7th International Joint Conference on NLP, pages 397–407, Beijing, China. find
2. Daniel Albright, Arrick Lanfranchi, Anwen Fredriksen, William F Styler IV, Colin Warner, Jena D Hwang, Jinho D Choi, Dmitriy Dligach, Rodney D Nielsen, James Martin, et al. 2013. Towards comprehensive syntactic and semantic annotations of the clinical narrative. Journal of the American Medical Informatics Association, 20(5):922–930. 86.7 83.2 80.1 80.5 77.3 *Table 3: Baseline performance on four main corpora annotated with PropBank v3.4.0 rolesets for English. The results include performance across all parts of speech. Follow latest updates and analysis at* https://leaderboard.propbank.org ↗
3. Summary and Discussion Olga Babko-Malaya, Ann Bies, Ann Taylor, Szu-ting Yi, Martha Palmer, Mitch Marcus, Seth Kulick, and Libin Shen. 2006. Issues in synchronizing the english treebank and propbank. In Proceedings of the Workshop on Frontiers in Linguistically Annotated Corpora 2006, pages 70–77. find
4. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of the 17th International Conference on Computational Linguistics (COLING/ACL-98), pages 86–90, Montreal. find our card
5. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse. find our card
6. Steven Bethard, Guergana Savova, Martha Palmer, and James Pustejovsky. 2017. SemEval-2017 task 12: Clinical TempEval. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 565–572, Vancouver, Canada. Association for Computational Linguistics. find
7. Rajesh Bhatt, Bhuvana Narasimhan, Martha Palmer, Owen Rambow, Dipti Misra Sharma, and Fei Xia. 2009. A multi-representational and multi-layered Treebank for Hindi/Urdu. In Proceedings of the Third Linguistic Annotation Workshop, pages 186– 189. Association for Computational Linguistics. find
8. Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English Web Treebank (LDC2012T13). Linguistic Data Consortium, Philadelphia, PA. find
9. Claire Bonial, Julia Bonn, Kathryn Conger, Jena Hwang, and Martha Palmer. 2014. PropBank: Semantics of New Predicate Types. In LREC, pages 3013–3019. find our card
10. Claire Bonial, Julia Bonn, Kathryn Conger, and Jena D. Hwang. 2012. English propbank annotation guidelines. find
11. Claire Bonial, Kathryn Conger, Jena D Hwang, Aous Mansouri, Yahya Aseri, Julia Bonn, Timothy O’Gorman, and Martha Palmer. 2017. Current directions in English and Arabic PropBank. In Handbook of Linguistic Annotation, pages 737–769. Springer. find
12. Julia Bonn, Martha Palmer, Zheng Cai, and Kristin Wright-Bettner. 2020. Spatial AMR: Expanded spatial annotation in the context of a grounded Minecraft corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 4883– 4892, Marseille, France. European Language Resources Association. find
13. Xavier Carreras and Lluís Màrquez. 2005. Introduction to the CoNLL-2005 shared task: Semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning, pages 152–164. Association for Computational Linguistics. find our card
14. David Dowty. 1991. Thematic proto-roles and argument selection. language, pages 547–619. find
15. R Duerr, A Thessen, CJ Jenkins, M Palmer, S Myers, and S Ramdeen. 2016. The ClearEarth Project: Preliminary findings from experiments in applying the CLEARTK NLP pipeline and annotation tools developed for biomedicine to the earth sciences. In AGU Fall Meeting Abstracts. find
16. Ruth Duerr, Skatje Myers, Martha Palmer, Chris J Jenkins, Anne Thessen, and James Martin. 2015. Natural language processing and machine learning (nlp/ml): Applying advances in biomedicine to the earth sciences. In AGU Fall Meeting Abstracts, volume 2015, pages IN51A–1784. find
17. Magali Sanches Duran and Sandra Maria Aluísio. 2011. Propbank-br: a brazilian portuguese corpus annotated with semantic role labels. In Proceedings of the 8th Symposium in Information and Human Language Technology, Cuiabá/MT, Brazil. find
18. Christiane Fellbaum. 2010. WordNet. In Theory and applications of ontology: computer applications, pages 231–243. Springer. find
19. Jennifer Garland, Stephanie Strassel, Safa Ismael, Zhiyi Song, and Haejoong Lee. 2012. Linguistic resources for genre-independent language technologies: user-generated content in BOLT. In Workshop Programme, page 34. find
20. John J Godfrey, Edward C Holliman, and Jane Mc-Daniel. 1992. Switchboard: Telephone speech corpus for research and development. In Acoustics, Speech, and Signal Processing, 1992. ICASSP-92., 1992 IEEE International Conference on, volume 1, pages 517–520. IEEE. find
21. Jens Van Gysel, Meagan Vigus, Jayeol Chun, Kenneth Lai, Sarah Moeller, Jiarui Yao, Timothy J. O’Gorman, Andrew Cowell, W. Bruce Croft, Chu-Ren Huang, Jan Hajic, James H. Martin, Stephan Oepen, Martha Palmer, James Pustejovsky, Rosa Vallejos, and Nianwen Xue. 2021. Designing a uniform meaning representation for natural language processing. Künstliche Intell., 35:343–360. find
22. Katri Haverinen, Veronika Laippala, Samuel Kohonen, Anna Missilä, Jenna Nyblom, Stina Ojala, Timo Viljanen, Tapio Salakoski, and Filip Ginter. 2013. Towards a dependency-based PropBank of general Finnish. In Proceedings of the 19th Nordic Conference of Computational Linguistics (NODALIDA 2013); May 22-24; 2013; Oslo University; Norway. NEALT Proceedings Series 16, 085, pages 41–57. Linköping University Electronic Press. find
23. Micah Hodosh, Peter Young, and Julia Hockenmaier. 2013. Framing image description as a ranking task: Data, models and evaluation metrics. Journal of Artificial Intelligence Research, 47:853–899. find
24. Jena D Hwang, Archna Bhatia, Clare Bonial, Aous Mansouri, Ashwini Vaidya, Nianwen Xue, and Martha Palmer. 2010. PropBank annotation of multilingual light verb constructions. In Proceedings of the Fourth Linguistic Annotation Workshop, pages 82–90. Association for Computational Linguistics. find
25. Ishan Jindal, Alexandre Rademaker, Michał Ulewicz, Huyen Nguyen, Ha Linh, Khoi-Nguyen Tran, Huaiyu Zhu, and Yunyao Li. 2022. Universal proposition bank 2.0. Marseille, France. find
26. Paul Kingsbury and Martha Palmer. 2002. From Treebank to PropBank. In LREC, pages 1989–1993. find our card
27. Tao Li, Parth Anand Jawale, Martha Palmer, and Vivek Srikumar. 2020. Structured tuning for semantic role labeling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8402–8412, Online. find
28. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The NomBank project: An interim report. In HLT-NAACL 2004 workshop: Frontiers in corpus annotation, volume 24, page 31. find
29. Sameer S Pradhan, Wayne Ward, and James H Martin. 2008. Towards robust semantic role labeling. Computational linguistics, 34(2):289–310. find
30. GG Sahin. 2016. Verb sense annotation for Turkish PropBank via crowdsourcing. In Proceedings of 17th international conference on intelligent text processing and computational linguistics. CICLING. find
31. Azadeh Mirzaei and Amirsaeid Moloodi. 2016. Persian Proposition Bank. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 3828–3835, Portorož, Slovenia. find
32. Sarah Moeller, Irina Wagner, Martha Palmer, Kathryn Conger, and Skatje Myers. 2020. The Russian Prop-Bank. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5995– 6002. find
33. Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019. Collaborative dialogue in Minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5405–5415, Florence, Italy. find
34. Tim O’Gorman, Sameer Pradhan, Martha Palmer, Julia Bonn, Kathryn Conger, and James Gung. 2018. The New Propbank: Aligning Propbank with AMR through POS Unification. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. find
35. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005a. The Proposition Bank: An annotated corpus of semantic roles. Computational linguistics, 31(1):71–106. find our card
36. Martha Palmer, Shijong Ryu, Jinyoung Choi, Sinwon Yoon, and Yeongmi Jeon. 2006. Korean PropBank. LDC Catalog No.: LDC2006T03 ISBN, pages 1– 58563. find
37. Martha Palmer, Nianwen Xue, Olga Babko-Malaya, Jinying Chen, and Benjamin Snyder. 2005b. A parallel Proposition Bank II for Chinese and English. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 61–67. Association for Computational Linguistics. find
38. Sameer Pradhan, Kadri Hacioglu, Valerie Krugler, Wayne Ward, James Martin, and Dan Jurafsky. 2005. Support vector learning for semantic argument classification. Machine Learning Journal, 60(1):11–39. find
39. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 143–152, Sofia, Bulgaria. find
40. Karin Kipper Schuler. 2005. VerbNet: A broad-coverage, comprehensive verb lexicon. University of Pennsylvania. find
41. Zhiyi Song, Stephanie Strassel, Haejoong Lee, Kevin Walker, Jonathan Wright, Jennifer Garland, Dana Fore, Brian Gainor, Preston Cabe, Thomas Thomas, et al. 2014. Collecting natural SMS and chat conversations in multiple languages: The BOLT phase 2 corpus. In LREC, pages 1699–1704. find
42. Stephanie Strassel and Jennifer Tracey. 2016. Lorelei language packs: Data, tools, and resources for technology development in low resource languages. In LREC. find
43. William F Styler, Steven Bethard, Sean Finan, Martha Palmer, Sameer Pradhan, Piet C De Groen, Brad Er-ickson, Timothy Miller, Chen Lin, Guergana Savova, et al. 2014. Temporal annotation in the clinical domain. Transactions of the association for computational linguistics, 2:143–154. find
44. Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluís Màrquez, and Joakim Nivre. 2008. The CoNLL-2008 shared task on joint parsing of syntactic and semantic dependencies. In Proceedings of the Twelfth Conference on Computational Natural Language Learning, pages 159–177. find our card
45. Ann Taylor. 1996. Bracketing Switchboard: an addendum to the Treebank II bracketing guidelines. Linguistic Data Consortium. find
46. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A large training corpus for enhanced processing. In Joseph Olive, Caitlin Christianson, and John McCary, editors, Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation. Springer. find
47. Kristin Wright-Bettner, Martha Palmer, Guergana Savova, Piet de Groen, and Timothy Miller. 2019. Cross-document coreference: An approach to capturing coreference without context. In Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), pages 1–10, Hong Kong. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/pradhan-2022-propbank-comes-age-larger-smarter/