Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities

What it is about

Most SRL research and benchmarks assume that predicates are verbs. The authors test a state-of-the-art PropBank system (CN-22) trained on OntoNotes. Its argument F1 falls from 84.7 on verbs to 16.4 on nouns, and a noun-trained model falls from 72.8 to 11.2 on verbs, so it does not transfer knowledge across predicate types. The authors add PB-Examples (7,481 framesets, built from PropBank example sentences) and PB-Unseen (over 4,000 senses not in OntoNotes). On Challenge-SRL, 288 sentences with 1,898 predicate-argument pairs, annotators first disagreed on 32% of items and then on 6% after discussion. Here VerbAtlas, which groups roles across frames, beats PropBank on argument F1 for verbs (47.0 against 5.5), nouns (44.2 against 2.1) and adjectives (36.8 against 10.8). A word-sense disambiguator, mapped to VerbAtlas frames, beats the SRL system on predicate senses; an oracle that picks the better of the two reaches 41.5% against 26.0%.

Ideas

• I1. Cross-type transfer test for Mova: train the frame reader on verbal frames only, test on nominal and adjectival ones from a hand-made list, and report the drop as the paper does. The place is prag/src/roles.rs.
• I2. Sense step before roles: resolve the predicate's sense from en/src/dict.rs first, then take the role set of that sense. Measure predicate-sense accuracy on school sentences against a lemma-only baseline.
• I3. Cross-frame roles: one closed list of thematic roles shared by all frames, as VerbAtlas does, so that nouns and adjectives reuse the roles of verbs. Test the argument recall on nominal sentences.
• I4. Report the oracle best-of-two number (rule-based sense versus frame reader) on the school test, as the paper does, to see whether the dictionary carries information the frame reader misses.

For Mova

The chapters are 11. Nouns, 15. Adjectives and adverbs and 10. Word classes. The code is prag/src/roles.rs, en/src/dict.rs and the school evaluation in world/src/knowledge.rs. Measure role F1 by part of speech on school sentences, with the two-fold cv split.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-acl.783

@inproceedings{orlando2023exploring,
  title = {{Exploring Non-Verbal Predicates in Semantic Role Labeling: Challenges and Opportunities}},
  author = {Orlando, Riccardo and Conia, Simone and Navigli, Roberto},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.783},
  url = {https://doi.org/10.18653/v1/2023.findings-acl.783}
}

References

34 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The berkeley framenet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 1, ACL ’98/COLING ’98, page 86–90, USA. Association for Computational Linguistics. find our card
2. Michele Bevilacqua, Tommaso Pasini, Alessandro Raganato, and Roberto Navigli. 2021. Recent trends in word sense disambiguation: A survey. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI-21, pages 4330–4338. International Joint Conferences on Artificial Intelligence Organization. Survey Track. find
3. Xavier Carreras and Lluís Màrquez. 2005. Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 152–164, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
4. Simone Conia, Andrea Bacciu, and Roberto Navigli. 2021. Unifying cross-lingual semantic role labeling with heterogeneous linguistic resources. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 338– 351, Online. Association for Computational Linguistics. find
5. Simone Conia and Roberto Navigli. 2022. Probing for predicate argument structures in pretrained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4622–4632, Dublin, Ireland. Association for Computational Linguistics. find
6. Angel Daza and Anette Frank. 2020. X-SRL: A parallel cross-lingual semantic role labeling dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3904–3914, Online. Association for Computational Linguistics. find
7. Andrea Di Fabio, Simone Conia, and Roberto Navigli. 2019. VerbAtlas: a novel large-scale verbal semantic resource and its application to semantic role labeling. find our card
8. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 627–637, Hong Kong, China. Association for Computational Linguistics. ceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 486–492, New Orleans, Louisiana. Association for Computational Linguistics. find
9. Daniel Gildea and Daniel Jurafsky. 2002. Automatic labeling of semantic roles. Computational Linguistics, 28(3):245–288. find our card
10. Marco Maru, Simone Conia, Michele Bevilacqua, and Roberto Navigli. 2022. Nibbling at the hard core of Word Sense Disambiguation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4724–4737, Dublin, Ireland. Association for Computational Linguistics. find
11. Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martí, Lluís Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, Pavel Straňák, Mihai Surdeanu, Nianwen Xue, and Yi Zhang. 2009. The CoNLL-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL 2009): Shared Task, pages 1–18, Boulder, Colorado. Association for Computational Linguistics. find our card
12. Hardy Hardy and Andreas Vlachos. 2018. Guided neural language generation for abstractive summarization using Abstract Meaning Representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 768–773, Brussels, Belgium. Association for Computational Linguistics. find
13. Ishan Jindal, Alexandre Rademaker, Michał Ulewicz, Ha Linh, Huyen Nguyen, Khoi-Nguyen Tran, Huaiyu Zhu, and Yunyao Li. 2022. Universal Proposition Bank 2.0. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1700–1711, Marseille, France. European Language Resources Association. find
14. Ayal Klein, Jonathan Mamou, Valentina Pyatkin, Daniela Stepanov, Hangfeng He, Dan Roth, Luke Zettlemoyer, and Ido Dagan. 2020. QANom: Question-answer driven SRL for nominalizations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3069–3083, Barcelona, Spain (Online). International Committee on Computational Linguistics. find our card
15. Ilia Kuznetsov and Iryna Gurevych. 2020. A matter of framing: The impact of linguistic formalism on probing results. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 171–182, Online. Association for Computational Linguistics. find
16. George A. Miller. 1992. WordNet: A lexical database for English. In Speech and Natural Language: Proceedings of a Workshop Held at Harriman, New York, February 23-26, 1992. find our card
17. Roberto Navigli, Edoardo Barba, Simone Conia, and Rexhina Blloshmi. 2022. A tour of explicit multilingual semantics: Word sense disambiguation, semantic role labeling and semantic parsing. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing: Tutorial Abstracts, pages 35–43, Taipei. Association for Computational Linguistics. find
18. Riccardo Orlando, Simone Conia, Fabrizio Brignone, Francesco Cecconi, and Roberto Navigli. 2021. AMuSE-WSD: An all-in-one multilingual system for easy Word Sense Disambiguation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 298–307, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
19. Riccardo Orlando, Simone Conia, Stefano Faralli, and Roberto Navigli. 2022. Universal semantic annotator: the first unified API for WSD, SRL and semantic parsing. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2634–2641, Marseille, France. European Language Resources Association. find
20. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71– 106. ↗ our card
21. Kexin Liao, Logan Lebanoff, and Fei Liu. 2018. Abstract Meaning Representation for multi-document summarization. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1178–1190, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
22. Sameer Pradhan, Julia Bonn, Skatje Myers, Kathryn Conger, Tim O’gorman, James Gung, Kristin Wright-bettner, and Martha Palmer. 2022. PropBank comes of Age—Larger, smarter, and more diverse. In Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 278–288, Seattle, Washington. Association for Computational Linguistics. find our card
23. Diego Marcheggiani, Jasmijn Bastings, and Ivan Titov. 2018. Exploiting semantics in neural machine translation with graph convolutional networks. In Pro-Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguistics. find our card
24. Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2019. The MuCoW test suite at WMT 2019: Automatically harvested multilingual contrastive word sense disambiguation test sets for machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 470–480, Florence, Italy. Association for Computational Linguistics. find
25. Arka Sadhu, Tanmay Gupta, Mark Yatskar, Ram Nevatia, and Aniruddha Kembhavi. 2021. Visual Semantic Role Labeling for Video Understanding. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5585–5596. ISSN: 2575-7075. find
26. Karin Kipper Schuler and Martha Palmer. 2005. Verbnet: a broad-coverage, comprehensive verb lexicon. find
27. Peng Shi and Jimmy Lin. 2019. Simple BERT Models for Relation Extraction and Semantic Role Labeling. find
28. Linfeng Song, Daniel Gildea, Yue Zhang, Zhiguo Wang, and Jinsong Su. 2019. Semantic neural machine translation using AMR. Transactions of the Association for Computational Linguistics, 7:19–31. find our card
29. Kevin Stowe, Jenette Preciado, Kathryn Conger, Susan Windisch Brown, Ghazaleh Kazeminejad, James Gung, and Martha Palmer. 2021. SemLink 2.0: Chasing Lexical Resources. In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 222–227, Groningen, The Netherlands (online). Association for Computational Linguistics. find our card
30. Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluís Màrquez, and Joakim Nivre. 2008. find
31. The CoNLL 2008 shared task on joint parsing of syntactic and semantic dependencies. In CoNLL 2008: Proceedings of the Twelfth Conference on Computational Natural Language Learning, pages 159–177, Manchester, England. Coling 2008 Organizing Committee. find our card
32. Rocco Tripodi, Simone Conia, and Roberto Navigli. 2021. UniteD-SRL: A unified dataset for span- and dependency-based multilingual and cross-lingual Semantic Role Labeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2293–2305, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
33. Mark Yatskar, Luke S. Zettlemoyer, and Ali Farhadi. 2016. Situation recognition: Visual semantic role labeling for image understanding. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 5534–5542. IEEE Computer Society. find
34. Yanpeng Zhao and Ivan Titov. 2020. Unsupervised Transfer of Semantic Role Models from Verbal to Nominal Domain. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/orlando-2023-exploring-non-verbal-predicates-semantic/