Mova0.5 pre-release⌕
☰
home / docs / papers / English / Out-of-domain FrameNet Semantic Role Labeling

Out-of-domain FrameNet Semantic Role Labeling

What it is about

FrameNet role labeling is usually tested on one in-domain set (das-test). The authors test how it generalises. They build YAGS, a new test set of 1,415 sentences from Yahoo! Answers with 3,091 frame and 6,081 role annotations. Inter-rater agreement is Krippendorff's alpha 0.76 for frames and 0.62 for roles given matching spans. With the open-source Semafor system, frame identification accuracy is 82.09% on das-test but 39.52% on MASC and 62.17% on the Twitter sets, and role labeling collapses out of domain. With gold frames, the gap in role labeling disappears, so frame identification is the bottleneck. The authors propose a simpler frame identifier: averaged word vectors of the predicate and its context, with a neural classifier or WSABIE. It reaches 87.63% total accuracy in domain and 62.03% on YAGS, 53.73% on MASC and 68.67% on the Twitter average, above Semafor on all sets, least clearly on YAGS.

Ideas

For Mova

The chapters are 13. Verbs and 19. The lexicon: vocabulary, multiword expressions and register. The code is prag/src/frames.rs and the lexicon in en/src/dict.rs. The measure is frame accuracy on school sentences split by whether the predicate is in the lexicon, run twice to confirm identical output.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{hartmann2017out,
  title = {{Out-of-domain FrameNet Semantic Role Labeling}},
  author = {Hartmann, Silvana and Kuznetsov, Ilia and Martin, Teresa and Gurevych, Iryna},
  booktitle = {Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers},
  year = {2017},
  url = {https://aclanthology.org/E17-1045/}
}

References

37 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Oier López de Lacalle, Christiane Fellbaum, Shu-Kai Hsieh, Maurizio Tesconi, Monica Monachini, Piek Vossen, and Roxanne Segers. 2010. SemEval-2010 Task 17: All-Words Word Sense Disambiguation on a Specific Domain. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 75–80. Association for Computational Linguistics. find
  2. Collin Baker, Michael Ellsworth, and Katrin Erk. 2007. SemEval-2007 Task 19: Frame Semantic Structure Extraction. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval-2007), pages 99–104, Prague, Czech Republic, June. Association for Computational Linguistics. find
  3. Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. 2014. Modeling Biological Processes for Reading Comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1499–1510, Doha, Qatar. Association for Computational Linguistics. find
  4. John Blitzer, Ryan McDonald, and Fernando Pereira. 2006. Domain adaptation with structural correspondence learning. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 120–128, Sydney, Australia, July. Association for Computational Linguistics. find
  5. Xavier Carreras and Lluı́s Màrquez. 2005. Introduction to the CoNLL-2005 shared task: Semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning (CoNLL-2005), pages 152–164, Ann Arbor, Michigan, June. Association for Computational Linguistics. find our card
  6. Danilo Croce, Cristina Giannone, Paolo Annesi, and Roberto Basili. 2010. Towards open-domain semantic role labeling. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 237–246, Uppsala, Sweden, July. Association for Computational Linguistics. find
  7. Dipanjan Das and Noah A. Smith. 2011. Semi-Supervised Frame-Semantic Parsing for Unknown Predicates. In Proc. of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1435–1444, Portland, Oregon, USA. find
  8. Dipanjan Das, Nathan Schneider, Desai Chen, and Noah A. Smith. 2010. Probabilistic Frame-Semantic Parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 948–956, Los Angeles, California. Association for Computational Linguistics. find
  9. Dipanjan Das, Desai Chen, André F. T. Martins, Nathan Schneider, and Noah A. Smith. 2014. Frame-semantic parsing. Computational Linguistics, 40(1):9–56. find
  10. Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256–263, Prague, Czech Republic, June. Association for Computational Linguistics. find
  11. Katrin Erk and Sebastian Padó. 2006. SHAL-MANESER – A Toolchain For Shallow Semantic Parsing. In Proceedings of the 5th International Conference on Language Resources and Evaluation (LREC 2006), volume 6, pages 527–532, Genoa, Italy. ELRA. find
  12. Charles J. Fillmore, Christopher R. Johnson, and Miriam R.L. Petruck. 2003. Background to FrameNet. International journal of lexicography, 16(3):235–250. find
  13. Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 960–970, Lisbon, Portugal, September. Association for Computational Linguistics. find
  14. Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martı́, Lluı́s Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, Pavel Straňák, Mihai Surdeanu, Nianwen Xue, and Yi Zhang. 2009. The conll-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL 2009): Shared Task, pages 1–18, Boulder, Colorado, June. Association for Computational Linguistics. find our card
  15. Karl Moritz Hermann, Dipanjan Das, Jason Weston, and Kuzman Ganchev. 2014. Semantic frame identification with distributed word representations. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1448–1458, Baltimore, Maryland, June. Association for Computational Linguistics. find
  16. Fei Huang and Alexander Yates. 2010. Open-domain semantic role labeling by modeling word spans. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 968– 978, Uppsala, Sweden, July. Association for Computational Linguistics. find
  17. Ignacio Iacobacci, Mohammad Taher Pilehvar, and Roberto Navigli. 2016. Embeddings for Word Sense Disambiguation: An Evaluation Study. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 897–907, Berlin, Germany, August. Association for Computational Linguistics. find
  18. Anders Johannsen, Héctor Martı́nez Alonso, and Anders Søgaard. 2015. Any-language frame-semantic parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2062–2066, Lisbon, Portugal, September. Association for Computational Linguistics. find
  19. Richard Johansson and Pierre Nugues. 2008. The effect of syntactic representation on semantic role labeling. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 393–400, Manchester, UK, August. Coling 2008 Organizing Committee. find
  20. Meghana Kshirsagar, Sam Thomson, Nathan Schneider, Jaime Carbonell, Noah A. Smith, and Chris Dyer. 2015. Frame-semantic role labeling with heterogeneous annotations. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 218–224, Beijing, China, July. Association for Computational Linguistics. find
  21. Maciej Kula. 2015. Metadata embeddings for user and item cold-start recommendations. In Toine Bogers and Marijn Koolen, editors, Proceedings of the 2nd Workshop on New Trends on Content-Based Recommender Systems co-located with 9th ACM Conference on Recommender Systems (RecSys 2015), volume 1448 of CEUR Workshop Proceedings, pages 14–21, Vienna, Austria, September. CEUR-WS.org. find
  22. Omer Levy and Yoav Goldberg. 2014. Dependency-based word embeddings. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014, June 22-27, 2014, Baltimore, MD, USA, Volume 2: Short Papers, pages 302–308. The Association for Computer Linguistics. find
  23. Oren Melamud, Jacob Goldberger, and Ido Dagan. 2016. context2vec: Learning generic context embedding with bidirectional LSTM. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, CoNLL 2016, Berlin, Germany, August 11-12, 2016, pages 51–61. find
  24. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed Representations of Words and Phrases and Their Compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS ’13), pages 3111–3119, Lake Tahoe, Nevada, USA. find
  25. Alexis Palmer and Caroline Sporleder. 2010. Evaluating FrameNet-style semantic parsing: the role of coverage gaps in FrameNet. In Proceedings of the 23rd International Conference on Computational Linguistics: Posters, pages 928–936, Beijing, China, August. find
  26. Rebecca J. Passonneau, Collin F. Baker, Christiane Fellbaum, and Nancy Ide. 2012. The MASC Word Sense Corpus. In Proceedings of the Eight International Conference on Language Resources and Evaluation (LREC’12), pages 3025–3030, Istanbul, Turkey. find
  27. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532– 1543, Doha, Qatar, October. Association for Computational Linguistics. find
  28. Michael Roth and Mirella Lapata. 2015. Context-aware frame-semantic role labeling. Transactions of the Association for Computational Linguistics, 3:449–460. find
  29. Josef Ruppenhofer, Michael Ellsworth, Miriam R. L. Petruck, Christopher R. Johnson, and Jan Scheffczyk. 2010. FrameNet II: Extended Theory and Practice. Technical report, ICSI, University of California, Berkeley. find
  30. Anders Søgaard, Barbara Plank, and Héctor Martı́nez Alonso. 2015. Using Frame Semantics for Knowledge Extraction from Twitter. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 2447–2452, Austin, Texas, USA. find
  31. Anders Søgaard. 2013. Semi-supervised learning and domain adaptation in natural language processing. Synthesis Lectures on Human Language Technologies, 6(2):1–103. find
  32. Mihai Surdeanu, Richard Johansson, Adam Meyers, Lluı́s Màrquez, and Joakim Nivre. 2008. The conll 2008 shared task on joint parsing of syntactic and semantic dependencies. In CoNLL 2008: Proceedings of the Twelfth Conference on Computational Natural Language Learning, pages 159–177, Manchester, England, August. Coling 2008 Organizing Committee. find our card
  33. Mihai Surdeanu, Massimiliano Ciaramita, and Hugo Zaragoza. 2011. Learning to rank answers to non-factoid questions from web collections. Computational Linguistics, 37(2):351–383. find
  34. Kaveh Taghipour and Hwee Tou Ng. 2015. Semi-Supervised Word Sense Disambiguation Using Word Embeddings in General and Specific Domains. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 314–323, Denver, Colorado, May–June. Association for Computational Linguistics. find
  35. Jason Weston, Samy Bengio, and Nicolas Usunier. 2011. WSABIE: Scaling Up to Large Vocabulary Image Annotation. In Proceedings of the Twenty-Second International Joint Conference on Artificial Intelligence - Volume Volume Three, IJ-CAI’11, pages 2764–2770, Barcelona, Catalonia, Spain. AAAI Press. find
  36. Haitong Yang, Tao Zhuang, and Chengqing Zong. 2015. Domain adaptation for syntactic and semantic dependency parsing using deep belief networks. Transactions of the Association for Computational Linguistics, 3:271–282. find
  37. Seid Muhie Yimam, Richard Eckart de Castilho, Iryna Gurevych, and Chris Biemann. 2014. Automatic Annotation Suggestions and Custom Annotation Layers in WebAnno. In Kalina Bontcheva and Zhu Jingbo, editors, Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics. System Demonstrations, pages 91–96, Stroudsburg, PA 18360, USA. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/hartmann-2017-out-domain-framenet-semantic-role/