Question-Answer Driven Semantic Role Labeling: Using Natural Language to Annotate Natural Language

What it is about

The paper introduces question-answer driven semantic role labeling (QA-SRL): each verbal predicate is labeled with wh-questions, such as "Who finished something?", and each question is answered by phrases of the sentence. The questions come from a template of seven fields, so the roles are defined by the questions and not by a frame inventory. Non-expert annotators, hired on Upwork after a short screen, labeled over 3,000 sentences (nearly 8,000 verbs) at about 50 cents per verb. Agreement with PropBank is high: over all roles the precision is 81.4 and the recall 86.3, and for core arguments 85.9 and 89.8. Two baselines are given. Question generation uses a classifier over roles followed by templates; answer identification is a per-word classifier that finds the answer head, and it is right for about 80% of test questions, against a random baseline of about 26%.

Ideas

• I1. Answer head as a deterministic rule on the UD tree: the head of an answer span is a word whose governor lies outside the span, as the paper defines it. Implement it in en/src/parse.rs and measure head accuracy against gold spans on UD EWT dev.
• I2. Map the wh-words to role classes in prag/src/roles.rs: who and what as core, when, where, why and how as adjuncts. Report the agreement with the school's role labels, in the form of the paper's Table 5.
• I3. The mapping table of wh-field and voice to roles (the paper's Table 7) is a closed table. Put it in 28. Voice and the passive and test it on active and passive school sentences.
• I4. Negative control: choose the answer head at random among the children of the verb. The paper's random baseline is about 26%; Mova's rule should beat it by a wide margin on the same sentences.

For Mova

The chapters are 26. Questions and 28. Voice and the passive. The code is en/src/parse.rs and prag/src/roles.rs. The measures are the answer-head accuracy on UD EWT dev, the core and adjunct agreement on school sentences, and the random-choice control.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1076

@inproceedings{he2015question,
  title = {{Question-Answer Driven Semantic Role Labeling: Using Natural Language to Annotate Natural Language}},
  author = {He, Luheng and Lewis, Mike and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1076},
  url = {https://doi.org/10.18653/v1/d15-1076}
}

References

28 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (ucca). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 228–238. find our card
2. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of the 17th International Conference on Computational Linguistics, volume 1, pages 86–90. Association for Computational Linguistics. find our card
3. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In Proceedings of the Linguistic Annotation Workshop. find our card
4. Valerio Basile, Johan Bos, Kilian Evang, and Noortje Venhuizen. 2012. Developing a large semantically annotated corpus. In Proceedings of the 2012 International Conference on Language Resources and Evaluation, volume 12, pages 3196–3200. find
5. Claire Bonial, Olga Babko-Malaya, Jinho D Choi, Jena Hwang, and Martha Palmer. 2010. Propbank annotation guidelines. Center for Computational Language and Education Research, CU-Boulder. find
6. Desai Chen, Nathan Schneider, Dipanjan Das, and Noah A Smith. 2010. Semafor: Frame argument resolution with log-linear models. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 264–267. Association for Computational Linguistics. find
7. Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. Liblinear: A library for large linear classification. The Journal of Machine Learning Research, 9:1871–1874. find
8. Jan Hajič, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, Maria Antònia Martı́, Lluı́s Màrquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štěpánek, et al. 2009. The conll-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–18. Association for Computational Linguistics. find our card
9. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: the 90% solution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, pages 57–60. Association for Computational Linguistics. find
10. Richard Johansson and Pierre Nugues. Dependency-based semantic role labeling of propbank. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 69–78. Association for Computational Linguistics. find our card
11. Paul Kingsbury and Martha Palmer. 2002. From treebank to propbank. In Proceedings of the 2002 International Conference on Language Resources and Evaluation. Citeseer. find our card
12. Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, volume 1, pages 423–430. Association for Computational Linguistics. find our card
13. Joel Lang and Mirella Lapata. 2011. Unsupervised semantic role induction via split-merge clustering. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, volume 1, pages 1117–1126. Association for Computational Linguistics. find
14. Mike Lewis, Luheng He, and Luke Zettlemoyer. 2015. Joint a* ccg parsing and semantic role labeling. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
15. Ding Liu and Daniel Gildea. 2010. Semantic role features for machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 716–724. Association for Computational Linguistics. find
16. Bill MacCartney and Christopher D Manning. 2007. Natural logic for textual inference. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 193–200. Association for Computational Linguistics. find
17. Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2):313–330. find our card
18. Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The nombank project: An interim report. In HLT-NAACL 2004 workshop: Frontiers in corpus annotation, pages 24–31. find
19. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. ↗ our card
20. Martha Palmer, Daniel Gildea, and Nianwen Xue. 2010. Semantic role labeling. Synthesis Lectures on Human Language Technologies, 3(1):1–103. find
21. Simone Paolo Ponzetto and Michael Strube. 2006. Exploiting semantic role labeling, wordnet and wikipedia for coreference resolution. In Proceedings of the Human Language Technology Confer-2008. ence of the North American Chapter of the Association of Computational Linguistics, pages 192–199. Association for Computational Linguistics. find
22. Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. Transactions of the Association for Computational Linguistics, 3:475–488. find
23. Josef Ruppenhofer, Michael Ellsworth, Miriam RL Petruck, Christopher R Johnson, and Jan Scheffczyk. 2006. Framenet ii: Extended theory and practice. find
24. Asher Stern and Ido Dagan. 2014. Recognizing implied predicate-argument relationships in textual inference. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. find
25. Ivan Titov and Alexandre Klementiev. 2012. A bayesian approach to unsupervised semantic role induction. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pages 12–22. Association for Computational Linguistics. find
26. Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology, volume 1, pages 173–180. Association for Computational Linguistics. find
27. Nianwen Xue and Martha Palmer. 2004. Calibrating features for semantic role labeling. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 88–94. find
28. Deniz Yuret, Aydin Han, and Zehra Turgut. 2010. Semeval-2010 task 12: Parser evaluation using textual entailments. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 51–56. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/he-2015-question-answer-driven-semantic-role/