Large-Scale QA-SRL Parsing

What it is about

The paper releases QA-SRL Bank 2.0, which has 265,140 valid question-answer pairs for 64,018 sentences and 133,479 verbs across Wikipedia, Wikinews and science textbooks. A streamlined crowdsourcing interface, with an autocomplete automaton over the seven question slots and a separate validation stage with two validators, was used; the whole set took nine days and cost $43,647, or 16.5 cents per valid question. The parser is a pipeline: span detection (a BIO tagger, or a span-based model that scores every span) followed by question generation (a local slot classifier, or a sequential LSTM decoder). The final parser reaches 82.6% question accuracy and 77.6% span accuracy in a human evaluation. Re-running the parser to over-generate questions and validating them with the crowd adds 20% valid questions to the training and development sets.

Ideas

• I1. The seven-slot question template (WH, AUX, SBJ, TGT, OBJ, PREP, MISC) as a deterministic grammar for school questions in world/src/qa.rs. The paper's autocomplete automaton is a grammaticality filter, and it can be written as a finite-state check in Rust.
• I2. Span detection from UD subtrees: candidate answers are the subtrees of the verb's dependents, scored by a count-based rule, and kept above a threshold. Measure span F1 on dev with the paper's IOU of at least 0.5.
• I3. Data expansion as a deterministic loop: generate candidate question-answer pairs from the parser, keep those not already covered by an answer, and validate them by a second, independent source. Keep the held-out test untouched.
• I4. Negative control: shuffle the question slots of correct questions. Question accuracy must drop, otherwise the slots are not doing any work.

For Mova

The chapters are 26. Questions and 13. Verbs. The code is world/src/qa.rs and en/src/parse.rs for the subtrees. The measures are span F1 at IOU 0.5 on dev, question accuracy against a human-checked sample, and the negative control. Run each twice.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1191

@inproceedings{fitzgerald2018large,
  title = {{Large-Scale QA-SRL Parsing}},
  author = {FitzGerald, Nicholas and Michael, Julian and He, Luheng and Zettlemoyer, Luke},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1191},
  url = {https://doi.org/10.18653/v1/p18-1191}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (UCCA). In ACL 2013. find our card
2. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The Berkeley Framenet project. In ICCL 1998. find our card
3. Laura Banarescu, Claire Bonial, Shu Cai, Madalina Georgescu, Kira Griffitt, Ulf Hermjakob, Kevin Knight, Philipp Koehn, Martha Palmer, and Nathan Schneider. 2013. Abstract meaning representation for sembanking. In 7th Linguistic Annotation Workshop and Interoperability with Discourse. find our card
4. Valerio Basile, Johan Bos, Kilian Evang, and Noortje Venhuizen. 2012. Developing a large semantically annotated corpus. In LREC 2012. find
5. Claire Bonial, Olga Babko-Malaya, Jinho D Choi, Jena Hwang, and Martha Palmer. 2010. Propbank annotation guidelines. find
6. Ronan Collobert and Jason Weston. 2007. Fast semantic extraction using a novel neural network architecture. In ACL 2007. find
7. Manjuan Duan, Ethan Hill, and Michael White. 2016. Generating disambiguating paraphrases for structurally ambiguous sentences. In 10th Linguistic Annotation Workshop. find
8. Josef Ruppenhofer, Michael Ellsworth, Miriam RL Petruck, Christopher R Johnson, and Jan Scheffczyk. 2016. FrameNet II: Extended theory and practice. Institut für Deutsche Sprache, Bibliothek. find
9. Nicholas FitzGerald, Oscar Täckström, Kuzman Ganchev, and Dipanjan Das. 2015. Semantic role labeling with neural network factors. In EMNLP 2015. find
10. Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Training very deep networks. In NIPS 2015. find
11. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In NIPS 2016. find
12. Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In ACL 2018. find
13. Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. 2017. Deep semantic role labeling: What works and whats next. In ACL 2017. find our card
14. Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In EMNLP 2015. find our card
15. Luheng He, Julian Michael, Mike Lewis, and Luke Zettlemoyer. 2016. Human-in-the-loop parsing. In EMNLP 2016. find
16. Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In CVPR 2017. find
17. Gabriel Stanovsky and Ido Dagan. 2016. Creating a large benchmark for open information extraction. In EMNLP 2016. find
18. Swabha Swayamdipta, Sam Thomson, Chris Dyer, and Noah A Smith. 2017. Frame-semantic parsing with softmax-margin segmental rnns and a syntactic scaffold. arXiv preprint arXiv:1706.09528. ↗
19. Adam R Teichert, Adam Poliak, Benjamin Van Durme, and Matthew R Gormley. 2017. Semantic proto-role labeling. In AAAI 2017, pages 4459–4466. find
20. Chenguang Wang, Alan Akbik, Laura Chiticariu, Yunyao Li, Fei Xia, and Anbang Xu. 2018. Crowd-in-the-loop: A hybrid approach for annotating semantic roles. In EMNLP 2017. find
21. Aaron Steven White, Kyle Rawlins, and Benjamin Van Durme. 2017. The semantic proto-role linking model. In ACL 2017. find
22. Bishan Yang and Tom Mitchell. 2017. A joint sequential and relational model for frame-semantic parsing. In EMNLP 2017, pages 1247–1256. find
23. Jie Zhou and Wei Xu. 2015. End-to-end learning of semantic role labeling using recurrent neural networks. In ACL 2015. find
24. Kenton Lee, Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2017. End-to-end neural coreference resolution. In EMNLP 2017. find our card
25. Kenton Lee, Shimi Salant, Tom Kwiatkowski, Ankur Parikh, Dipanjan Das, and Jonathan Berant. 2016. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436. ↗
26. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The stanford corenlp natural language processing toolkit. In ACL 2014, pages 55–60. find
27. Julian Michael, Gabriel Stanovsky, Luheng He, Ido Dagan, and Luke Zettlemoyer. 2018. Crowdsourcing question-answer meaning representations. In NAACL 2018. find our card
28. Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The proposition bank: An annotated corpus of semantic roles. Computational Linguistics. find our card
29. Drew Reisinger, Rachel Rudinger, Francis Ferraro, Craig Harman, Kyle Rawlins, and Benjamin Van Durme. 2015. Semantic proto-roles. TACL. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/fitzgerald-2018-large-scale-qa-srl-parsing/