Controlled Crowdsourcing for High-Quality QA-SRL Annotation

What it is about

The paper shows that the crowdsourced QA-SRL data of 2018 has low coverage, with an estimated recall in the low 70s, and proposes a controlled protocol to fix it. Workers are screened: 30 were trained with two qualification rounds of 15 predicates each, and 11 were kept. Two workers generate questions and answers for each verb, and a third, the consolidator, reviews both sets, keeps or merges answers and removes redundant questions. The result is a gold set of 1,000 sentences each from Wikinews and Wikipedia, with 25% more roles than the 2018 data at a cost of 54.2 cents per predicate. Worker-against-worker agreement is an F1 of 79.8, and consolidated triplets agree at 84.1. Against an expert set, the new gold set reaches an unlabeled F1 of 84.4 against 68.3 for the 2018 dense set. A parser baseline reaches 63.7 unlabeled F1 on the test set.

Ideas

• I1. Use the consolidation stage for Mova's gold school sets: two independent annotators per item and a third adjudicates. Report the pairwise agreement before any set is used, as a precondition for trusting it.
• I2. A short training round with automatic comparison to expert answers, then keep the best third of the annotators. The same idea applies to any human labeling of lessons.
• I3. Strict match for question equivalence: two questions are equivalent only if they agree on wh-word, subject, object, negation, voice and modality. Use this as the equivalence test when Mova evaluates generated questions in world/src/qa.rs.
• I4. Use agreement with the PropBank roles as an outside check on Mova's frames, on a sample of sentences from the school corpus.

For Mova

The relevant chapters are 26. Questions and 19. The lexicon: vocabulary, multiword expressions and register. The code is world/src/qa.rs for the equivalence check and the evaluation of question generation. The measures are the pairwise agreement before a set is accepted, and the strict-match accuracy of generated questions against the gold set. The gold set stays held out and is used once per release.

Topics: Semantic role labeling

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.626

@inproceedings{roit2020controlled,
  title = {{Controlled Crowdsourcing for High-Quality QA-SRL Annotation}},
  author = {Roit, Paul and Klein, Ayal and Stepanov, Daniela and Mamou, Jonathan and Michael, Julian and Stanovsky, Gabriel and Zettlemoyer, Luke and Dagan, Ido},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.626},
  url = {https://doi.org/10.18653/v1/2020.acl-main.626}
}

References

11 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Omri Abend and Ari Rappoport. 2013. Universal conceptual cognitive annotation (ucca). In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 228–238. find our card
2. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The berkeley framenet project. In Proceedings of the 17th International Conference on Computational Linguistics - Volume 1, COLING ’98, pages 86–90, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
3. Shany Barhom, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019. Revisiting joint modeling of cross-document entity and event coreference resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4179–4189, Florence, Italy. Association for Computational Linguistics. find
4. Yun-Nung Chen, William Yang Wang, and Alexander I Rudnicky. 2013. Unsupervised induction and filling of semantic slots for spoken dialogue systems using frame-semantic parsing. In 2013 IEEE Workshop on Automatic Speech Recognition and Understanding, pages 120–125. IEEE. find
5. Pengxiang Cheng and Katrin Erk. 2018. Implicit argument prediction with event knowledge. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 831–840. find
6. Quynh Ngoc Thi Do, Steven Bethard, and Marie-Francine Moens. 2017. Improving implicit semantic role labeling by predicting semantic frame arguments. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 90–99. find
7. Nicholas Fitzgerald, Julian Michael, Luheng He, and Luke S. Zettlemoyer. 2018. Large-scale qa-srl parsing. In ACL. find our card
8. Luheng He, Mike Lewis, and Luke S. Zettlemoyer. 2015. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In EMNLP. find our card
9. Martha Palmer, Dan Gildea, and Paul Kingsbury. 2005. The proposition bank: a corpus annotated with semantic roles. Computational Linguistics Journal, 31(1). find
10. Gabriel Stanovsky and Ido Dagan. 2016. Creating a large benchmark for open information extraction. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2300–2305. find
11. Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2015. Machine comprehension with syntax, frames, and semantics. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), volume 2, pages 700–706. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/roit-2020-controlled-crowdsourcing-high-quality-qa/