Towards Semi-Automated Annotation for Prepositional Phrase Attachment

What it is about

The paper asks whether untrained crowd workers can label PP attachment well enough to make training data. The system turns Penn Treebank parses into multiple-choice questions: the PP and its alternative noun or verb heads. The study used 1000 questions from Wall Street Journal sentences, each answered by three Amazon Mechanical Turk workers, at a cost of $135 and 49 seconds per task. Workers picked the right attachment in 86.7% of the 3000 responses. The majority vote was right for 90.4% of questions, and 92.2% after the authors removed 18 of 96 wrong-majority cases caused by their own question generator. A unanimous answer was correct in 97% of cases, a bare majority in 82%.

Ideas

I1. annot: keep a silver PP head only when three independent sources agree (a large model, a tools/treeagree rule, and the en parse), and drop the rest. Measure the precision of the kept heads on a 300-item gold sample from EWT, and the share kept. Use EWT only, since it is CC BY-SA.
I2. Error map: the workers failed mostly when the PP attached to a verb or an adjective, and rarely to a noun. Report PP head accuracy in en parse-eval split by the gold head's part of speech, on EWT test.
I3. Two-head check: the paper found items with two correct attachments, which the answer key treated as one. Mark PPs with more than one defensible head in the EWT gold set and score any of them as correct. Measure the ceiling before tuning any model.
I4. Only the agreement rule is taken from this work. Mova does not use a crowd, and the Penn Treebank data is test only.

For Mova

The code is en/src/parse.rs for the PP scores and annot for the silver rule. The chapter is 22. Phrases: noun, adjective, adverb and prepositional phrases. The paper's questions come from WSJ text (LDC-licensed), so any measurement on that text is test only. Its 92.2% is a majority of three people on 1000 questions, so it is not a parser score and must not be read as one.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{rosenthal2010semi,
  title = {{Towards Semi-Automated Annotation for Prepositional Phrase Attachment}},
  author = {Rosenthal, Sara and Lipovsky, William and McKeown, Kathleen and Thadani, Kapil and Andreas, Jacob},
  booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)},
  year = {2010},
  url = {https://aclanthology.org/L10-1572/}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Eneko Agirre, Timothy Baldwin, and David Martinez. 2008. Improving parsing and PP attachment performance with sense information. In Proceedings of ACL, pages 317–325, June. find
Michaela Atterer and Hinrich Schütze. 2007. Prepositional phrase attachment without oracles. Computational Linguistics, 33(4):469–476. find
Eric Brill. 1995. Transformation-based error-driven learning and natural language processing: a case study in part-of-speech tagging. Computational Linguistics, 21(4):543–565. find
Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of ACL, pages 132–139. find
Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational Linguistics, 29(4):589–637. find our card
Daniel Gruhl, R. Guha, David Liben-Nowell, and Andrew Tomkins. 2004. Information diffusion through blogspace. In WWW ’04: Proceedings of the 13th international conference on World Wide Web, pages 491–501. find
Liangjie Hong and Brian D. Davison. 2009. A classification-based approach to question answering in discussion boards. In SIGIR ’09: Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 171– 178. find
Gilly Leshed and Joseph ’Jofish’ Kaye. 2006. Understanding how bloggers feel: recognizing affect in blog posts. In CHI ’06 extended abstracts on Human factors in computing systems, pages 1019–1024. find
Mitchell P. Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993. Building a large annotated corpus of english: the Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. The nombank project: An interim report. In Adam Meyers, editor, Proceedings of the HLT-NAACL Workshop on Frontiers in Corpus Annotation, pages 24– 31, Boston, Massachusetts, USA, May. find
Martha Palmer, Daniel Gildea, and Paul Kingsbury. 2005. The Proposition Bank: An annotated corpus of semantic roles. Computational Linguistics, 31(1):71–106. ↗ our card
Sameer S. Pradhan and Nianwen Xue. 2009. Ontonotes: the 90% solution. In Proceedings of NAACL, pages 11– 12. find
Adwait Ratnaparkhi, Jeff Reynar, and Salim Roukos. 1994. A maximum entropy model for prepositional phrase attachment. In Proceedings of HLT, pages 250–255. find our card
Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast—but is it good?: evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP, pages 254–263. find
Jiri Stetina and Makoto Nagao. 1997. Corpus based pp attachment ambiguity resolution with a semantic dictionary. In Proceedings of the Workshop on Very Large Corpora, pages 66–80. find
Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task: chunking. In Proceedings of the CoNLL workshop on Learning Language in Logic, pages 127–132. find
Alexander S. Yeh and Marc B. Vilain. 1998. Some properties of preposition and subordinate conjunction attachments. In Proceedings of COLING, pages 1436–1442. find
Jakub Zavrel, Walter Daelemans, and Jorn Veenstra. 1997. Resolving PP attachment ambiguities with memory-based learning. In Proc. of the Workshop on Computational Language Learning (CoNLL’97), ACL, pages 136–144. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2010-rosenthal-semi-automated-annotation-prepositional-phrase/