Mova0.5 pre-release⌕
☰
home / docs / papers / English / Parser Evaluation over Local and Non-Local Deep Dependencies in a Large Corpus

Parser Evaluation over Local and Non-Local Deep Dependencies in a Large Corpus

What it is about

The authors ask how well parsers recover the dependencies that matter for interpretation. They take ten linguistic phenomena that are reasonably frequent in running text, such as tough adjectives, right-node raising, expletive it, particle verbs, bare relatives and absolutives, and sample 100 sentences each from parsed English Wikipedia. They write gold dependencies for these 1,000 sentences. Seven parsers are scored by mapping their outputs to the targets with 364 regular expressions. Even the best parsers miss many of the relevant dependencies, and the per-phenomenon, per-dependency results (their Fig. 1) are the main finding. The paper argues for construction-focused evaluation in place of one aggregate score.

Ideas

For Mova

Touches 23. Complementation (tough adjectives, raising), 25. Relative clauses (bare relatives) and 29. Ellipsis, word order and information structure (right-node raising). The construction set (I1) is the first thing to build; it gives the per-construction gate that the single LAS number hides.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{bender2011parser,
  title = {{Parser Evaluation over Local and Non-Local Deep Dependencies in a Large Corpus}},
  author = {Bender, Emily M. and Flickinger, Dan and Oepen, Stephan and Zhang, Yi},
  booktitle = {Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing},
  year = {2011},
  url = {https://aclanthology.org/D11-1037/}
}

References

36 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jennifer E. Arnold, Thomas Wasow, Anthony Losongco, and Ryan Ginstrom. 2000. Heaviness vs. newness: The effects of structural complexity and discourse status on constituent ordering. Language, 76(1):28–55. find
  2. Ann Bies, Mark Ferguson, Karen Katz, and Robert Mac-Intyre. 1995. Bracketing guidelines for treebank II style Penn treebank project. Technical report, University of Pennsylvania. find
  3. Ted Briscoe, John Carroll, and Rebecca Watson. 2006. The second release of the RASP system. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 77–80, Sydney, Australia. find
  4. Aoife Cahill, John T. Maxwell III, Paul Meurer, Christian Rohrer, and Victoria Rosén. 2008. Speeding up LFG parsing using c-structure pruning. In Coling 2008: Proceedings of the workshop on Grammar Engineering Across Frameworks, pages 33–40, Manchester, England, August. Coling 2008 Organizing Committee. find
  5. John Carroll, Ted Briscoe, and Antonio Sanfilippo. 1998. Parser evaluation: A survey and a new proposal. In Proceedings of the International Conference on Language Resources and Evaluation, pages 447–454, Granada. find
  6. Daniel Cer, Marie-Catherine de Marneffe, Daniel Jurafsky, and Christopher D. Manning. 2010. Parsing to Stanford dependencies: Trade-offs between speed and accuracy. In 7th International Conference on Language Resources and Evaluation (LREC 2010), Malta. find our card
  7. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 173–180, Ann Arbor, Michigan. find our card
  8. Eugene Charniak. 1997. Statistical parsing with a context-free grammar and word statistics. In Proceedings of the Fourteenth National Conference on Artificial Intelligence, pages 598 – 603, Providence, RI. find
  9. Stephen Clark and James R. Curran. 2007. Wide-coverage efficient statistical parsing with CCG and log-linear models. Computational Linguistics, 33(4):493–552. find our card
  10. Alexander Clark. 2001. Unsupervised induction of stochastic context-free grammars using distributional clustering. In Proceedings of the 5th Conference on Natural Language Learning, pages 105–112. Toulouse, France. find
  11. Michael Collins. 1999. Head-Driven Statistical Models for Natural Language Parsing. Ph.D. thesis, University of Pennsylvania. find our card
  12. Dan Flickinger, Stephan Oepen, and Gisle Ytrestøl. 2010. WikiWoods. Syntacto-semantic annotation for English Wikipedia. In Proceedings of the 6th International Conference on Language Resources and Evaluation, Valletta, Malta. find
  13. Dan Flickinger. 2000. On building a more efficient grammar by exploiting types. Natural Language Engineering, 6 (1) (Special Issue on Efficient Processing with HPSG):15 – 28. find our card
  14. Sumukh Ghodke and Steven Bird. 2010. Fast query for large treebanks. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 267–275, Los Angeles, California, June. Association for Computational Linguistics. find
  15. Aria Haghighi and Dan Klein. 2006. Prototype-driven grammar induction. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 881–888, Sydney, Australia, July. Association for Computational Linguistics. find
  16. Julia Hockenmaier and Mark Steedman. 2007. CCGbank: A Corpus of CCG Derivations and Dependency Structures Extracted from the Penn Treebank. Computational Linguistics, 33(3):355–396. find our card
  17. Rodney Huddleston and Geoffrey K. Pullum. 2002. The Cambridge Grammar of the English Language. Cambridge University Press, Cambridge, UK. find our card
  18. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to-dependency conversion for English. In In Proceedings of NODALIDA 2007, pages 105–112, Tartu, Estonia. find our card
  19. Ron Kaplan, Stefan Riezler, Tracy H King, John T Maxwell III, Alex Vasserman, and Richard Crouch. 2004. Speed and accuracy in shallow and deep stochastic parsing. In Susan Dumais, Daniel Marcu, and Salim Roukos, editors, HLT-NAACL 2004: Main Proceedings, pages 97–104, Boston, Massachusetts, USA, May. Association for Computational Linguistics. find
  20. Dan Klein and Christopher D. Manning. 2003. Fast exact inference with a factored model for natural language parsing. In Advances in Neural Information Processing Systems 15, pages 3–10, Cambridge, MA. MIT Press. find
  21. Dan Klein and Christopher Manning. 2004. Corpus-based induction of syntactic structure: Models of dependency and constituency. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics, pages 478–485, Barcelona, Spain. find
  22. Dekang Lin. 1995. A dependency-based method for evaluating broad-coverage parsers. In Proceedings of IJCAI-95, pages 1420–1425, Montreal, Canada. find
  23. Robert Malouf. 2000. Verbal gerunds as mixed categories in HPSG. In Robert Borsley, editor, The Nature and Function of Syntactic Categories, pages 133–166. Academic Press, New York. find
  24. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19:313–330. find our card
  25. Ryan McDonald, Fernando Pereira, Kiril Ribarov, and Jan Hajic. 2005. Non-Projective Dependency Parsing using Spanning Tree Algorithms. In Proceedings of the 2005 Conference on Empirical Methods in Natural Language Processing, pages 523–530, Vancouver, Canada. find our card
  26. Yusuke Miyao, Takashi Ninomiya, and Jun’ichi Tsujii. 2004. Corpus-oriented grammar development for acquiring a Head-driven Phrase Structure Grammar from the Penn Treebank. In Proceedings of the 1st International Joint Conference on Natural Language Processing, pages 684–693, Hainan Island, China. find
  27. Joakim Nivre, Laura Rimell, Ryan McDonald, and Carlos Gómez Rodrı́guez. 2010. Evaluation of dependency parsers on unbounded dependencies. In Proceedings of the 23rd International Conference on Computational Linguistics, pages 833–841, Beijing, China. find our card
  28. Ruth O’Donovan, Michael Burke, Aoife Cahill, Josef Van Genabith, and Andy Way. 2004. Large-scale induction and evaluation of lexical resources from the penn-ii treebank. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics, pages 367–374, Barcelona, Spain. find
  29. Stephan Oepen, Daniel Flickinger, Kristina Toutanova, and Christopher D. Manning. 2004. LinGO Redwoods. A rich and dynamic treebank for HPSG. Journal of Research on Language and Computation, 2(4):575 – 596. find our card
  30. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 433–440, Sydney, Australia. find our card
  31. Stefan Riezler, Tracy H. King, Ronald M. Kaplan, Richard Crouch, John T. Maxwell III, and Mark Johnson. 2002. Parsing the Wall Street Journal using a Lexical-Functional Grammar and discriminative estimation techniques. In Proceedings of the 40th Meeting of the Association for Computational Linguistics, Philadelphia, PA. find
  32. Laura Rimell, Stephen Clark, and Mark Steedman. 2009. Unbounded dependency recovery for parser evaluation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 813–821, Singapore. Association for Computational Linguistics. find our card
  33. Ivan Sag, Timothy Baldwin, Francis Bond, Ann Copestake, and Dan Flickinger. 2002. Multiword expressions. A pain in the neck for NLP. In Alexander Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, volume 2276 of Lecture Notes in Computer Science, pages 189–206. Springer, Berlin, Germany. find our card
  34. David Vadas and James R. Curran. 2008. Parsing noun phrase structure with CCG. In Proceedings of ACL-08: HLT, pages 335–343, Columbus, Ohio, June. Association for Computational Linguistics. find
  35. L. van der Beek, Gosse Bouma, Robert Malouf, and Gertjan van Noord. 2002. The Alpino Dependency Treebank. In Mariet Theune, Anton Nijholt, and Hendri Hondorp, editors, Computational Linguistics in the Netherlands, Amsterdam, The Netherlands. Rodopi. find
  36. Luke Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Proceedings of the Twenty-First Annual Conference on Uncertainty in Artificial Intelligence, pages 658–666, Arlington, Virginia. AUAI Press. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2011-bender-parser-evaluation-over-local-non/