Mova0.5 pre-release⌕
☰
home / docs / papers / English / Formalism-Independent Parser Evaluation with CCG and DepBank

Formalism-Independent Parser Evaluation with CCG and DepBank

What it is about

Parsers that use different formalisms produce different output, so their scores cannot be compared directly. The authors evaluate a CCG parser on DepBank, a set of grammatical relations that does not match the parser's own output. The main difficulty is the conversion from CCG output to DepBank relations, so they measure the conversion itself: parse the gold analyses, convert them, and compare. This gives an upper bound on what the parser can score. The CCG parser reaches an F-score of 81.9% on labelled dependencies against an upper bound of 84.8%, and beats the RASP parser by over 5% overall. The paper's message is to evaluate on a resource that was not used to develop the parser, and to report the conversion ceiling next to the score.

Ideas

For Mova

Touches 20. Syntax: the Universal Dependencies frame. The conversion ceiling is a cheap check that the mova dialect does not lose information, which the converters must keep. Measure I1 first; it is the quickest of the three and needs no new data.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{clark2007formalism,
  title = {{Formalism-Independent Parser Evaluation with CCG and DepBank}},
  author = {Clark, Stephen and Curran, James},
  booktitle = {Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics},
  year = {2007},
  url = {https://aclanthology.org/P07-1032/}
}

References

22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rens Bod. 2003. An efficient implementation of a new DOP model. In Proceedings of the 10th Meeting of the EACL, pages 19–26, Budapest, Hungary. find
  2. Ted Briscoe and John Carroll. 2006. Evaluating the accuracy of an unlexicalized statistical parser on the PARC DepBank. In Proceedings of the Poster Session of COLING/ACL-06, Sydney, Australia. find
  3. Ted Briscoe, John Carroll, and Rebecca Watson. 2006. The second release of the RASP system. In Proceedings of the Interactive Demo Session of COLING/ACL-06, Sydney, Australia. find
  4. Sabine Buchholz, Jorn Veenstra, and Walter Daelemans. 1999. Cascaded grammatical relation assignment. In Proceedings of EMNLP/VLC-99, pages 239–246, University of Maryland, June 21-22. find
  5. A. Cahill, M. Burke, R. O’Donovan, J. van Genabith, and A. Way. 2004. Long-distance dependency resolution in automatically acquired wide-coverage PCFG-based LFG approximations. In Proceedings of the 42nd Meeting of the ACL, pages 320–327, Barcelona, Spain. find
  6. John Carroll, Ted Briscoe, and Antonio Sanfilippo. 1998. Parser evaluation: a survey and a new proposal. In Proceedings of the 1st LREC Conference, pages 447–454, Granada, Spain. find
  7. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine nbest parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the ACL, University of Michigan, Ann Arbor. find
  8. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the 1st Meeting of the NAACL, pages 132– 139, Seattle, WA. find
  9. Stephen Clark and James R. Curran. 2004a. The importance of supertagging for wide-coverage CCG parsing. In Proceedings of COLING-04, pages 282–288, Geneva, Switzerland. find our card
  10. Stephen Clark and James R. Curran. 2004b. Parsing the WSJ using CCG and log-linear models. In Proceedings of the 42nd Meeting of the ACL, pages 104–111, Barcelona, Spain. find
  11. Michael Collins. 2003. Head-driven statistical models for natural language parsing. Computational Linguistics, 29(4):589–637. find our card
  12. James R. Curran and Stephen Clark. 2003. Investigating GIS and smoothing for maximum entropy taggers. In Proceedings of the 10th Meeting of the EACL, pages 91–98, Budapest, Hungary. find
  13. Julia Hockenmaier and Mark Steedman. 2002. Generative models for statistical parsing with Combinatory Categorial Grammar. In Proceedings of the 40th Meeting of the ACL, pages 335–342, Philadelphia, PA. find
  14. Julia Hockenmaier. 2003. Data and Models for Statistical Parsing with Combinatory Categorial Grammar. Ph.D. thesis, University of Edinburgh. find
  15. Ron Kaplan, Stefan Riezler, Tracy H. King, John T. Maxwell III, Alexander Vasserman, and Richard Crouch. 2004. Speed and accuracy in shallow and deep stochastic parsing. In Proceedings of the HLT Conference and the 4th NAACL Meeting (HLT-NAACL’04), Boston, MA. find
  16. Tracy H. King, Richard Crouch, Stefan Riezler, Mary Dalrymple, and Ronald M. Kaplan. 2003. The PARC 700 Dependency Bank. In Proceedings of the LINC-03 Workshop, Budapest, Hungary. find
  17. Robert Malouf and Gertjan van Noord. 2004. Wide coverage parsing with stochastic attribute value grammars. In Proceedings of the IJCNLP-04 Workshop: Beyond shallow analyses - Formalisms and statistical modeling for deep analyses, Hainan Island, China. find
  18. Joakim Nivre and Mario Scholz. 2004. Deterministic dependency parsing of English text. In Proceedings of COLING-2004, pages 64–70, Geneva, Switzerland. find our card
  19. Judita Preiss. 2003. Using grammatical relations to compare parsers. In Proceedings of the 10th Meeting of the EACL, pages 291–298, Budapest, Hungary. find
  20. Anoop Sarkar and Aravind Joshi. 2003. Tree-adjoining grammars and its application to statistical parsing. In Rens Bod, Remko Scha, and Khalil Sima’an, editors, Data-oriented parsing. CSLI. find
  21. Mark Steedman. 2000. The Syntactic Process. The MIT Press, Cambridge, MA. find
  22. Kristina Toutanova, Christopher Manning, Stuart Shieber, Dan Flickinger, and Stephan Oepen. 2002. Parse disambiguation for a rich HPSG grammar. In Proceedings of the First Workshop on Treebanks and Linguistic Theories, pages 253–263, Sozopol, Bulgaria. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2007-clark-formalism-independent-parser-evaluation-ccg/