Task-oriented Evaluation of Syntactic Parsers and Their Representations

What it is about

Parsers are usually compared on their own accuracy, and parse representations differ from framework to framework, so the scores are hard to compare. The authors evaluate eight parsers (dependency, phrase-structure and deep) with five representations by plugging each into a protein-protein interaction extraction system for biomedical papers. They measure the effect on the task, not the parser's own score. Accuracy levels across parsers turn out to be similar, but the gains from retraining on the biomedical domain vary a lot: some parsers improve clearly and others barely change. The reranking parser is slightly best, at a much higher parsing cost. An ensemble over parsers and representations is also tested. The authors conclude that one task is not enough to rank parsers in general.

Ideas

• I1. Extrinsic test for Mova: run the prag event extraction on the outputs of en, gparse and one witness on 200 hand-annotated sentences (test only where the data is NC), and measure event-role F1 per parser next to LAS.
• I2. Ensemble: combine the arc scores of en and gparse into one tree with Eisner decoding, and measure LAS on EWT dev against the better single parser, two runs each.
• I3. treeagree: where the two parsers agree on a whole sentence, measure the agreement rate and the event-role F1 of the agreed subset, to decide which silver sentences to keep.

For Mova

Touches 37. Register and variation (domain shift) and 20. Syntax: the Universal Dependencies frame. The main lesson for Mova is to judge a parser by what the world layer does with it, not only by LAS. Start with I1, since it is the measure the paper argues for.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{miyao2008task,
  title = {{Task-oriented Evaluation of Syntactic Parsers and Their Representations}},
  author = {Miyao, Yusuke and Sætre, Rune and Sagae, Kenji and Matsuzaki, Takuya and Tsujii, Jun'ichi},
  booktitle = {Proceedings of ACL-08: HLT},
  year = {2008},
  url = {https://aclanthology.org/P08-1006/}
}

References

43 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. D. M. Bikel. 2004. Intricacies of Collins’ parsing model. Computational Linguistics, 30(4):479–511. find
2. T. Briscoe and J. Carroll. 2006. Evaluating the accuracy of an unlexicalized statistical parser on the PARC DepBank. In COLING/ACL 2006 Poster Session. find
3. R. Bunescu and R. J. Mooney. 2004. Collective information extraction with relational markov networks. In ACL 2004, pages 439–446. find
4. R. C. Bunescu and R. J. Mooney. 2005. Subsequence kernels for relation extraction. In NIPS 2005. find
5. E. Charniak and M. Johnson. 2005. Coarse-to-fine nbest parsing and MaxEnt discriminative reranking. In ACL 2005. find
6. E. Charniak. 2000. A maximum-entropy-inspired parser. In NAACL-2000, pages 132–139. find
7. S. Clark and J. R. Curran. 2004. Parsing the WSJ using CCG and log-linear models. In 42nd ACL. find
8. S. Clark and J. R. Curran. 2007. Formalism-independent parser evaluation with CCG and DepBank. In ACL 2007. find our card
9. A. B. Clegg and A. J. Shepherd. 2007. Benchmarking natural-language parsers for biological applications using dependency graphs. BMC Bioinformatics, 8:24. find
10. M. Collins and N. Duffy. 2002. New ranking algorithms for parsing and tagging: Kernels over discrete structures, and the voted perceptron. In ACL 2002. find
11. M. Collins. 1997. Three generative, lexicalised models for statistical parsing. In 35th ACL. find
12. M.-C. de Marneffe, B. MacCartney, and C. D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In LREC 2006. find our card
13. J. M. Eisner. 1996. Three new probabilistic models for dependency parsing: An exploration. In COLING 1996. find our card
14. G. Erkan, A. Ozgur, and D. R. Radev. 2007. Semi-supervised classification for extracting protein interaction sentences using dependency parsing. In EMNLP 2007. find
15. D. Gildea. 2001. Corpus variation and parser performance. In EMNLP 2001, pages 167–202. find our card
16. C. Giuliano, A. Lavelli, and L. Romano. 2006. Exploiting shallow linguistic information for relation extraction from biomedical literature. In EACL 2006. find
17. T. Hara, Y. Miyao, and J. Tsujii. 2007. Evaluating impact of re-training a lexical disambiguation model on domain adaptation of an HPSG parser. In IWPT 2007. find
18. R. Johansson and P. Nugues. 2007. Extended constituent-to-dependency conversion for English. In NODALIDA 2007. find our card
19. R. M. Kaplan, S. Riezler, T. H. King, J. T. Maxwell, and A. Vasserman. 2004. Speed and accuracy in shallow and deep stochastic parsing. In HLT/NAACL’04. find
20. S. Katrenko and P. Adriaans. 2006. Learning relations from biomedical corpora using dependency trees. In KDECB, pages 61–80. find
21. J.-D. Kim, T. Ohta, Y. Teteisi, and J. Tsujii. 2003. GE-NIA corpus — a semantically annotated corpus for bio-textmining. Bioinformatics, 19:i180–182. find
22. D. Klein and C. D. Manning. 2003. Accurate unlexicalized parsing. In ACL 2003. find our card
23. D. Lin. 1998. Dependency-based evaluation of MINI-PAR. In LREC Workshop on the Evaluation of Parsing Systems. find
24. M. Marcus, B. Santorini, and M. A. Marcinkiewicz. 1994. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313–330. find our card
25. T. Matsuzaki, Y. Miyao, and J. Tsujii. 2005. Probabilistic CFG with latent annotations. In ACL 2005. find
26. R. McDonald and F. Pereira. 2006. Online learning of approximate dependency parsing algorithms. In EACL 2006. find our card
27. T. Mitsumori, M. Murata, Y. Fukuda, K. Doi, and H. Doi. 2006. Extracting protein-protein interaction information from biomedical text with SVM. IEICE - Trans. Inf. Syst., E89-D(8):2464–2466. find
28. Y. Miyao and J. Tsujii. 2008. Feature forest models for probabilistic HPSG parsing. Computational Linguistics, 34(1):35–80. find our card
29. Y. Miyao, K. Sagae, and J. Tsujii. 2007. Towards framework-independent evaluation of deep linguistic parsers. In Grammar Engineering across Frameworks 2007, pages 238–258. find
30. A. Moschitti. 2006. Making tree kernels practical for natural language processing. In EACL 2006. find
31. J. Nivre and J. Nilsson. 2005. Pseudo-projective dependency parsing. In ACL 2005. find
32. S. Petrov and D. Klein. 2007. Improved inference for unlexicalized parsing. In HLT-NAACL 2007. find
33. S. Pyysalo, T. Salakoski, S. Aubin, and A. Nazarenko. 2006. Lexical adaptation of link grammar to the biomedical sublanguage: a comparative evaluation of three approaches. BMC Bioinformatics, 7(Suppl. 3). find
34. S. Pyysalo, F. Ginter, J. Heimonen, J. Björne, J. Boberg, J. Järvinen, and T. Salakoski. 2007a. BioInfer: a corpus for information extraction in the biomedical domain. BMC Bioinformatics, 8(50). find
35. S. Pyysalo, F. Ginter, V. Laippala, K. Haverinen, J. Heimonen, and T. Salakoski. 2007b. On the unification of syntactic annotations under the Stanford dependency scheme: A case study on BioInfer and GENIA. In BioNLP 2007, pages 25–32. find
36. C. Quirk and S. Corston-Oliver. 2006. The impact of parse quality on syntactically-informed statistical machine translation. In EMNLP 2006. find
37. E. K. Ringger, R. C. Moore, E. Charniak, L. Vanderwende, and H. Suzuki. 2004. Using the Penn Treebank to evaluate non-treebank parsers. In LREC 2004. find our card
38. R. Sætre, K. Sagae, and J. Tsujii. 2007. Syntactic features for protein-protein interaction extraction. In LBM 2007 short papers. find
39. K. Sagae and J. Tsujii. 2007. Dependency parsing and domain adaptation with LR models and parser ensembles. In EMNLP-CoNLL 2007. find
40. K. Sagae, Y. Miyao, T. Matsuzaki, and J. Tsujii. 2008. Challenges in mapping of syntactic representations for framework-independent parser evaluation. In the Workshop on Automated Syntatic Annotations for Interoperable Language Resources. find
41. D. D. Sleator and D. Temperley. 1993. Parsing English with a Link Grammar. In 3rd IWPT. find our card
42. Y. Tsuruoka, Y. Tateishi, J.-D. Kim, T. Ohta, J. Mc-Naught, S. Ananiadou, and J. Tsujii. 2005. Developing a robust part-of-speech tagger for biomedical text. In 10th Panhellenic Conference on Informatics. find
43. A. Yakushiji, Y. Miyao, Y. Tateisi, and J. Tsujii. 2005. Biomedical information extraction with predicate-argument structure patterns. In First International Symposium on Semantic Mining in Biomedicine. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2008-miyao-task-oriented-evaluation-syntactic-parsers/