Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics

What it is about

The paper builds a new test set of 1,000 hand-corrected British National Corpus parse trees, chosen so that each sentence has a verb that is unseen in the WSJ training sections. Four versions of the Charniak parser are scored on it with three metrics: Parseval, Leaf-Ancestor and a dependency score (LAS, UAS, label accuracy). Reranking and self-training on BNC parses help, the BNC self-trained parser best of all: Parseval F rises from 82.5 to 85.4, LAS from 82.5 to 84.2, UAS from 85.8 to 87.4, and Leaf-Ancestor from 0.8807 to 0.8900. Per-label F-scores show that ADV, DEP, IOBJ, PRN and PRT are weak for all four parsers, and that coordination is weaker for the newspaper self-trained parser.

Ideas

• I1. A difficult-by-design test subset: select EWT sentences that contain a verb form not seen in en training, using en lexicon for the check, and measure en parse LAS on this subset against the full EWT test. The paper's selection rule is the model here. Test only, since the subset is used for measurement.
• I2. Per-label F table in en ud-eval: report F-score for each dependency label, as the paper's breakdown does. Measure first whether the same labels are weak in Mova on EWT test before any model change.
• I3. Path-to-root lineage score: a Leaf-Ancestor analogue for dependency trees. For each word, compare the gold and predicted path to the root by edit distance, and report the mean beside LAS in en parse-eval. Measure on EWT test and check that it ranks models the same way as LAS.
• I4. Self-training on target-genre parses: add silver en parses of raw open text from the target genre to the perceptron training data, and test on a held-out sample of that genre. The gain must beat a control that adds the same number of sentences from the original genre.

For Mova

The code is en/src/parse.rs and the en parse-eval and en ud-eval commands. The chapters are 21. The simple clause and 37-register-and-variation.md. The numbers are Charniak-parser figures on a BNC set, so they are not Mova numbers; Mova's own reference is LAS 79.8 on EWT test. The BNC is non-commercial and the test set is test only, never training data.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/2kobxa2925pm

@inproceedings{foster2008parser,
  title = {{Parser Evaluation and the BNC: Evaluating 4 constituency parsers with 3 metrics}},
  author = {Foster, Jennifer and Genabith, Josef van},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/2kobxa2925pm},
  url = {https://doi.org/10.63317/2kobxa2925pm}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ann Bies, Mark Ferguson, Karen Katz, and Robert MacIntyre. 1995. Bracketing guidelines for Treebank II style, Penn Treebank project. Technical Report Tech Report MS-CIS-95-06, University of Pennsylvania, Philadelphia, PA. find
2. Ezra Black, Steve Abney, Dan Flickinger, Claudia Gdaniec, Robert Grishman, Philip Harrison, Donald Hindle, Robert Ingria, Fred Jelinek, Judith Klavans, Mark Liberman, Mitchell Marcus, Salim Roukos, Beatrice Santorini, and Tomek Strzalkowski. 1991. A procedure for quantitatively comparing the syntactic coverage of english grammars. In Proceedings of the 1991 DARPA Speech and Natural Language Workshop, pages 306– 311. find
3. Lou Burnard. 2000. User reference guide for the British National Corpus. Technical report, Oxford University Computing Services. find
4. John Carroll, Anette Frank, Dekang Lin, Detlef Prescher, and Hans Uszkoreit, editors. 2002. Proceedings of the “Beyond Parseval - Towards Improved Evaluation Measures for Parsing Systems” Workshop at the 3rd International Conference on Linguistic Resources and Evaluation (LREC-02), Las Palmas, Gran Canaria. find
5. Eugene Charniak and Mark Johnson. 2005. Course-to-fine n-best-parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting of the ACL (ACL-05), pages 173–180, Ann Arbor, Michigan, June. find
6. Eugene Charniak. 2000. A maximum-entropy-inspired parser. In Proceedings of the Annual Meeting of the North American Association for Computational Linguistics (NAACL-00), pages 132–139, Seattle, Washington. find
7. Grzegorz Chrupała, Nicolas Stroppa, Josef van Genabith, and Georgiana Dinu. 2007. Better training for function labeling. In Proceedings of the Recent Advances in Natural Language Processing Conference (RANLP-07), pages 133–138, Borovets, Bulgaria. find
8. Jennifer Foster, Joachim Wagner, Djamé Seddah, and Josef van Genabith. 2007. Adapting WSJ-trained parsers to the British National Corpus using in-domain self-training. In Proceedings of the Tenth International Workshop on Parsing Technologies (IWPT-07), pages 33–35, Prague, Czech Republic. find our card
9. Richard Johansson and Pierre Nugues. 2007. Extended constituent-to-dependency conversion for English. In Joakim Nivre, Heiki-Jaan Kaalep, Kadri Muischnek, and Mare Koit, editors, Proceedings of NODALIDA 2007, pages 105–112, Tartu, Estonia. find our card
10. Sandra Kübler and Heike Telljohann. 2002. Towards a dependency-oriented evaluation for partial parsing. In Carroll et al. (Carroll et al., 2002), pages 9–16. find
11. Dekang Lin. 1998. Dependency-based evaluation of MINIPAR. In John Carroll, editor, Proceedings of The Evaluation of Parsing Systems Workshop at the 3rd International Conference on Linguistic Resources and Evaluation (LREC), Cognitive Science Research Papers 489, pages 48–56. University of Sussex, Brighton, England. find
12. Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: Annotating Predicate Argument Structure. In Proceedings of the ARPA Workshop on Human Language Technology, pages 110–115, Princeton, NJ. find our card
13. David McClosky, Eugene Charniak, and Mark Johnson. 2006a. Effective self-training for parsing. In Proceedings of the Human Language Technology Conference and North American chapter of the ACL annual meeting (HLT-NAACL-06), pages 152–159, New York, June. find our card
14. David McClosky, Eugene Charniak, and Mark Johnson. 2006b. Reranking and self-training for parser adaptation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL (COLING-ACL-06), pages 337–344, Sydney, Australia, July. find our card
15. Joakim Nivre, Johan Hall, Sandra Kübler, Ryan Mac Donald, Jens Nilsson, Sebastian Riedel, and Deniz Yuret. 2007. find
16. The CoNLL 2007 shared task on dependency parsing. In Proceedings of the CoNLL Shared Task Session of EMNLP-CoNLL 2007, pages 915–932, Prague, Czech Republic, June. find
17. Slav Petrov, Leon Barrett, Romain Thibaux, and Dan Klein. 2006. Learning accurate, compact, and interpretable tree annotation. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 433–440, Sydney, Australia. find our card
18. Ines Rehbein and Josef van Genabith. 2007. Treebank annotation schemes and parser evaluation for German. In Proceedings of the Joint EMNLP-CoNLL 2007, pages 630–639, Prague, Czech Republic. find
19. Geoffrey Sampson and Anna Babarczy. 2002. A test of the leaf-ancestor metric for parse accuracy. In Carroll et al. (Carroll et al., 2002). find
20. Joachim Wagner, Djamé Seddah, Jennifer Foster, and Josef van Genabith. 2007. C-structures and f-structures for the British National Corpus. In Proceedings of the 12th International Workshop on Lexical Functional Grammar, Stanford, CA. find
21. Hiroyasu Yamada and Yuji Matsumoto. 2003. Statistical dependency analysis with support vector machines. In Proceedings of the 8th International Workshop on Parsing Technologies (IWPT-03), pages 195–206, Nancy, France. find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2008-foster-parser-evaluation-bnc-evaluating-4/