Using the Penn Treebank to Evaluate Non-Treebank Parsers

What it is about

The question is how to compare a parser that was never trained on the Penn Treebank (NLPWin, built at Microsoft Research) with Treebank-trained parsers such as Charniak's, when the two attach modifiers in a different order and use different brackets. The authors score both on unlabelled precision and recall of maximal projections of heads (MPH), a bracket set that focuses on "maximal projections of heads" (p. 1). Head labels for the gold trees come from Charniak's head rules, plus a small set of reassignment rules; single-word brackets, the sentence bracket and punctuation are dropped, and unary nodes are flattened. On WSJ section 23, NLPWin reaches P 74.32 and R 74.25, and Charniak's parser P 86.18 and R 85.72. On the restricted Brown test BR1 the gap is smaller (NLPWin P 69.54, R 70.56; Charniak P 77.71, R 77.73). The authors argue that raw brackets would reward high PP attachment, and that a dependency tree with ordered dependents is isomorphic to an MPH tree, so the same metric could score dependency parsers.

Ideas

• I1. tools/parsecmp: add an MPH score next to the labelled scores. In a UD tree, each word and all its dependents form one span, which is the same bracket the paper's MPH trees give, so the metric needs no head-rule set. Score the en output and the constituency runs already listed in parsecmp (stanza-con, benepar, supar-con) against the same Haiku reference, and check whether the MPH ranking of systems agrees with the LAS ranking. Non-projective trees give spans with gaps; report their count separately.
• I2. Negative control for that metric: in a copy of the reference, move every PP from verb attachment to noun attachment by a fixed rule. MPH precision and recall must fall on the same sentences. A metric that stays flat under this change is not measuring attachment.
• I3. Bias check, as the paper warns about high attachment: on EWT test, count how often en attaches a PP to the verb and to the noun, and how often gold does, and report the MPH score on the PP subset alone. This shows whether the SLM has the bias the paper describes, and in which direction.

For Mova

The chapters are 22. Phrases: noun, adjective, adverb and prepositional phrases (NP structure, PP attachment) and 21. The simple clause. The code is tools/parsecmp (the SYSTEMS and CONSTITUENCY lists) and en parse-eval. The Penn Treebank is LDC data under non-commercial terms: use it, if at all, for tests only, never in a release. The EWT measurements are CC BY-SA, so they are measurements, not material carried into a build.

Topics: Dependency parsing · Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{ringger2004using,
  title = {{Using the Penn Treebank to Evaluate Non-Treebank Parsers}},
  author = {Ringger, Eric K. and Moore, Robert C. and Charniak, Eugene and Vanderwende, Lucy and Suzuki, Hisami},
  booktitle = {Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04)},
  year = {2004},
  url = {https://aclanthology.org/L04-1115/}
}

References

15 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Bies, A., M. Ferguson, K. Katz, R. MacIntyre, V. Tredinnick, G. Kim, M.A. Marcinkiewicz, B. Schasberger. (1995) "Bracketing Guidelines for Treebank II Style Penn Treebank Project". TR, University of Pennsylvania. WSJ 23 BR 0 BR 1 8903 84.82 85.77 13131 74.25 74.32 15163 85.72 86.18 8489 71.55 70.49 9415 79.33 79.66 8217 70.56 69.54 9054 77.73 77.71 41.23 0.32 80.91 27.11 0.7 68.34 45.16 0.29 82.41 32.95 0.67 73 45.75 0.4 80.55 32.8 0.67 73.2 44.3 0.47 79.05 97.03 90.44 97.14 91.3 95.65 90.85 94.5 find
2. Black, E et al. (1991) “A Procedure for Quantitatively Comparing the Syntactic Coverage of English Grammars”. In Proceedings of the February 1991 DARPA Speech and Natural Language Workshop. pp. 306-311. find
3. Charniak, E. (2000) “A Maximum-Entropy-Inspired Parser”. In Proceedings of NAACL-2000. pp. 132–139. find
4. Collins, M. (1999) “Head-Driven Statistical Models for Natural Language Parsing”. Ph.D. thesis, U. Pennsylvania. find our card
5. Collins, M. & N. Duffy. (2002) “New Ranking Algorithms for Parsing and Tagging: Kernels over Discrete Structures, and the Voted Perceptron”. In Proceedings of ACL 2002. find
6. Gaizauskas, R., M. Hepple, & C. Huyck. (1998a) “A Scheme for Comparative Evaluation of Diverse Parsing Systems”. In Proceedings of 1st International Conference on Language Resources and Evaluation (LREC'98), pp. 143-149. find
7. Gaizauskas, R., M. Hepple, & C. Huyck. (1998b) “Modifying Existing Annotated Corpora for General Comparative Evaluation of Parsing”. In Proceedings of the LREC'98 Workshop on Evaluation of Parsing Systems. find
8. Heidorn, G. (2000). “Intelligent writing assistance”. In Dale et al. Handbook of Natural Language Processing, Marcel Dekker. find
9. Kucera, H. & W.N. Francis. (1967) Computational Analysis of Present-Day American English. Providence: Brown University Press. find
10. Lin, D. (1995) “A dependency-based method for evaluating broad-coverage parsers”. In Proceedings of IJCAI-95, pp.1420-1425. find
11. Lin, D. (1998) “Dependency-based Evaluation of MINIPAR”. In Workshop on the Evaluation of Parsing Systems. find
12. Magerman, D. (1995) “Statistical decision-tree models for parsing”. In ACL 33, pp. 276–283. find
13. Marcus, M., B. Santorini, M.A. Marcinkiewicz. (1993) “Building a Large Annotated Corpus of English: The Penn Treebank”. Computational Linguistics 19 (2), 313-330. find our card
14. Riezler, S., T. King, R. Kaplan, R. Crouch, J. Maxwell III, M. Johnson. (2002) “Parsing the Wall Street Journal using a Lexical-Functional Grammar and Discriminative Estimation Techniques”. In Proceedings of ACL 2002. pp. 271-278. find
15. Sekine, S. & M. Collins. (1997) EvalB: a bracket scoring program. URL: http://www.cs.nyu.edu/cs/projects/proteus/evalb/ ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/bank-2004-ringger-penn-treebank-evaluate-non-treebank/