Building a Large Annotated Corpus of English: The Penn Treebank

What it is about

The paper reports how the first phase of the Penn Treebank was built, from 1989 to 1992: over 4.5 million words tagged for part of speech and about half of them given a skeletal bracketing. The tagset is the Brown tagset pared down by removing forms that are lexically recoverable (the verb forms of be, do and have). Tagging is semi-automatic: a tagger proposes tags and humans correct them. Bracketing works the same way, with a parser proposing a skeleton that annotators correct. Semi-automatic tagging beat manual tagging on speed, consistency and accuracy. Correcting took 20 minutes per 1,000 words against 44 for tagging, and mean inter-annotator disagreement was 7.2% for tagging against 4.1% for correcting; the estimated POS error rate of the automatic tagger was 3 to 5%.

Ideas

• I1. Pre-tag, then correct: in annot, compare the Haiku or Opus silver produced from a Rust draft with the same teacher working from scratch, on 300 EWT sentences; measure agreement with gold, and time or tokens per sentence, since the paper finds correction both faster and more consistent.
• I2. Recoverable-form check in en/src/ud.rs: list the UFeats errors on EWT test that come from be, do and have forms, and check whether the lemma alone decides the tag in every case; this tests the paper's rule for dropping tags that are lexically recoverable.
• I3. Two-number report per batch: record disagreement separately for the automatic and the corrected stage of annot, as the paper does, so that the cost of teacher errors is visible.

For Mova

The code is annot (the batch and gates), en/src/ptag.rs and en/src/ud.rs. The chapters are 10. Word classes and 13-verbs.md. The paper's main lesson is that the teacher's draft should be checked and corrected rather than trusted or replaced, and that the cost of correction should be measured, not assumed.

Topics: Treebanks · Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.21236/ada273556

@misc{marcus1993building,
  title = {{Building a Large Annotated Corpus of English: The Penn Treebank}},
  author = {Marcus, Mitchell P. and Santorini, Beatrice and Marcinkiewicz, Mary Ann},
  year = {1993},
  doi = {10.21236/ada273556},
  url = {https://doi.org/10.21236/ada273556}
}

References

18 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Brill, Eric (1991). "Discovering the lexical features of a language." In Proceedings, 29th Annual Meeting of the Association for Computational Linguistics. Berkeley CA. find
2. Brill, Eric; Magerman, David; Marcus, Mitchell P.; and Santorini, Beatrice (1990). "Deducing linguistic structure from the statistics of large corpora." In Proceedings, DARPA Speech and Natural Language Workshop. June 1990, 275-282. find
3. Church, Kenneth W. (1980). Memory limitations in natural language processing. Master's dissertation, Massachusetts Institute of Technology, Cambridge MA. find
4. Church, Kenneth W. (1988). "A stochastic parts program and noun phrase parser for unrestricted text." In Proceedings, Second Conference on Applied Natural Language Processing. 136--143. find
5. Francis, W. Nelson (1964). "A standard sample of present-day English for use with digital computers." Report to the U.S Office of Education on Cooperative Research Project No. E-007. Brown University, Providence RI. find
6. Francis, W. Nelson, and Ku~era, Henry (1982). Frequency Analysis of English Usage: Lexicon and Grammar. Houghton Mifflin. find
7. Garside, Roger; Leech, Geoffrey; and Sampson, Geoffrey (1987). The Computational Analysis of English: A Corpus-Based Approach. Longman. find
8. Hindle, Donald (1983). "User manual for Fidditch." Technical memorandum 7590-142, Naval Research Laboratory. find
9. Hindle, Donald (1989). "Acquiring disambiguation rules from text." In Proceedings, 27th Annual Meeting of the Association for Computational Linguistics. find
10. Lewis, Bil; LaLiberte, Dan; and the GNU Manual Group (1990). The GNU Emacs Lisp reference manual. Free Software Foundation, Cambridge, MA. find
11. Magerman, David, and Marcus, Mitchell P. (1990). "Parsing a natural language using mutual information statistics." In Proceedings of AAAI-90. find
12. Meteer, Marie; Schwartz, Richard; and Weischedel, Ralph (1991). "Studies in part of speech labelling." In Proceedings, Fourth DARPA Speech and Natural Language Workshop. February 1991. find
13. Niv, Michael (1991). "Syntactic disambiguation." In The Penn Review of Linguistics, 14, 120-126. find
14. Pereira, Fernando, and Schabes, Yves (1992). "Inside-outside reestimation from partially bracketed corpora." In Proceedings, 30th Annual Meeting of the Association for Computational Linguistics. find
15. Santorini, Beatrice (1990). "Part-of-speech tagging guidelines for the Penn Treebank Project." Technical report MS-CIS-90-47, Department of Computer and Information Science, University of Pennsylvania. find
16. Santorini, Beatrice, and Marcinkiewicz, Mary Ann (1991). "Bracketing guidelines for the Penn Treebank Project." Unpublished manuscript, Department of Computer and Information Science, University of Pennsylvania. find
17. Veilleux, N. M., and Ostendorf, Mari (1992). "Probabilistic parse scoring based on prosodic features." In Proceedings, Fifth DARPA Speech and Natural Language Workshop. February 1992. find
18. Weischedel, Ralph; Ayuso, Damaris; Bobrow, R.; Boisen, Sean; Ingria, Robert; and Palmucci, Jeff (1991). "Partial parsing: a report of work in progress." In Proceedings, Fourth DARPA Speech and Natural Language Workshop. February 1991. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/bank-1993-marcus-building-large-annotated-corpus-english/