Mova0.5 pre-release⌕
☰
home / docs / papers / English / Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics?

Part-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics?

What it is about

Taggers report about 97% token accuracy, but only about 56% of sentences are fully right, and a single error can break a downstream parser. Manning asks what it would take to reach close to 100%. He improves the Stanford tagger with richer features (word shapes and distributional similarity), reaching 97.28% token and 56.79% sentence accuracy in the best row of his table. Then he analyses the remaining errors by class. Inconsistent or missing standards account for 28%, difficult linguistics for 19.5%, cases a tagger could plausibly get right for 16%, gold-standard errors for 15.5%, underspecified categories for 12%, and lexicon gaps and unknown words for 4.5% each. His conclusion is that the largest remaining gains come from better linguistic taxonomies and consistent conventions, not from better learners, though he also notes limits to that route.

Ideas

For Mova

Touches 10. Word classes, 07. Punctuation, capital letters, abbreviations and numbers and 19. The lexicon: vocabulary, multiword expressions and register. The paper is a strong argument for Mova's suspect-label approach: errors in the gold labels are themselves a class to measure. Start with I1 and I2 on dev, not on test, and report two runs.

Topics: Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.1007/978-3-642-19400-9_14

@inproceedings{manning2011part,
  title = {{Part-of-Speech Tagging from 97\% to 100\%: Is It Time for Some Linguistics?}},
  author = {Manning, Christopher D.},
  booktitle = {Computational Linguistics and Intelligent Text Processing, 171-189},
  year = {2011},
  doi = {10.1007/978-3-642-19400-9\_14},
  url = {https://doi.org/10.1007/978-3-642-19400-9_14}
}

References

34 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Toutanova, K., Klein, D., Manning, C.D., Singer, Y.: Feature-rich part-of-speech tagging with a cyclic dependency network. In: NAACL 3. (2003) 252–259 find
  2. Shen, L., Satta, G., Joshi, A.: Guided learning for bidirectional sequence classification. In: ACL 2007. (2007) find
  3. Spoustová, D.j., Hajič, J., Raab, J., Spousta, M.: Semi-supervised training for the averaged perceptron POS tagger. In: Proceedings of the 12th Conference of the European Chapter of the ACL (EACL 2009). (2009) 763–771 find
  4. Søgaard, A.: Simple semi-supervised training of part-of-speech taggers. In: Proceedings of the ACL 2010 Conference Short Papers. (2010) 205–208 find
  5. Subramanya, A., Petrov, S., Pereira, F.: Efficient graph-based semi-supervised learning of structured tagging models. In: Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. (2010) 167–176 find
  6. Collins, M.: Discriminative training methods for Hidden Markov Models: Theory and experiments with perceptron algorithms. In: EMNLP 2002. (2002) find our card
  7. Marcus, M.P., Santorini, B., Marcinkiewicz, M.A.: Building a large annotated corpus of English: The Penn treebank. Computational Linguistics 19 (1993) 313– 330 find our card
  8. Finkel, J., Dingare, S., Manning, C., Nissim, M., Alex, B., Grover, C.: Exploring the boundaries: Gene and protein identification in biomedical text. BMC Bioinformatics 6 (Suppl 1) (2005) find
  9. Collins, M.: Ranking algorithms for named entity extraction: Boosting and the voted perceptron. In: ACL 40. (2002) 489–496 find
  10. Tsuruoka, Y., Tsujii, J.: Bidirectional inference with the easiest-first strategy for tagging sequence data. In: Proceedings of HLT/EMNLP 2005. (2005) 467–474 find
  11. Clark, A.: Combining distributional and morphological information for part of speech induction. In: EACL 2003. (2003) 59–66 find
  12. Klein, D., Manning, C.D.: Accurate unlexicalized parsing. In: ACL 41. (2003) 423–430 find our card
  13. MacKinlay, A.: The effects of part-of-speech tagsets on tagger performance. Honours thesis, Department of Computer Science and Software Engineering, University of Melbourne (2005) find
  14. Church, K.W.: Current practice in part of speech tagging and suggestions for the future. In Mackie, A.W., McAuley, T.K., Simmons, C., eds.: For Henry Kučera: Studies in Slavic philology and computational linguistics. Number 6 in Papers in Slavic philology. Michigan Slavic Studies, Ann Arbor (1992) 13–48 find
  15. Magerman, D.M.: Natural language parsing as statistical pattern recognition. PhD thesis, Stanford University (1994) find
  16. Ratnaparkhi, A.: A maximum entropy model for part-of-speech tagging. In: EMNLP 1. (1996) 133–142 find
  17. Abney, S., Schapire, R.E., Singer, Y.: Boosting applied to tagging and PP attachment. In Fung, P., Zhou, J., eds.: Proceedings of the 1999 Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora. (1999) 38–45 find
  18. Voutilainen, A., Järvinen, T.: Specifying a shallow grammatical representation for parsing purposes. In: 7th Conference of the European Chapter of the Association for Computational Linguistics. (1995) 210–214 find
  19. Samuelsson, C., Voutilainen, A.: Comparing a linguistic and a stochastic tagger. In: Proceedings of the 35th Annual Meeting of the Association for Computational Linguistics. (1997) 246–253 find
  20. Dickinson, M., Meurers, W.D.: Detecting errors in part-of-speech annotation. In: Proceedings of the 10th Conference of the European Chapter of the Association for Computational Linguistics (EACL-03). (2003) find
  21. Levy, R., Andrew, G.: Tregex and Tsurgeon: tools for querying and manipulating tree data structures. In: 5th International Conference on Language Resources and Evaluation (LREC 2006). (2006) find
  22. Rohde, D.L.T.: Tgrep2 user manual. ms., MIT (2005) find
  23. Santorini, B.: Part-of-speech tagging guidelines for the Penn treebank project. 3rd Revision, 2nd printing, Feb. 1995. University of Pennsylvania (1990) find our card
  24. Moore, D.S.: Statistics: Concepts and Controversies. 3rd edn. W. H. Freeman, New York (1991) find
  25. Ross, J.R.: A fake NP squish. In Bailey, C.J.N., Shuy, R.W., eds.: New Ways of Analyzing Variation in English. Georgetown University Press, Washington (1973) 96–140 find
  26. Quirk, R., Greenbaum, S., Leech, G., Svartvik, J.: A Comprehensive Grammar of the English Language. Longman, London (1985) find
  27. Aarts, B.: Syntactic gradience: the nature of grammatical indeterminacy. Oxford University Press, Oxford (2007) find
  28. Abney, S.: Statistical methods and linguistics. In Klavans, J., Resnik, P., eds.: The Balancing Act. MIT Press, Cambridge, MA (1996) find
  29. Maling, J.: Transitive adjectives: A case of categorial reanalysis. In Heny, F., Richards, B., eds.: Linguistic Categories: Auxiliaries and Related Puzzles. Volume 1. D. Reidel, Dordrecht (1983) 253–289 find
  30. Harnad, S., ed.: Categorical perception : the groundwork of cognition. Cambridge University Press, Cambridge (1987) find
  31. Radford, A.: Transformational Grammar. Cambridge University Press, Cambridge (1988) find
  32. Manning, C.D., Schütze, H.: Foundations of Statistical Natural Language Processing. MIT Press, Boston, MA (1999) find
  33. Bies, A., Ferguson, M., Katz, K., MacIntyre, R., and colleagues: Bracketing guidelines for Treebank II style: Penn treebank project. ms, University of Pennsylvania (1995) find
  34. Huddleston, R.D., Pullum, G.K.: The Cambridge Grammar of the English Language. Cambridge University Press, Cambridge (2002) find our card

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/enud-2011-manning-part-speech-tagging-97-100/