Mova0.5 pre-release⌕
☰
home / docs / papers / English / CCOHA: Clean Corpus of Historical American English

CCOHA: Clean Corpus of Historical American English

What it is about

The paper cleans the downloadable version of the Corpus of Historical American English (COHA), about 406 million words from 1810 to 2009. The known problems are: every 200 tokens, 10 are replaced by "@" for copyright reasons; malformed tokens such as "&c.;"; NUL characters; inconsistent and malformed lemmas; escaped HTML; and lost sentence boundaries. The cleaning runs in two passes. The first pass fixes tokens without context and unifies lemmas; the second re-tags with NLTK, splits sentences with Punkt and cleans malformed tokens at boundaries, marking each sentence end. The check is task-based: extract the sentence context of 50 target words and compare counts. For condominium the web portal gives 524 occurrences, the downloadable corpus 473 and the cleaned corpus 498. The cleaned annotated corpus has about 25 million more word tokens, and the lemma types rise from 795,806 to 2,246,898, partly because of new inconsistent lemmas.

Ideas

For Mova

The relevant code is tools/gutenberg (text cleaning) and the sentence splitter in en/src/tok.rs. The relevant chapter is 1. The period, the register and the texts Mova reads. The measures: boundary counts before and after cleaning, and the target-word counts against the reference.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{alatrash2020ccoha,
  title = {{CCOHA: Clean Corpus of Historical American English}},
  author = {Alatrash, Reem and Schlechtweg, Dominik and Kuhn, Jonas and Walde, Sabine Schulte im},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference},
  year = {2020},
  url = {https://aclanthology.org/2020.lrec-1.859/}
}

References

25 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Baroni, M., Chantree, F., Kilgarriff, A., and Sharoff, S. (2008). Cleaneval: A Competition for Cleaning Web Pages. In Proceedings of the 6th International Conference on Language Resources and Evaluation, Marrakech, Morocco. European Language Resources Association. find
  2. Bird, S. and Loper, E. (2004). NLTK: The Natural Language Toolkit. In Proceedings of the ACL Interactive Poster and Demonstration Sessions, pages 214–217, Barcelona, Spain. Association for Computational Linguistics. find
  3. Blank, A. (1999). Why Do New Meanings occur? A Cognitive Typology of the Motivations for Lexical Semantic Change. In Andreas Blank et al., editors, Historical Semantics and Cognition, Cognitive Linguistics Research, pages 61–89. Mouton de Gruyter, Berlin / New York. find
  4. Bowern, C. (2019). Semantic Change and Semantic Stability: Variation is Key. In Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change, pages 48–55, Florence, Italy. Association for Computational Linguistics. find
  5. Dubossarsky, H., Hengchen, S., Tahmasebi, N., and Schlechtweg, D. (2019). Time-Out: Temporal Referencing for Robust Modeling of Lexical Semantic Change. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 457–470, Florence, Italy. Association for Computational Linguistics. find
  6. Faaß, G. and Eckart, K. (2013). SdeWaC: A Corpus of Parsable Sentences from the Web. In Language Processing and Knowledge in the Web, pages 61–68. Springer. find
  7. Fromkin, V., Rodman, R., and Hyams, N. (2018). An Introduction to Language. Cengage Learning, 11th edition. find
  8. Graën, J., Batinić, D., and Volk, M. (2014). Cleaning the Europarl Corpus for Linguistic Applications. In Josef Ruppenhofer et al., editors, Proceedings of the 12th Conference on Natural Language Processing, pages 222– 227, Hildesheim. Universitätsverlag Hildesheim. find
  9. Hill, M. J. and Hengchen, S. (2019). Quantifying the Impact of Dirty OCR on Historical Text Analysis: Eighteenth Century Collections Online as a Case Study. Digital Scholarship in the Humanities. find
  10. Imamura, K. and Sumita, E. (2002). Bilingual Corpus Cleaning Focusing on Translation Literality. In Proceedings of the 7th International Conference on Spoken Language Processing. find
  11. Koehn, P. and Knight, K. (2003). Empirical Methods for Compound Splitting. In Proceedings of the 10th Conference of the European Chapter of the Association for Computational Linguistics, pages 187–193, Budapest, Hungary. Association for Computational Linguistics. find
  12. Kutuzov, A., Øvrelid, L., Szymanski, T., and Velldal, E. (2018). Diachronic Word Embeddings and Semantic Shifts: A Survey. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1384–1397, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  13. Macherey, K., Dai, A. M., Talbot, D., Popat, A. C., and Och, F. (2011). Language-Independent Compound Splitting with Morphological Operations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1395–1404. Association for Computational Linguistics. find
  14. Marcus, M., Kim, G., Marcinkiewicz, M. A., MacIntyre, R., Bies, A., Ferguson, M., Katz, K., and Schasberger, B. (1994). The Penn Treebank: Annotating Predicate Argument Structure. In Proceedings of the Workshop on Human Language Technology, pages 114–119. Association for Computational Linguistics. find our card
  15. Norvig, P. (2009). Natural Language Corpus Data. In Toby Segaran et al., editors, Beautiful Data, chapter 14, pages 219–242. O’Reilly Media. find
  16. Perrone, V., Palma, M., Hengchen, S., Vatri, A., Smith, J. Q., and McGillivray, B. (2019). GASC: Genre-Aware Semantic Change for Ancient Greek. In Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change, pages 56–66, Florence, Italy. Association for Computational Linguistics. find
  17. Rayson, P. and Garside, R. (1998). The CLAWS Web Tagger. ICAME Journal, 22:121–123. find
  18. Reynaert, M. (2006). Corpus-Induced Corpus Clean-Up. In Proceedings of the 5th Conference on Language Resources and Evaluation, pages 461–464, Genoa, Italy. European Language Resources Association. find
  19. Rossum, G. (1995). Python Reference Manual. CWI (Centre for Mathematics and Computer Science), Amsterdam, The Netherlands. find
  20. Schlechtweg, D., Hätty, A., del Tredici, M., and Schulte im Walde, S. (2019). A Wind of Change: Detecting and Evaluating Lexical Semantic Change across Times and Domains. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 732–746, Florence, Italy. Association for Computational Linguistics. find
  21. Tahmasebi, N., Borin, L., and Jatowt, A. (2018). Survey of Computational Approaches to Diachronic Conceptual Change. arXiv preprint arXiv:1811.06278. ↗
  22. Tang, X. (2018). A State-of-the-Art of Semantic Change Computation. Natural Language Engineering, 24(5):649–676. 9. find
  23. Language Resource References Baroni, M., Bernardini, S., Ferraresi, A., and Zanchetta, E. (2009). The WaCky Wide Web: A Collection of Very Large Linguistically Processed Web-Crawled Corpora. Language Resources and Evaluation, 43(3):209–226. find
  24. Davies, Mark. (2012). Expanding Horizons in Historical Linguistics with the 400-Million Word Corpus of Historical American English. Edinburgh University Press. find
  25. Google. (2010). Google Books Ngram Viewer. Google, Google Books Ngram Datasets, 1.0. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/alatrash-2020-ccoha-clean-corpus-historical-american/