Mova0.5 pre-release⌕
☰
home / docs / papers / English / MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950)

MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950)

What it is about

The paper trains a BERT-base model from scratch on about 3.9 billion tokens of English from 1450 to 1950 (EEBO, ECCO, COHA, CLMET and other collections). It compares this model with BERT, multilingual BERT, and TuringBERT (a model adapted to 1760–1900). The tasks are built on the Oxford English Dictionary: word-in-context classification over 2,700 lemmas and 246,048 quotations, parameter-free word sense disambiguation, and periodization of a quotation by year. On held-out word-in-context pairs the model scores 63.0% against 59.5% for BERT. On parameter-free sense assignment the total accuracy is 42.3% against 36.0% for BERT, and the gain is largest on function words (50.4% against 34.8%; content words 42.0% against 36.1%). The periodization error is about 50 years on average, the lowest of the compared models. The authors conclude that pre-training on historical text from scratch is more robust than adapting a modern model.

Ideas

For Mova

Chapters: 1. The period, the register and the texts Mova reads (periods and the texts that define them), 2. Spelling, print conventions and tokens (tokenisation of old spellings), 10. Vocabulary: archaic words and false friends (sense change). Code: en/src/tok.rs for the tokenizer, en/src/emode.rs for the closed table of Early Modern forms, world/src/dictionary.rs for senses. Measures: MAE in years on a held-out split of dated quotations, accuracy by content and function words, and a negative control in which the dates are shuffled (it must score worse). Run twice and check the output is identical.

Topics: Word sense disambiguation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{arevalo2021macberth,
  title = {{MacBERTh: Development and Evaluation of a Historically Pre-trained Language Model for English (1450-1950)}},
  author = {Arevalo, Enrique Manjavacas and Fonteyn, Lauren},
  booktitle = {Proceedings of the Workshop on Natural Language Processing for Digital Humanities},
  year = {2021},
  url = {https://aclanthology.org/2021.nlp4dh-1.4/}
}

References

43 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Humanities Research (CHR 2020), volume 2723 of CEUR Workshop Proceedings, pages 257–268. find
  2. David Bamman and Patrick J Burns. 2020. Latin BERT: A contextual language model for classical philology. find
  3. Mario Giulianelli, Marco Del Tredici, and Raquel Fernández. 2020. Analysing lexical semantic change with contextualised word representations. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 3960–3973, Online. Association for Computational Linguistics. find
  4. David Bamman and Gregory Crane. 2011. Measuring historical word sense variation. In Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, pages 1–10. find
  5. Kaspar Beelen, Federico Nanni, Mariona Coll Ardanuy, Kasra Hosseini, Giorgia Tolfo, and Barbara McGillivray. 2021. When time makes sense: A historically-aware approach to targeted sense disambiguation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2751–2761. Association for Computational Linguistics. find
  6. Edouard Grave. 2017. Language Identification · fast-Text. find
  7. Xiaochuang Han and Jacob Eisenstein. 2019. Unsupervised domain adaptation of contextualized embeddings for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4238–4248, Hong Kong, China. Association for Computational Linguistics. find
  8. Steven Bird. 2006. Nltk: the natural language toolkit. In Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, pages 69–72. find
  9. Yoonseok Heo, Sangwoo Kang, and Jungyun Seo. 2020. Hybrid sense classification method for large-scale word sense disambiguation. IEEE Access, 8:27247– 27256. find
  10. Marcel Bollmann. 2019. A large-scale comparison of historical text normalization systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3885–3898, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  11. Kasra Hosseini, Kaspar Beelen, Giovanni Colavizza, and Mariona Coll Ardanuy. 2021a. Neural Language Models for Nineteenth-Century English. Journal of Open Humanities Data, 7:22. find
  12. Emanuela Boros, Elvys Linhares Pontes, Luis Adrián Cabrera-Diego, Ahmed Hamdi, José Moreno, Nicolas Sidère, and Antoine Doucet. 2020. Robust named entity recognition and linking on historical multilingual documents. In Conference and Labs of the Evaluation Forum (CLEF 2020), volume 2696, pages 1–17. CEUR-WS Working Notes. find
  13. Kasra Hosseini, Kaspar Beelen, Giovanni Colavizza, and Mariona Coll Ardanuy. 2021b. Neural Language Models for Nineteenth-Century English (dataset; language model zoo). find
  14. Renfen Hu, Shen Li, and Shichen Liang. 2019. Diachronic sense modeling with deep contextualized word embeddings: An ecological view. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3899–3908, Florence, Italy. Association for Computational Linguistics. find
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  16. Eric Huang, Richard Socher, Christopher Manning, and Andrew Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 873–882, Jeju Island, Korea. Association for Computational Linguistics. find
  17. Maud Ehrmann, Matteo Romanello, Alex Flückiger, and Simon Clematide. 2020. Overview of CLEF HIPE 2020: Named entity recognition and linking on historical newspapers. In Experimental IR Meets Multilinguality, Multimodality, and Interaction, pages 288–310. Springer International Publishing. find
  18. Leonard Konle and Fotis Jannidis. 2020. Domain and Task Adaptive Pretraining for Language Models. In Proceedings of the Workshop on Computational Humanities Research (CHR 2020), volume 2723 of CEUR Workshop Proceedings, pages 248–256. find
  19. Susan Fitzmaurice, Justyna A Robinson, Marc Alexander, Iona C Hine, Seth Mehl, and Fraser Dallachy. 2017. Linguistic dna: Investigating conceptual change in early modern english discourse. Studia Neophilologica, 89(sup1):21–38. find
  20. Anthony Kroch, Beatrice Santorini, and Lauren Delfs. 2004. Penn-helsinki parsed corpus of early modern english. find
  21. Lauren Fonteyn. 2020. What about grammar? Using BERT embeddings to explore functional-semantic shifts of semi-lexical and grammatical constructions. In Proceedings of the Workshop on Computational British Library Labs. 2014. Digitised books. c. 1510 - c. 1900. json (ocr derived text). find
  22. Kai Labusch, Clemens Neudecker, and David Zellhöfer. 2019. Bert for named entity recognition in contemporary and historical german. In Proceedings of the 15th Conference on Natural Language Processing, Erlangen, Germany, pages 8–11. American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1267–1273, Minneapolis, Minnesota. Association for Computational Linguistics. find
  23. Adam Langley and Dan S. Bloomberg. 2007. Google Books: making the public domain universally accessible. In Document Recognition and Retrieval XIV, volume 6500, pages 148 – 157. International Society for Optics and Photonics, SPIE. find
  24. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  25. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics. find
  26. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint arXiv:2107.13586. ↗
  27. Enrique Manjavacas, Ákos Kádár, and Mike Kestemont. 2019. Improving lemmatization of non-standard languages with joint learning. In NAACL HLT 2019 -2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, volume 1, pages 1493–1503. Association for Computational Linguistics. find
  28. Ville Satopaa, Jeannie Albrecht, David Irwin, and Barath Raghavan. 2011. Finding a ”kneedle” in a haystack: Detecting knee points in system behavior. In 2011 31st International Conference on Distributed Computing Systems Workshops, pages 166–171. find
  29. Dominik Schlechtweg, Barbara McGillivray, Simon Hengchen, Haim Dubossarsky, and Nina Tahmasebi. 2020. SemEval-2020 task 1: Unsupervised lexical semantic change detection. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1–23, Barcelona (online). International Committee for Computational Linguistics. find
  30. Jani Marjanen, Lidia Pivovarova, Elaine Zosa, and Jussi Kurunmaki. 2019. Clustering Ideological Terms in Historical Newspaper Data with Diachronic Word Embeddings. In The 5th International Workshop on Computational History (HistoInformatics 2019), volume 2461 of CEUR Workshop Proceedings, pages 21–29. find
  31. Mike Schuster and Kaisuke Nakajima. 2012. Japanese and korean voice search. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5149–5152. IEEE. find
  32. Carlos Martinez-Ortiz, Tom Kenter, Melvin Wevers, Pim Huijnen, and Joris van Eijnatten. 2019. Design and implementation of ShiCo: Visualising shifting concepts over time. In The 5th International Workshop on Computational History (HistoInformatics 2019), volume 1632 of CEUR Workshop Proceedings, pages 11–19. find
  33. Stefan Schweter. 2020. Europeana bert and electra models. find
  34. Stefan Schweter and Johannes Baiter. 2019. Towards robust named entity recognition for historic German. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 96–103, Florence, Italy. Association for Computational Linguistics. find
  35. Martin Mueller, Philip R Burns, and Craig A Berry. 2016. Collaborative curation and exploration of the eebo-tcp corpus. In Laura Estill, Diane K. Jakacki, and Michael Ullyot, editors, Early Modern Studies after the Digital Turn, chapter 7, pages 147–167. Iter and the Arizona Center for Medieval and Renaissance Studies. find
  36. Stefan Schweter and Luisa März. 2020. Triple e-effective ensembling of embeddings and language models for ner of historical german. In Conference and Labs of the Evaluation Forum (CLEF 2020), volume 2696. CEUR-WS Working Notes. find
  37. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics. find
  38. John Simpson and Edmund Weiner. 1989. Oxford English Dictionary. Oxford University Press. find
  39. Pia Sommerauer and Antske Fokkens. 2019. Conceptual Change and Distributional Semantic Models: An Exploratory Study on Pitfalls and Possibilities. In Proceedings of the 1st International Workshop on Computational Approaches to Historical Language Change, pages 223–233. Association for Computational Linguistics. find
  40. Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North Nina Tahmasebi, Lars Borin, Adam Jatowt, et al. 2018. Survey of computational approaches to diachronic conceptual change. arXiv preprint arXiv:1811.06278. ↗
  41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  42. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-ofthe-art natural language processing. arXiv preprint arXiv:1910.03771. ↗
  43. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19–27. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/arevalo-2021-macberth-development-evaluation-historically-pre/