How BERT Speaks Shakespearean English? Evaluating Historical Bias in Contextual Language Models

What it is about

The paper asks whether contextual language models know the difference between Early Modern English and Modern English. It builds a test of 60 sentences with one masked word each: 20 marked for Early Modern English, 20 for Modern English and 20 neutral. Each sentence gets a temporal score on a five-point scale, set by hand from the linguistic markedness of the sentence. Three BERT-family models fill each mask, and each candidate word receives a score for its era in its own sentence context, not in isolation. The results are weighted into a bias score and an adequacy score. MacBERTh, trained on 1500 to 1950 texts, is most aligned with the early sentences. Plain BERT is most aligned with the modern ones. On neutral sentences all models do well. The authors call the study preliminary: one test set, three models, hand-set labels.

Ideas

• I1. Era audit of the standard. Score the standard encoder's chunk candidates for each concept with an era value taken from the author corpora, using a count ratio: how often the word appears in early-author text against modern text. On the Bible pairs this shows which standard choices are archaic, such as hath or unto (open problem 6).
• I2. Score in context, not globally. The paper shows that you is neutral in general but archaic in a particular sentence. The encoder already picks the most specific context seen often enough, so the era audit should use the same context key.
• I3. Use the adequacy score to check an author delta. After a delta applies, output on held-out verses should move toward the target era. A drop would show the delta is not doing what its labels say.
• I4. Treat the masked model as a teacher only, labelling candidate sets offline. The SLM answers at test time from counts.

For Mova

The code is tools/translate (profile.rs, measure.rs). The neural masked-language model is not for Mova, but the era-scoring idea carries over in count form. It gives the translator a way to find archaic choices in the standard without hand lists, and an adequacy check for deltas on held-out verses.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2402.05034

@misc{cuscito2024how,
  title = {{How BERT Speaks Shakespearean English? Evaluating Historical Bias in Contextual Language Models}},
  author = {Cuscito, Miriam and Ferrara, Alfio and Ruskov, Martin},
  year = {2024},
  eprint = {2402.05034},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.05034}
}

References

14 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Jaimeen Ahn and Alice Oh. 2021. Mitigating language-dependent ethnic bias in BERT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 533–549, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
2. Lera Boroditsky. 2011. How Language Shapes Thought. Scientific American, 304(2):62–65. find
3. Jerome Bruner. 1984. Pragmatics of Language and Language of Pragmatics. Social Research, 51(4):969– 984. find
4. Penelope J. Corfield. 2022. Fleeting gestures and changing styles of greeting: researching daily life in British towns in the long eighteenth century. Urban History, 49(3):555–567. find
5. Daniel de Vassimon Manela, David Errington, Thomas Fisher, Boris van Breugel, and Pasquale Minervini. 2021. Stereotype and skew: Quantifying gender bias in pre-trained and fine-tuned language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2232–2242, Online. Association for Computational Linguistics. find
6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
7. Alba Morollon Diaz-Faes, Carla Murteira, and Martin Ruskov. 2023. Explicit references to social values in fairy tales: A comparison between three european cultures. In The Joint 3rd International Conference on Natural Language Processing for Digital Humanities & 8th International Workshop on Computational Linguistics for Uralic Languages, Tokyo, Japan. Association for Computational Linguistics. find
8. Enrique Manjavacas and Lauren Fonteyn. 2022. Adapting vs. Pre-training Language Models for Historical Languages. Journal of Data Mining & Digital Humanities, NLP4DH. find
9. Lev Manovich. 2020. Cultural analytics. The MIT Press, Cambridge, Massachusetts. find
10. Stefano Montanelli and Francesco Periti. 2023. A Survey on Contextualised Semantic Shift Detection. ArXiv:2304.01666. ↗
11. Marzieh Mozafari, Reza Farahbakhsh, and Noël Crespi. 2020. Hate speech detection and racial bias mitigation in social media based on BERT model. PLOS ONE, 15(8):e0237861. find
12. Andrew Piper, Hao Xu, and Eric D. Kolaczyk. 2023. Modeling Narrative Revelation. In Proceedings of the Computational Humanities Research Conference 2023, volume 3558 of CEUR Workshop Proceedings, pages 500–511, Paris, France. CEUR. find
13. Martin Ruskov. 2023. Who and How: Using Sentence-Level NLP to Evaluate Idea Completeness. In Artificial Intelligence in Education. Posters and Late Breaking Results, Workshops and Tutorials, Industry and Innovation Tracks, Practitioners, Doctoral Consortium and Blue Sky, Communications in Computer and Information Science, pages 284–289, Cham. Springer Nature Switzerland. find
14. L.W.C. van Lit. 2019. Among Digitized Manuscripts. Philology, Codicology, Paleography in a Digital World. Brill, Leiden, The Netherlands. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/cuscito-2024-bert-speaks-shakespearean-english-evaluating/