Normalisation of Historical Text Using Context-Sensitive Weighted Levenshtein Distance and Compound Splitting

What it is about

The paper normalizes historical Swedish spelling to modern forms by comparing each historical token with a modern dictionary. The comparison uses a weighted Levenshtein distance, where some edits cost less because they are common spelling variation in the period, and the weights depend on context. Compounds are split before lookup, since historical Swedish writes many compounds apart or together differently. A small set of hand-checked normalizations is kept as a cache that overrides the automatic choice. Without any manually annotated training data, the share of tokens spelled as in the gold standard rises from a baseline of 64.6 percent to 77.0 percent. With the edit weights and the validated cache, it reaches 86.9 percent. Verb identification with a modern tagger also improves after normalization, which is the practical goal.

Ideas

• I1. Set edit costs by counting. For each edit (a letter change in a context, say), the cost falls with its frequency in aligned pairs. The counts are kept in BTreeMap order, so the costs are deterministic. This suits spelling deltas for an author and the Hebrew transliteration of names.
• I2. Treat the validated cache as the shape of the author delta. Override the standard only where the author's form is hand-checked, and keep an override only if it fixes at least three more tokens. Compare the two thresholds on dev.
• I3. Split compounds before a merge, so phrase concepts such as unto_thou are broken into parts that keep their function words. This targets open problem 2 directly: merges that drop function words are the noise the lexicon needs to avoid.
• I4. Bound candidate pairs by length difference. The paper shows that most true normalizations differ in length by only a little. For the lexicon, a length-ratio bound on candidate merges would cut noise merges cheaply.

More ideas (a second reading)

• I5. en/src/morph.rs: rank candidate modern forms for archaic English words that are not in the dictionary, by a weighted edit distance with the paper's limits (length change from +1 to -3, distance at most 4). Measure the exact-match rate on held-out tools/gutenberg lessons against the archaic rules alone.
• I6. Learn the edit weights from pairs: count the edits that separate archaic forms from their modern lemmas in the training lessons, and store them in a BTreeMap. Two identical runs are needed before any gain is claimed.
• I7. en/src/lexgen.rs: split unknown words into two dictionary parts, as the compound splitter does, and report how many unknown historical words are compounds. Use the count to decide whether the step is worth building.

For Mova

The code is tools/translate (meta.rs for the merges and compound handling, profile.rs for the cache-as-delta). The weighted edit idea is count-based and fits Mova's rules. The paper's compound split and length bound address open problem 2 directly, and the validated cache is the delta shape already in use.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{pettersson2013normalisation,
  title = {{Normalisation of Historical Text Using Context-Sensitive Weighted Levenshtein Distance and Compound Splitting}},
  author = {Pettersson, Eva and Megyesi, Beáta and Nivre, Joakim},
  booktitle = {Proceedings of the 19th Nordic Conference of Computational Linguistics (NODALIDA 2013)},
  year = {2013},
  url = {https://aclanthology.org/W13-5617/}
}

References

17 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Ågren, M., Fiebranz, R., Lindberg, E., and Lindström, J. (2011). Making verbs count. The research project ’Gender and Work’ and its methodology. Scandinavian Economic History Review, 59(3):271–291. Forthcoming. find
2. Baron, A. and Rayson, P. (2008). Vard2: A tool for dealing with spelling variation in historical corpora. In Postgraduate Conference in Corpus Linguistics, Aston University, Birmingham. find
3. Black, A. W. and Taylor, P. (1997). Festival speech synthesis system: system documentation. Technical report, University of Edinburgh, Centre for Speech Technology Research. find
4. Bollmann, M. (2012). (semi-)Automatic Normalization of Historical Texts using Distance Measures and the norma tool. In Proceedings of the Second Workshop on Annotation of Corpora for Research in the Humanitites (ACRH-2). find
5. Bollmann, M., Petran, F., and Dipper, S. (2011). Rule-based normalization of historical texts. In Proceedings of the Workshop on Language Technologies for Digital Humanities and Cultural Heritage, pages 34–42, Hissar, Bulgaria. find our card
6. Borin, L., Forsberg, M., and Lönngren, L. (2008). Saldo 1.0 (svenskt associationslexikon version 2). Språkbanken, University of Gothenburg. find
7. Brants, T. (2000). TnT - a statistical part-of-speech tagger. In Proceedings of the 6th Applied Natural Language Processing Conference (ANLP), Seattle, Washington, USA. find our card
8. Ejerhed, E. and Källgren, G. (1997). Stockholm Umeå Corpus. Version 1.0. Produced by Department of Linguistics, Umeå University and Department of Linguistics, Stockholm University. ISBN 91-7191-348-3. find
9. Halácsy, P., Kornai, A., and Oravecz, C. (2007). HunPos - an open source trigram tagger. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics, pages 209–212, Prague, Czech Republic. find
10. Jurish, B. (2008). Finding canonical forms for historical German text. In Storrer, A., Geyken, A., Siebert, A., and Würzner, K.-M., editors, Text Resources and Lexical Knowledge: Selected Papers from the 9th Conference on Natural Language Processing (KONVENS 2008), pages 27–37. Mouton de Gruyter, Berlin. find
11. Jurish, B. (2010). More Than Words: Using Token Context to Improve Canonicalization of Historical German. Journal for Language Technology and Computational Linguistics, 25(1):23– 39. find
12. Kukich, K. (1992). Techniques for automatically correcting words in text. ACM Computing Surveys (CSUR), 24(4):377–439. find
13. Levenshtein, V. (1966). Binary Codes Capable of Correcting Deletions, Insertions and Reversals. Soviet Physics Doklady, 10(8):707–710. find
14. Pettersson, E., Megyesi, B., and Nivre, J. (2012). Rule-based normalisation of historical text - a diachronic study. In Proceedings of the First International Workshop on Language Technology for Historical Text(s), Vienna, Austria. find
15. Rayson, P., Archer, D., and Nicholas, S. (2005). VARD versus Word – A comparison of the UCREL variant detector and modern spell checkers on English historical corpora. In Proceedings from the Corpus Linguistics Conference Series on-line e-journal, volume 1, Birmingham, UK. find
16. Stymne, S. (2008). German compounds in factored statistical machine translation. In Ranta, A. and Nordström, B., editors, Proceedings of GoTAL, 6th International Conference on Natural Language Processing, volume 5221, pages 464–475, Gothenburg, Sweden. Springer LNCS/LNAI. find
17. Stymne, S. and Holmqvist, M. (2008). Processing of Swedish Compounds for Phrase-Based Statistical Machine Translation. In Proceedings of the 12th EAMT conference, Hamburg, Germany. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/pettersson-2013-normalisation-historical-text-context-sensitive/