Rule-Based Normalization of Historical Texts

What it is about

The paper normalizes Early New High German spelling to modern forms using only unsupervised learning and freely available resources. The training material is two aligned versions of the Luther Bible, which give word pairs that differ in spelling. From these the method derives context-aware rewrite rules over character sequences, including identity rules that leave a character unchanged, and weights each rule by its frequency. To normalize a word, the rules are applied to each character and the candidates are checked against a modern lexicon. The evaluation finds that the rules reach 83 to 91 percent exact matches, against 65 percent for leaving the text unchanged. The main open issue is the multi-path problem: one historical word can be reached from the source by several rule sequences, and the method does not yet choose the right one reliably.

Ideas

• I1. Derive word-level rewrite rules from aligned versions of the Bible, for example KJV against WEB at verse level. A rule is a change with its context and a frequency weight, and identity rules keep unchanged words. This gives the author delta as an explicit rule table. Measure exact matches on dev verses against copying the standard.
• I2. Handle the multi-path problem with the paper's own suggestion: keep the derivation with the fewest rules. Count ties in BTreeMap order, so the choice is deterministic, and report how often ties occur. The same ambiguity appears in Hebrew word alignment.
• I3. Frequency-weighted identity rules are a count-based form of the standard-by-default rule. A delta then needs a rule with real counts behind it, which matches the threshold on overrides already used in tools/translate.
• I4. Use the rule count as a size measure of a delta. A small rule table for an author that is close to the standard is a sign that the author delta is well founded; a large one points to noise.

More ideas (a second reading)

• I5. en/src/morph.rs: derive rewrite rules from aligned archaic and modern forms in the tools/gutenberg lessons (for example hast to has, doth to does, -eth to -s), rank them by frequency, and measure the exact-match rate on held-out lessons against the archaic function.
• I6. Keep both rankers: a best-rule ranker for unseen words and a best-probability ranker for seen words. The 3% of words where the two disagree show where a second method adds something. Test the combination with en folds.
• I7. Clitic contractions: the paper's example soltu for sollst du is a rule family of verb-plus-pronoun forms. Count how often the thou-forms (art, dost, hast) in tools/gutenberg lessons are fixed by the archaic function, and add rules only for the misses.

For Mova

The code is tools/translate (the delta side, and profile.rs for the standard). The method is rule-based and count-based, so it fits Mova's design directly. It addresses open problem 6 by deriving the delta from the data, and open problem 2 through the multi-path choice.

Topics: Machine translation

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{bollmann2011rule,
  title = {{Rule-Based Normalization of Historical Texts}},
  author = {Bollmann, Marcel and Petran, Florian and Dipper, Stefanie},
  booktitle = {Proceedings of the Workshop on Language Technologies for Digital Humanities and Cultural Heritage},
  year = {2011},
  url = {https://aclanthology.org/W11-4106/}
}

References

11 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alistair Baron, Paul Rayson, and Dawn Archer. 2009. Automatic standardization of spelling for historical text mining. In Proceedings of Digital Humanities. find
2. Karin Beijering, Charlotte Gooskens, and Wilbert Heeringa. 2008. Predicting intelligibility and perceived linguistic distances by means of the Levenshtein algorithm. Linguistics in the Netherlands, pages 13–24. find
3. Werner Besch. 2000. Die Rolle Luthers für die deutsche Sprachgeschichte. In Werner Besch et al., editor, Sprachgeschichte. Ein Handbuch zur Geschichte der deutschen Sprache und ihrer Erforschung, pages 1713–1745. de Gruyter, 2nd edition. find
4. Fabienne Braune and Alexander Fraser. 2010. Improved unsupervised sentence alignment for symmetrical and asymmetrical parallel corpora. In Proceedings of COLING, Poster Volume, pages 81–89. find
5. Andrea Ernst-Gerlach and Norbert Fuhr. 2006. Generating search term variants for text collections with historic spellings. In Proceedings of the 28th European Conference on Information Retrieval Research (ECIR 2006). Springer. find
6. Andreas W. Hauser and Klaus U. Schulz. 2007. Unsupervised learning of edit distance weights for retrieving historical spelling variations. In Proceedings of the First Workshop on Finite-State Techniques and Approximate Search, pages 1–6. find
7. Bryan Jurish. 2010. More than words: Using token context to improve canonicalization of historical German. Journal for Language Technology and Computational Linguistics, 25(1):23–39. find
8. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of MT Summit. find
9. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
10. Jan Strunk. 2003. Information retrieval for languages that lack a fixed orthography. Seminar Paper, Stanford University. http://www.linguistics. rub.de/˜strunk/LSreport.pdf. ↗
11. Norbert Richard Wolf. 2000. Handschrift und Druck. In Werner Besch et al., editor, Sprachgeschichte. Ein Handbuch zur Geschichte der deutschen Sprache und ihrer Erforschung, pages 1705–1713. de Gruyter, 2nd edition. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/bollmann-2011-rule-based-normalization-historical-texts/