Language Classification and Segmentation of Noisy Documents in Hebrew Scripts

What it is about

Jewish manuscripts from the Cairo Genizah mix Hebrew, Aramaic, Judeo-Arabic and other languages written in the Hebrew script, often switching language several times on one page, and the text they come with is the output of noisy OCR on handwriting. The authors cut a text into arbitrary fixed-size segments and classify each one by the cosine distance between its character bigram distributions; bigrams beat unigrams and usually trigrams. Classification is then refined with neighbouring segments as features, a boundary search after each language switch, and an "unknown character" symbol for OCR gaps. The test documents are synthetic: segments from pairs of languages, with a controlled average segment length k. Hebrew against Aramaic is the hard pair; Hebrew against Judeo-Arabic is easy to classify but hard to segment. Neighbour features help short segments, farther neighbours add nothing, and post-processing improves word accuracy at every segment length. For the noise, simply ignoring unknown bigrams does about as well as estimating them, while error correction and restricting to the most common n-grams do not help. The figures carry the exact numbers; they were not readable in the extracted text, so none are quoted here.

Ideas

• I1. A segmenter for mixed-language text in tools/langseg: cut at word boundaries into windows, score each window against bigram profiles of hbo, arc, tmr and jpa, then merge neighbours with a penalty for each language switch. Keep the profiles as count tables built from existing corpora, so the whole thing is deterministic and each decision can say which profile won. Measure on synthetic mixes built as in the paper, with k from about 50 to 250 characters.
• I2. Boundary refinement as an exact search: after each detected switch, try a few split points in the joined two windows and keep the split with the best combined score. It is cheap, and the paper's post-processing result says it helps at every length.
• I3. An unknown-character symbol for OCR input in tools/ocrfix: a marker for an unreadable letter, with every bigram that contains it ignored in the score. This is the simplest variant the paper found as good as the others; start with it.

For Mova

I1 and I2 belong to tools/langseg, where the language classes already exist (hbo, arc, tmr, jpa), and their test is a set of synthetic mixed documents shared with tools/hbo. I3 belongs to tools/ocrfix. The Aramaic side is covered by languages/arc/ and languages/tmr/, which the classes must match.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{zhicharevich2012language,
  title = {{Language Classification and Segmentation of Noisy Documents in Hebrew Scripts}},
  author = {Zhicharevich, Alex and Dershowitz, Nachum},
  booktitle = {Proceedings of the 6th Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities},
  year = {2012},
  url = {https://aclanthology.org/W12-1016/}
}

References

6 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Gökhan Bakır. 2007. Predicting Structured Data. MIT Press, Cambridge, MA. find
2. Freddy Y. Y. Choi. 2000. Advances in domain independent linear text segmentation. Proc. 1st North American Chapter of the Association for Computational Linguistics Conference, pp. 26-33. find
3. Juha Häkkinen and Jilei Tian. 2001. N-gram and decision tree based language identification for written words. Proc. Automatic Speech Recognition and Understanding (ASRU '01), Italy, pp. 335-338. find
4. Marti A. Hearst. 1993. TextTiling: A quantitative approach to discourse segmentation. Technical Report, Sequoia 93/24, Computer Science Division. find
5. Sylvain Lamprier, Tassadit Amghar, Bernard Levrat, and Frédéric Saubion. 2007. On evaluation methodologies for text segmentation algorithms. Proc. 9th IEEE International Conference on Tools with Artificial Intelligence (ICTAI), volume 2, pp. 19-26. find
6. Karen Kukich. 1992. Techniques for automatically correcting words in text. ACM Computing Surveys 24(4): 377-439. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2012-zhicharevich-language-classification-segmentation-noisy-documents/