D-Nikud: Enhancing Hebrew Diacritization with LSTM and Pretrained Models

What it is about

Another Hebrew vocaliser: frozen TavBERT character embeddings, two Bi-LSTM layers and three classifier heads per letter (vowel, dagesh, shin/sin dot). Training data are Nakdimon's corpora (about 1.8 million pre-modern tokens: Ben-Yehuda, Mishneh Torah from Mamre, short stories; and 1.5 million modern tokens auto-pointed by Dicta and corrected) plus Dicta's material, all in full spelling (ktiv male). On Nakdimon's test set it leads on decision and character accuracy (about 98.3 / 97) but trails Dicta on whole-word accuracy; it is best on texts addressed to women, where the gender of second-person forms hides in the vowels. By genre, law and modern prose score highest (CHA ≈ 97–99), while biblical text is worst by far (CHA 90.7, word accuracy 75.7).

Ideas

I1. The biblical score (three words in four right) shows that modern vocalisers do not transfer to Biblical Hebrew; quote it in the comparison chapter as one more measured reason why Modern Hebrew tools misread the Bible.
I2. Mova's fully pointed Torah is itself a gold test for any vocaliser: strip the points (keep consonants and the matres as written), run the tool, score by the paper's DEC/CHA/WOR/VOC. A cheap measure for a neural teacher, never a part of Mova's answers.
I3. Second-person gender lives in the vowels (ךָ vs ךְ, אַתָּה/אַתְּ, qatal תָּ/תְּ). Check that every second-person form in the trees carries Gender and that it agrees with the addressee in prag; a disagreement is a converter or data defect.

For Mova

I1 is a sentence in 12. Biblical, Mishnaic and Modern Hebrew compared. I2 is a measure described in 13-mova-tools.md, run only with the operator's word if a model must be installed. I3 is a FEATS check in tools/macula and an agreement test between the trees and tools/macula/src/prag.rs (addressee), with the rule in 05-pronouns-particles.md.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2402.00075

@misc{rosenthal2024d,
  title = {{D-Nikud: Enhancing Hebrew Diacritization with LSTM and Pretrained Models}},
  author = {Rosenthal, Adi and Shaked, Nadav},
  year = {2024},
  eprint = {2402.00075},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2402.00075}
}

References

3 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Elazar Gershuni and Yuval Pinter, Restoring Hebrew Diacritics Without a Dictionary, 2022, March, https:// arxiv.org/pdf/2105.05209.pdf. [5] Sepp Hochreiter and Jurgen Schmidhuber, LONG SHORT-TERM MEMORY, 1997, April, https://www.bioinf.jku.at/publications/older/2604.pdf, MIT Press. ↗ our card
Avi Shmidman and Shaltiel Shmidman and Moshe Koppel and Yoav Goldberg, Nakdan: Professional Hebrew Diacritizer, 2020, March, https://arxiv.org/pdf/2005.03312.pdf. [6] Stav Klein and Reut Tsarfaty, Getting the life out of living: How Adequate Are Word-Pieces for Modelling Complex Morphology?, 2020, July, https://aclanthology.org/2020. sigmorphon-1.24. ↗ our card
Omri Keren and Tal Avinari and Reut Tsarfaty and Omer Levy, Breaking Character: Are Subwords Good Enough for MRLs After All?, 2022, April, https://arxiv.org/pdf/2204.04748.pdf. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/hbo-2024-rosenthal-d-nikud-enhancing-hebrew-diacritization/