NLP in the DH pipeline: Transfer-learning to a Chronolect

What it is about

The authors build a diacritizer (vowel marks) for Emergent Modern Hebrew, the Hebrew of the early twentieth century, using a hand-vocalised goldset of 144,446 words from the Jerusalem Corpus of Emergent Modern Hebrew (129,453 words of literature, 14,993 of ephemera). They use 120K words for training and 11K for testing. They compare an LSTM trained from scratch at four sizes (50K to 120K words) with fine-tuning Dicta's LSTM for Modern Hebrew, and also with fine-tuning a model for Rabbinic Hebrew. Fine-tuning from Modern Hebrew is better at every size; unconstrained word accuracy rises from 78.30% with no target-chronolect training to 81.98% at 120K, while training from scratch reaches 77.29%. A wordlist that restricts each word to known valid vocalisations adds 6 to 11 points in every setting (87.38% for the best fine-tuned model). Transfer helps only within the genre of the training data: on the ephemera test set, fine-tuning lowers accuracy from 80.66% to 74.01%.

Ideas

• I1. Constrain by a wordlist: in unpointed mode, restrict each token's candidates to forms in Mova's lexicon, then score them. The paper's largest single gain (6 to 11 points) comes from this lookup step, not from a bigger model. It fits the unpointed mode of tools/hbo, with the lex table as the list.
• I2. A learning curve as a planning tool: each 20K to 25K extra words gave about half a point to one and a half points. Use the same stepwise test before annotating any new Hebrew or Aramaic variety (for example languages/tmr/ or languages/jpa/) to decide how much gold to build first.
• I3. A held-out genre test: the transfer failed on the genre not seen in training. Keep a held-out genre in every test of a model trained on Biblical or Mishnaic text, so that a score is not read as success on a new genre.

For Mova

I1 goes into tools/hbo (unpointed mode) and is documented in The lexicon. I2 is a planning note for languages/tmr/ and languages/jpa/. I3 goes into 11. Mishnaic Hebrew (hbo-x-mishnaic) and 12. Biblical, Mishnaic and Modern Hebrew compared.

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{rubinstein2021nlp,
  title = {{NLP in the DH pipeline: Transfer-learning to a Chronolect}},
  author = {Rubinstein, Aynat and Shmidman, Avi},
  booktitle = {Proceedings of the Workshop on Natural Language Processing for Digital Humanities},
  year = {2021},
  url = {https://aclanthology.org/2021.nlp4dh-1.12/}
}

References

21 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Alistair Baron and Paul Rayson. 2008. VARD2: a tool for dealing with spelling variation in historical corpora. Postgraduate Conference in Corpus Linguistics ; Conference date: 22-05-2008. find
2. Ze’ev Ben-Hayyim. 1992. The struggle for a language. The Academy of the Hebrew Language, Jerusalem. In Hebrew. find
3. Haim Blanc. 1968. The Israeli koine as an emergent national standard. In Joshua A. Fishman, editor, Language problems of developing nations, pages 237–251. Wiley, New York. find
4. Yaacov Choueka and Yoni Neeman. 1995. Nakdan-text,(an in-context text-vocalizer for modern hebrew). In BISFAI-95, The Fifth Bar Ilan Symposium for Artificial Intelligence. find
5. Edit Doron, Malka Rappaport Hovav, Yael Reshef, and Moshe Taube. 2019a. Introduction. In Edit Doron, Malka Rappaport Hovav, Yael Reshef, and Moshe Taube, editors, Language Contact, Continuity and Change in the Genesis of Modern Hebrew, pages 1–31. John Benjamins, Amsterdam. find
6. Edit Doron, Malka Rappaport Hovav, Yael Reshef, and Moshe Taube, editors. 2019b. Language Contact, Continuity and Change in the Genesis of Modern Hebrew. John Benjamins, Amsterdam. find
7. Ya’akov Gal. 2002. An hmm approach to vowel restoration in arabic and hebrew. In Proceedings of the ACL-02 workshop on Computational approaches to semitic languages, pages 1–7. Association for Computational Linguistics. find our card
8. Eva Pettersson and Joakim Nivre. 2015. Improving verb phrase extraction from historical text by use of verb valency frames. In Proceedings of the 20th Nordic Conference of Computational Linguistics (NODALIDA 2015), pages 153– 161, Vilnius, Lithuania. Linköping University Electronic Press, Sweden. find
9. Chaim Rabin. 1999. What was the revival of the Hebrew language? Linguistic Studies, pages 359–376. In Hebrew. find
10. Yael Reshef. 2013. Revival of Hebrew: Grammatical structure and lexicon. In Geoffrey Khan, editor, Encyclopedia of Hebrew Language and Linguistics, volume 3, pages 397– 405. Brill, Leiden. find
11. Yael Reshef. 2015. Hebrew in the Mandate period. The Academy of the Hebrew Language, Jerusalem. In Hebrew. find
12. Yael Reshef. 2016. Written Hebrew of the revival generation as a distinct phase in the evolution of Modern Hebrew. Journal of Semitic Studies, 61(1):187–213. find
13. Haiim Rosén. 1956. Ha-‘ivrit šelanu. Am Oved, Tel Aviv. In Hebrew. find
14. Aynat Rubinstein. 2019. Historical corpora meet the digital humanities: the Jerusalem Corpus of Emergent Modern Hebrew. Language Resources and Evaluation, 53(4):807– 835. find
15. Gerold Schneider. 2020. Spelling normalisation of Late Modern English. In Merja Kytö and Erik Smitterberg, editors, Late Modern English: Novel encounters, pages 244—-268. John Benjamins, Amsterdam. find
16. Avi Shmidman, Shaltiel Shmidman, Moshe Koppel, and Yoav Goldberg. 2020. Nakdan: Professional Hebrew diacritizer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 197–203, Online. Association for Computational Linguistics. find our card
17. Reut Tsarfaty, Amit Seker, Shoval Sadde, and Stav Klein. 2019. What’s wrong with Hebrew NLP? and how to make it right. CoRR, abs/1908.05453. ↗ our card
18. Shuly Wintner. 1998. Towards a linguistically motivated computational grammar for Hebrew. In Computational Approaches to Semitic Languages. find our card
19. Elazar Gershuni and Yuval Pinter. 2021. Restoring hebrew diacritics without a dictionary. find our card
20. Viktor Golinets. 2013. Masora, Tiberian. In Geoffrey Khan, editor, Encyclopedia of Hebrew Language and Linguistics. Brill, Leiden. find
21. Jirka Hana, Anna Feldman, and Katsiaryna Aharodnik. 2011. A low-budget tagger for old Czech. In Proceedings of the 5th ACL-HLT Workshop on Language Technology for Cultural Heritage, Social Sciences, and Humanities, pages 10–18, Portland, OR, USA. Association for Computational Linguistics. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2021-rubinstein-nlp-dh-pipeline-transfer-learning/