Automatic training of lemmatization rules that handle morphological changes in pre-, in- and suffixes alike

What it is about

Most lemmatizers only cut suffixes, which fails when a word changes in its prefix or in its middle. The author trains lemmatization rules automatically from full-form and lemma pairs. The rules can change prefixes, infixes and suffixes, and the lemmatizer applies them to a new full form. The method is trained on ten languages: Danish, Dutch, English, German, Greek, Icelandic, Norwegian, Polish, Slovene and Swedish. Compared with a suffix-only lemmatizer, it improves by 24% for Polish, 2.3% for Dutch, 1.5% for English, 1.2% for German and 1.0% for Swedish. Icelandic gets worse by 1.9%. The paper also reports how the number of rules grows with the number of training pairs.

Ideas

• I1. en/src/morph.rs: learn lemma rules that can change a prefix or infix from data/forms.tsv and data/lemmas.tsv, and use them only for forms that the dictionary does not cover. Measure lemma accuracy on the uncovered forms of EWT test, against the suffix-only rules.
• I2. Count the rules against the training pairs, as the paper does, and print the rule count per run. A rule count that keeps growing with the pairs is a sign of over-fitting, and it can be checked before a claim.
• I3. Test the paper's English gain of 1.5% on Mova's own form table: remove a random tenth of the English forms, learn the rules, and measure how many removed lemmas the rules recover. The result is a Mova number, not the paper's.

For Mova

The code is en/src/morph.rs and en/src/lexgen.rs, with en morph-eval. The chapter is 06. Spelling rules: inflection, derivation, compounds and national standards, and for prefixes 16. Derivation I: prefixes. English gains here are small, so the value is the method for prefix changes, which English has in few words. Measure first on EWT test, twice, with a two-fold check (en folds).

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.3115/1687878.1687900

@inproceedings{jongejan2009automatic,
  title = {{Automatic training of lemmatization rules that handle morphological changes in pre-, in- and suffixes alike}},
  author = {Jongejan, Bart and Dalianis, Hercules},
  booktitle = {Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP},
  year = {2009},
  doi = {10.3115/1687878.1687900},
  url = {https://doi.org/10.3115/1687878.1687900}
}

References

24 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Per Bak, Chao Tang and Kurt Wiesenfeld. 1987. Self-Organized Criticality: An Explanation of 1/f Noise, Phys. Rev. Lett., vol. 59,. pp. 381-384, 1987 find
2. Per Bak, Chao Tang and Kurt Wiesenfeld . 1988. Phys. Rev. A38, (1988), pp. 364-374 find
3. Andrzej Białecki, 2004, Stempel - Algorithmic Stemmer for Polish Language http://www.getopt.org/stempel/ ↗
4. Remco R. Bouckaert and Eibe Frank. 2000. Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms. In H. Dai, R. Srikant, & C. Zhang (Eds.), Proc. 8th Pacific-Asia Conference, PAKDD 2004, Sydney, Australia, May 26-28, 2004 (pp. 3-12). Berlin: Springer. find
5. Johan Carlberger, Hercules Dalianis, Martin Hassel, and Ola Knutsson. 2001. Improving Precision in Information Retrieval for Swedish using Stemming. In the Proceedings of NoDaLiDa-01 - 13th Nordic Conference on Computational Linguistics, May 21-22, Uppsala, Sweden. find
6. Celex: http://celex.mpi.nl/ ↗
7. Hercules Dalianis and Bart Jongejan 2006. Hand-crafted versus Machine-learned Inflectional Rules: the Euroling-SiteSeeker Stemmer and CST's Lemmatiser, in Proceedings of the International Conference on Language Resources and Evaluation, LREC 2006. find
8. F. Çuna Ekmekçioglu, Mikael F. Lynch, and Peter Willett. 1996. Stemming and N-gram matching for term conflation in Turkish texts. Information Research, 7(1) pp 2-6. find
9. Niklas Hedlund 2001. Automatic construction of stemming rules, Master Thesis, NADA-KTH, Stockholm, TRITA-NA-E0194. find
10. IFD: Icelandic Centre for Language Technology, http://tungutaekni.is/researchsystems/rannsoknir\_1 2en.html Bart Jongejan and Dorte Haltrup. 2005. The CST Lemmatiser. Center for Sprogteknologi, University of Copenhagen version 2.7 (August, 23 2005) http://cst.dk/online/lemmatiser/cstlemma.pdf ↗
11. Jakub Kanis and Ludek Müller. 2005. Automatic Lemmatizer Construction with Focus on OOV Words Lemmatization in Text, Speech and Dialogue, Lecture Notes in Computer Science, Berlin / Heidelberg, pp 132-139 find
12. Ron Kohavi. 1995. A study of cross-validation and bootstrap for accuracy estimation and model selection. Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence 2 (12): 1137–1143, Morgan Kaufmann, San Mateo. find
13. Prasenjit Majumder, Mandar Mitra, Swapan K. Parui, Gobinda Kole, Pabitra Mitra, and Kalyankumar Datta. 2007. YASS: Yet another suffix stripper. ACM Transactions on Information Systems , Volume 25 , Issue 4, October 2007. find
14. Juršič Matjaž, Igor Mozetič, and Nada Lavrač. 2007. Learning ripple down rules for efficient lemmatization In proceeding of the Conference on Data Mining and Data Warehouses (SiKDD 2007), October 12, 2007, Ljubljana, Slovenia Morfologik: Polish morphological analyzer http://mac.softpedia.com/get/Wor… ↗
15. Douglas W. Oard, Gina-Anne Levow, and Clara I. Cabezas. 2001. CLEF experiments at Maryland: Statistical stemming and backoff translation. In Cross-language information retrieval and evaluation: Proceeding of the Clef 2000 workshops Carol Peters Ed. Springer Verlag pp. 176-187. 2001. find
16. Georgios Petasis, Vangelis Karkaletsis , Dimitra Farmakiotou , Ion Androutsopoulos and Constantine D. Spyropoulo. 2003. A Greek Morphological Lexicon and its Exploitation by Natural Language Processing Applications. In Lecture Notes on Computer Science (LNCS), vol.2563, "Advances in Informatics - Po… find
17. Joël Plisson, Nada Lavrač, and Dunja Mladenic. 2004, A rule based approach to word lemmatization, Proceedings of the 7th International Multiconference Information Society, IS-2004, Institut Jozef Stefan, Ljubljana, pp.83-6. find
18. Martin F. Porter 1980. An algorithm for suffix stripping. Program, vol 14, no 3, pp 130-130. find
19. John W. Ratcliff and David Metzener, 1988. Pattern Matching: The Gestalt Approach, Dr. Dobb's Journal, page 46, July 1988. find
20. SCARRIE 2009. Scandinavian Proofreading Tools http://ling.uib.no/~desmedt/scarrie/ ↗
21. STO: http://cst.ku.dk/sto\_ordbase/ ↗
22. SUC 2009. Stockholm Umeå corpus, http://www.ling.su.se/staff/sofia/suc/suc.html ↗
23. Pieter Theron and Ian Cloete 1997 Automatic acquisition of two-level morphological rules, Proceedings of the fifth conference on Applied natural language processing, p.103-110, March 31-April 03, 1997, Washington, DC. find
24. Ellen M. Voorhees. 2000. Variations in relevance judgments and the measurement of retrieval effectiveness, J. of Information Processing and Management 36 (2000) pp 697-716 find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/aclbulk-2009-jongejan-automatic-training-lemmatization-rules-that/