Evaluating Lemmatization Models for Machine-Assisted Corpus-Dictionary Linkage

What it is about

The task is to link each word token of a Syriac corpus to the right dictionary headword (corpus-dictionary linkage), here reduced to stem-to-baseform linkage, since segmentation already reaches about 98.9% accuracy. The authors adapt the DirecTL+ discriminative string transducer to this task and compare it with Morfette (a maximum-entropy edit-script classifier) and with a baseline that memorises the most frequent baseform per stem. The data is the Peshitta New Testament (109,640 tokens, 3,038 baseform types). With local sentence context, DirecTL+ reaches 96.92% overall accuracy on the blind test, against 96.06% for Morfette with local context, but it trains about 25 times slower (100 minutes against 3.7). The baseline is at 98.2% on known stems and 26% on unknown ones; the discriminative models are at about 60% on unknown stems. The error analysis splits stems into known or unknown and stable or ambiguous cases.

Ideas

• I1. Report lemmatizer accuracy in the same six error classes (known stable unambiguous, known latent ambiguous, known stable ambiguous, known dynamic ambiguous, unknown unambiguous, unknown ambiguous). For tools/hbo and macula dict, this shows whether a gain comes from known stems or from unknown ones, which a single overall figure hides.
• I2. A hybrid lemmatizer: a lookup table of stem-to-baseform pairs seen in the dictionary or in macula dict first, a rule-based or statistical guesser only for unknown stems. The paper shows the table is near-perfect on seen stems and the guesser is needed for the rest.
• I3. Train time as a selection criterion: a model that is 1% better but 25 times slower to train loses in an iteratively refined assist loop. Mova's choice of lemmatizer should be scored on accuracy and on the cost of each update.

For Mova

I1 is a test table for tools/hbo and for the Hebrew lemma work in The lexicon. I2 belongs to The lexicon of Biblical Aramaic (the Aramaic lemma table) and to macula dict for Hebrew. I3 is a decision rule for any future lemmatizer comparison.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/3xt99g4swge5

@inproceedings{black2014evaluating,
  title = {{Evaluating Lemmatization Models for Machine-Assisted Corpus-Dictionary Linkage}},
  author = {Black, Kevin and Ringger, Eric and Felt, Paul and Seppi, Kevin and Heal, Kristian and Lonsdale, Deryle},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/3xt99g4swge5},
  url = {https://doi.org/10.63317/3xt99g4swge5}
}

References

17 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Carmen, M., Felt, P., Haertel, R., Lonsdale, D., McClanahan, P., Merkling, O., Ringger, E., and Seppi, K. (2010). Tag dictionaries accelerate manual annotation. In Proceedings of LREC-10, Valletta, Malta. ELRA. ↗
2. Chrupała, G., Dinu, G., and van Genabith, J. (2008). Learning morphology with Morfette. In Proceedings of LREC-08, Marrakech, Morocco. ELRA. ↗ our card
3. Collins, M. (2002). Discriminative training methods for Hidden Markov Models: Theory and experiments with perceptron algorithms. In Proceedings of the EMNLP-02, pages 1–8, Stroudsburg, PA. ACL. ↗ our card
4. Felt, P., Ringger, E. K., Seppi, K., Heal, K. S., Haertel, R. A., and Lonsdale, D. (2013). Evaluating machine-assisted annotation in under-resourced settings. Language Resources and Evaluation. ↗
5. Habash, N. and Rambow, O. (2005). Arabic tokenization, part-of-speech tagging and morphological disambiguation in one fell swoop. In Proceedings of ACL-05, pages 573–580, Stroudsburg, PA. ACL. ↗ our card
6. Jiampojamarn, S., Kondrak, G., and Sherif, T. (2007). KLA Correct 61 ± 40 39 ± 42 40 ± 42 41 ± 42 UU Correct 344 ± 20 79 ± 14 208 ± 14 205 ± 15 KSA Acc. N/A 90.70 ± 0.98 90.17 ± 0.93 93.48 ± 0.86 UA Acc. N/A 27.77 ± 31.03 36.11 ± 28.70 27.77 ± 31.03 KSA Correct 1297 ± 55 1176 ± 57 1170 ± 57 1212 ± 56 UA Correct 1.3 ± 1.1 0.40 ± 0.70 0.50 ± 0.71 0.40 ± 0.70 Applying many-to-many alignments and hidden Markov models to letter-to-phoneme conversion. In Proceeding of NAACL-07, pages 372–379, Rochester, NY. ACL. find
7. Jiampojamarn, S., Cherry, C., and Kondrak, G. (2008). Joint processing and discriminative training for letter-to-phoneme conversion. In Proceedings of ACL-08: HLT, pages 905–913, Columbus, OH. ACL. find
8. Jiampojamarn, S., Bhargava, A., Dou, Q., Dwyer, K., and Kondrak, G. (2009). DirecTL: a language independent approach to transliteration. In Proceedings of NEWS-09, pages 28–31, Suntec, Singapore. ACL. ↗
9. Jiampojamarn, S., Cherry, C., and Kondrak, G. (2010). Integrating joint n-gram features into a discriminative training framework. In Proceedings of NAACL-10, pages 697–700, Los Angeles, CA. ACL. find
10. Kilgarriff, A. and Rundell, M. (2002). Lexical profiling software and its lexicographic applications - a case study. In Proceedings of EURALEX-02, pages 807–818, København, Denmark. Center for Sprogteknologi. find
11. Kilgarriff, A. (2005). Linking dictionary and corpus. In Proceedings of ASIALEX-05, Singapore. find
12. Kiraz, G. A. (1994). Automatic concordance generation of Syriac texts. Orientalia Christiana Analecta, 247:461– 475. find
13. Kiraz, G. A. (2000). Multitiered nonlinear morphology using multitape finite automata: A case study on Syriac and Arabic. Computational Linguistics, 26(1):77–105. ↗ our card
14. Lindgren, L. (2011). dkrMorph: A Syriac morphological analyzer. Bachelor’s thesis, Uppsala University, Department of Informatics and Media. find
15. McClanahan, P., Busby, G., Haertel, R., Heal, K., Lonsdale, D., Seppi, K., and Ringger, E. (2010). A probabilistic morphological analyzer for Syriac. In Proceedings of EMNLP-10, pages 810–820, Cambridge, MA. ACL. find our card
16. Payne Smith, J. (1903). A compendious Syriac dictionary founded upon the Thesaurus Syriacus of R. Payne Smith. find
17. Ringger, E., Carmen, M., Haertel, R., Seppi, K., Lonsdale, D., McClanahan, P., Carroll, J., and Ellison, N. (2008). Assessing the costs of machine-assisted corpus annotation through a user study. In Proceedings of LREC-08, pages 3318–3324, Marrakech, Morocco. ELRA. ↗

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lingbulk-2014-black-evaluating-lemmatization-models-machine-assisted/