Very-large Scale Parsing and Normalization of Wiktionary Morphological Paradigms

What it is about

The paper describes a language-independent extractor that reads the paradigm tables of Wiktionary straight from its HTML. It separates inflected forms from grammatical descriptors by token frequency across lexemes, and maps the descriptors to one annotation scheme with 212 features in 23 dimensions. The output has 952,530 unique noun, verb and adjective lemmas across 350 languages, and about 1,000,000 paradigms overall. The authors report 123 usable language-POS pairs in 88 languages, against 5 pairs in 3 languages for an earlier language-specific extractor. The English edition often gives forms only as human-readable instructions, so its tables need separate handling.

Ideas

• I1. en/src/lexgen.rs: the cross-lexeme test (a token that recurs across many lemma tables is a descriptor, not a form) can flag suspect entries in data/forms.tsv. Measure how many entries it flags and hand-check a sample of 100 flagged entries before any entry is removed.
• I2. en/src/morph.rs: a descriptor table that maps each written tag to one feature set, as the paper's unified scheme does. Measure on EWT test the share of gold morphological features the table can express, and report it per feature dimension.
• I3. Layout consistency check: group English lemma tables by layout and count how many layouts give more than one feature reading. Use this as an offline check on the lexicon build, not as a training source.

For Mova

The code is en/src/lexgen.rs and en/src/morph.rs, with en/src/udreg.rs for the feature names. The check is en morph-eval. Wiktionary text is CC BY-SA, and the extracted resource is for tests and comparisons only; never carry the extracted tables into the build. Measure first: the lemma and feature accuracy of en morph-eval on EWT test, with and without the flagged entries removed, on the two-fold check.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/5nt2dyri2un4

@inproceedings{kirov2016very,
  title = {{Very-large Scale Parsing and Normalization of Wiktionary Morphological Paradigms}},
  author = {Kirov, Christo and Sylak-Glassman, John and Que, Roger and Yarowsky, David},
  booktitle = {LREC 2016 (Portorož)},
  year = {2016},
  doi = {10.63317/5nt2dyri2un4},
  url = {https://doi.org/10.63317/5nt2dyri2un4}
}

References

7 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Choi, J., de Marneffe, M.-C., Dozat, T., Ginter, F., Goldberg, Y., Hajič, J., Manning, C., McDonald, R., Nivre, J., Petrov, S., Pyysalo, S., Silveira, N., Tsarfaty, R., and Zeman, D. (2015). Universal Dependencies. Accessible at: http://universaldependencies.github.io/docs/, January. ↗
2. Durrett, G. and DeNero, J. (2013). Supervised learning of complete morphological paradigms. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1185–1195. Association for Computational Linguistics, A… find our card
3. Liebeck, M. and Conrad, S. (2015). IWNLP: Inverse Wiktionary for natural language processing. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP), pages 414–418, Beijing, Jul… find
4. Sylak-Glassman, J., Kirov, C., Post, M., Que, R., and Yarowsky, D. (2015a). A universal feature schema for rich morphological annotation and fine-grained cross-lingual part-of-speech tagging. In Cerstin Mahlow et al., editors, Proceedings of the 4th Workshop on Systems and Frameworks for Computation… find
5. Sylak-Glassman, J., Kirov, C., Yarowsky, D., and Que, R. (2015b). A language-independent feature schema for inflectional morphology. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (A… find
6. Zeman, D. (2008). Reusable tagset conversion using tagset drivers. In Proceedings of LREC 2008, pages 213–218. find
7. Lang English Italian Finnish French Serbo-Croatian Japanese Spanish German Portuguese Dutch Esperanto Latin Russian Swedish Hungarian Greek Polish Georgian Irish Catalan Latvian Armenian Icelandic Romanian Norwegian Bokmål Korean Scottish Gaelic Norwegian Nynorsk Ido Old French Old Armenian Danish… find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/lex-2016-kirov-wiktionary-paradigms/