A Survey of Corpora for Germanic Low-Resource Languages and Dialects

What it is about

The paper surveys corpora for Germanic languages and dialects that are low-resource and often non-standard: Scots, Frisian, Low German, Dutch dialects, Faroese, Yiddish and others. The authors searched repositories and ACL and arXiv papers, and kept corpora that are accessible, easy to download and of good quality. Corpora are sorted by annotation type, curation profile, size and data format. The main finding is that manual annotation is sparse: apart from geolocation, most annotated resources cover only morphosyntax (POS tags, lemmas, dependencies). The survey lists over 80 corpora with a companion website, and the authors give recommendations for releasing such data. The English entries are few but real: a POS-tagged Scots corpus, the Tyneside corpus (DECTE), the Helsinki Corpus of British English Dialects, and the SCOTS subset.

Ideas

I1. 08. National varieties: add a table of English dialect corpora from the survey (Scots, Tyneside, British dialects) with their annotation type, so Mova's variety rules have a list of known test sets. Only list them; do not carry any text (licence check first).
I2. Use the Scots POS corpus as a measure for en/src/tag.rs: run the English tagger on it, report accuracy by tag and by the words that are not in en/src/dict.rs, and keep the result as a note on what the dictionary misses.
I3. The survey's own rule (use only corpora that download easily and are clean) is a good gate for Mova's data intake; copy it into docs/ as a written intake check, with each source's licence recorded beside it.

For Mova

The chapters are 08. National varieties (regional and dialect English) and 37. Register and variation (variation). The code is en/src/tag.rs and en/src/dict.rs. The measure: tagger accuracy on the Scots sample against the English sample, by tag. Nothing here is a training set for Mova's build: the survey is a source list for tests and for planning which dialect gaps to measure first.

Topics: Part-of-speech tagging

Cite

The paper itself, not this card. BibTeX from the library's record of the work.

@inproceedings{blaschke2023survey,
  title = {{A Survey of Corpora for Germanic Low-Resource Languages and Dialects}},
  author = {Blaschke, Verena and Schuetze, Hinrich and Plank, Barbara},
  booktitle = {Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)},
  year = {2023},
  url = {https://aclanthology.org/2023.nodalida-1.41/}
}

References

1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Julien Abadji, Pedro Ortiz Suarez, Laurent Romary, and Benoît Sagot. 2022. Towards a cleaner document-oriented multilingual crawled corpus. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4344–4355, Marseille, France. European Language Resources Association. find

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/papers/blaschke-2023-survey-corpora-germanic-low-resource/