Mova0.6 pre-release

Okinawan: the flag of Okinawa Prefecture, a white ring in a red disc on white 32. Central Okinawan in Mova

Mova uses ryu for Central Okinawan. It has neither an annotated Central Okinawan treebank nor a parser model for the language. The grammar manual supplies descriptions that Squid can turn into claims. Language identification, grammatical annotation and implementation each have their own status.

The Central Okinawan package

What every language gets asks every language package to establish the same parser baseline. For Central Okinawan, the package code identifies the language for which that baseline would be measured. A package code, a grammar manual, an annotated corpus and a trained model serve different purposes.

ComponentRoleCentral Okinawan status
Language codeIdentifies the language covered by the package and manualryu
Grammar manualDescribes the language and supplies grammatical points to SquidAvailable as a source of claims
Annotated UD treebankSupplies training sentences and reference annotations for evaluationMova has none for ryu
Level-two modelSupplies the trained parser for spans assigned to the languageNo model exists for ryu
Parser baselineMeasures attachment accuracy against a development splitUnmeasured for ryu

The package code is used in the convention for locating a language's model. Recognizing the language of an input span is a separate operation with its own coverage. This distinction matters when reading a multilingual result. A span must first be assigned to a language before the parser can look for that language's model.

Script coverage and the language segmentation stage

At level one, tools/langseg assigns a language or fallback category to each sentence and span. A span is a portion of the input assigned a label for the next stage. This stage decides which language model to look for. It supplies no Central Okinawan word boundaries or grammatical annotations.

Input covered by level oneDistinctions available
Latin-script English and FrenchEnglish and French, distinguished using a Latin table, a lexicon for each language and a character 4-gram model
Hebrew scriptVarieties written in Hebrew script
Scholarly transliteration of Hebrew or AramaicHebrew or Aramaic
Scripts outside that coverageother

Central Okinawan kana and kanji fall outside the supported script coverage. Central Okinawan romanization uses Latin characters, but the existing Latin-script language distinction is between English and French. The same Central Okinawan expression can therefore be presented in two scripts without either spelling establishing a ryu identification route.

Form うちなー · Romanization Uchinaa · Gloss Okinawa · Translation “Okinawa” (constructed).

To add a Latin-script language to this stage, Mova's mechanism calls for both a lexicon and a Lang code. langseg latin-lexicon builds the lexicon from an open UD training split. The resulting lexicon belongs to language identification. The model that supplies grammatical analyses at level two is a separate file.

Level-two model availability and Parse=None output

tools/mova-parse looks up a model for each span using the language assigned at level one. The lookup follows <code>.bin, with the span's language code in place of <code>. The lookup is per span, so model availability is evaluated separately for spans assigned to different languages in the same input.

When a span's language has no model file, the # parse comment reports that language. The output uses the following values for the affected token rows.

FieldOutputAnalysis left unavailable
LEMMA_Lemma
UPOS_Universal part of speech
XPOS_Language-specific part of speech
FEATS_Morphological features
HEAD_Syntactic head
DEPREL_Dependency relation
MISCParse=NoneRecords that the parse is unavailable

This is the missing-model output for spans treated as ryu. Its interpretation depends on the language label. Where the writing receives other, that label reflects the coverage of level one. Where ryu is the span's language, Parse=None reflects the lack of a Central Okinawan model at level two.

Read the six underscores together with Parse=None. They mark missing annotations across the row. In particular, an underscore in FEATS gives no evidence about whether the word carries any morphological features, and the paired underscores in HEAD and DEPREL leave its attachment unspecified. A token row in this output does not establish that Mova has applied Central Okinawan tokenization.

Treebank requirements and baseline measurements

The baseline requested by What every language gets is a train-to-development evaluation on Universal Dependencies (UD) data. The training split supplies sentences with reference word boundaries, tags and dependency structures. The development split supplies annotated sentences against which to check the parser's output. For each evaluated word, the reference tree gives a head and a dependency relation. The interpretation of both attachment scores depends on consistent annotations within and across the two splits.

Mova's tree parser and tagger are in the en crate. en xpos-fill precedes training on the UD training split. en train trains the parser and tagger, and en parse-eval measures attachment on the development split.

MeasureWhat counts as a correct attachmentSplit used for scoring
UAS, unlabeled attachment scoreThe predicted head matches the reference headDevelopment
LAS, labeled attachment scoreBoth the predicted head and the dependency relation match the referenceDevelopment

The two measures distinguish errors in the choice of head from errors in labeling an attachment. A word attached to the right head with the wrong relation counts as correct for UAS and incorrect for LAS.

Mova has models for English (en.bin) and French (fr.bin). The Central Okinawan entries for UAS and LAS are unmeasured. Mova has no ryu treebank to supply the training and development splits needed for the baseline. Parse=None is a status in parser output and supplies no attachment score.

Squid claims and implementation status

Squid reads the grammar manual and turns each grammatical point into a claim. Formalizable rules, closed lists and statistics enter its queue for a place in Mova's code. A rule can describe the conditions under which a grammatical pattern holds. A closed list can give all the members of a paradigm. A statistic can record a distribution. In each case the queued claim records material that may inform an implementation.

A claim's presence in the queue leaves its implementation pending. The manual can therefore describe a form and its function in detail while the parser still has no Central Okinawan annotations to return for it. None of the Central Okinawan rules described in the manual is implemented in Mova yet.

The first claims relevant to this gap are those needed for a tokenizer, a tagger and a UD treebank baseline. For example, a description of how a particle combines with a host can inform a decision about word boundaries. A description of the particle's grammatical distribution can inform which tag to assign after those boundaries are determined. A treebank makes those decisions explicit in annotated sentences and also supplies the head and relation judgments needed to train and evaluate dependency parsing. Each step calls for a representation that the relevant tool can use.

How Mova handles Central Okinawan across the pipeline

tools/langseg assigns a language or fallback category to sentences and spans at level one. Its distinctions cover English and French in Latin script, varieties in Hebrew script, and Hebrew or Aramaic in scholarly transliteration. Other scripts are other. A further Latin-script language can be added with a lexicon from an open UD training split using langseg latin-lexicon, together with a Lang code. The available script and lexicon coverage limits which Central Okinawan spans Mova can identify.

At level two, tools/mova-parse looks for the model belonging to each span's language. There is no model for ryu. The parser reports the missing model in the # parse comment, leaves LEMMA, UPOS, XPOS, FEATS, HEAD and DEPREL as _ on the affected rows, and writes Parse=None in MISC. Mova does not tokenize, tag, lemmatize or parse Central Okinawan.

The baseline in What every language gets calls for en xpos-fill and en train on a UD training split, followed by UAS and LAS measurement on the development split with en parse-eval. Mova has no annotated Central Okinawan corpus and has measured no treebank baseline for the language. The frames, pragmatics and world components, prag and world, read English trees. They have no Central Okinawan parsed input.

Squid turns the manual's grammatical points into claims and queues the formalizable rules, closed lists and statistics. The Central Okinawan rules remain unimplemented. The first points needed are those for tokenization, tagging and a UD treebank baseline.

Sources for this chapter

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/languages/ja/ryu/32-mova-tools/