Mova0.6 pre-release⌕
☰
home / docs / tools / mova-parse

mova-parse

A two-level multilingual parser. Level 1 decides the language of every sentence and of every span inside it: it is langseg with its Latin table (English and French by dictionary, a character model for unknown words, a switch cost for spans; tools/langseg), plus the dialect modifier. Level 2 is purely language-specific: each span is parsed by its own language's model, a file of en::annotate::Annotator (en.bin, fr.bin). The models stay separate; nothing in level 2 knows about other languages.

How it works

  1. Sentences. Each input line is a paragraph, cut after . ? ! … followed by whitespace and a capital, a digit or an opening quote, except after a one-letter or listed abbreviation (Dr., M., etc.).
  2. Level 1. langseg::Segmenter::segment gives word spans with languages; Hebrew transliteration goes with English. The sentence's language is the one with most letters (a sentence without letters takes the language of the sentence before). The spans become pieces that cover the sentence: the text between two pieces is cut at its last whitespace, an opening quote going with the piece it opens.
  3. Level 2. The pieces in the sentence's language are tokenized by its model and parsed together as one sentence, the switched pieces taken out, so the matrix clause stays whole. A switched piece of --min-span words or more (default 2) is tokenized and parsed by its own language's model; its root is attached to the matrix root with dep and CSAttach=root in MISC (a placeholder, not a parse of the attachment). A shorter switched piece (one word: a loan, a name) stays in the matrix parse and is marked Lang=xx|CSParse=matrix.
  4. No model. A language without a model file is reported on stderr and in # parse; its tokens get _ in LEMMA, UPOS, XPOS, FEATS, HEAD and DEPREL and Parse=None in MISC. French is never parsed by the English model, except a one-word span under --min-span, which says so.

Output: one CoNLL-U document. Every sentence has

# sent_id = doc-1
# text = He said « je ne sais pas » and left.
# lang = en
# dialect = unknown                    (or a label, with # dialect_confidence and # dialect_markers)
# style = …                            (only when a style detector answers)
# parse = en=en.bin; fr=fr.bin         (or fr=none (no model for fr))
# spans = en 0-8, fr 8-29, en 29-38    (byte offsets into # text, when there is more than one piece)
...
6	sais	…	2	dep	_	Lang=fr|CSAttach=root

A label is (language, dialect, style). The dialect is langseg::variety (US/GB/CA for English, FR/CA/BE/CH for French, by marker words, unknown without enough evidence). The style is filled by an implementation of langseg::variety::StyleTagger set as Dispatcher::style; none is built yet (Mova's register lexicons global/data/register-*.tsv are the first candidates).

Run

cargo build --release -p mova-parse
mova-parse [--model xx=PATH]... [--min-span N] [--doc ID] [FILE...] > out.conllu

Models default to <xx>.bin for every language of the Latin table. Library use:

let mut d = mova_parse::Dispatcher::new(2);
d.load(langseg::Lang::En, &mova_parse::default_model(langseg::Lang::En))?;
let sents = d.parse(text);                       // Vec<OutSent>: lang, dialect, style, spans, tokens
let doc = mova_parse::conllu("doc", &sents);

Readers that take raw text in more than one language (the Elephant's Wikipedia graph, the tree store's writers) call Dispatcher::parse and read OutSent::lang and the Lang= MISC values, or run mova-parse and read # lang; a sentence whose # parse says none has no tree.

A new language in level 2

  1. A Lang code in langseg (Lang::code, Lang::parse).
  2. A lexicon for the Latin table (langseg latin-lexicon <out.tsv> <train.conllu>... from open treebanks, licences checked; dev and test splits stay out) or, for another script, its own level-1 classifier; then a LexLang in latin::default_table, and a re-measured switch cost on a dev half.
  3. A model file <code>.bin: en train on the language's UD train after en xpos-fill (What every language gets item 6). Nothing else changes in mova-parse.
  4. Dialect markers in langseg::variety (optional), from the language manual's chapter on national varieties.

Measurements

Level 1: see langseg (2026-11-06: 97.77% of 3 455 UD dev sentences, 96.93% of the tokens of 3 158 code-switched sentences, switch-point F1 86.01%; shuffled lexicons 55.02% against a majority class of 57.28%). Level 2 adds no decision of its own: a sentence reaches the model of the language level 1 chose, so its parse quality is the model's (the baselines in What every language gets). The output of two runs on the same text is identical (2026-11-06, 60 Global Voices lines).

Limits

The Elamur team

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/tools/mova-parse/