📘 Open parsers: witnesses and tests

This page covers the open parsers Mova uses as witnesses and test references: for English, French, Ukrainian, Latin, Ancient Greek, Biblical Hebrew and Modern Hebrew. It says which licences allow what, how the parsers are used, and which parser serves which role. The papers behind the parsers are in the library.

What it is

Open parsers and their models are supplied separately; Mova does not ship them. They serve three purposes:

• Witnesses for the teacher's trees: where independent parsers agree on a head and the teacher disagrees, the word is a suspect to check first.
• Test references for Mova's own en parser and for the parsers of the language packages.
• Candidates for transparent rule-based components (Link Grammar, ACE + ERG, Grammatical Framework).

A witness runner is configured through CYCLE_NEURAL (tools/cycle). Its inputs and outputs must keep token alignment, and each model's licence is checked apart from its code's licence.

Licences: build vs test

By the data-licence rule (Data and model licences), share-alike, non-commercial and research-only models are for tests and measurements only, never built into Mova or shipped. A permissive code licence does not settle it: a model trained on restricted treebanks needs a data-licence check before any build use.

Code only, no pretrained English model: Enju (HPSG), TurboParser, Hexatagger, DiaParser. CLTK has no parser of its own (it wraps Stanza or a large model) and is not set up.

How they are used

• School witnesses. tools/cycle runs the neural parsers over school sentences (runner path in CYCLE_NEURAL), pretokenized so trees align token by token; tools/parsecmp compares them and tools/treeagree merges their agreement with the teacher's trees (Training Mova). Stanza and Trankit (UD, GPU) are the witnesses; a word where both agree on another head than the teacher is a suspect, and spaCy trf agreeing as well puts it at the top of the list. Agreement with a participating witness is not independent accuracy evidence; linguistic quality is measured on an admissible held-out gold set.
• Convention, not error. spaCy (ClearNLP) and SuPar (Stanford dependencies) make prepositions and copulas heads of their phrases. Low agreement with UD trees from them is a head-rule difference; use them as witnesses only after conversion. Constituency parsers (Stanza, benepar, SuPar CRF) serve bracket comparisons.
• Language packages. For each language, dev sets are parsed with gold syntactic words (multiword tokens already split, tokenizer off) and scored on all words, punctuation included: LAS on the full label and LAS-b on the label before the colon, each on the parser's own tags. A model trained on the same treebank is not a fair witness; a model from another treebank is fair but measures convention differences as much as quality.
• Mixed-language text. Mova's own parsers are one model per language; tools/mova-parse cuts a text into language spans with langseg (English and French by dictionary) and sends each span to its language's model, writing # lang per sentence and Lang= on switched tokens. The neural witnesses above stay single-language.

Which parser serves which role:

Not measured here: raw-text parsing with multiword-token recovery.

French witnesses (measured 2026-11-06)

Dev sets of UD 2.18 French GSD (1476 sentences, 35721 words) and Sequoia (412 sentences, 9999 words), gold tokens (multiword tokens split into syntactic words, tokenizer off), LAS and UAS over all words, punctuation included, on each parser's own output. Speed is sentences per second on the RTX 4000 SFF Ada (GSD dev, model already loaded, one warm-up batch excluded; batch 500 sentences for Stanza and Trankit, 64 for hopsparser, 1 for COMBO). GPU is the peak of the process as seen by nvidia-smi (it includes the allocator cache, so it overstates the need). spaCy lg runs on the CPU. Runner: tools/treeagree/bridge/fr_neural.py; raw results results.jsonl in the run directory fr-parsers-2026-11-06. One run per parser, not repeated; the numbers are deterministic up to GPU arithmetic.

Bold marks a model trained on the treebank of that column (in-treebank style; a development set may have been used for model selection, so these are slightly optimistic); the rest are fair witnesses that carry the convention gap of another treebank. Sorted by the sum of the two LAS.

Reading the table:

• Treebank version drift. The hopsparser models were trained on UD 2.9 (model cards: GSD LAS 94.5 for camembertav2-base-gsd). On GSD dev 2.18 they reproduce that (94.5), but the Sequoia models drop to 87 on Sequoia dev 2.18 (UAS 90.9) although the runner reproduces each card's own parsed dev file exactly (99.99% agreement): the Sequoia annotation changed between the versions, so the Sequoia-trained models are measured against a moved target and Stanza sequoia (retrained on a later version) is the better Sequoia-style witness. Compare these columns only for the version stated.
• Spoken and small treebanks (Rhapsodie, ParTUT, ParisStories, Trankit spoken) are useful only as register witnesses; they score about 75 LAS on written text.
• Agreement of the two best GSD-style GPU models (hopsparser camembertav2-base-gsd and COMBO, both trained on GSD, so not independent): mutual UAS 97.3 on GSD dev; both right on 95.4% of words; at least one right on 97.8%. On Sequoia dev, hopsparser camembertav2-base-sequoia with Stanza sequoia agree on 88.9% (UAS).
• Not measured: raw-text parsing, the 2.19 restoration of GSD features, Rhapsodie/ParisStories dev as test sets, UDPipe 2 (needs its server; NC).

Models, revisions and checksums of the hopsparser downloads are in hops/manifest.tsv in the parsers directory of the data tree (outside Git; its own environment with hopsparser 0.8.0; COMBO has another). The COMBO batch mode fails on pretokenized input with combo-nlp 4.0.14 (a lemma-head size mismatch), so the runner parses one sentence at a time.

Recommended French witness set for the Eel's agreement step. GPU pair: hopsparser camembertav2-base-gsd (fast, 94.5 LAS on GSD) and COMBO (94.7); a third GSD-style vote from Stanza gsd if three are wanted. Fair witness: Stanza sequoia (other treebank, 87.0 LAS / 92.6 UAS on GSD dev, and the best Sequoia-style model at 91.6); Trankit french is the fast cheap alternative (488 sent/s, 2.6 GB). Agreement with a participating witness is not independent accuracy evidence (see above); the table says which witnesses carry the GSD convention and which carry another one.

Sources: hopsparser (grobol-2021), CamemBERT 2.0 (antoun-2024), Stanza (qi-2020), Trankit (nguyen-2021), spaCy (honnibal-2020), UDPipe 2 (straka-2018), COMBO (klimaszewski-2021), Universal Dependencies v2 (nivre-2020); GSD (guillaume-2019), Sequoia (candito-2012), Rhapsodie (lacheret-2014).

Ukrainian witnesses (measured 2026-11-06)

Treebanks: UD 2.18 Ukrainian IU (dev 673 sentences, 12606 words; CC BY-NC-SA 4.0 by its README and LICENSE.txt: tests and counts only; the README offers a commercial licence on request) and ParlaMint (dev 739 sentences, 10842 words; CC BY-SA 4.0: tests, as little as possible). Both are the r2.18 tags of UniversalDependencies/UD_Ukrainian-IU and UD_Ukrainian-ParlaMint, the only Ukrainian treebanks in that release, kept outside Git in the research raw-data tree (raw/uk-ud/ud-treebanks-v2.18/). Measured as in the French section: gold tokens (Ukrainian has no multiword tokens), tokenizer off, LAS and UAS over all words, punctuation included, each parser on its own output. Speed is sentences per second on the RTX 4000 SFF Ada (IU dev, model already loaded, one warm-up batch excluded; batch 500 for Stanza and Trankit, 64 for spaCy trf, 1 for COMBO and the goeswith models). GPU is the process peak seen by nvidia-smi (it includes the allocator cache, so it overstates the need). spaCy lg runs on the CPU. Runner: tools/treeagree/bridge/uk_neural.py; raw results results.jsonl in the run directory uk-parsers-2026-11-06. One run per parser, not repeated; the numbers are deterministic up to GPU arithmetic.

Bold marks a model trained on the treebank of that column (in-treebank style; a development set may have been used for model selection, so these are slightly optimistic); the rest are fair witnesses that carry the convention gap of another treebank. Sorted by the sum of the two LAS.

Reading the table:

• Two conventions. The IU-trained models (Trankit, COMBO IU, Stanza iu_*) lose 7 to 9 LAS on ParlaMint at nearly the same UAS (88.7 to 92.8), and the ParlaMint-trained ones lose 7 to 13 LAS on IU: the head structure transfers, the labels do not. We have not analysed which relations differ. Compare LAS columns only with the training treebank in mind; UAS is the fairer cross-treebank number.
• spaCy trf was not trained on a gold treebank but on the silver Ukr-Synth set (MIT licence; encoder ukr-models/xlm-roberta-base-uk) and still scores 91.9 LAS on IU, level with the IU-trained Trankit and COMBO, which suggests the silver labels follow the IU convention. We did not trace how Ukr-Synth was labelled, so the model is neither build-cleared nor independent of IU.
• Yasuoka's goeswith models (2025; ModernBERT large and base, RoBERTa large WECHSEL) were trained on all Ukrainian UD treebanks together (the repository's maker.py clones every UD_Ukrainian-*), so they sit between the two conventions. They parse raw text and split words themselves: with space-joined gold tokens, 4 to 22 sentences per run (up to 3%) came back with a different token count, or crashed in the repository's ud.py, and are scored as wrong, which lowers their figures by up to about 2 LAS (the mismatch field of results.jsonl). They are slow (3 to 6 sent/s: one masked forward pass per subword) and run remote code (trust_remote_code). Their cards say CC BY 4.0 or MIT although the training data includes the NC-SA IU: treat them as NC-SA.
• Stanza without the character language model loses 2.3 LAS on IU (85.4 against 87.7) and 1.2 on ParlaMint. Stanza peaks at 6 to 9 GB because of its allocator cache.
• Agreement (UAS between parsers, IU dev, gold tokens): COMBO IU with Trankit 95.7, Trankit with spaCy trf 95.2, COMBO IU with spaCy trf 94.2, Stanza iu_charlm with the others 91.3 to 91.7, modernbert-large with COMBO IU or Trankit 91.6. All five of {COMBO IU, Trankit, Stanza iu_charlm, spaCy trf, modernbert-large} are right on 83.3% of words and at least one is right on 98.2%. ParlaMint dev, with COMBO ParlaMint and Stanza parlamint_charlm in place of the first and third: COMBO ParlaMint with Trankit 93.6, with Stanza parlamint_charlm 92.4; all right 81.7%, at least one right 98.5%. Models that share a training treebank agree because of the shared data and convention, not because they are independent.
• Mova's own parser is the en tree parser trained in memory on each Ukrainian train set after en xpos-fill (nothing is saved; the IU run is a test-only measurement and no model of it is kept). It is about 19 LAS behind the best neural parser on IU with its own tags (14 with gold tags) and 12 behind on ParlaMint (10 with gold tags); it has no Ukrainian lexicon or morphology.
• Not measured: raw-text parsing; UDPipe 2 (uk_all of the 2.15 release is extracted, CC BY-NC-SA 4.0, but its prediction mode reads word embeddings from a separate embedding service on TensorFlow, which is not cheap to set up); Stanza ner languk; lang-uk/flair-uk-pos (POS only, downloaded); Yasuoka's embeds variants and its BERT-large and RoBERTa-base models.

Models, revisions, licences, training data and checksums are in uk/manifest.tsv in the parsers directory of the data tree (outside Git; Stanza and Trankit are under stanza/resources/uk and trankit/cache, COMBO under combo/hf, the Yasuoka models under uk/hf). The Hugging Face search for 2023 to 2026 Ukrainian UD parsers found the COMBO models (October 2026) and Yasuoka's families (2025) used above, and only encoders otherwise (lang-uk/electra-base-ukrainian-*, Goader/modern-liberta-*), which are not parsers.

Recommended Ukrainian witness set for the Eel's agreement step. GPU trio in the IU convention: Trankit ukrainian (92.0 LAS, 306 sent/s, 3 GB), COMBO IU (92.0, 15 sent/s, 3.5 GB) and spaCy uk_core_news_trf (91.9, 134 sent/s, 2 GB; the only one with an MIT licence on the weights); their mutual UAS is 94 to 95, but they share the IU convention. For a second convention add COMBO ParlaMint (91.0 on ParlaMint; UAS 90.9 on IU) or the faster Stanza parlamint_charlm (88.5), and, when a slow vote is acceptable, Yasuoka modernbert-large (another architecture, trained on both treebanks). The fast cheap pair is Trankit plus spaCy trf. All are tests-only: the IU-trained models and Yasuoka's carry the NC-SA data, and the silver data of spaCy is unchecked for build use. Agreement with a witness that shares the training treebank is not independent accuracy evidence.

Sources: Stanza (qi-2020), Trankit (nguyen-2021), spaCy (honnibal-2020), COMBO (klimaszewski-2021), UDPipe 2 (straka-2018), Universal Dependencies v2 (nivre-2020); the Ukrainian treebanks, Ukr-Synth and Yasuoka's models are listed in docs/usage.tsv.

Teacher and dispute interfaces

The annotation teacher is Codex CLI through mova-teacher (training). treeagree also has a dispute mode (dispute-ask, dispute-apply): only the words whose head the parsers dispute are sent, as short multiple-choice questions. The teacher uses the full mode (a little world context, the sentence, and the SLM's tree and pragmatics); dispute mode stays available. Neither interface establishes annotation quality or authorizes a teacher run.

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/repo-docs/parsers/