📘 Measured: what the works did in Mova

Measured in the development notes of en: TnT alone scores 93,12% (EWT), 93,12% (PUD), 92,84% (GUM) tagging accuracy; the perceptron tagger that uses TnT priors for unknown words reaches 94,35 / 94,04 / 93,82%. Downstream, greedy parsing with TnT tags gives EWT UAS/LAS 82,3/77,3 versus 83,4/79,0 with the perceptron tags. TnT-based generation is about 4 times faster (~10 500 vs ~2 600 sentences/s) at slightly lower quality (54,6% vs 56,6% exact, chrF 92,4 vs 92,8).

Early update is the default training scheme of the solver and is not ablated on its own. The alternative max-violation update (Huang, Fayong, Guo 2012; MATH_MAXVIOL=1) was measured against it: SVAMP 57.3% → 56.0%, i.e. worse, so early update was kept.

Used only as a benchmark reference, so there is nothing to measure for the paper itself. The comparison as recorded in the development notes of en (EWT test, gold tokens): STEPS (XLM-R) 91.91 LAS; UDPipe 2 (EWT 2.17 model, Mova's own measurement) 92.37 LAS; Mova small model plus large-model edits 91.90 LAS on 300 random sentences, where UDPipe scores 92.11; Mova's small model alone 80.24 LAS, and 81.15 LAS when also trained on GUM converted to Mova's annotation dialect.

Measured (beam with early update vs greedy, same features): EWT test LAS 79.00 → 79.77, dev 77.33 → 79.17, PUD 74.48 → 76.33. In the en crate summary with Mova's own tags: EWT UAS/LAS 84.2/79.8 with the beam vs 83.4/79.0 greedy.

This is an evaluation metric, not a modelling idea, so it has no gain of its own. It is the measuring instrument for all coreference results, e.g. the sieve resolver on GUM test with gold trees: CoNLL F1 73.04 with singletons (MUC / B³ / CEAF-e = 69.18 / 75.26 / 74.68), 60.31 without singletons (B³ 58.79), per the development notes of coref.

Measured on GUM dev with gold trees (the development notes of coref), Hobbs order versus pure recency: pronoun precision 64.2% vs 55.4%; CoNLL F1 73.43 / 63.34 vs 72.23 / 61.33 (with / without singletons).

The cue list is not ablated separately, so the contribution of this source is not measured on its own. The hedge field as a whole (from the prag README), five-fold cross-validation on 2,986 sentences, accuracy / macro-F1 %: majority baseline 97.3 / 49.3, perceptron 98.1 / 76.3, IGTree 98.0 / 75.5, k-NN 97.8 / 73.4. On the held-out test (538 sentences): baseline 96.8 / 49.2, IGTree best at 98.5 / 84.2. Shuffled-label controls stay at baseline level.

Measured on GUM test (UD 2.18, head match), from the development notes of coref. CoNLL F1 of the default sieve (12 sieves): 73.04 with singletons / 60.31 without on gold trees, 67.90 / 54.03 on Mova's own trees. Baselines on gold trees: exact string match only 53.07 / 27.88; pronoun → nearest agreeing mention 57.73 / 32.67. Cumulative sieve contributions (CoNLL F1 without singletons, gold trees): speaker +21.88, exact +20.94, strict_a +5.18, pronoun +6.38. Dev decisions: Hobbs order vs pure recency gave pronoun precision 64.2% vs 55.4% (CoNLL 73.43 / 63.34 vs 72.23 / 61.33); a pronoun window of 1–4 sentences made no difference (±0.1), so 3 was kept as in Lee et al.; dropping indefinites from exact match was worse (−0.4 / −0.9), so the paper's rule was kept. For reference, the CRAC 2025 mBERT baseline on en_gum is 61.7 (settings differ, comparison is only indicative).

It is an evaluation metric, not a modelling idea, so it has no "gain" of its own. It is part of every coref measurement, e.g. the default system's CEAF-e F1 on GUM test is 74.68 with singletons and 52.89 without on gold trees (the development notes of coref).

Not an idea that changes scores, so not measured as a gain. Its role is correctness of the measuring instrument: the scorer reproduces the paper's worked example, and every reported coreference result (for example CoNLL F1 73.04 with singletons and 60.31 without on GUM test, gold trees) depends on it.

Measured on GUM test (UD 2.18), head match, CoNLL F1 with / without singletons, gold trees: the default sieve (12 passes) scores 73.04 / 60.31, versus exact string match only 53.07 / 27.88 and "pronoun → nearest agreeing mention" 57.73 / 32.67; on Mova's own predicted trees 67.90 / 54.03. On dev, Hobbs-style candidate ordering beats pure recency: pronoun precision 64.2% vs 55.4%, CoNLL 73.43 / 63.34 vs 72.23 / 61.33. Two relaxed sieves were disabled because their precision on dev was too low (strict_c 31%, relaxed_head 24%).

It is an evaluation metric, not a modelling idea, so there is no gain to measure. It is a component of the reported results, e.g. on GUM test (UD 2.18, gold trees, with singletons) the default sieve system scores CoNLL F1 73.04 with MUC F1 69.18, B³ 75.26, CEAF-e 74.68.

Measured (architecture status) on FairytaleQA feeling questions, comparing the story reader before and after adding the ATOMIC xReact table: test 0.130 → 0.158, validation 0.105 → 0.118 (inferred answers 28 → 71). Overall reader score 0.212 → 0.215, identical on a repeat run (deterministic), with no change on fables. Adding oReact: validation 0.118 → 0.127, test unchanged at 0.158. The ObjectUse/CapableOf tables pass their negative-control test but have no downstream measurement yet.

Measured in the commit that introduced the module: SVAMP 49.7% with the reranking judge (45.7% without), and GSM8K 7.0%, both with canonical step order and with the dynamic oracle. The model did not learn long problems: the space of pool pairs is too large for the beam. The tree-based solver therefore stayed the working default, and the DAG solver is kept as an experiment. For reference, project notes list the neural DeductReasoner at 47.3 on SVAMP.

The arith model has been measured as an extra signal for the main SVAMP solver's judge (its answer as a feature, computed fairly across folds): SVAMP 56.3% versus 57.3% without it, i.e. no gain. An earlier record of 58.3% turned out to be a lucky run caused by non-deterministic map ordering in arith (results varied between 55.3% and 58.3%); after making it deterministic the result is a stable 56.3%. The standalone accuracy of the arith model is not reported separately.

• Non-lexical features (verb classes, rates, binding to the question; together with Roy & Roth 2018): SVAMP 44.7% → 48.0% (improvement).
• Hard unit constraint (MATH_UNITHARD=1): SVAMP 57.3% → 46.7% — harmful. Cases like "more cups of flour than sugar" subtract different things measured in the same unit, and the unit extraction is too coarse for a hard ban (the paper uses a trained classifier of units and rates). The idea works only as soft features.

Measured (from the architecture status table): on the contrast variants the bAbI reader first scored 74.6%. This exposed parser defects ("attic" tagged ADJ, "Taras" lemmatised as "tara", "Following" and "is" read as verbs) and two vocabulary gaps (hurry, abandon). After these were fixed, the contrast set reached 99.7% with all 20 tasks at 95% or higher, and the original test did not drop (99.8%). The method found real bugs that the in-distribution test (already 99.8%) could not show.

Measured on SpartQA-Human (test): 59.3% overall (majority baseline 34.9%; first version 50.5%), by type YN 60.1%, FB 67.1%, CO 58.2%, FR 50.6%; training split 61.3%. Fixes were made only from training-split errors. Mova's test split is the 2022 version (YN 143, FR 77 questions), so the overall figure is not comparable with the 2021 BERT transfer result on the 510-question test. Comparable figures: YN 60.1% with DK in the gold labels, or 68.5% two-class (DK mapped to No), versus BERT+Q-Chain 59.44, Llama-3-8B CoT 67.83, PistaQ 75.52 and GPT-4 77.62; FR 50.6% (exact set match) versus BERT with SpaRTUN 50.64. This is achieved with no neural network and no training on this data. The ceiling is limited: questions were written from images, so some answers are not derivable from (or contradict) the text.

From the "Ideas from papers through the bench" table in Mova's development notes: 96.3% on the full test set (k=1 98.2% → k=10 94.7%), deterministic. The suspicion detector found 2 phrasings that are contradictory in the data (one sentence carries 2–3 different labels, a generator defect); without them accuracy is 98.0–99.6% on all k, so long chains barely degrade because composition is exact. The headline number remains 96.3% on the full test; 11,553 questions excluded from the "clean" figure are listed with reasons. For comparison, the paper reports far lower neural results (e.g. TP-MANN averaging about 53% for k=1–5), though setups differ (Mova trains its reader only on level 1 and composes symbolically).

From the "Ideas from papers through the bench" table in Mova's development notes: test 99.8%, 20/20 tasks ≥ 95%; training split 99.7% (not tuned to). Motivations were learned from the training split (375/375). Majority voting in induction was worse than using the freshest example (97.6% vs 99.5%). On contrast variants accuracy first dropped to 74.6%, exposing parser defects (attic tagged ADJ, Taras lemmatised as "tara", Following as a verb, is as a main verb) and two lexicon gaps (hurry, abandon); after fixing them contrast accuracy reached 99.7%, 20/20, with no drop on the original (99.8%).

Measured in the development notes of prag (accuracy / macro-F1, %). Held-out silver test (538 sentences), k-NN vs majority baseline vs averaged perceptron:

• form 91,6 / 63,5 (baseline 77,3 / 12,5; perceptron 94,1 / 70,9)
• act 67,5 / 29,4 (baseline 33,3 / 3,6; perceptron 68,8 / 30,0)
• indirect 90,1 / 52,5; hedge 98,0 / 77,5; polarity 53,5 / 37,5; voice 98,9 / 98,9

5-fold cross-validation on all silver data (2 986 sentences): form 92,5 / 68,4, act 68,1 / 26,0, indirect 92,5 / 53,9, hedge 97,8 / 73,4, polarity 54,3 / 38,9, voice 98,2 / 98,2. The negative control (same model on shuffled labels) stays at baseline level (e.g. form 77,0 / 12,4). k-NN is close to, but generally below, the perceptron; its value in Mova is example-based explanations.

Measured: SVAMP 57.3% → 52.7% (identical in two runs), i.e. a loss, so the idea was not adopted. Diagnosis: the artificial distractor always sits in the same position, so the model learns the position rather than the principle; in the paper, distractors are tailored to the problem type (similar object, same topic).

Measured on SVAMP: 57.3% → 55.7% with the staged curriculum, i.e. no gain (recorded as "zero"). A naive variant that simply sorts all training problems by difficulty dropped to 50.0%, because it breaks perceptron averaging. The flag is therefore off by default.

Measured on SVAMP (architecture notes, table of ideas from papers): reranking with a judge over top-K raised accuracy from 47,7% to 54,3% — one of the confirmed gains.

Measured on UD English EWT test (recorded in the module header of en/src/ptag.rs): the TnT HMM tagger gives 93.12%; the greedy averaged perceptron 93.74%; plus the TnT prior for unknown words 94.13%; plus neighbouring-word shape features and beam 94.35%. The dependency parser on these tags improved from LAS 77.27 to 79.00. With 5% corrupted training tags, the robust variant raised accuracy from 94.00 to 94.10 and found the corrupted labels with 95.8% precision and 63% recall.

Note: the code cites "Collins 2002"; this refers to the EMNLP 2002 perceptron paper above, not Collins & Duffy 2002 (kernel-based reranking, ACL 2002).

Measured in the development notes of prag (accuracy / macro-F1, %). Held-out silver test (538 sentences), IGTree vs majority baseline vs averaged perceptron:

• form 88,7 / 55,8 (baseline 77,3 / 12,5; perceptron 94,1 / 70,9)
• act 65,6 / 25,1 (baseline 33,3 / 3,6; perceptron 68,8 / 30,0)
• hedge 98,5 / 84,2 — best of the three models on this field (perceptron 97,4 / 74,3)
• indirect 88,7 / 54,0; polarity 48,9 / 31,1; voice 98,5 / 98,5

5-fold cross-validation (2 986 sentences): form 89,6 / 64,5, act 64,1 / 23,9, indirect 92,2 / 58,2, hedge 98,0 / 75,5, polarity 49,0 / 31,8, voice 97,6 / 97,6. Shuffled-label negative control stays at baseline level. Overall IGTree trails the perceptron except on hedging in the held-out split.

Measured on SVAMP (architecture notes, table of ideas from papers): max-violation lowered accuracy from 57,3% to 56,0%, so it stays off by default and early update remains the training regime.

This paper supplies a metric rather than a model idea, so its value is in the defects it exposed. From the mlab README (minimum LRE, before → after fixes): backslash on Pontius 4.9 → 13.5, Longley 9.1 → 11.4, Wampler2 10.3 → 12.5; on Filip (degree-10 polynomial) the solver went from silently returning 0 correct digits to a "rank deficient" warning; anova1 on SmLs07–09 improved from 2.7 / 2.0 / 0.0 to 4.0 / 3.9 / 3.9, and on AtmWtAg/SmLs04–06 from 6.5–8.5 to 9.9–10.2.

Measured and harmful: SVAMP 57.3% → 52.7% (same result in two runs). The artificial distractor always sits in the same place, so the model learns the position rather than the principle; in the papers distractors are tailored to the problem (similar object, same topic). The idea is not used by default.

Measured, no gain: +440 solutions added over 2 rounds, GSM8K 10.6% → 10.4%. The cause is the known STaR weakness — the answer matches but is computed from the wrong numbers. Not used by default.