Word problems: exact core, step solver, world scripts, world reader, SVAMP / GSM8K

Crates: math (exact core, step solver mathsolve, world reader qread) and world (world scripts: format, checker, optional LLM writer). Both belong to the root Cargo workspace; pick one with -p or --manifest-path. math has three binaries (math, mathsolve, vmmtests) and no default-run, so --bin is required:

cargo run --release --manifest-path math/Cargo.toml --bin mathsolve -- <cmd> ...
cargo run --release --manifest-path math/Cargo.toml --bin math      -- <cmd> ...
cargo run --release --manifest-path world/Cargo.toml                -- <cmd> ...

0. Setup (read once)

Gist

Three things to know before running anything beyond calc and controls.

Steps

1. Parser model. mathsolve (steps, qread, qread-text) loads the en UD model shipped in en/models/en.bin (the path is fixed at build time). To use your own model, train it with en train (see retrain-the-parser.md) and replace that file, or use MATH_EN_MODEL with the math binary.
2. Run directory. Every mathsolve command creates its run directory at startup: MATH2_RUN, or math2-2026-09-26. Set it explicitly:

   export MATH2_RUN=runs/math2
   

3. Dataset root. mathsolve steps|qread and world quant-* read datasets from raw (MOVA_DATA defaults to data in the current folder). Call that directory <DATA_ROOT> below. The expected files are listed in section 6. The world scripts come with the corpus: train/get-corpus.sh puts them in corpus/math-scripts/.

Pitfalls

• If the model file is missing, steps and qread fail at load time. This is not a data problem.
• mathsolve steps --set tiny also reads a TinyGSM sample from tinygsm. It is optional; skip it unless you have that data.

1. Exact core

Gist

All arithmetic is exact:

• v1 (math/src/rat.rs, i128) for units and temperatures. An overflow is an explicit Overflow error, never a silent f64.
• v2 (math/src/big.rs) for word problems: arbitrary-size integers and rationals via dashu (Q = RBig, always reduced).

calc.rs evaluates expressions on v2. Every result gets an independent check: the same expression computed with different arithmetic (f64, or a fingerprint modulo 2^61−1 for huge integers). A failed check blocks the answer.

When to use

• Checking a number or formula.
• Debugging why a solver step produced a value.
• Running the gate's negative controls after touching the core.

Steps

# expression -> exact value + independent check
cargo run --release --manifest-path math/Cargo.toml --bin mathsolve -- calc "3^40 + 17" "(16 - 3 - 4) * 2"

# negative controls: each must be RED (caught); exits non-zero if any passes
cargo run --release --manifest-path math/Cargo.toml --bin mathsolve -- controls

# English question -> expression tree -> steps, checks, answer (units, %, stats, probability)
cargo run --release --manifest-path math/Cargo.toml --bin math -- eval "What is 15% of 80?"

# frozen suite: id <TAB> question <TAB> expected
cargo run --release --manifest-path math/Cargo.toml --bin math -- test math/data/suite-v1.tsv

Example

expr:    3^40+17
  ✓ independent path: fingerprint modulo 2^61−1 fingerprint 628450412988459063
answer: 12157665459056928818

controls ends with negative controls red: 7 of 7.

In math test, the "expected" column takes one of:

• a value with an optional unit (12, ≈ 22.2222 °C, 50/9 Δ°C);
• NU (an honest "I don't understand");
• ERR:dim, ERR:temp, ERR:overflow or ERR:div0.

The summary line # TOTAL <n> <hit> <honest NU> <wrong> gives the totals.

Pitfalls

• Round only on output. A non-terminating decimal prints as ≈ … (exactly a/b).
• Temperature is affine: point − point = difference, and point + point is an error with an explanation.
• Don't edit suite-v1.tsv to make it pass. A new suite version goes in a new file.

2. Step solver: mathsolve steps

Gist

A transition system reads the numbers in text order and, at each one, chooses SHIFT, SKIP or REDUCE (op + direction). The features come from the en UD tree (unit noun, verb class, subject, cue words, question words). The model is a structured perceptron with beam search and early update. The exact core computes every subtree.

With --exact, the gold trees come from SVAMP equations (( 290.0 / 2.0 )) or GSM8K calculator steps (<<48/2=24>>). Without it, the gold trees are guessed by weak supervision. Training happens in memory on every run; no model file is written.

When to use

• Training and evaluating the word-problem solver on SVAMP or GSM8K.
• Trying a feature or idea as a flag and measuring it against the baseline.

Steps

The flags come from the "steps" arm of math/src/bin/mathsolve.rs. Defaults are in brackets.

Environment variables:

• MATH_NOLEX=1|2: non-lexical features (used by the benchmark command below).
• MATH_IGNORE=0..4: the "ignore extra numbers" trigger.
• MATH_DUMP=<file>: per-item dump, one id<TAB>ok(0/1)<TAB>value<TAB>tree line per item.
• MATH_SHOW_MISS=N: show N items where the right answer was ranked but not chosen.
• MATH_HUMBLE=<th> [0.7]: confidence threshold for the self-assessment.
• MATH_UNITHARD, MATH_MAXVIOL, MATH_IMPLIED: experiments.

Example

The configuration used as the benchmark baseline:

export MATH2_RUN=runs/math2
MATH_NOLEX=1 cargo run --release --manifest-path math/Cargo.toml --bin mathsolve -- \
  steps --set svamp --exact --epochs 20 --beam 32 --rerank 5 --world corpus/math-scripts/scripts.jsonl

# GSM8K, same settings without --world
MATH_NOLEX=1 cargo run --release --manifest-path math/Cargo.toml --bin mathsolve -- \
  steps --set gsm8k --exact --epochs 20 --beam 32 --rerank 5

Add --qread to the SVAMP command to use the world reader as a judge feature.

What the output reports (in order):

1. <set> (exact gold): training with gold N; no tree …; no exact tree X; tree not in text order Y. This is the training data and the excluded items with their reasons: no exact tree, or a tree not buildable in text order.
2. epoch k: correct on training a/N: training accuracy per epoch.
3. <set> test: SLM alone (v4) correct acc/n (p%): the headline test accuracy.
4. top-K: correct among them …: how often the right answer is among the top-K candidates.
5. Self-assessment:
   • accuracy when only answering above a confidence threshold (threshold);
   • accuracy when the first model and the judge agree (opinions agree);
   • accuracy for the most confident 10/20/30/50% (most confident);
   • trigger counts, e.g. v2: answered — correct / wrong and world: … (world agreement).

Pitfalls

• Determinism first. Run the same command twice and diff the MATH_DUMP files before claiming a gain. An unordered map can produce a "gain" that is only run-to-run noise.
• Report the exclusion counts from line 1 together with accuracy.
• For a small SVAMP gain, compare per-item dumps (e.g. a McNemar test on the items that changed), not just percentages.
• Without --exact, --world, --princ and --qread silently have no training items to attach to.

3. World scripts: format

Gist

A world script turns a word problem into a quantity world:

• the state is entity.thing → exact number;
• every command is anchored to the sentence that justifies it (@N) and labelled with the principle it applies (k=).

The executor (world/src/quant.rs: parse_line, run, check) runs scripts with the exact core. It accepts a script only if it passes every gate.

When to use

• Reading or writing training data for the world writer.
• Checking a hand-written script against a problem.

Steps

The line grammar comes from parse_line. Blank lines and lines starting with # are skipped.

@N set   E.x = EXPR          k=KIND
@N add   E.x = EXPR          k=KIND
@N sub   E.x = EXPR          k=KIND
@N mul   E.x = EXPR          k=KIND
@N div   E.x = EXPR          k=KIND
@N move  A.x -> B.x = EXPR   k=KIND      (from A to B; A must exist)
@N ask   EXPR                            (k= optional)

• N: a 1-based sentence number. Sentences are split at ., ? or ! when followed by a space or the end of text, so $2.50 is not split.
• Cells are entity.thing: lowercased, with non-empty parts around the dot, e.g. janet.egg or tom.$.
• EXPR: numbers, cell references, + - * / and parentheses.
• KIND is one of given total gain loss transfer rate compare part convert combine unit. Any other value is an error.

Gates (each reports an error):

• the sentence @N exists;
• every number in EXPR appears in sentence N or an earlier one. Digits and number words both count, a percentage counts both as 20 and as 0.2, and the constants 0 1 2 3 4 7 10 12 24 30 52 60 100 365 1000 and their reciprocals are always allowed;
• add/sub/mul/div and move apply only to an existing cell (use set first);
• no division by zero;
• there is an ask;
• the answer equals the gold answer.

Corpus file (corpus/math-scripts/, JSONL, one problem per line; written by quant-vmm, read by load_done, quant-stats and steps --world/--princ):

{"id":"svamp-train-chal-777","set":"svamp-train","question":"There are 87 oranges ... How big is each group of bananas?",
 "gold":"145","script":"@1 set philip.orange = 87 k=given\n...\n@3 ask philip.banana_per_group\n",
 "ok":true,"errors":[],"round":1}

• ok is true only if the gates found no errors.
• round is 1 for the first attempt and 2 if the single retry fixed the script.
• steps uses only ok: true records and matches them to training problems by the exact question text. For SVAMP that text is Body + space + Question, with ". " inserted if Body lacks final punctuation.

Example

Script file s.txt for SVAMP train item chal-777:

@1 set philip.orange = 87 k=given
@1 set philip.banana = 290 k=given
@2 set philip.banana_group = 2 k=given
@3 set philip.banana_per_group = philip.banana / philip.banana_group k=unit
@3 ask philip.banana_per_group

cargo run --release --manifest-path world/Cargo.toml -- quant-check svamp-train svamp-train-chal-777 s.txt
# answer Some("145"); errors 0
cargo run --release --manifest-path world/Cargo.toml -- quant-stats corpus/math-scripts
# green counts per set, plus principle and verb histograms (reads <dir>/scripts.jsonl)

A script that uses 300 instead of 290 fails with two errors: number 300 not from the text up to sentence @1 and answer 300 ≠ reference 145.

Pitfalls

• k= must be preceded by a space; it is split off with rsplit_once(" k=").
• Set names for quant-check/quant-vmm are svamp-train, svamp-test, gsm8k-train and gsm8k-test. SVAMP ids are <set>-<ID>; GSM8K ids are <set>-<index>.
• Negative controls live in world/src/quant.rs tests and must stay red: cargo test --release --manifest-path world/Cargo.toml --lib quant.

4. Training the world writer

Gist

The small model learns to write the quantity world itself (math/src/qworld.rs):

• WorldModel labels each number with an action (=, +, -, rate) and a cell;
• AskModel picks a query template (State, Diff, Sum, Ratio, Product).

Labels come from verified world scripts (section 3). Without trained models, a rule-based writer (qworld::write) is used.

When to use

You have verified scripts (shipped in corpus/math-scripts/ or generated yourself) and want the world as a third opinion or as a judge feature.

Steps

1. Optional: generate more scripts with an LLM. quant-vmm sends batches to an LLM through a CLI (claude -p), checks each script, retries red ones once with the error list (the gold answer is never shown), and appends to <dir>/scripts.jsonl. LLM costs are logged in <dir>/calls.jsonl. It resumes by skipping ids already present. This step is paid and optional:

   MOVA_SUBSCRIPTION=1 cargo run --release --manifest-path world/Cargo.toml -- \
     quant-vmm svamp-train runs/quant --limit 20 --batch 10 --par 2 --cap 2
   

   MOVA_SUBSCRIPTION=1 makes the CLI use its own login instead of looking for a locally stored API key. --cap is the spend ceiling in USD [10]. The model and effort come from PRAG_MODEL / PRAG_EFFORT [medium].
2. Train and use the writer inside steps (it is not saved; it is retrained on every run):

   cargo run --release --manifest-path math/Cargo.toml --bin mathsolve -- \
     steps --set svamp --exact --rerank 5 --world corpus/math-scripts/scripts.jsonl --world-feat
   

3. Read the report lines:
   • world: training problems with scripts N, number labels M: training problems matched to scripts, and labelled numbers;
   • query: templates from scripts …: template histogram;
   • at test time, world: correct, agrees with judge and similar lines count how often the world was right and whether it agreed with the judge.

Pitfalls

• Scripts match only if the question text is byte-identical to the problem text built by the loader.
• Few scripts give little signal. Treat more scripts as more training and test data, not as a guaranteed accuracy gain.

5. World reader: qread (world v2 for numbers)

Gist

math/src/qread.rs reads a problem as events in time:

• who: the subject; a pronoun maps to the previous subject;
• what: the noun with its modifiers;
• how much: a number, or an unknown variable for "some" / "several";
• verb class: have → set, get/make → add, lose → subtract, give → transfer.

"He had 492 left" or "if he has 116 now" are not events but conditions on the state. The world solves for the unknowns from them (linear equations). The question becomes a query: state now / initially, sum of a class of events, difference, total.

If the world doesn't understand something, it returns None with a reason in the log and lists the numbers it could not read (unread). It does not guess. Rates and shares ("each", "per", "half", …) are left to steps.

When to use

• Additive and transfer problems with unknown starting amounts.
• As a judge feature (steps --qread).

Steps

# probe one problem (the last sentence is the question)
cargo run --release --manifest-path math/Cargo.toml --bin mathsolve -- qread-text \
  "Paul had some crayons. He gave 52 crayons to his friends and lost 535 crayons. He had 492 crayons left. How many crayons did he have at first?"

# coverage and accuracy on SVAMP: qread [svamp|svamp-train] [N = wrong answers to print]
cargo run --release --manifest-path math/Cargo.toml --bin mathsolve -- qread svamp 5

• QREAD_STRICT=1: only answer when every number was read.
• QREAD_DUMP=<file>: write index<TAB>ok<TAB>value for every answered item.

Example

Some(1079.0) unread []
t0: paul have(have) crayon Set None
t1: paul give(give) crayon Sub Some(52.0)
t1: paul lose(lose) crayon Sub Some(535.0)
t2: paul have(have) crayon Is Some(492.0)
state initially Some("paul") Some(("crayon", "crayon"))

unread lists the numbers the world could not read; state initially is the state at the start.

qread svamp prints svamp: the world answered A/300, of them correct K (p%): the world answered A of 300, and K of those were right. Report both coverage (A/300) and accuracy (K/A).

Pitfalls

• High accuracy on low coverage is not a solver. Quote both numbers.
• As a judge feature (--qread), the reader's answer is used only when unread is empty.

6. Evaluating on SVAMP / GSM8K

Gist

The datasets are not in the repo. Download them (both MIT) and convert them to the JSON arrays the loaders expect under <DATA_ROOT> (section 0).

Steps

1. SVAMP: take the 700/300 split from Hugging Face ChilleD/SVAMP (train.json, test.json). The original 1000 problems are on GitHub at arkilpatel/SVAMP (same fields, no split). Each item needs ID, Body, Question, Equation, Answer.

   <DATA_ROOT>/mwpdata-svamp/svamp-train.json
   <DATA_ROOT>/mwpdata-svamp/svamp-test.json
   

2. GSM8K: take the socratic files from GitHub openai/grade_school_math (grade_school_math/data/train_socratic.jsonl, test_socratic.jsonl). The loaders want a JSON array, not JSONL. Fields: question, and answer with <<…>> steps and a final #### N.

   jq -s . train_socratic.jsonl > <DATA_ROOT>/mwpdata-gsm8k-socratic/gsm8k-socratic-train.json
   jq -s . test_socratic.jsonl  > <DATA_ROOT>/mwpdata-gsm8k-socratic/gsm8k-socratic-test.json
   

3. Run the commands from section 2 (steps --set svamp|gsm8k …) and section 5 (qread svamp). Each command prints its own summary; see the output lists above.
4. Before claiming any change:
   • run twice and confirm identical results (diff MATH_DUMP);
   • report the excluded training items (line 1 of steps) and the reader's coverage;
   • compare per item against the baseline.

Example

SVAMP test (300 problems) with the benchmark configuration plus --qread:

• 179/300 (59.7%) correct;
• against a baseline with the same lexicon: 19 items gained, 6 lost (McNemar p = 0.015);
• against a stronger baseline the gain is not significant (p = 0.27).

GSM8K test (1319) was around 10–11% with the same solver. Treat these figures as a snapshot, not a guarantee.

Pitfalls

• SVAMP "test" in this guide means the 300-item split. Don't compare it directly with numbers computed on all 1000 SVAMP problems.
• GSM8K training problems without an exact tree (implicit numbers) or with trees that reuse numbers are excluded unless you use --dag. Say so whenever you quote a number.
• Never tune on the test split. Make fixes based on errors in the train split only.

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/train/math-world/