Tests
Unit tests
tests/run.sh # all crates
tests/run.sh math world # some crates
Each crate is built and tested with cargo test --release. All
crates share one build directory (target/ at the repository root, or CARGO_TARGET_DIR).
tests/secret-guard-control.sh is the negative control of secret-guard.sh: in a throwaway repo with a fake
local.md it checks that a staged machine path or secret is refused and that a variable, or a path inside tera/, passes.
Data folder
Commands that read datasets, write runs or databases look under $MOVA_DATA (default: ~/mova-data, the same as install.sh): raw/ for downloaded datasets, runs/ for outputs, db/ for databases, ud/ for Universal Dependencies treebanks (or set EN_UD_DIR). Commands that call a large language model read ANTHROPIC_API_KEY=… from local.md in the current folder (or the file named by MOVA_LOCAL), a local file that .gitignore keeps out of git; none of them is needed for the tests and benchmarks above.
Benchmarks
The numbers below were measured for release 0.4 (2026-10-16), with the model en.bin from install.sh.
Build once at the root (cargo build --release); W=target/release/world.
The benchmark data is not in this repository (each set has its own license); download it and point the command to it. Results are deterministic: the same command gives the same number on every run.
| Benchmark | What it tests | Command | Mova's result |
|---|---|---|---|
| bAbI tasks 1–20, v1.2 (CC BY 3.0) | keeping the world of a story: who is where, who has what, relations | $W babi <path>/tasks_1-20_v1-2/en test | 99.4% on test, 19 of 20 tasks ≥ 95% (task 18 92.3%; with the 0.3 model 99.8%, 20 of 20); no training on the questions |
| bAbI contrast set | the same stories with new names, places, things and synonym verbs | $W babi-contrast <bAbI en dir> <out dir>, then $W babi <out dir> test | 99.4%, 19 of 20 |
| StepGame clean (MIT) | relative positions on a grid over 1–10 steps | $W stepgame <path to the json files> | 96.3% on the full test (k=1 98.2% … k=10 94.7%); 98.0–99.6% on questions without the two expressions that the dataset labels inconsistently |
| StepGame corrected (Li, Hogg & Cohn 2024, MIT) | the same task with the wrong templates removed; clean and noise variants | $W stepgame <dir with qaK_{train,test}.txt> (unpack data.zip; one dir per variant) | 100.0% at every k=1..10, clean and noise; negative control: test labels shifted by one class → 0.0% |
| StepGame contrast set | phrasings the StepGame generator never produced (incl. the reverse view "B has A on its left"), zero overlap with training | $W stepgame-contrast <StepGame json dir> 500 --test | 100.0% at every k; without the lexical reader (STEPGAME_NO_LEXICAL=1) 17.1% |
| SpartQA-Human (data: non-commercial) | spatial questions about human-written scene descriptions | $W spartqa <path>/human_test.json | YN 75.5%, FB 79.7%, CO 75.9%, FR 63.6%; total 74.1% |
| SVAMP (MIT) | arithmetic word problems | cd math && MATH_NOLEX=1 ../target/release/mathsolve steps --set svamp --exact --epochs 20 --beam 32 --rerank 5 --qread (expects svamp-{train,test}.json) | 64.3% (193 of 300) |
| SVAMP, the school's maths student | the same test read into a world of actors (once per release) | $W school actors held <path>/svamp-test.json 0; decisions are made on $W school actors cv (train only) | 160 of 300; cv: world reader 513, book's way 1116 |
| Parsing, tales test and UD English EWT test | LAS of en.bin with the shipped repair rules | $W state las origin <tales-odd.conllu> tale, $W state las origin <en_ewt-ud-test.conllu> real | tales 79.69, EWT 80.68 |
| Primary-school courses | 25 knowledge courses, 26 992 test questions | $W knowledge course <course> for each of the 25 courses listed in world/src/knowledge.rs (world/data/school/<course>/) | 11 730 right (43.5%); with the GPU parser (MOVA_EXEC=gpu) 11 817 |
world babi-probe "sentence" "question?" shows how Mova reads single
sentences (set BABI_DEBUG=1 to print the dependency trees).
This page in Markdown → https://md.mova.elamur.ai/free/en/docs/tests/