Story worlds: bAbI, StepGame, SpartQA

Gist

The world crate (world) reads short stories into a temporary story world (who is where, who holds what, which thing relates to which) and answers questions from that world. It does not train on the questions, and its code has no trigger words. When it cannot answer, it prints ? and the reason. A gap like that tells you what knowledge to add next.

Three readers are covered here:

Global knowledge lives in the seed files global/seeds/**/*.md. At compile time, global/build.rs turns their fenced blocks into Rust tables (RELATIONS, verb_class(), CONCEPTS, ...). The seed file is the source of the knowledge, and the compiled tables are what the readers query.

When to use

• A probe or an eval prints a gap such as verb "sprint" has no class at the global level or gap: where people go when sleepy. Add a verb or a relation to a seed.
• You want to learn a fact table from training data and freeze it as a seed (world babi-learn).
• You changed the parser (en) or a seed, and you need to show that nothing regressed.

Setup

• One workspace. The crates share the root Cargo.toml, Cargo.lock and target/, and depend on each other through relative paths (world → ../global, ../en, ../prag, ../math). Pick a crate with -p <crate> or --manifest-path.
• Parser model. bAbI needs the en tagger/parser model. It ships in en/models/en.bin and is used by default (WORLD_EN_MODEL overrides it). Without a model the reader fails with an error; it does not silently fall back to rules. To train your own from the UD English EWT and ESL-Spok treebanks (CC BY-SA 4.0):

  cargo run --release --manifest-path en/Cargo.toml -- train <en_ewt-ud-train.conllu> <en_eslspok-ud-train.conllu> -- models/en.bin
  

  StepGame and SpartQA do not use the model.
• Datasets are not in the repo. Get them from their public sources:
  • bAbI tasks 1-20 v1.2 (Facebook Research, CC BY 3.0). The archive is tasks_1-20_v1-2.tar.gz; there is also a mirror on Hugging Face as facebook/babi_qa. Use the en/ folder, which holds files qaN_<name>_{train,test}.txt.
  • StepGame (MIT): GitHub ShiZhengyan/StepGame. You need a folder with qa1_train.json and qa1_test.json ... qa10_test.json.
  • SpartQA-human: GitHub HLR/SpartQA_generation. The archive SpartQA_Human.zip contains human_test.json and human_train_annotation.json. No data license is stated, so treat it as non-commercial: use it for measurement only, and copy nothing from it into code or seeds.

The commands below use these placeholder paths: data/babi/en, data/stepgame, data/SpartQA_Human/human_test.json.

Commands (verified against world/src/main.rs)

All arguments are positional. The code has a home-directory fallback when the folder argument is missing, so always pass the folder explicitly.

W="cargo run --release --manifest-path world/Cargo.toml --"

$W babi <en-dir> [test|train] [N]   # all files *_<split>.txt in the folder; N = errors shown per task (default 0)
$W babi-probe "<sentence>" ... "<question?>"   # any argument containing '?' is a question; the others are read into the story
$W babi-learn <qa20_..._train.txt>  # learn "state -> go=place" from training data; prints relation lines
$W babi-contrast <src-dir> <dst-dir>  # writes contrast copies: new names, places, things, synonym verbs
$W stepgame <dir> [epochs] [N]      # trains on qa1_train.json (default 10 epochs), tests qa1..qa10_test.json
$W spartqa <file.json> [N]          # accuracy per question type (YN / FB / CO / FR)

Environment switches (all read in the source):

The key output words are total, solved (≥95%), without doubtful, and gap (missing knowledge).

Seed blocks the readers use

Verb classes (```verbs, in Possession and changes in quantity). Each line has three parts: <class> <effect> <lemma>.... The effect is one of =, +, -, ±, which mean no change, more, less, and transfer of quantity. Multi-word verbs are written with an underscore (pick_up), and the reader forms them from a verb plus its particle (compound:prt).

get + get grab pick_up get_on buy receive find collect ...
give ± give pass hand_over sell lend pay send ...
lose - lose eat spend use break drink throw ... drop discard abandon put_down ...
move = move hurry choose come start gnaw wake run walk drive travel journey ...

The bAbI reader acts on these classes:

• move: the subject goes to the first nominal oblique.
• get or have with an object: the subject now holds the thing.
• lose: the subject drops the thing.
• give: the thing moves from the subject to the to-oblique or the indirect object.

Bare be with in or at is also read as a location. Any other verb gives verb "<lemma>" has no class at the global level.

Relations (```relation, one per line as name: key=value ...). From Relations of space, size and time:

north_of: inverse=south_of vec=0,1 step=n
bigger_than: inverse=smaller_than transitive=yes
fit_inside: same=smaller_than
morning: order=1

From State → where they go:

hungry: go=kitchen
tired: go=bedroom

Keys that babi.rs actually reads:

symmetric=yes (in scene.md) is documentation only: no reader looks it up at the time of writing. The SpartQA relation words (left, above, near, touching, ...) are a match in spartqa.rs. SpartQA does read the scene_color, scene_size, scene_shape and scene_any categories from scene.md.

Build gate. global/build.rs panics, so the build fails, and the message names the seed file, if any of these hold:

• a relation line has no :, or a field has no =;
• a relation name is defined twice, in any seed;
• a ```verbs line has no effect, or the effect is not one of = + - ±;
• a verb class name collides with a concept;
• a ```link line points to an undeclared concept, or a link is duplicated.

When a word appears in two verb classes, the first one in build order wins, and nothing warns you. Files are walked in sorted path order.

Steps: add a verb or a relation

1. Find the gap.

   $W babi-probe "Mary sprinted to the attic." "Where is Mary?"
   

   Output:

   Mary sprinted to the attic. → Some("verb «sprint» has no class at the global level")
   ? Where is Mary? → ("?", "where mary is — unknown")
   

   Use BABI_DEBUG=1 to make sure the parse is sane (here sprinted is lemmatised to sprint and is the VERB root). If the tree is wrong, the fix belongs in en, not in a seed.
2. Edit the seed. Append the lemma to the right class line in Possession and changes in quantity, for example move = move hurry ... sprint. For a new relation, add a line to a ```relation block. Give it inverse= (or same=) so the reader recognises it. Add step= on both directions if path questions need it.
3. Rebuild and probe again. build.rs re-runs when anything under seeds/ changes, so the next cargo run recompiles global and world. Rerun the same probe. The sentence should now read as None (understood) and the question should be answered.
4. Check for regressions. Run cargo test --release --manifest-path global/Cargo.toml. It includes verb_class assertions, such as eat → lose. Then run the full evals below, on both the original files and the contrast files. Verb classes are also used by math, which reads quantity word problems, so run its checks too.
5. Write down why in the seed's prose (gist, conditions, examples, sources). The block is the data, and the prose is the explanation.

Steps: learn knowledge from training data into a seed

world babi-learn reads a bAbI task-20 training file. For every Where will X go? question, it takes X's last stated attribute ("Jason is thirsty") and counts each attribute → gold-place pair. Then it prints the majority place for each attribute, with its support. It never opens the test file.

$W babi-learn data/babi/en/qa20_agents-motivations_train.txt

Output on the 1k training split:

bored: go=garden    # 98/98
hungry: go=kitchen    # 89/89
thirsty: go=kitchen    # 97/97
tired: go=bedroom    # 91/91

To turn it into a seed:

1. Decide. A person (or a gate) decides what to keep. Keep only rows with clean support (here 375/375). A split count is a signal to look at the data, not a rule to freeze.

2. Strip the # n/all comments. Inside a block, # is a field without =, so the build fails with relation "...": "#" without "=". Move the counts into the prose.

3. Check the names are new. For example, hungry already exists in motivations.md. A second definition makes the build fail with relation "hungry" is repeated.

4. Write the seed at global/seeds/shortcuts/<topic>.md:

   # State → where people go
   
   **Gist.** A person's state suggests a goal: hungry or thirsty → kitchen, tired → bedroom, bored → garden.
   
   **Conditions and exceptions.** A house with typical rooms; another world (forest, city) has other goals.
   Learned with `world babi-learn` from the bAbI task-20 training split: 375 of 375 questions, no exceptions;
   the test split was not opened.
   
   **Examples.** "Jason is thirsty. Where will Jason go?" → kitchen.
   
   **Sources.** Weston et al. 2015, bAbI task 20 (CC BY 3.0), training split.
   
   ```relation
   hungry: go=kitchen
   thirsty: go=kitchen
   tired: go=bedroom
   bored: go=garden
   ```
   

5. Confirm the seed is what answers. Rebuild, then probe "Jason is thirsty." "Where will Jason go?". The answer should be ("kitchen", "thirsty → kitchen (first level)"). Then do an ablation: build with GLOBAL_EXCLUDE=motivations and qa20 should drop. This shows the score comes from the seed. The run only needs the env var; do not delete the seed file.

babi-learn handles only the task-20 question shape. To learn another table, copy its pattern from learn_motivations in babi.rs: read the story, count (state, gold) pairs on training data only, print a block line with its support, and let a person decide what to keep.

Evaluate

# bAbI: test split; also run train (should be close: nothing was fitted to test)
$W babi data/babi/en test 0
$W babi data/babi/en train 0
# Contrast set: same stories with new names, places, things and synonym verbs (some deliberately not in the seeds)
$W babi-contrast data/babi/en data/babi-contrast
$W babi data/babi-contrast test 0

# StepGame: full test, plus the list of excluded items
STEPGAME_EXCLUDED=stepgame-excluded.tsv $W stepgame data/stepgame 10 0

# SpartQA-human test
$W spartqa data/SpartQA_Human/human_test.json 5

Output formats:

• bAbI prints one line per task, qaN_name ok/n acc%, and ends with total ok/n (acc%), solved (≥95%) s/20. With N > 0, each error is printed with the last 12 context sentences, the answer, the gold answer, and the reason.
• StepGame prints the training accuracy for each epoch on stderr. On stdout it prints one line per k: the full accuracy and then without doubtful. After training, the reader flags any sentence template that has contradictory labels in qa1_train.json (at least 10 examples, with the top label under 60%). Test questions whose story uses a flagged template count as "doubtful".
  • The headline number is the full test. The clean number is secondary, and every excluded item is listed by name in the TSV, so the clean number cannot be read as quietly pruning the test.
  • TSV columns: k, id in qa{k}_test.json, the flagged template numbers, and whether the answer was right (1 or 0). The templates themselves are listed in # lines at the end of the file.
• SpartQA prints TYPE: ok/n (acc%) for each type, then a total. The test split here is the 2022 version (YN 143, FR 77, with FB and CO also reduced). The total is therefore not comparable with results on the original 510-question test. Compare only per type, under matching rules.

Results for release 0.4 (2026-10-16; the full table is in the README). These runs reproduce them:

• bAbI (1k): test 99.4%, 19 of 20 tasks at ≥95% (task 18: 92.3%); contrast set 99.4%, 19 of 20.
• StepGame: 96.3% on the full test (k=1 98.2% → k=10 94.7%). The reader flags two contradictory templates; without them every k scores 98.0-99.6%, and 11,553 questions are excluded by name.
• SpartQA-human: YN 75.5%, FR 63.6% (majority baseline 34.9%).

Runtime is seconds: about 14 s for the bAbI test, about 8 s for StepGame, and under 1 s for SpartQA.

Pitfalls

• Determinism before claims. Run the same eval twice and get identical output before you claim a gain. The readers use BTreeMap/BTreeSet, and the StepGame perceptron uses a fixed visiting order and integer weights. Any new code on these paths must stay deterministic: no HashMap iteration order, no RNG.
• Tune on train, measure on test. Choose fixes by looking at training-split errors (babi ... train N, SpartQA training annotations). babi-learn must only ever see _train files.
• A seed fix does not fix a parse. If BABI_DEBUG shows a wrong tag or head (e.g. attic as ADJ, a name lemmatised to a common noun), adding words to seeds hides the problem. The contrast set exists to expose this.
• Global seeds are shared. A word added to a verb class changes math and world's quantity reader as well. A word that already sits in an earlier class is silently ignored.
• A relation the reader cannot see. It needs inverse= or same=. A path needs step= on both directions.
• SpartQA data is non-commercial. It is measurement only. Nothing from it may go into seeds or code.
• Positional arguments are not checked. babi-learn, babi-contrast and spartqa panic if the path is missing. babi and stepgame fall back to a built-in folder, so pass the folder explicitly.

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/train/story-worlds/