📘 Elephant — mova-elephant

🐘 The Elephant is one of Mova's animals (Rust on Mova): the keeper of the word graph (wordgraph) and of the approach nodes over papers (wordgraph/tamp). See the animals.

It keeps the graph incrementally, because its program keeps changing: new passes, new parts of a running pass and new papers are added without a full rebuild, and a changed program recounts only the parts it affects. It reads English with Mova, works in a standing loop under Crab, and asks two consultant terminals only what the counts cannot decide; their answers are review-only proposals, never graph writes.

Commands

elephant init STORE [K K_CORE MIN_COUNT MIN_OOV]   a new, empty store (default 32 128 3 20)
elephant add STORE PASS_DIR [PART…]               count the finished parts of a pass (all, or those named)
elephant retract STORE PASS/PART                  take a counted part out (subtracted at the next tamp)
elephant tamp STORE [FAN_IN [major]]              merge the FAN_IN smallest units (8), or all with major
elephant view STORE                               build the graph of what is counted (reused when unchanged)
elephant status STORE                             the store as JSON
elephant compare A.wg B.wg                        the same graph, every section but provenance?
elephant lists STORE WRONG_TSV RIGHT_TSV          keep a snapshot of a version of the hand lists
elephant plan STORE KEY_JSON                      what a new program key means for the counted parts
elephant reprogram STORE KEY_JSON PLAN_JSON       attest, retract and rekey by a plan
elephant suspects STORE [N]                       suspicious edges of the view (nothing is asked)
elephant ask CONFIG DEPUTY_CONFIG ID < text       read English, answer from the graph or the deputy
elephant serve CONFIG DEPUTY_CONFIG SOCKET LOCK   answer English on a private socket (one per LOCK)
elephant client SOCKET ID [urgent] < text         send English to a serving Elephant
elephant work CONFIG [CONSULTANTS|- [STEPS|- [SECS]]]   the standing loop (SECS 300)
elephant work-summary CONFIG                      events of the loop

Build and test:

cargo build --release --manifest-path tools/elephant/Cargo.toml --offline --locked
MOVA_OCTOPUS_TEST_MODEL=$MOVA_DATA/…/en.bin cargo test --release --manifest-path tools/elephant/Cargo.toml --offline --locked

Store commands that write take the store's writer lock (STORE/writer.lock); a running loop holds it.

The incremental graph

Model. The graph is a function of the multiset of pass parts it counts, G = build(Σ parts) (wordgraph::compile::build). PMI needs the global marginals (N_r, c(h,r,·), c(·,r,d)), and any new event of a relation changes N_r and so every PMI of that relation: no edge of G can be updated locally. What is incremental is the counting. Counts form a commutative group under addition, so the merge of parts can be done in any order and grouping, and a part can be subtracted exactly.

The store is a log-structured merge (LSM) store of counts with the graph as its index:

• runs: immutable canonical merges of parts, written by the Elephant (src/merge.rs): the summed counts restricted to positive ones, in key order, with exactly the out-of-dictionary words their records use, in the part format, so the wordgraph build reads a run like any part;
• deltas: parts added since, read in place from their pass directories (never copied);
• retractions: parts inside a run to subtract at the next tamp (a recount replaces an old part);
• the view: the graph built from runs + deltas. Queries read it: one wordgraph::Graph file, memory-mapped, Clone + Send + Sync, the same API as any graph.

Operations. add hashes the part, checks its key and appends a delta (O(part bytes)). tamp merges the fan_in smallest units (runs and deltas) into one run — size-tiered compaction, so a record is rewritten O(log_fan_in N) times over the store's life and a view build reads a bounded number of inputs; major merges everything. view stages the runs and deltas as one pass of symbolic links and builds the graph; it is named by its lineage (the hash of the sorted part hashes, the key and the parameters) and reused while the lineage is unchanged, also across tamps.

Invariants.

• I1, counted once: a part's sha256 occurs in at most one run or delta; a part with a counted hash is skipped, and a part with a counted name and other contents is refused (retract the old one first).
• I2, one program: every counted part has the store's key, or an attestation for it (below).
• I3, exactness: the run of any merge tree over parts P equals the run of merging P at once (byte for byte), so a view over runs + deltas equals the wordgraph build over the original parts in every section but provenance, given the same OOV registry state (registry ids are append-only: a word keeps the id it got first).
• I4, one graph per lineage.

Every change writes a new generation: history/<generation>.json, then manifest.json replaced atomically, then a line in journal.jsonl. Nothing is deleted; superseded runs are listed in the manifest as reclaimable.

Reprogramming. A part's key is what decides its records: part format, extraction rules version, the fingerprint of the hand lists it was extracted with, the parse models, language and dictionary fingerprint. plan sorts every counted part under a new key into ok, attest and recount. Another rules version, model or format: recount. Only the hand lists changed: only compound_reading entries change extraction (the other kinds act at query time). A new entry (h, r, d) suppresses only emissions of the triple (h, r, d) itself — and the role swaps it would take part in, which exist only if it is emitted — so a part holding no record of (h, r, d) under any UPOS is provably unchanged; a removed entry changes a part only where it suppressed something, which the part's sidecar counts. Those parts are attested; the rest are retracted and come back recounted in a new pass. reprogram applies the plan and adopts the key. The list versions are kept as snapshots in the store (lists/<fingerprint>/).

Cost model. Add: one hash per part. Tamp: two streaming reads of its inputs' records (O(R log k)), two of their OOV tables, one write; memory 4 bytes per input OOV entry plus 9 bytes per word of their union, no strings or records held. View: one wordgraph build over the staged inputs. Prospective prediction (2026-11-06, before the first real run): on the 50 000-article pass (100 parts, 103 M events) a tamp of 8 parts in seconds, a view over a few runs plus deltas about as long as the full build (≈ 1 min), memory below the full build's.

Measurements (2026-11-06)

Pass wiki-t1-000-100-r2 (50 000 Wikipedia tier-1 articles, 100 parts, 103.1 M events, extraction rules 2, part format 1), k = 32, k_core = 128, min_count = 3, min_oov = 20; a shared box (load 11–15 on 20 cores), heavy steps under tools/jobs with a 4 GB limit.

• The view over runs + deltas equals the full build (elephant compare: every section but provenance).
• Three stores with fan-in 8, 4 and 8 (two merge trees, two versions of the merge's memory layout) wrote the same final run, byte for byte (SHA-256 acad1f59…).
• English through serve on that view: 107–159 ms a question (client process, graph open, parse, answer); the ask command loads the model per call (≈ 2 s).
• suspects on that view: 7 977 suspicious edges (5 190 compound shadows) in 0.36 s, identical on two runs. Of the 23 edges of the hand list of the wrong (compound_reading, branch word-graph-all) present in this graph, 10 are among them (ranks 44, 101, 112, 735, 1612, 1833, 2495, 2554, 2988, 4233); the class-outlier signal alone found 1 of 23. The top shadows are mostly participial compounds («supporting actor», «backing vocals», «highest-grossing film», «recurring role», «defending champion», «failed attempt», «best-selling album») next to real pairs («speak a language», «increase the risk»): telling them apart is the consultants' question.

The predictions held for the view (39 s against ≈ 1 min) and the memory; the tamps of the larger runs took longer than «seconds» (up to 23 s).

The whole-Wikipedia pass (wiki-all-v3, extraction rules 3, part format 2, running): its first 11 parts (≈ 223 000 articles, 245.7 M events, 126.0 M records) were added in 0.8 s and tamped into one run of 77.8 M records, 2.0 M OOV words and 300 MB in 69 s at 0.10 GB. Built with wordgraph of branch word-graph-all (which reads format 2), the view from that run (54 s, 0.65 GB) equals the view from the 11 original parts (60 s, 0.68 GB). Prospective prediction for the finished pass (about 30 times as many articles): a major tamp of 30–60 min under 2 GB, a run of 3–6 GB; the view's memory is the wordgraph build's, not the Elephant's.

In English

A prompt is read first by the question reader (src/ask.rs) over Mova's own parse (world::tree::Tree::parse_cased on the en annotator), with a small table of frames, each one graph query:

The deputy's capabilities are a closed vocabulary (typed slots take URLs, hosts, identifiers and quoted text), so a question about arbitrary words cannot be a capability; the frame table plays that role with the same native parse. What the reader does not take — statements, negations, several sentences, unlisted frames — goes to Mova's native deputy with the Elephant's hands (deputy.example.json, handler elephant): status and proposals observe ("How many words do you know?", "How big is the graph?", "Is the graph tamped?", "How many proposals do you have?"); tamp and refresh act ("Tamp the graph.", "Please refresh the graph.") by leaving a request for the standing loop — the English service never writes the store. "Delete the graph." and "Do not tamp the graph." are refused with a review candidate.

Answers on the 50 000-article view (2026-11-06): «In my graph a dog most often goes, eats, barks, bites, skips and dies.» «Those who most typically elect in my graph: Assembly, voter, member, council, constituency and Congress. «elect a president» is in my graph, seen about 470 times.» «I find «the food fruits the man» unlikely. I have no evidence for «food» as the subject of «fruit». …» Absence is weak evidence and the answer says so; «the dog bites the man» comes out unlikely and «the man bites the dog» plausible on this graph (no «bite → man» object edge was kept, while «man» as a subject of «bite» and «dog» as its object were), which is a fact about these counts, not about dogs.

Another language

An Elephant keeps one language: language in its configuration (en by default, or fr), and a store counts parts of one key, whose language is part of it; the loop stops when the store's language is not the configuration's. The questions stay English and are read with the English en.bin. A word of the graph's language is quoted: a quoted single word is read in place of an English stand-in (a noun phrase, else a verb), so «What does "chien" do?» is the does frame on «chien», «What is "Paris"?» the isa frame, «Who "élire"?» the subjects frame. Answers about a non-English graph say «my French graph» and quote its lemmas, without English articles or inflection; an English word is not a word of a French graph («I do not know the word «dog» in my French graph»). The consultants' question names the graph's language and gives a French example of a compound reading.

Measured 2026-11-06 on a store of the top 1 000 French articles (one part, 3.67 M events, 12 367 words): «In my French graph «roi» is most often the subject of «mourir», «acheter», «accorder», «construire», «accepter» and «décider».»; «In my French graph «Paris» is most often «ville», «capitale» or «siège».»; «…«guerre» is most often the subject of «éclater», «opposer», «reprendre» …». Quoted words must be lemmas (there is no French lemmatiser in the reader). tests/english.rs checks these frames on a synthetic French store.

Consultants

consultants.example.json: two profiles (Sol through the Codex CLI in our deployment), each with its own mova-terminal (terminal-elephant-{1,2}.example.json, held by crab-tmux-worker with the consultant prompt tools/terminal/prompts/animal-terminal.md), purposes edges and approaches. The loop asks one bounded question at a time (at most 10 edges or 3 approach nodes, under 24 000 bytes), round-robin over the terminals whose pacing (600 s) passed, alternating the purposes:

• edges (src/suspect.rs): is a suspicious edge a parse error, a compound reading, figurative, real or unsure? Two graph-only signals choose them: the compound shadow — a verb–noun core edge whose noun is also modified by the verb itself (amod: «eating disorder» counted as eat + object disorder) — and the class outlier — a strong dependent that shares no isa class with the other typical dependents of its head.
• approaches: do the members of an approach node (elephant-tamp build, nodes.jsonl) explain the same approach? Lowest mean fit first.

Rules (as the teachers of Nautilus and Sea Turtle): no question without operator_word in the configuration; the terminal must be the profile's own; every question is journaled before and after with its hash and size; a question whose hash is in the journal is never asked again, and an uncertain one is never replayed. Answers are parsed strictly (one line per item, fixed columns; a compound reading must end with one of the edge's words) and kept in WORK/proposals/proposals.jsonl with the question's and the answer's hashes, the hand-list kinds also as wrong.proposed.tsv in the lists' six columns. Nothing is admitted: a person moves a reviewed line into wordgraph/seeds/wrong.tsv or right.tsv, which is a reprogramming (above).

The standing loop

work is the store's only writer. One step: take the English service's requests; add up to add_per_step finished parts of the configured passes; tamp when more than fan_in units wait (or a retraction, or a request); rebuild the view when the events outside it reach view_share of it (at most once per view_min_secs, or on request); when papers are configured and at least min_new are new, run elephant-tamp read (cached per text: only new papers are read) and elephant-tamp build into a new folder; ask one consultant question when one is due. Each step is one line of WORK/work.jsonl; an error stops the loop (fail fast, nothing retried); STOP in the work folder ends it.

crab-workers.example.json lists the English service, the loop and the two terminal holders, all autostart: false.

Tests

tests/lsm.rs: a store fed one pass, tamped in pairs, then another pass as deltas equals the full build; a major tamp reuses the view and its rebuild is equal again; another order, merge tree and fan-in give the same run bytes; a retraction equals the full build without that part; reprogramming on a list change attests the parts without the new compound's edge and recounts only the one that holds it. tests/work.rs: the loop adds, tamps and rebuilds, takes a request once, finds the planted suspicious edge, holds the question without the operator's word, keeps the answer as one review-only proposal and leaves the graph unchanged. tests/english.rs (needs the model): the frames on a synthetic graph, the deputy's status, proposals, tamp and refresh, and the refusals. Unit tests: part formats, signed merges, keys and list changes, consultant rules, strict answer parsing, quotations.

Negative controls (each red, restored, 2026-11-06): the comparison itself (a graph missing a pass differs); a part under another rules version, a file that does not hash to its sidecar, a retraction of what is not counted, a changed part at tamp time — refused; an entry on an edge every part holds attests nothing; merging by max instead of sum (the equality test red); the counted-hash skip removed (double counting, red); the negation check of the reader removed (the English tests red).

Sources

Library ids are folders of the paper library; works marked «outside» have no card yet (docs/usage.tsv).

• LSM store with size-tiered compaction: O'Neil et al. 1996 (outside); Bigtable, Chang et al. 2006 (outside); a GPU LSM dictionary, ashkiani-2017. Sorted runs merged k-way with a heap: Knuth 1998 vol. 3 (outside).
• Signed counts as a commutative group (retraction = weight −1): no card; the invariants I1–I4 are this crate's own.
• Part compression: zstd, Collet & Kucherawy 2021, RFC 8878 (outside).
• Scores behind the graph: see wordgraph — the word graph (church-1990, dunning-1993, evert-2005).
• Class-outlier suspects: resnik-1993, rooth-1999.

Limits

• The view is exact but not free: after new parts it is rebuilt by one wordgraph build over runs + deltas (no edge is updated in place, since PMI is global). Between rebuilds queries read the last view; the status says when it lags.
• Retraction needs the original part file as counted (same hash). Superseded runs are not deleted by the Elephant.
• The store keeps one part format per store: wordgraph on main reads format-1 parts only; the whole-Wikipedia pass writes format 2 (branch word-graph-all), which this crate reads and merges, and whose build reads it.
• wordgraph compact (branch word-graph-all) tamps a pass's consecutive parts once before one build; the Elephant's runs are the standing form of the same idea (retraction, lineage, program keys, any merge tree). Both write parts the build reads.
• Approach nodes are rebuilt whole (≈ 1 min for 1 763 papers); only the reading of papers is incremental.
• The question frames are a fixed table; what they do not cover goes to the deputy and is refused there.
• The suspicious-edge signals choose what to ask; they are not judgements. Their precision is unmeasured until reviewed answers exist.
• Another language: the quoted words must be lemmas; the plausibility frame parses its quoted sentence as English.

Sources

• The graph's scores and lists: wordgraph (evert-2005 and the works listed there).
• Log-structured merging (size-tiered compaction) as in LSM stores: O'Neil et al. 1996 (no card).
• The French graph: UD French GSD (guillaume-2019) through the French pass, French Wikipedia (CC BY-SA 4.0, internal).

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/tools/elephant/