prag — Mova pragmatics: LLM silver and a transparent SLM

prag is Mova's pragmatics: the frames reader (who did what to whom, why, what came next), speech acts and their labeller, implicatures, irony candidates, expressions read by their meaning, and the agreer that checks our labels against open witnesses. Pragmatics: frames — who did what to whom, why, and what came next is the guide to the frames.

The small model learns pragmatics with memory models (something like a graph) next to a perceptron. The criterion: if the perceptron gives better quality and even speed, the little graph is not pulling its weight yet.

A separate crate that depends on en, not the other way round. It holds:

• LLM silver (Opus) — 2,986 sentences of open texts;
• three SLM models in Rust behind a shared interface: an averaged perceptron, IGTree (memory as a graph, a prefix tree) and k nearest neighbours (IB1);
• evaluation, predictions in the database and decision traces.

Schema (Smart annotator — layers, context, memory)

A sentence with direct speech inside a narrator frame ("‘Pay me my wages,’ said Hans.") is labelled by the speech itself: voice=character, the frame is ignored. Full definitions are in the prompt (src/silver.rs, RULES). Closed sets are enums (src/schema.rs).

Data

Only texts with a commercial licence. The gate is en::license::commercial_gate; the sources are added to en/data/train-licenses.tsv:

• gutenberg-tales — fairy tales from the database, public domain. 1,999 sentences:
  • 209 windows — consecutive whole paragraphs of one tale, 10–18 sentences;
  • 150 tales, book shares proportional to size: Grimm 831, Andersen 543, Kipling 310, Aesop 304, Potter 11;
  • 719 sentences with quotation marks, i.e. with dialogue; the rest is narration;
  • context for the LLM — paragraphs and 1–2 sentences before the window.
• tatoeba-eng-cc0 — 1,000 standalone sentences (CC0): one sentence, 1–25 words, different texts. The UD bronze of these sentences (en) is loaded into the database, run prag-2026-09-26:tatoeba-ud, document tatoeba. Otherwise pragmatics would have nothing to refer to.
• GUM (CC BY-NC-SA) — only for evaluating form against s_type. Negative control of the gate: GUM in training means a stop (test license_gate_blocks_gum).

LLM silver and its cost

Run parameters:

• Opus claude-opus-5-5, effort high via claude -p --settings '{"env":{"CLAUDE_CODE_EFFORT_LEVEL":"high"}}'; the call works as in annot/src/llm.rs, shared module src/opus.rs;
• 57 batches of 23–60 sentences, 4 calls in parallel;
• output — 8-column TSV with numbers local to the batch.

Cache writes are mostly the built-in system prompt of claude -p, ~20k tokens per call. Our prompt is ~4k.

Gates (silver::gate), all with negative controls in tests:

• values only from the closed sets;
• every sentence has a row;
• an extra or repeated number rejects the whole batch, because the alignment is in doubt;
• voice consistent with the source: tale — narrator|character; Tatoeba without quotation marks — none.

Rejects go to rejects.tsv, with no repeated calls. All 13 are about voice:

• 5 tale sentences with none: the moral of a fable, a narrator's digression;
• 8 Tatoeba sentences with narrator/character without quotation marks.

There were no format failures and no number shifts.

Label distribution (2,986):

• form: decl 78.9 % · imp 6.9 · wh 4.6 · q 4.2 · frag 3.3 · excl 1.0 · intj 0.9 · other 0.2
• act: narrate 32.5 % · assert 29.2 · evaluate 9.1 · request 7.1 · ask 5.9 · express 4.9 · disagree 2.6 · suggest 2.1 · warn 1.6 · offer 1.3 · promise 1.3 · agree 1.0 · greet 0.5 · other 0.5 · thank 0.1 · apologize 0.1
• indirect 7.3 % · hedge 2.7 % · polarity: none 52.6 · neg 28.5 · pos 19.0 · voice: narrator 34.3 · none 33.1 · character 32.6
• means is not "-" in 1,404 sentences (47 %). Indirect acts: request 55, evaluate 48, suggest 26, disagree 21, assert 19, express 15, offer 13…

Database (mova.duckdb, schema v2 — migration in db/schema.sql, command db migrate):

• run prag-2026-09-26, sets pragmatics / silver / opus: 2,986 sets and rows;
• provenance is in runs.params: model, effort, accounting of calls and tokens;
• view prag_labels — labels with source, run and text;
• comparison in SQL — sql/silver-vs-rust.sql.

SLM

Features (src/feats.rs) — 32 slots with symbolic values, the same for all three models:

• words:
  • w1 w2 w3 wl — the first three words and the last one;
  • u1 — UPOS of the first word;
  • end — final punctuation;
  • qm — "?" in the sentence;
  • excl — the construction what a…! / how…!;
  • len — length.
• UD tree (en::annotate, model ud-ewt-eslspok.bin):
  • root: rupos, rform (Mood=Imp, Fin, Inf…), rlem, tense;
  • aux1 — auxiliary at the start; inv — inversion of auxiliary and subject;
  • wh — wh-word at the start; modal — modal;
  • neg — negation on the root or elsewhere;
  • subject: subj (pronoun or part of speech), sperson (person), you, p1;
  • voc — vocative.
• cues (data/cues.tsv) — our own small list after Lakoff 1973, Hyland 1998, Searle 1969/1975 and Brown & Levinson 1987:
  • please;
  • hedge — a hedge word;
  • eval — evaluative words: pos, neg, mix;
  • cue — act marker: thank, apol, greet, promise, warn, suggest, offer, yes, no, intj.
• context:
  • quote — none, whole, framed (with a narrator frame) or cont (speech continues from the previous sentence);
  • sayv — speech verb in the frame;
  • ppos — position in the paragraph;
  • prev — act of the previous sentence: in training from the silver, in prediction the model's own prediction;
  • genre — tale or Tatoeba.

If a sentence contains direct speech, the words and the tree are taken from the quoted text. Quotes are tracked across the sentences of a paragraph. "’" between letters is an apostrophe. Titles in quotes ("the title of “Gentlemen Weavers.”") are not treated as direct speech.

Models (src/model.rs) — a shared trait Classifier (predict → class, confidence, explanation; bytes; info). The models run side by side on the same features:

• perceptron — averaged (Collins 2002), "slot=value" indicators, 12 epochs. Explanation — the 3–5 features contributing most to the margin of the chosen class over the runner-up.
• IGTree (Daelemans, van den Bosch & Weijters 1997; TiMBL):
  • slots ordered by gain ratio;
  • each node holds a default label and counts;
  • leaves with the parent's label are pruned;
  • prediction — walking the graph while values match. Explanation — the path walked.
• k-NN / IB1 (Aha et al. 1991; TiMBL):
  • the whole memory; distance — overlap weighted by gain ratio;
  • the k nearest distances vote, k chosen by leave-one-out on training data: form 3, act 5, indirect 11, hedge 3, polarity 11, voice 7;
  • explanation — the most similar sentences in memory with the same label.

Results

Held-out silver (80/20: tales split by document, Tatoeba by sentence; test 538)

Accuracy / macro-F1, %. Baseline — majority class of the training set. Control — the same model on shuffled labels: it must land at baseline level.

The negative control goes red as it should: with shuffled labels the accuracy of all three models is at baseline level (no more than 1.1 pp above) or below, far from the real models.

Cross-validation on the whole silver (5 folds, 2,986 sentences)

A test of 538 sentences is small for rare acts, so cross-validation gives a more stable comparison. Each sentence is predicted by a model that has not seen it. These predictions are what is written to the database: sets pragmatics / rust, runs prag-2026-09-26:perceptron|igtree|knn, with decision traces in pragmatics.trace.

Speed and size (all 6 fields together)

Shared feature extraction takes 0.45 ms per sentence: tokens, the en tagger and parser, cues. The en model loads in 0.11 s. The whole prag eval takes ~40 s: three models, controls, 5 folds and GUM.

Agreement of perceptron and memory — the "ask the LLM" signal

Cross-validated, 2,986 sentences. "All three" means agreement of the three models; in that case the last two columns show the perceptron's accuracy outside agreement.

Act:

• the three agree on two thirds of the sentences, and there accuracy is 84.5 %;
• on the remaining 979 sentences (33 %) the perceptron is right only 41 % of the time;
• these sentences hold 576 of the 887 act errors, i.e. 65 %.

Model disagreement is a natural "ask the LLM" trigger: a third of the sentences go to Opus and take two thirds of the errors with them. On disagreements the perceptron is mostly more accurate than memory (exception — indirect: 46.8 vs 53.2 for k-NN), so on disagreement it is the one to trust, or the LLM.

Full tables (all pairs, confusion of each model) — report.md.

Confusion matrix for act — perceptron, cross-validated

Rows — silver, columns — prediction; the 12 most frequent of the 16 acts are shown.

What the matrix shows:

• the model recognises narrate, assert, ask and direct request by form;
• the main pit is evaluate and express, which become assert (139 and 50);
• offer, suggest, warn and promise get confused with request and assert: the difference is in intent, not syntax;
• k-NN barely sees agree/disagree (0 and 2 hits vs 9 and 30 for the perceptron): short "No!" and "Yes" drown among neighbours in memory.

form vs GUM test s_type (evaluation only)

Label mapping:

• decl, q, wh, imp, intj, frag, other — one to one;
• ger, inf, sub → frag (no finite main clause);
• multiple (51) — excluded from the comparison: we have one form per utterance;
• GUM has no excl.

We compare 1,413 of 1,464 sentences.

On the main sentence types the transfer from tales and Tatoeba to GUM genres is good: 95 %. The main discrepancy is in the definition of a fragment:

• 208 of 235 GUM frag (together with sub, ger, inf) are called decl by the model. In GUM these are headings, captions and standalone subordinate clauses ("Because it was cold."), while in our silver fragments are mostly dialogue lines;
• GUM intj ("Yeah.", "Okay.") is mostly frag for us: that is how the LLM labelled it under our definition.

To compute GUM metrics, the frag/intj definitions need to be aligned with the GUM scheme, or labels mapped at the schema level.

Decision examples (test and cross-validated predictions)

Numbers next to perceptron features are the contribution to the margin of the chosen class over the runner-up. In an IGTree path — the node's default label and count.

Correct, all three. The Innkeeper said, “Why do you howl so fearfully?” — silver: wh, ask, character.

• perceptron → ask: end=? +17.6 · w3=you +16.5 · qm=yes +13.7 · modal=- +8.7 · neg=- +6.2
• IGTree → ask, path: root → narrate 790/2448 · qm=yes → ask 143/215 · inv=- → ask 55/88 · end=? → ask 55/87 · … · sayv=said → ask 6/7
• k-NN → ask, because the nearest are: "‘Where are you going?’ said Rose-red…" (ask, 1.73) · "“And you,” they said, “how many sons have you at a birth?”" (ask, 1.83) · "Said the peasants: ‘Are there any more there?’" (ask, 1.87)

Indirect request. The Hare said to him, “I wish you would act sincerely by me, and show yourself in your true colors.” — silver: decl, request, indirect=1; means: "the Hound should stop acting two ways and show his true intent".

• perceptron → express: subj=i +11.1 · w2=wish +10.2 · rlem=wish +10.2 · w1=i +7.5 · prev=narrate +5.3 — it sees the literal "I wish";
• IGTree → request, but indirect=0, path: … sayv=said → request 14/43 · you=yes → request 6/21 · quote=framed → request 6/21 · w1=i → request 3/6. A hit based on 3–6 examples is luck rather than knowledge;
• k-NN → express, because the nearest are: "…he said, ‘I wish I had someone to bring the cart after me…’" (express, 1.05) · "I wish you had been made to run the soles off your shoes!’" (express).

An indirect "I wish you would…" as a request is a job for the LLM. So far the SLM has learned only the classic "Can you…?":

Indirect request where there is none (teasing). ‘Can your name be RUMPELSTILTSKIN?’ said the lady slyly. — silver: q, ask, hedge=1; means: "she already knows his name and is teasing him".

• perceptron → request, indirect=1: wh=- +15.0 · modal=can +11.0 · you=yes +9.1; for indirect — qm=yes +11.5;
• that is, "can + you + ?", the learned cue of an indirect request, fired on "Can your name be…";
• IGTree → assert, k-NN → assert.

A perceptron error that memory fixes. How magnificent was the sight from this height! — silver: excl, evaluate, narrator.

• perceptron → decl, narrate: quote=none +21.0 · genre=tale +20.8 · sayv=- +14.2 · tense=Past +10.9. Strong narration features outweighed the rare excl=yes;
• k-NN → excl, evaluate, because the nearest is: "‘What a noble heart that good man has!’ thought he." (evaluate, 2.23).

An error of all three. I will climb up yonder rock.” — silver: promise.

• all say assert;
• perceptron: you=- +12.0 · aux1=- +8.4 · quote=cont +8.2 — the cue cue=promise ("I will") is weak, because "I will…" in the silver is more often an assertion;
• k-NN: nearest are "I can climb trees." · "I can crack shells." (assert).

What distinguishes a promise from an intention is context and addressee, not form.

Irony. …but the horse let him sing on, and made his way quietly over the fields… — silver: narrate, indirect=1; means: "the lion's furious roaring is ironically called singing".

• all three: narrate, indirect=0 (perceptron: genre=tale +23.9 · quote=none +22.8).

Irony is beyond the SLM with these features; it stays with the LLM.

More examples with full explanations from all three models — examples.md.

Conclusion by field

Honestly, by the stated criterion. The perceptron wins on quality for form, act, polarity, voice. On hedge it is a tie; on indirect nobody wins.

The little IGTree graph is faster at bare prediction (0.008 vs 0.016 ms) and 12 times smaller. But prediction for every model except k-NN is pennies next to the shared feature extraction (0.45 ms), so the speed advantage gives nothing in practice. Therefore the little graph is not pulling its weight yet.

The reason is visible in the IGTree paths. Gain ratio puts binary slots at the top (qm, inv, end, genre…), while lexical cues (w1, rlem) end up deep down, where only a few examples reach.

k-NN is almost level with the perceptron on form and useful as a second opinion. But it is 100 times slower and loses rare short acts.

Practical summary:

• the main model is the perceptron;
• memory (k-NN or IGTree) runs alongside for agreement;
• disagreement goes to the LLM.

How to run

export PATH=$HOME/.cargo/bin:$PATH
cargo build --release -p prag
P=target/release/prag; R=$MOVA_DATA/runs/prag-2026-09-26; M=$MOVA_DATA/en/models/en.bin
$P select $MOVA_DATA/db/mova.duckdb $MOVA_DATA/annot/src/tatoeba-eng-cc0.tsv $M $R   # items.jsonl, tatoeba-ud.conllu, select.md
$P prompt $R 5                          # batch prompt without a call
$P silver $R --batches 5,37 --jobs 2    # pilot; then without --batches — the rest (finished batches are skipped)
$P gate $R                              # gates only: silver.tsv, rejects.tsv, silver-params.json
$P eval $R $M --gum $MOVA_DATA/en/ud/ud-treebanks-v2.18/UD_English-GUM/en_gum-ud-test.conllu   # report.md, results.json, examples.md, rust-*.tsv
D=target/release/db; B=$MOVA_DATA/db/mova.duckdb
$D migrate $B
$D load $B bronze ud-2.18 --kind prag --docs idprefix --match auto --run prag-2026-09-26:tatoeba-ud --origin prag-2026-09-26/tatoeba-ud $R/tatoeba-ud.conllu
$D prag $B $R/silver.tsv --run prag-2026-09-26 --source silver --tool opus --origin prag-2026-09-26/silver --params $R/silver-params.json --path $R
$D prag $B $R/rust-perceptron.tsv --run prag-2026-09-26:perceptron --source rust --origin prag-2026-09-26/perceptron --params $R/results.json   # likewise igtree, knn
$D sql $B sql/silver-vs-rust.sql

Variables: PRAG_MODEL (claude-opus-5-5), PRAG_EFFORT (high), DUCKDB. Tests: cargo test --release. They check:

• gates and closed sets;
• quotation marks;
• selection by whole paragraphs;
• the three models on a rule and the negative control with shuffled labels;
• the licence gate.

What next

1. Routing to the LLM: model disagreement plus a low margin → an Opus request. Measure the trigger's precision and recall, and its token cost per thousand sentences.
   • Flywheel. At some point the third will start to shrink. LLM answers on disagreements become SLM training data, so disagreement decreases. We measure the share of LLM calls per thousand sentences across versions — this is also the "data-centre offloading" curve. It will not reach zero: new domains and rare phenomena.
   • Agreement audit. Agreement hides errors: on the act field 15.5% of agreed sentences are wrong. So a small random sample of agreed sentences also goes to the LLM. Otherwise the SLM will learn errors on which all three models agree: examples are not truth (Smart annotator — layers, context, memory, "Small model and Opus: division of labour").
2. Memory that pulls its weight: TRIBL2 (IGTree on top, IB1 below) or a different slot order in IGTree. For the perceptron — feature pairs (aux1×sperson, rform×you).
3. A second silver wave for rare acts (thank, apologize, greet, promise, offer, warn): selection by cues, ~2–3k sentences. Only on assignment: this is an expensive run.
4. polarity: evaluative words from the silver itself (pos/neg shares per lemma), negation scope over the evaluative word (neg × eval).
5. Align frag/intj with GUM and maybe add multiple.
6. A shared LLM-call crate for annot and prag (src/opus.rs is already generic).

Licence

Elamur Cosmic License (Elamur Cosmic License). Third-party data keeps its own terms, as noted where it is used.

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/prag/