parallel — parallel texts in one TSV (English–French, English–Ukrainian, French–Ukrainian)

Converts free parallel corpora to one aligned TSV (id, en, fr, prov) and aligns Project Gutenberg book pairs. Plain Rust, deterministic (two runs give identical bytes); the only dependencies serve tr-run (mova-teacher, serde_json).

Build and run

cargo build --release -p parallel
parallel moses   <en.txt> <fr.txt> <out.tsv> <source-tag>      # OPUS/Moses line-aligned files
parallel tatoeba <links.tsv> <eng.tsv> <fra.tsv> <out.tsv> [la lb]  # Tatoeba exports (bzcat them first)
parallel bible   <en-vpl.txt> <fr-vpl.txt> <out.tsv> <tag> [la lb]  # verse-per-line files; chapters with different verse counts go to <out>.mismatch
parallel align   <out.tsv> <tag> <en.txt[,..]> <fr.txt[,..]>   # Gutenberg texts: chapters, paragraphs, sentences
parallel choices <list.tsv> <out.tsv> <examples.tsv> <la> <lb> <corpus.tsv[@count]>...   # count listed renderings, pick and verify examples
parallel choices-md <counted.tsv> <examples.tsv> <manual.md> <out.md> <date>   # render the entries into a manual
parallel choices-queue <out.tsv> <pair>:<list>=<counted.tsv>...   # the entries that still need an example
parallel tr-run <queue.tsv> <translation-dir> <work-dir> <date> [--teacher terra|sol] [--pair P] [--calls N] [--batch 20] [--parse-bin P --parse-en P --parse-fr P] [--verify hard|all|none] [--dry-run]
parallel tr-report <queue.tsv> <translation-dir> <work-dir>      # accepted/rejected/left per pair and teacher, tokens, rejection reasons

tatoeba and bible take two optional column labels (en uk, fr uk; default en fr): they name the text columns of the header and the keys in prov, so the same converters build the English–Ukrainian and French–Ukrainian corpora in en-uk and fr-uk/. Without them the output is byte-identical to before (checked on the Tatoeba EN–FR pairs, 2026-11-06).

Existing output files are never overwritten. Data and per-source READMEs live in en-fr (not in git).

How the aligner works

Project Gutenberg boilerplate is cut at the *** START/END OF markers. Headings (CHAPTER, CHAPITRE, Roman numerals, ...) split the books into chapters; when both sides have the same number of chapters they are paired in order, otherwise the whole book is one block. Inside a block, paragraphs are aligned with the Gale–Church length model (1993): beads 1-1, 1-0, 0-1, 2-1, 1-2, 2-2, 3-1, 1-3 with their priors, a banded dynamic programme, and the ratio of the two sides' total length as the length constant. Each aligned paragraph bead is split into sentences (abbreviation list for Mr., M., Mme, initials) and aligned the same way. Every row keeps para_bead, para_p, sent_bead, sent_p; consumers filter on them (the corpus README names the filter used for pairs-good.tsv).

Cost: O(n · band) per block, memory O(n · band) bytes; Les Misérables (12,000 paragraph beads) aligns in seconds.

Counting renderings: `choices`

choices serves the false-friend lists (languages/translation/false-friends/<pair>.tsv) and the context lists (languages/translation/context/<pair>.tsv). Each list row names a source word (src_match), optionally a context the source sentence must also contain (ctx_match), the recommended renderings (good_match) and the look-alike or rendering to avoid (bad_match), as patterns: | between alternatives, spaces between consecutive tokens, * for a prefix, matched on lowercased tokens (apostrophes stay inside Ukrainian words). For every row it counts the sentence pairs with the source word (n_src), and among them those whose target side contains a recommended rendering (n_good) or the avoided one (n_bad), per corpus too (by_corpus), and sorts the rows by n_src. An empty example_id is filled with the shortest quotable row (25–220 characters) that has the source word and a recommended rendering but not the avoided one; a given id is verified (example_ok = yes, no-match or missing). A corpus given as path@count is counted but never quoted (learn/test-only sources); Tatoeba rows with an unknown author are never quoted. It also writes frequency_rank (1 + the number of rows with a higher n_src), example_status (corpus for a verified id, constructed for an id ours-tr:<pair>:<n> of our own translated sentences, needs_example) and keeps the authored difficulty (plain or hard:<reason>; empty becomes plain). Rows whose text carries placeholder characters (DGT has Surname: 愦灭※) are never quoted. The examples file keeps the pair's column order (id, <la>, <lb>, prov), so one row can serve entries of both directions. On a rerun a picked id is verified, so example_ok turns from picked to yes; after that, reruns are byte-identical. choices-md fills the blocks  …  of a manual with the matching rows, most frequent first, and fails when a row lands in no block or in two. choices-queue lists the rows with example_status = needs_example of several lists, with the suggested teacher (Terra for plain, Sol for hard) for the planned example pipeline; it calls no teacher.

The counts are co-occurrence, not word alignment: a recommended rendering in the same sentence pair is evidence, not proof, that it translates the source word; duplicated rows (DGT) count each time. Cost: one pass over the corpora, O(tokens) with a first-token index; the English–French corpora (about 10 million pairs) take about two minutes on one core.

Our own example pairs: `tr-run`

tr-run fills the needs_example entries of the false-friend and context lists with our own sentence pairs (queue and plan). It is the only subcommand that calls a teacher: Codex CLI through mova-teacher, no Claude API, no retry. Terra (gpt-6.1-terra, the mova-teacher default) takes the queue's plain entries and Sol (gpt-6.1-sol) the hard:* ones, both at medium effort; MOVA_TR_TERRA_MODEL and MOVA_TR_SOL_MODEL override the names.

Queue to rows. examples-queue.tsv names (pair, list, entry key, context); the n-th queue entry takes the n-th matching list row, so duplicate keys stay distinct. Entries are grouped by pair and teacher, in frequency order, and cut into batches of 20 (--batch). A batch mixes directions and lists; each entry states its own.
One call per batch. The prompt gives, per entry, the headword, context, recommended and avoided renderings, the list's note, and its patterns (src_match, ctx_match, good_match, bad_match). The teacher answers JSON {"items":[{"n","src","tgt","used","note"}]}: one sentence of its own, its translation, the rendering used, a one-line note.
Gates (every failing reason is recorded): both texts non-empty, no tabs or line breaks, script of the language, one sentence, length and length-ratio bounds, the source matches src_match (and ctx_match), the translation matches good_match and not bad_match, no duplicate source. A pattern is a form list, so an inflected form outside it is rejected; that costs yield, not correctness.
Mova's parse of the English and French sides (mova-parse, en.bin, fr.bin; Ukrainian has no model): the findings no-root, multi-root, parse-none, verb-not-root(root=X), detected-<lang> and sentence-count are recorded, never a rejection: a parse failure is a finding about the model.
Second call for the hard entries (--verify hard, default): Terra back-translates each Sol pair without being asked to match the source, then judges verdict (same, close, different), trap_ok and natural; different, a misused trap or an unnatural side rejects. --verify all also checks the plain ones, none checks nothing.
Accept. An accepted pair gets ours-tr:<pair>:<n> (one counter per pair across both lists), a row in languages/translation/<kind>/<pair>.ours.tsv (columns id src_lang tgt_lang src tgt note headword context list teacher model verify parse prov; Elamur Cosmic License) and its list row becomes example_id = that id, example_ok = ours, example_status = constructed. choices-md reads the sibling <pair>.ours.tsv and renders the pair in the entry; re-render the manuals after a run.

State lives outside the repo, in the work directory: results.tsv (append-only; the last row per entry wins; statuses accepted, rejected, held), calls/NNN-<teacher>-<gen|verify>.json (prompt, reply, token usage) and codex-calls.jsonl. A rejected entry is not requeued by a rerun. If a verification call fails after generation, the passed rows stay held and the next run verifies them first. --calls N caps the calls of one run; a batch starts only if its generation and verification fit. The first transport or limit error stops the run.

parallel tr-run languages/translation/examples-queue.tsv languages/translation $MOVA_RUNS/tr-examples-2026-11-06 2026-11-06 --teacher terra --pair en-uk --calls 1
parallel tr-report languages/translation/examples-queue.tsv languages/translation $MOVA_RUNS/tr-examples-2026-11-06

Checks: unit tests with a negative control for every gate (a gate that cannot fail is not a gate), the reply decoder, the parse-finding reader, choices-md with our pairs (and a pair of a foreign language refused), and two tests on the real lists: every list rewrites byte for byte, and the whole queue resolves to distinct rows. A run with a fake codex (MOVA_CODEX_BIN) returning one good, one wrong-context and one avoided-form answer accepted 3 and rejected 2 as expected.

Checks

Unit tests: sentence splitting with abbreviations, 1-1 on equal lengths, 2-1 when a target merges two sources, and a negative control (a 5x length mismatch scores below 0.001); for choices, tokenisation (French apostrophes split, Ukrainian kept), exact, prefix and multi-word patterns, counts and example choice on a tiny corpus, and a negative control (a wrong pre-filled example id must show no-match).

Limits

Length-only alignment: no lexical anchors, so a mistranslated or cut passage can drift until the next chapter anchor; books with different chapter counts or long notes align poorly (Candide, Zadig and The Black Tulip were measured and excluded). The dialogue dash and front matter survive in a few rows. Alignment quality is checked by eye on samples, not against a gold standard. The aligner and choices.rs have no groupant sidecars yet and are not registered in models/contracts.json: a known gap.

Sources and citations

Gale and Church 1993 (the aligner); Koehn 2005 Europarl; Tiedemann 2012 OPUS; Steinberger et al. 2006 JRC-Acquis and 2012 DGT-TM; Eisele and Chen 2010 MultiUN; Ziemski et al. 2016 UN Parallel Corpus (ziemski-2016-united-nations-parallel-corpus-v1); Schwenk et al. 2021 WikiMatrix. Data licences: Data and model licences.

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/tools/parallel/

📘 parallel — parallel texts in one TSV (English–French, English–Ukrainian, French–Ukrainian)