📘 Data, libraries and licences

This page covers where Mova's data lives, the licence rules every source passes through, the open mathematics and science library and its fetch tool, the Wiktionary dumps and their tools, the paper cards and how a paper's ideas reach the code, the manual usage log, the evaluation lists in eval/, and the competitions Mova can enter.

Where data lives

The repository holds code, rules, seeds and small data. Everything large lives outside it, under $MOVA_DATA, and so do run results:

Every download is recorded in a manifest in the tools/sources format, one JSON line per item: size, md5, sha256, license, license_url, key, content_type, fetched, and for compacted sources also orig_size, orig_sha256, text_files, dropped_files. sources verify DIR MANIFEST checks size, md5 and signature. Raw catalog pages and XML are kept next to the data, so catalogs can be rebuilt without the network.

Licence rules

Data and model licences is the one inventory of source terms and release rules, and the saved-model training gate is en::license::commercial_gate. Only legal open sources are fetched: no shadow libraries, no lent books, no datasets taken down by DMCA.

Licence classes used by the fetch tools (raw::License for arXiv, books::BookLicense for textbooks):

Negative controls in the tests: NC, ND, standard arXiv and unknown licences do not pass, and neither does the build gate.

Mathematics and science library

An open library of mathematics, statistics, physics and chemistry at university-student level, for the SLM's mathematical skills. Student-level textbooks come first, then papers that teach solving problems: olympiad and competition mathematics, math.HO, symbolic computation (cs.SC), student-level methods, number theory, combinatorics, algebra, inequalities, geometry and analysis. LaTeX and text are preferred; PDF is taken only where nothing else exists.

Sources

Adjacent arXiv categories need no separate harvest: arXiv aliases already place them in mathematics (math-ph ≡ math.MP, stat.TH ≡ math.ST, cs.IT ≡ math.IT, cs.NA ≡ math.NA). cs.DM and cs.LO without a mathematics cross-listing are not harvested; --set cs:cs:DM adds one.

Selection rules

• Only the open class is downloaded (for textbooks also CC BY-SA and GFDL, as the copyleft class).
• A book whose sources disagree on its licence (one says closed terms, another an open licence) is not downloaded.
• Lab manuals, workbooks, answer keys and companions are filtered out of textbook PDFs; pre-university level (pre) is not downloaded. Solutions manuals of strong books may be added by hand (manual-list.tsv).
• Major current textbook editions often moved to CC BY-NC-SA; then only the earlier open editions qualify.
• The arXiv extension list (sql/extend.sql) takes open papers with sources up to 30 MB, in tiers: problem solving and competitions, math.HO, cs.SC and student-level methods, math.GM, surveys and elementary papers, then the remaining short papers.
• Problem sets with answers for evaluation (GSM8K, SVAMP, AQuA-RAT, MathQA, MMLU mathematics, TheoremQA, ProofNet, Lila, DeepMind Mathematics, open problem books) are checked for licence before use. Competition-derived sets whose compilers do not hold the problems' rights (Omni-MATH, OlympiadBench, AIME collections, NuminaMath) are at most for internal evaluation with a provenance mark, by explicit decision. hendrycks/competition_math (MATH) was taken down by DMCA; it and every copy or derivative (MATH-500, PRM800K, MetaMathQA and others) are forbidden. Lila must be checked for MATH problems before use.

tools/mathlib

B=target/release/mathlib; M=$MOVA_DATA/raw/math; DB=$MOVA_DATA/db/math.duckdb
# 1. metadata, subset by subset (state in state.tsv; a rerun resumes; a STOP file pauses)
$B oai $M/arxiv-oai/math-AG --set math:math:AG        # likewise for every math.* subset and cs:cs:SC; --from DATE for updates
# 2. catalog: arXivRaw -> TSV -> DuckDB; textbooks
$B parse $M/arxiv.tsv $M/arxiv-oai/math-?? $M/arxiv-oai/cs-SC
$B books-fetch $M/textbooks/catalog                    # or otl:<subject> (physics, chemistry ...)
$B books $M/textbooks/catalog $M/books.tsv && $B books-pdf $M/textbooks/catalog $M/books.tsv
$B sql $DB tools/mathlib/sql/load.sql arxiv=$M/arxiv.tsv books=$M/books.tsv "files=$M/*/manifest*.tsv"
$B sql $DB tools/mathlib/sql/stats.sql --mode markdown
# 3. download: list -> polite fetch -> manifest -> verification
$B sql $DB tools/mathlib/sql/books-pick.sql out=$M/books-list.tsv
$B fetch $M/books-list.tsv $M/textbooks $M/textbooks/manifest.jsonl --delay-ms 2000 --max-bytes 2000000000 --expect-pdf --kind textbook
$B sql $DB tools/mathlib/sql/extend.sql out=$M/arxiv-src/list.tsv gcs_out=$M/arxiv-pdf/list.tsv
$B fetch $M/arxiv-src/list.tsv $M/arxiv-src $M/arxiv-src/manifest.jsonl --delay-ms 3500 --compact-src
target/release/sources verify $M/textbooks $M/textbooks/manifest.jsonl

How it behaves:

• Licences. fetch downloads only allowed licences. A list with any other licence is an error before the first request.
• Politeness. OAI-PMH: one connection, at least 3 s between requests (the tool pauses 4 s), Retry-After respected on 503; 403 or 429 stops. The first request to every host reads its robots.txt (RFC 9309: the * group or ours, longest match, */$, Crawl-delay as the per-host pause; 4xx means no restrictions, 5xx closes the host). A 403 or 429 closes the host for the rest of the run. No blind retries.
• No GitHub. robots::github_host refuses any GitHub host by name or by address (GitHub Pages under custom domains), before any HTTP request; the textbook catalog marks such books github=true.
• Safeguards. Free disk space is checked before the start and every 10 files (--min-free-gb, default 150); --max-bytes caps the total and --file-max a single file; a STOP file next to the manifest pauses before the next file, and a rerun does not download existing files again.
• Compact sources. --compact-src keeps only text files (.tex, .bbl, .bib, .sty ...) from a source archive and drops figures.
• Log. Every row is appended to manifest.log: ok, miss, bad-signature, cap, stop-disk. The manifest is checkpointed every 100 files.
• resumptionToken expires at midnight UTC; on badResumptionToken restart that subset. parse merges cross-listed duplicates by id, keeping the later datestamp.
• books adds a science column (math, statistics, physics, chemistry, other); BOOKS_ALL=1 also includes OpenStax science books.

Papers and books never go into git; only the code and this page do.

Not collected

• LaTeX sources of open mathematics papers in bulk: needs the S3 route (ranged GETs over tar headers, or filtering inside the same AWS region), an explicit decision.
• More open books from OAPEN (Springer Open Access, OBP), catalogued by Thema classification and MARC licence.
• LaTeX sources of textbooks published outside GitHub.
• Physics and chemistry textbooks, statistics papers (JSS, R Journal), CODATA and the PubChem periodic table: their licences are checked and the lists are ready.

Wiktionary

Wiktionary is Mova's first open dictionary source: all languages, English and Ukrainian first. Everything is built from the raw dumps; ready-made parses (Kaikki, wiktextract) are not used.

Data. <run> mirrors the dumps.wikimedia.org layout, one snapshot run for all editions; provenance, sha1 and reproduction commands are in SOURCES.md beside it. The format is pages-articles-multistream with an offset:page_id:title index. It holds articles, templates (ns 10) and Lua modules (ns 828): everything needed to expand templates ourselves. Streams hold 100 pages, so a single page can be fetched by title and the dump can be parsed in parallel. Large editions are split into parts, each with its own index. Processing is streaming only; decompressed XML is never written to disk. The licence is CC BY-SA 4.0 plus GFDL, so derived data falls under the CC BY-SA rule above.

Tools.

Article structure. Grammar is recorded in template parameters, not in the text, so a parser must read template calls with their parameters and, for inflection, reproduce the logic of templates and Lua modules rather than parse the generated tables.

• English Wiktionary puts every language with a given spelling on one page (==Ukrainian==, ==Russian== ...), then ===Etymology N===, ====Noun==== and so on. Inflection templates ({{uk-noun|...}}, {{uk-ndecl|...}}) pass a specification in <...> to a Lua module that derives the paradigm from the stressed lemma. Pronunciation, etymology, examples and labels are templates too ({{uk-IPA}}, {{inh}}, {{ux}}, {{lb}}). Large articles move translations to a <word>/translations subpage ({{t}}, {{t+}}, {{tt}}, {{tt+}}, {{t-check}}).
• Ukrainian Wiktionary uses one = {{-uk-}} = block with ===...=== sections. Inflection follows Zaliznyak's scheme in a noun template (declension type, gender, animacy), with separate templates for morphemes, transcriptions with stress and audio, examples, labels, related words and translations.

The two editions encode the same grammar differently, so cross-checking them is a ready quality control.

The English module. wikt-extract reads the English entries of the English edition from the raw dump and writes the add-on module wiktionary-en (Add-on modules): a dictionary layer in the school layers' six columns plus labels, synonyms, antonyms, forms, form-of links, template gaps and provenance (page id, revision id, dump file, stream offset), the list of templates it had no handler for, a report and module.json. Each dump stream is decoded on its own from the index's offsets, on a fixed number of threads; the rows are sorted, so the layer does not depend on the thread count. Templates are emulated, not executed: a template with no handler leaves its text out and is counted in the row and in gaps.tsv, never dropped silently. The module is CC BY-SA 4.0 and GFDL like its source; it lives in <version>, is distributed apart from Mova, and the tool refuses to write it inside a Git work tree.

wikt-extract check validates a built module and cover reports how much of a word list it covers; extraction counts, resource use and template-gap reports stay with the run. The layer is sorted, so it does not depend on the thread count. A dictionary entry or an acronym match does not establish that its sense fits the document at hand. What the module changes in the pupil: building the dictionary.

Other editions. The stream reader, the template parser and the row format serve every edition; what differs is the section and template vocabulary, so each edition needs its own entry reader and template handlers.

Paper cards

Every paper, book or manual Mova reads gets a short card, so that what the literature says reaches the grammar (languages/<code>/) and the code. The cards live in the Research library (its cards/ folder) and are published on research.elamur.ai; this repo keeps only the lists of the works Mova uses: Papers Mova uses (the main list, each row linked to its card there) and papers/measured.tsv (what came of each measured work), with papers/aliases.tsv for merged duplicate cards.

Where. Cards Mova already uses in code are at the top of the cards folder. Other cards are in <code>/<slug>.md, one folder per language or group: en (English and Germanic), hbo (Ancient Hebrew), he (Modern Hebrew), arc (Aramaic, Syriac, Neo-Aramaic), sem (general Semitic and Arabic morphology), and the topic folder mtrans (machine translation). The slug is the library's: <venue-or-lang>-<year>-<first author>-<title words>.

Format.

# <Title>

**Authors:** ... · **Year:** ... · **Venue:** ...
**Link:** <open URL or library path> · **Licence:** <CC BY 4.0 / public domain / copyrighted — summary only>

## What it is about
Three to six sentences in our own words: the question, the data, the method, the result.

## Ideas
- I1. <one idea: what could be done, and where>
- I2. ...

## For Mova
Where the ideas would apply: a chapter of `languages/<code>/`, a crate or tool, a measure to run.

Rules.

• Own words only. Quote at most a short phrase, with its page. A copyrighted paper is summarised, never copied.
• An idea is something that can be tried, measured or written into the grammar, not a restatement of the abstract. Ideas are numbered I1, I2 ... within the card.
• A paper with nothing to take still gets a card, with "Ideas: none for Mova" and why.
• The card says what the paper says. Whether an idea worked is recorded where it is tried: in that commit and in the chapter or README it changes.

How ideas reach the code. Where code applies a paper's idea, a comment cites the card and idea, for example (ning-2020 I3). The main list names each work's role for Mova (implements, measured, rules, background).

Hubs. Pages per concept, per author and per cited card are generated from the library's metadata, citations and cards, never stored in this repo: research.elamur.ai publishes them, and an internal tool can write them as Markdown into a build output. Two runs on the same inputs give the same bytes.

The usage log

usage.tsv is the manual log of where papers and books are applied. It holds only what the repository text does not show as a reference: a work mentioned only by title or by author and year, a method without a reference, reading that influenced a decision. References already in the text (ACL ids, papers/... and books/... paths, aliases from tools/papers/aliases.tsv, "author year" with a work in the library) are collected automatically and do not belong there.

Columns, tab-separated: date (YYYY-MM-DD), work (ACL id, library folder, or "outside the library: author year title"), where (repo file, optionally #section), how (rule source, knowledge source, bar or metric, method, data, format, precedent, review, mention), who, note. A bad row (unknown id, folder, file or section; bad date, how or who) stops the collector before any output is written. The log feeds back-citations, the bibliography and system papers.

Evaluation lists

eval/ holds small lists that qualify an evaluation, such as stepgame-excluded.tsv: StepGame test questions whose story contains a statement contradictory in the training data, so a clean score can leave them out. The benchmark files themselves stay outside the repository under their own licences.

Competitions

Mova enters linguistic shared tasks where possible. Candidates, each with papers in the library:

Rules that concern Mova:

• The closed track accepts only the competition data; the open or LLM track accepts any resources. The SLM alone goes to the closed track; the SLM with a large model goes to the open one.
• Parsing is measured with the CoNLL 2018 script (LAS, MLAS, BLEX), so scores are comparable.
• Competition data is for research and sometimes has NC parts. The competition build is separate and never enters the production model; commercial_gate stops it.
• A system paper is usually required; it is written from the usage log.
• Calendars and deadlines are checked when the time comes.

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/repo-docs/data/