hbo

Mova's own parser for Ancient (Biblical) Hebrew: raw text → words → trees. macula gives trees only for the text MACULA analysed; hbo reads Hebrew that MACULA never saw (quoted in papers, commentaries, other books).

raw text → tokens (spaces, maqaf, sof pasuq) → segmenter (seg.tsv) → gparse (GPU) → CoNLL-U, hbo

Data

macula converts the whole Hebrew Bible (all 39 books of MACULA's WLC lowfat files): 22 945 Hebrew sentences, 468 356 MACULA words. The Aramaic chapters (Daniel 2:4b–7:28, Ezra 4:8–6:18 and 7:12–26, Jeremiah 10:11) go to arc/, 268 sentences and 7 555 words, and are not read here. A sentence goes by the language of most of its words, so Genesis 31:47 (two Aramaic words in a Hebrew verse) stays Hebrew; 2 sentences mix the two.

hbo split holds books out whole (2026-10-18):

The test covers all of PTNK's test split (Genesis 19–30 and its verses of Deuteronomy), so PTNK test can be read through MACULA ids. FORM is normalized to a level of pointing: pointed drops the accents (teʿamim), meteg and rafe and keeps the vowels; unpointed keeps letters only. XPOS is MACULA's Type (the verb form: wayyiqtol, qatal, yiqtol, weqatal, imperative …; the kind of a particle), or the UPOS where it has none.

The segmenter

A written word is cut into MACULA's words: prefixes (ו ה ב כ ל מ ש …), the host, suffixes (pronoun suffixes, the directional and paragogic ה and נ). A split is a list of pieces, each a number of letters (a letter with its marks) or H, the article a preposition swallowed (בָּאָרֶץ = בְּ + ה + אָרֶץ, the ה not written). A suffix can take the vowel of the letter before it: MACULA cuts כֻּלָּם as כֻּלּ + ָם, so a piece marked ^ takes the vowels of the letter before it and leaves the dagesh.

The model is learned from the training trees (seg.tsv, plain TSV):

1. lex: every written word seen in training, with its splits and their counts. A seen word takes its commonest split.
2. pre: the first k letters of a word (k = 1 … max_k) → how its prefixes split, counted over the rarer words (seen at most rare times), which behave like unseen ones. An unseen word takes the split of its longest beginning seen at least min times.
3. rest: what follows the prefixes, as seen in training → its suffixes; if that was never seen, suf: the last k letters of it, as for pre.
4. chain, sufx, host: the prefix chains (the letters of the prefixes: ו, ו.ב, ב.H …), suffixes and hosts of training, with counts.
5. ctx: for the written words the lexicon splits more than one way, their splits by the word before and the word after.

From these tables a word gets a list of candidate splits, best first (Model::candidates); the segmenter takes the first. Every candidate must pass the consonant check: its words, the swallowed article left out, spell the written word's letters back (Naaijer 2023); a word no candidate passes stays whole. The switches, written in the header of seg.tsv (with=) or given with --with:

Every word says why it was cut so, in MISC: Seg=lex:1772/1772 (seen 1 772 times, always so), Seg=pre:וּבְ→1.1:31/33,suf:יו→^2:40/52, Seg=lex:30/41,p:אל:6/6 (the word before decided) and the like.

Parameters chosen on dev and on the two-fold check of train (hbo seg cv, books alternating between the halves), 2026-10-18: pointed max_k 4, min 10, rare 2, with=hostlex:0.3,partial, unpointed max_k 3, min 3, rare 5, with=ctx:0.5. Learning twice gives the same file.

Measured (2026-10-18)

Share of written words split exactly right; «unseen» are the words not in the lexicon.

Word F1 (pieces as spans) on test: pointed 98.33 (DEU) and 97.87 (GEN 19–30); unpointed 95.93 and 95.19. The consonant check fails on no word of dev, test or the two-fold check: it guards cut.

What each switch gives (dev / two-fold check of train, split right):

The errors split two ways (Seker et al. 2020): the gold split not among the candidates at all, or among them but not chosen. On test the second kind is three quarters of the errors (pointed DEU 62 missing / 175 not chosen; unpointed DEU 165 / 494), and the gold split is among the first two candidates for 97.6–98.9% of the words (dev, cv). The commonest errors pointed are a ו or ל taken for a prefix in an unseen word or the reverse; unpointed, the swallowed article (1.H.3 → 1.3): without points the lexicon itself is ambiguous (בבית: בְּ + בַיִת or בַּ + ה + בַּיִת), which ctx addresses.

Partly pointed text (dev with a share of the points dropped, deterministically, seg eval --thin X; scored on the letters cut, as a dropped vowel hides which suffix takes it): points dropped 0 / 25 / 50 / 100%: pointed model alone 97.48 / 81.82 / 69.28 / 48.84; with partial and the unpointed fallback 97.63 / 95.37 / 94.61 / 94.21; the unpointed model 94.21 throughout.

Ablations of the tables on dev (switches off): without rest, pointed 96.93 (−0.45) and unpointed 93.05 (−1.16); without the lexicon (rules only), pointed 87.97 (−9.41) and unpointed 84.29 (−9.92).

Several segmentations (lattice)

hbo lattice writes each sentence in several readings: the segmenter's best, and per written word each other candidate among the first k (3) with at least ratio (0.05) of the best's share. gparse annotate with GP_SCORE=1 adds each reading's tree score (# gp_score: the log-probabilities of head and relation and half those of the two tags). hbo choose gives each word the candidate whose reading gains most, tree gain + λ · log share ratio + μ · words added (--study writes every reading's gains against gold), and the text is parsed again.

It does not pay (2026-10-18, end-to-end LAS): unpointed dev 77.09 → 76.42 … 77.39 over λ, μ; checked on 1–2 Chronicles with the layers models, pointed 85.24 → 85.10 … 85.27 and unpointed 79.73 → 79.58 … 79.99, the setting best on dev −0.10. On dev a gold candidate gains +0.13 tree score on average and a wrong one replacing a gold base −0.41: the parser, trained on gold words only, scores a wrong segmentation nearly as confidently as a right one. The commands stay for study; nothing uses them by default.

The parser

gparse (the GPU parser, gparse — the SLM parser trained by gradient descent on the GPU) is trained on the normalized trees as they are; its predicted XPOS carries the verb form. GP_LEX (the training trees, lex.conllu) gives LEMMA and FEATS for a word whose form and tags were seen, the commonest FEATS of its tags otherwise.

GP_LR=0.002 GP_TOKENS=1000 GP_EPOCHS=30 GP_SEED=42 gparse train pointed/gparse data-pointed/train.conllu -- data-pointed/dev-RUT.conllu data-pointed/dev-JON.conllu

30 epochs of about 100 s each on the shared RTX 4000 (jobs run --mem-gb 8, nice); the best dev epoch was the 29th.

Measured (2026-10-18)

On the test books against MACULA's converted trees. «Gold words»: the parser gets the gold words (LEMMA … DEPREL cleared, hbo blank). «End to end»: the written words cut by our segmenter (hbo resegment), scored as F1 with words aligned within each written word. LAS counts the full relation (with subtypes).

XPOS (with the verb form) is 98.24 on Deuteronomy and 97.19 on Genesis 19–30. gparse eval gives the same UAS and LAS on gold words.

Without points (unpointed, the same settings; best dev epoch 22), LAS:

What the segmenter's switches give end to end (test read once, 2026-10-18; parsed on the CPU, which gives the same trees as the GPU here), LAS:

On gold words, losing the points costs the parser only 2.2–2.4 LAS; end to end it costs 7.1–7.6, because the segmenter is the weak step without points (words F1 counts here only the written words split wholly right).

PTNK test (UD 2.18, CC BY-NC, read once), through MACULA ids (macula compare, LAS on the relation without its subtype), gold words: UAS 91.53, LAS 87.95 on 13 004 words. The rule-based conversion itself scores UAS 96.51, LAS 93.58 there (macula): the parser loses about 5 points against the trees it learned from.

By layer of the language (2026-10-18)

Biblical Hebrew was written over a thousand years in different grammars, so the scores are given by layer. A second model (layers-pointed, the same settings, its own segmenter) is trained without the archaic poems, without Chronicles (the whole of that late book unseen), without Amos and Micah, and without Genesis 19–30; Ezra–Nehemiah, Esther, the Hebrew of Daniel and Ecclesiastes stay in training. Gold words / end to end, LAS:

The segmenter column is the share of written words split right. Reading it:

• Classical prose is the parser's home: 90–91 LAS on gold words, 87 end to end.
• A late book never seen loses about 2.5 points against classical prose, and nearly all of it is in 1 Chronicles, whose genealogies and lists are built unlike narrative; 2 Chronicles, a narrative that retells Samuel–Kings, is as good as Genesis.
• Prophecy loses 6 points, archaic poetry 7.5; Judges 5, the oldest and hardest poem, 17. Psalm 18 and 2 Samuel 22 are the same poem twice and score highest among the poems, the only ones near prose.
• Archaic poetry gets the fewest seen words (22.7% of its written words unseen, against 12% in prose) and the lowest segmentation.
• FEATS fall with the layer too (prose 95, poetry 90): an unseen form gets the commonest FEATS of its tags.

The layer table is read with the layers model's own segmenter without switches. 1–2 Chronicles also served to check the segmenter's switches and the lattice (2026-10-18), so it is no longer an untouched measure: with hostlex:0.3,partial the pointed segmenter splits 98.37% right there and end to end LAS is 85.24 (from 84.96). A third model, layers-unpointed (the same held-out parts without points; best dev epoch 29), reads Chronicles end to end at LAS 78.43, and 79.73 with ctx:0.5.

Two runs of annotate give the same files. Parsing runs at about 35 000 words/s.

Run

macula <lowfat-dir> $MOVA_DATA/models/hbo/trees
hbo split $MOVA_DATA/models/hbo/trees $MOVA_DATA/models/hbo/data-pointed pointed --dev RUT,JON --test DEU,GEN.19-30
hbo seg learn pointed $MOVA_DATA/models/hbo/pointed/seg.tsv data-pointed/train.conllu --max-k 4 --min 10 --rare 2 --with hostlex:0.3,partial
hbo seg learn unpointed $MOVA_DATA/models/hbo/unpointed/seg.tsv data-unpointed/train.conllu --max-k 3 --min 3 --rare 5 --with ctx:0.5
hbo seg cv unpointed data-unpointed/train.conllu --max-k 3 --min 3 --rare 5 --with ctx:0.5
hbo seg eval pointed/seg.tsv data-pointed/test-DEU.conllu [--without lex,rest] [--with …] [--fallback unpointed/seg.tsv] [--thin 0.5]
HBO_GPARSE=<path to gparse> hbo parse $MOVA_DATA/models/hbo in.txt out.conllu

hbo parse takes one or more sentences a line; a sentence ends at sof pasuq (׃) or «.», «?», «!». A model dir holds seg.tsv, gparse/ and lex.conllu; given the dir above both, it takes pointed/ when the text has vowel points and unpointed/ otherwise. The output keeps the written words as multiword tokens (forms as written, accents included), with Seg= and Swallowed=ה in MISC.

The crate is also a library (hbo::load_seg, hbo::segment_line, hbo::annotate, hbo::lattice, hbo::text, hbo::seg, hbo::conllu): roundtrip mixed parses its Hebrew pieces through it, with the same segmenter switches and fallback as hbo parse.

Other commands: hbo segment (words only), hbo resegment (gold sentences cut by the segmenter; --with, --fallback), hbo lattice and hbo choose (several segmentations, above), hbo blank, hbo merge (a parser's columns back onto a file with multiword tokens and MISC), hbo score, hbo pick (the sentences of some books or chapters).

Models and data live in hbo (trees/, data-*/, pointed/, unpointed/, layers-pointed/, layers-unpointed/, runs/, log/), not in git; the segmenters before the switches are kept beside them as seg.before-2026-10-18.tsv.

Limits

• Gold is our conversion. The trees are read off MACULA's groups by macula's rules; the parser learns those rules' habits along with the language.
• Sentences are MACULA's. Raw text is cut at sof pasuq; a verse MACULA splits into two sentences is one here.
• Unseen words. One written word in seven on test is unseen in pointed form; the segmenter gets 82–88% of them right, and the parser has only hashed prefixes and suffixes for them.
• FEATS by lookup. FEATS are those of the commonest analysis of the form with its predicted tags; an unseen form gets the commonest FEATS of its tags, a guess.
• Pointing. The pointed model expects Tiberian vowels. Partly pointed words are segmented by partial and the unpointed fallback (94–95% split right with a quarter to all of the points gone), but the parser still sees their forms as written: a form whose points the pointed parser never saw is an unseen word to it. A text pointed by other conventions (modern Hebrew) is not handled.
• One reading. The segmenter commits to one split per written word; letting the parser choose among several did not pay, because the parser was trained on gold words only.
• Licence. The trees come from MACULA Hebrew (CC BY 4.0); PTNK (CC BY-NC) is only a measure.

Licence

Elamur Cosmic License (Elamur Cosmic License). MACULA Hebrew: © Biblica, Inc., CC BY 4.0.

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/tools/hbo/