📘 pageviews — Wikipedia page views for every language at once

pageviews turns the Wikimedia monthly pageview_complete user dumps into per-language article view counts, so that every language's Wikipedia pass can walk its articles from the most viewed down. One pass over each monthly dump covers all *.wikipedia projects; the raw .bz2 is never stored.

Source: Wikimedia, Pageview complete dumps (https://dumps.wikimedia.org/other/pageview_complete/readme.html, data under https://dumps.wikimedia.org/other/pageview_complete/monthly/), CC0. Line format project title page_id access count hourly; documentation: https://wikitech.wikimedia.org/wiki/Data_Platform/Data_Lake/Traffic/Pageview_actor and the dump readme above. The English-only precursor is pvtitle of the Grand Spider.

How it works

1. month streams curl --limit-rate 40M into lbzip2/pbzip2 when installed, else bzip2 -dc, into the parser. The dump is sorted by project, so one project is aggregated at a time in a hash map and flushed when the project changes: memory is that of the largest project of the month. The grouping is checked on the real stream; a project that returns after it was left is an error (the tool stops; the external-bucketing fallback was not needed and is not implemented). Service titles are dropped (see below). Output per project and month, title-sorted, zstd level 6: months/<ym>/<project>.tsv.zst, rows title<TAB>page_id<TAB>views (page id 0 = unknown, as in the dump's null; views summed over access methods). months/<ym>/_stats.tsv lists per project titles, views, lines, dropped lines. The month appears under its final name only after the whole stream ended cleanly (until then it is <ym>.part).
2. merge per project: a k-way merge of the twelve monthly files by title sums views, an external sort (zstd spill runs of --chunk-mb, default 512 MB, removed afterwards) orders them, and <project>-12m.tsv.zst is written. The order is views_order: views descending, ties by title bytes ascending. summary-12m.tsv lists project, articles, total views, months.
3. ns-fetch builds data/namespaces.tsv from each wiki's action=query&meta=siteinfo (local, canonical and alias names of every non-main namespace, and the main-page title); 364 Wikipedias on 2026-11-06. It is compiled in.

Titles are those of the dump (requested title, redirects included, underscores); page_id is the target page, so redirect titles share their target's id. Titles with no page (id 0) are scanner noise and 404s: filter on page_id > 0 or resolve against the dump's article list. The Grand Spider's file counts every title the same way.

Run

cargo build --release --manifest-path tools/pageviews/Cargo.toml
jobs run --name pv-2025-10 --mem-gb 4 -- pageviews month 2025-10 --root $MOVA_DATA/views
pageviews merge --root $MOVA_DATA/views [--projects fr.wikipedia,uk.wikipedia] [--threads 3] [--chunk-mb 512]
pageviews export $MOVA_DATA/views/fr.wikipedia-12m.tsv.zst > fr-views.tsv      # title<TAB>views, plain
pageviews compare A.tsv.zst B.tsv N                                           # top-N overlap

At most two downloads at a time (the server answers 429 to more). cargo test covers parsing, grouping (with a negative control for an ungrouped dump), determinism and the external sort against the in-memory path.

Using the views order in a pass

wikicorpus graph already takes a plain title<TAB>views file (its views argument; the score is ln(1 + inbound) + ln(1 + views), redirect views are added to their article). Replace the empty/dummy views file of a French or Ukrainian graph with pageviews export <project>-12m.tsv.zst. For a pure views order read the .tsv.zst in file order (it is already sorted by views_order) and keep the titles that are articles of the dump.

Limits

• Service namespaces come from each wiki's namespace table as of 2026-11-06; a wiki founded later, or a namespace added later, falls back to the English canonical names. A main-namespace title that begins with a namespace name is dropped.
• Pageviews include automated traffic that the user agent type did not catch, and 404 titles; views are not ranks of importance. The 12 months are 2025-10 … 2026-09, as in the English file.
• Bots/spider and automated agent types are not in the user dump by design.

Measurements (2026-11-06)

Box under load 38–47 on 20 cores, everything at nice 19 under tools/jobs with a 4 GB limit; no multi-threaded bzip2 was installed, so each month is one bzip2 -dc core.

Top Wikipedias of the 12 months (articles = distinct titles, views in billions): en 38.4 M / 75.6, ja 2.9 M / 10.1, de 5.8 M / 7.9, ru 5.5 M / 7.3, fr 5.3 M / 6.7, es 4.6 M / 5.9, it 3.7 M / 4.4, zh 6.9 M / 4.3, pl 2.6 M / 2.1, pt 2.5 M / 2.0; the full table is summary-12m.tsv.

The top of a pure views order contains automated and noise titles (fr: «Cookie_(informatique)» 172 M, «XXX»; uk: «Xnxx.com», wiki.phtml with page id 0). Filter page_id > 0 and article status, or combine with in-links as the wikicorpus graph score does.

This page in Markdown → https://md.mova.elamur.ai/free/en/docs/tools/pageviews/