bench 1.0.0 · generated 2026-11-10
Scored against Word, failures included.
The oracle is Word’s own PDF export of each document (SHA-pinned). A converter’s PDF is scored 0–100 by pixel similarity; redlines are additionally opened in Word and compared against Word’s own compare. jubarte is author-affiliated (†) and plays by the same rules. Medians and means count a failed document as 0, so no tool gains by skipping a hard one.
Redline vs Word compare · median, the 600-pair set
83.67vs docxodus 15.0.0 80.37
600 pairs of the fixed redline set (set 2026-11-03.1), one Word compare each; paired 95% CI of the difference [0.0, 0.3412]. Set list and shas: release_info/sample_redline_0.18.0_11-10-26_19-21.csv.
DOCX → PDF · median, the 860-document set
81.72vs LibreOffice 26.8.2.2 9dea406ab7695df6325b11b0e19fdca8db4b7276 55.39
860 documents of the fixed conversion set (set 2026-11-03.1); 0 jubarte failures. Set list and shas: release_info/sample_conversion_0.18.0_11-10-26_19-21.csv.
Redlines vs Word's compare — the 600-pair set
600 pairs · one Word compare each · pixel scorer · opened in Word · set 2026-11-03.1 · 2026-11-10The fixed 600-pair redline set (every path and sha256 in release_info/sample_redline_0.18.0_11-10-26_19-21.csv): every jubarte.pro case pair plus pairs drawn from the release sample, state-balanced, one Word compare each. Each tool redlines the pair; Word opens that redline and exports it to PDF, scored against Word's own compare. jubarte is scored by this run, intent-to-treat — a pair without a scored PDF counts 0; the comparators are the vendor refresh's runs on the same set, the newest version per tool, and a pair a comparator did not run is not run, never 0.
Redlines accepted or rejected by Word
Each tool redlines a pair; Word accepts — or rejects — every change in that redline and exports the result to PDF, scored with the pixel scorer against Word's own compare of the pair resolved the same way. A pair with no scored PDF counts 0 (intent-to-treat); the calibration row is Word's own compare against itself.
DOCX → PDF vs Word's own PDF — the 860-document set
860 docs · 380/375/22/83 by state · set 2026-11-03.1 · scorer torch-mps · 2026-11-10The fixed 860-document conversion set (every path and sha256 in release_info/sample_conversion_0.18.0_11-10-26_19-21.csv): each converter turns the same DOCX into PDF, scored pixel-wise against Word's own export of the same file. jubarte converts and is scored by this run, intent-to-treat — a document it fails scores 0; the comparators are the vendor refresh's runs on the same set, the newest version per tool, and a document a converter did not run is not run, never 0. Sorted by median; 95% CI is a percentile bootstrap of the median (2,000 resamples, seed 42).
Source DOCX → PDF, no redlines
Document set `c947065c59b2` (398 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.6`.Each converter turns the clean source DOCX documents, with no tracked changes in them, into PDF; its PDF is scored pixel-wise against the SHA-pinned PDF Word itself exported from the same source. A failed conversion scores 0 (intent-to-treat).
The same documents under docxide-pdf's own metrics
Document set `a89ffc217bba` (6427 documents), renderer `oracle:microsoft_word`, scorer `docxide-150dpi`, bench `1.0`.The same documents as the corpus-wide table, scored instead with docxide-pdf's own ink Jaccard (the ranked number) and text-boundary metrics at 150 DPI against the same Word PDF exports, so no converter is measured only by this bench's pixel scorer.
Below 3 pages: every converter on the same short documents
Document set `27122f4d945e` (2626 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.7`.The corpus-wide conversion benchmark restricted to the documents Word lays out in 1 to 3 pages, so a converter that renders only the first 3 pages (unlicensed PyMuPDF Pro) is compared with every other converter on the same documents. The oracle stays Word's own PDF export of each document, scored pixel-wise; a failed conversion scores 0 (intent-to-treat).
By corpus state, the 860-document conversion set
median · documents per state in bracketsclean (380)
tracking, no comments (375)
comments, clean (22)
comments + tracking (83)
Per-case viewer
Open this case full-size →Every published case is browsable: Word’s page next to each engine’s page, with overlay, difference and multiply modes, page-count mismatches, the score table and the files themselves on Hugging Face. Below is page one of tracking_without_comments__dc2d9eeccb_file_40__vs__42df4a13aa_file_41_redline_cb0c07e105 (2 pp in Word), a case whose jubarte score sits at its published median.
Oracle: real Word
Every reference PDF is exported by Microsoft Word and SHA-pinned. Redlines are opened in Word too: if Word refuses the file, the tool scores 0 for that pair.
Intent-to-treat
Failures are not dropped. A crash, a timeout or an unopenable file counts as 0 in ITT mean and median, so a tool cannot improve its rank by skipping hard documents.
Paired bootstrap
Ranks tie (n=) when the paired bootstrap interval of the median difference to the row above includes 0. 2,000 resamples, seed 42.
Fixed sets
From 0.18.0 the tables on this page are the bench's fixed evaluation sets (set version 2026-11-03.1), the same documents for every release; every row's inputs are sha256-listed in the engine's release_info/. The comparators are the vendor refresh's runs on the same sets, newest version per tool — a tool the refresh did not run shows as not run, never 0. Full-corpus runs remain in the bench's RESULTS.md.
Author-affiliated, same rules
jubarte is marked † everywhere. Pins, run dates and sample lists are published so any row can be re-run; the harness is public.
Figures are project-maintained and reproducible, not a neutral third-party benchmark. Competitors move: Docxodus in particular has evolved since several stamps above, so read the pins. The two headline tables are this release’s 600-item samples — state-balanced, every file’s path and sha256 listed in the engine’s release_info/ — scored on 2026-11-10. The remaining tables and the full-corpus runs live in RESULTS.md.