bench 1.0.0 · generated 2026-11-10

Scored against Word, failures included.

The oracle is Word’s own PDF export of each document (SHA-pinned). A converter’s PDF is scored 0–100 by pixel similarity; redlines are additionally opened in Word and compared against Word’s own compare. jubarte is author-affiliated (†) and plays by the same rules. Medians and means count a failed document as 0, so no tool gains by skipping a hard one.

Redline vs Word compare · median, the 600-pair set

83.67vs docxodus 15.0.0 80.37

600 pairs of the fixed redline set (set 2026-11-03.1), one Word compare each; paired 95% CI of the difference [0.0, 0.3412]. Set list and shas: release_info/sample_redline_0.18.0_11-10-26_19-21.csv.

DOCX → PDF · median, the 860-document set

81.72vs LibreOffice 26.8.2.2 9dea406ab7695df6325b11b0e19fdca8db4b7276 55.39

860 documents of the fixed conversion set (set 2026-11-03.1); 0 jubarte failures. Set list and shas: release_info/sample_conversion_0.18.0_11-10-26_19-21.csv.

Redlines vs Word's compare — the 600-pair set

600 pairs · one Word compare each · pixel scorer · opened in Word · set 2026-11-03.1 · 2026-11-10

The fixed 600-pair redline set (every path and sha256 in release_info/sample_redline_0.18.0_11-10-26_19-21.csv): every jubarte.pro case pair plus pairs drawn from the release sample, state-balanced, one Word compare each. Each tool redlines the pair; Word opens that redline and exports it to PDF, scored against Word's own compare. jubarte is scored by this run, intent-to-treat — a pair without a scored PDF counts 0; the comparators are the vendor refresh's runs on the same set, the newest version per tool, and a pair a comparator did not run is not run, never 0.

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1=
jubarte-0.18.0 †
jubarte 0.18.0
= 100: 52 · ≥ 90: 121
83.67 74.69 600 3
1=
docxodus
docxodus 15.0.0
= 100: 46 · ≥ 90: 98 · not run: 57
80.37 71.67 543 0
3
superdoc
superdoc 2.25.0
= 100: 1 · ≥ 90: 1 · not run: 533
61.89 61.98 67 0
4
apryse
apryse 12.2.1
= 100: 0 · ≥ 90: 4 · not run: 264
47.50 50.76 336 0

Redlines accepted or rejected by Word

Each tool redlines a pair; Word accepts — or rejects — every change in that redline and exports the result to PDF, scored with the pixel scorer against Word's own compare of the pair resolved the same way. A pair with no scored PDF counts 0 (intent-to-treat); the calibration row is Word's own compare against itself.

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
word-identity
Word's own compare (calibration) · accept all
= 100: 400 · ≥ 90: 400 · < 50: 0 · ink 1.00 · text 1.00
100.00 100.00 400 0
2
docxodus
docxodus 12.6.4 (npm) · accept all
= 100: 166 · ≥ 90: 299 · < 50: 16 · ink 0.85 · text 0.96
99.87 90.07 394 6
3
jubarte-rust †
jubarte 0.9.3 (673aff74) · accept all
= 100: 181 · ≥ 90: 261 · < 50: 33 · ink 0.80 · text 0.94
99.70 87.56 400 0
4
superdoc
superdoc-sdk 2.15.0 · accept all
= 100: 4 · ≥ 90: 52 · < 50: 1 · ink 0.95 · text 0.98
0.00 13.95 58 342
1
word-identity
Word's own compare (calibration) · reject all
= 100: 400 · ≥ 90: 400 · < 50: 0 · ink 1.00 · text 1.00
100.00 100.00 400 0
2
jubarte-rust †
jubarte 0.9.3 (673aff74) · reject all
= 100: 186 · ≥ 90: 260 · < 50: 35 · ink 0.77 · text 0.91
99.85 86.22 400 0
3
docxodus
docxodus 12.6.4 (npm) · reject all
= 100: 155 · ≥ 90: 275 · < 50: 26 · ink 0.80 · text 0.94
99.80 85.38 385 15
4
superdoc
superdoc-sdk 2.15.0 · reject all
= 100: 1 · ≥ 90: 36 · < 50: 1 · ink 0.83 · text 0.95
0.00 11.07 49 351

DOCX → PDF vs Word's own PDF — the 860-document set

860 docs · 380/375/22/83 by state · set 2026-11-03.1 · scorer torch-mps · 2026-11-10

The fixed 860-document conversion set (every path and sha256 in release_info/sample_conversion_0.18.0_11-10-26_19-21.csv): each converter turns the same DOCX into PDF, scored pixel-wise against Word's own export of the same file. jubarte converts and is scored by this run, intent-to-treat — a document it fails scores 0; the comparators are the vendor refresh's runs on the same set, the newest version per tool, and a document a converter did not run is not run, never 0. Sorted by median; 95% CI is a percentile bootstrap of the median (2,000 resamples, seed 42).

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte †
jubarte 0.18.0
[80.88, 82.36]
81.72 78.02 860 0
2
apryse
apryse-sdk 12.2.1 (PDFNet 12.2.0-079a1c0268, demo key: output watermarked)
[74.79, 76.42]
75.55 69.26 860 0
3
docxide
docxide-pdf v0.18.4
[56.51, 61.21]
58.57 61.01 860 0
4
soffice
LibreOffice 26.8.2.2 9dea406ab7695df6325b11b0e19fdca8db4b7276
[53.3, 57.76]
55.39 58.07 860 0
5
docxodus
@docxodus/export 15.0.0 (docxodus 15.0.0)
[42.19, 43.91]
43.04 46.12 860 38

Source DOCX → PDF, no redlines

Document set `c947065c59b2` (398 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.6`.

Each converter turns the clean source DOCX documents, with no tracked changes in them, into PDF; its PDF is scored pixel-wise against the SHA-pinned PDF Word itself exported from the same source. A failed conversion scores 0 (intent-to-treat).

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
office2pdf
office2pdf 0.6.7
[79.83, 86.03]
84.34 73.82 390 8
1=
pdfitdown
pdfitdown 4.0.0
[79.83, 86.03]
84.34 73.82 390 8
3
libreoffice_convert_rust
LibreOffice Convert Rust v0.1.0
[72.14, 79.47]
75.26 73.98 398 0
4
jubarte †
jubarte 0.7.0
[66.94, 68.10]
67.39 66.25 398 0
4=
docxide-pdf
docxide-pdf v0.17.0
[63.27, 64.73]
63.97 65.65 398 0
6
rdocx
rdocx 0.7.0
[53.38, 55.31]
54.49 59.11 398 0
7
dxpdf
dxpdf 0.5.1
[52.15, 52.94]
52.52 56.73 381 17

The same documents under docxide-pdf's own metrics

Document set `a89ffc217bba` (6427 documents), renderer `oracle:microsoft_word`, scorer `docxide-150dpi`, bench `1.0`.

The same documents as the corpus-wide table, scored instead with docxide-pdf's own ink Jaccard (the ranked number) and text-boundary metrics at 150 DPI against the same Word PDF exports, so no converter is measured only by this bench's pixel scorer.

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte †
jubarte 0.17.0+c607894
[70.20, 71.23] · text 100.00
70.68 66.84 6,397 30

Below 3 pages: every converter on the same short documents

Document set `27122f4d945e` (2626 documents), renderer `oracle:microsoft_word`, scorer `v1`, bench `0.7`.

The corpus-wide conversion benchmark restricted to the documents Word lays out in 1 to 3 pages, so a converter that renders only the first 3 pages (unlicensed PyMuPDF Pro) is compared with every other converter on the same documents. The oracle stays Word's own PDF export of each document, scored pixel-wise; a failed conversion scores 0 (intent-to-treat).

#Tool · pinMedian · 95% CIMedian (fail = 0)Mean (fail = 0)ScoredFailed
1
jubarte †
jubarte 0.9.3
[83.01, 83.28]
83.15 81.35 2,626 0
2
soffice
LibreOffice 26.8.0.3 bce0998afefdbc355585ca324285661a2170ba77
[67.84, 68.83]
68.40 66.26 2,611 15
3
PyMuPDF Pro
pymupdf-pro 1.28.2 (unlicensed, first 3 pages)
[59.09, 60.77]
60.06 60.02 2,601 25
4
docxide-pdf
docxide-pdf v0.17.1
[53.81, 54.89]
54.24 56.00 2,626 0

By corpus state, the 860-document conversion set

median · documents per state in brackets

clean (380)

jubarte †87.17
soffice62.26

tracking, no comments (375)

jubarte †80.77
soffice57.85

comments, clean (22)

jubarte †80.02
soffice40.79

comments + tracking (83)

jubarte †79.74
soffice43.21

Every published case is browsable: Word’s page next to each engine’s page, with overlay, difference and multiply modes, page-count mismatches, the score table and the files themselves on Hugging Face. Below is page one of tracking_without_comments__dc2d9eeccb_file_40__vs__42df4a13aa_file_41_redline_cb0c07e105 (2 pp in Word), a case whose jubarte score sits at its published median.

jubarte.pro/use-cases/convert#cases%2Frelease-0.18.0%2Ftracking_without_comments__dc2d9eeccb_file_40__vs__42df4a13aa_file_41_redline_cb0c07e105
Page one as Word rendered it
Wordreference
Page one as jubarte 0.18.0 rendered it
jubarte 0.18.081.75
Page one as LibreOffice 26.8.2.2 rendered it
LibreOffice 26.8.2.265.98

Oracle: real Word

Every reference PDF is exported by Microsoft Word and SHA-pinned. Redlines are opened in Word too: if Word refuses the file, the tool scores 0 for that pair.

Intent-to-treat

Failures are not dropped. A crash, a timeout or an unopenable file counts as 0 in ITT mean and median, so a tool cannot improve its rank by skipping hard documents.

Paired bootstrap

Ranks tie (n=) when the paired bootstrap interval of the median difference to the row above includes 0. 2,000 resamples, seed 42.

Fixed sets

From 0.18.0 the tables on this page are the bench's fixed evaluation sets (set version 2026-11-03.1), the same documents for every release; every row's inputs are sha256-listed in the engine's release_info/. The comparators are the vendor refresh's runs on the same sets, newest version per tool — a tool the refresh did not run shows as not run, never 0. Full-corpus runs remain in the bench's RESULTS.md.

Author-affiliated, same rules

jubarte is marked † everywhere. Pins, run dates and sample lists are published so any row can be re-run; the harness is public.

Figures are project-maintained and reproducible, not a neutral third-party benchmark. Competitors move: Docxodus in particular has evolved since several stamps above, so read the pins. The two headline tables are this release’s 600-item samples — state-balanced, every file’s path and sha256 listed in the engine’s release_info/ — scored on 2026-11-10. The remaining tables and the full-corpus runs live in RESULTS.md.