[wittjeff]

Summary

• Fold the "symbol charset" private-use block (U+F020–U+F0FF) of a /ToUnicode map back through the font's own encoding when the base font is Symbol/SymbolMT or ZapfDingbats. Office exporters embed these fonts with maps that send every glyph to U+F000 + code, so bullets, operators, Greek letters and even parentheses reached the text as invisible private-use characters (<U+F0B7> Welcome Address, <U+F03D>1, <U+F028>…<U+F029>). They now read • Welcome Address, =1, (…).
• The fold runs once, in init_cmap() after the /ToUnicode stream is parsed (fold_symbol_font_private_use()), and uses the shipped Symbol.afm / ZapfDingbats.afm glyph names and glyphlist.dat, the same tables that already decode a non-embedded /Symbol without /ToUnicode. Codes the encoding lacks keep the map's value, and every other font is untouched, so CJK gaiji and Arabic presentation fonts that use the block for real text are unaffected. The space glyph (U+F020) folds to a space, which the contraction already treated as a boundary.
• Adds tests/test_unit_symbol_font_private_use.py: in-memory Identity-H fonts with private-use /ToUnicode maps for SymbolMT, ZapfDingbats, an uncovered code, a non-symbol font, and the non-embedded /Symbol route.

Measurements

On the 940 regression pages (7.22.3), 23 documents carry private-use code points in their text; Symbol/SymbolMT fonts account for 815 characters in 10 of them (9 in the Office block U+F0xx; the tenth, 19246029784032167-4.pdf, carries the U+F8xx bracket pieces that the glyph list itself assigns, which the fold leaves alone), Wingdings/Webdings for 27 in 6, and 9 documents use the block legitimately (NotoSansCJK gaiji, TraditionalArabic, EuclidMathOne). With this change 11 pages in 9 documents change: 137 words are replaced one for one (the word count of every page is unchanged, so no word boundary moves) and no other page differs. MuPDF, poppler and pdfium report the same private-use values as 7.22.3 on these pages, so this is a deliberate improvement over parity, matching what docling-project/docling#4610 asks for on the DOCX side.

Wingdings/Webdings have no glyph-name route and need the Unicode L2/12-368 table as a resource; that is left for a follow-up.

The ground truth of the affected pages changes (private-use characters become •, -, =, (, ), ≤, Greek letters; the sanitizer maps the folded U+2212 to the ASCII hyphen-minus as it does for every other source); they are listed below.

• 11033738705194660817-2.pdf p15: <U+F02D> → - ×4
• 11033738705194660817-2.pdf p16: <U+F02D> → - ×5
• 13828292398120620566-1.pdf p1: <U+F0B7> → • ×6
• 16293520921922171261-1.pdf p1: <U+F0B7> → • ×6
• 16293520921922171261-1.pdf p2: <U+F0B7> → • ×2
• 16687286043211380927-1.pdf p15: <U+F0B7> → • ×3
• 6439951649404873078-1.pdf p1: <U+F0B7> → • ×3
• 8313873609013042575-4.pdf p5: <U+F0BE> → ⎯ ×11
• b766d7d18edf92b1_0002.pdf p1: <U+F03D> → = ×17, <U+F02D> → - ×11, <U+F071> → θ ×9, <U+F067> → γ ×5, <U+F073> → σ ×5, <U+F02B> → + ×3
• eac6df0b2a6f9cf0_0002.pdf p1: <U+F074> → τ ×4, <U+F02B> → + ×4, <U+F028> → ( ×3, <U+F029> → ) ×3, <U+F072> → ρ ×3, <U+F029><U+F028> → )( ×2
• redp5110_sampled.pdf p14: <U+F0AE> → →, HR_SCHEMA<U+F0AE> → HR_SCHEMA→

closes: #416

🤖 Generated with Claude Code

[github-actions]

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

[mergify]

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

• title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

[cau-git]

Thanks for this contribution. This PR is on hold until it has been consolidated as described in #428. Please resubmit it as part of the topic PR listed there, with the evidence the issue asks for.

[wittjeff]

Superseded by #431, which consolidates the font and ToUnicode decoding fixes as described in #428. This fix is commit 5 of 6 there, rebased on current main.