Skip to content

fix: follow a declared base encoding where a font-specific font has its glyph - #370

Open
Boyukkhan wants to merge 1 commit into
docling-project:mainfrom
Boyukkhan:fix-symbol-declared-encoding
Open

Boyukkhan wants to merge 1 commit into
docling-project:mainfrom
Boyukkhan:fix-symbol-declared-encoding

Conversation

@Boyukkhan

Copy link
Copy Markdown

Problem

Since 7.21.0 (#351), a font whose metrics file declares EncodingScheme FontSpecific decodes every code
that /Differences and /ToUnicode leave open through its curated map, even when the font dictionary
declares a base encoding. The rule was added for legacy symbol fonts that declare WinAnsi while using
their own codes (AdvPSSym, AdvPSMP10), but FontSpecific also covers the standard Symbol font and the
metrics under fonts/custom/Microsoft/ (Arial, Times New Roman, Georgia, Trebuchet, ...). Their declared
encodings are then ignored where they are right:

BT /F1 24 Tf 72 700 Td (a\327b) Tj ET      % /F1: /BaseFont /XXXXXX+Symbol /Encoding /WinAnsiEncoding

A renderer draws α×β: WinAnsi names 0xD7 multiply, which Symbol has; it names 0x61 and 0x62 a and
b, which Symbol lacks, so the font's own codes apply. 7.20 decoded a×b; 7.21 and later decode α⋅β
(Symbol's own 0xD7 is dotmath). The same happens to TrueType Arial, Times New Roman, Georgia and
SymbolMT fonts declared /MacRomanEncoding: German text in Georgia reads TŠnzer for Tänzer, a
degree sign reads ¡ (macroman_encoding_bug_demo.pdf), bullets read ¥ or ∞.

Change

In get_correct_character, the curated map now applies to the codes whose declared glyph the font does
not have. For a code whose declared glyph it has, the declared encoding governs -- the order
freetype_font_cache::resolve_glyph_indices already uses to pick the glyph it draws (the declared
glyph name, then the font's own code). base_font::has_glyph_name answers the lookup from the metrics
file. A legacy font that declares WinAnsi falsely keeps its curated mapping, because the WinAnsi names
are not glyphs it has (test_adv_ps_mp10_uses_font_specific_mapping_over_winansi is unchanged). The
check is against the metrics file's glyph set, not an embedded subset's.

Tests

test_symbol_follows_declared_winansi_where_it_has_the_glyph (in tests/test_unit_gid_glyph_names.py)
decodes the example above and expects α×β. It fails on 7.22.1 (α⋅β) and on 7.20.0 (a×b).

Verification

  • tests/test_unit_*.py: 202 passed, 4 skipped.
  • Parse regression tests: 81 passed, 2 failed -- the two reference comparisons, on 16 pages of 6
    reference PDFs: 11939885835628903831_004.pdf (pages 1, 3, 5, 6, 12), 5236588965309071374-1.pdf (2),
    6480366468566741514.pdf (1), 921843749409059316-11.pdf (3, 5, 10, 11),
    macroman_encoding_bug_demo.pdf (1) and multi_page.pdf (1-4). 98 glyphs change, every one in a
    TrueType font declared /MacRomanEncoding with the Nonsymbolic flag, now decoded as MacRoman:
    Georgia Š š Ÿ § Ž Ý -> ä ö ü ß é ›, SymbolMT ϒ/∞ -> °/•, ArialMT ¥/¡ -> •/°,
    TimesNewRomanPSMT Ð -> -. Each was checked against the rendered page (n°15, bullets, Tänzer,
    100°F, – Gale, María, ›), and 96 of the 98 equal 7.20.0's reading (the other two differ only by
    the double-quote folding added in 7.21). The references recorded the regression; could you regenerate
    those pages in the dataset?
  • On a private set of PDFs, compared with 7.22.1: only Symbol-font ⋅ become ×, nothing else
    changes, and the ×/⋅ counts per document return to those of 7.16; changed glyphs were checked on
    rendered pages.

🤖 Generated with Claude Code

…ts glyph

Since 7.21.0 a font whose metrics declare EncodingScheme FontSpecific
decodes every code that /Differences and /ToUnicode leave open through its
curated map, even when the font dictionary declares a base encoding. That
also covers the standard Symbol font: declared as WinAnsi, code 0xD7 paints
`multiply`, but decoded as Symbol's own `dotmath`.

The curated map now applies only to codes whose declared glyph the font
does not have, which keeps it for a legacy font that declares WinAnsi
falsely. Where the font has the declared glyph, the declared encoding
governs, the order freetype_font_cache::resolve_glyph_indices already uses
for the glyph it draws.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Boyukkhan <beyukxan12@gmail.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @Boyukkhan, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Sep 30, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant