Skip to content

fix: consult base-font metrics for character decoding when the font dict carries a /Name label - #327

Closed
wittjeff wants to merge 3 commits into
docling-project:mainfrom
wittjeff:fix/base14-name-entry-builtin-encoding
Closed

wittjeff wants to merge 3 commits into
docling-project:mainfrom
wittjeff:fix/base14-name-entry-builtin-encoding

Conversation

@wittjeff

Copy link
Copy Markdown
Contributor

Fixes #317.

A base-14 font dictionary that carries the obsolete /Name entry (<< /BaseFont /Symbol /Name /F5 >>, as ReportLab emits) reports the label as its font name. get_correct_character() matched only that name against the base-14 metrics, so Symbol/ZapfDingbats text missed the built-in encoding tables (Symbol.afm, ZapfDingbats.afm) and fell back to StandardEncoding: Symbol's arrowright (0256) decoded as the fi ligature, ZapfDingbats' a35 star (0110) as H. The bogus fi then merged into the neighbouring Helvetica line under the sanitator's cross-font ligature exemption, which is why the issue presented as a merging bug.

The fix mirrors the font_name-or-base_font fallback that every width/bbox lookup in page_font.h already uses.

  • src/parse/pdf_resources/page_font.h: one hunk in get_correct_character()
  • tests/test_font_name_resolution.py: parse-side regression tests (Symbol arrow, ZapfDingbats star, labelled and unlabelled) alongside the existing render-side /Name-shadowing test

Full test suite: 221 passed, 1 skipped, 3 failed — the three failures are a single groundtruth mismatch surfaced in test_parse/test_threaded_parse/test_threaded_render: annots_01.pdf groundtruth expects l where the fix now yields ●. That document's /F3 is << /BaseFont /ZapfDingbats /Name /F3 >> with no /Encoding — the same bug pattern — so the groundtruth l is itself the misdecode (ZapfDingbats code 108, a71, is the black-circle bullet). The groundtruth dataset needs a revision bump for that file, as with the ActualText fix (#300).

🤖 Generated with Claude Code

…ies a /Name label

A base-14 font dictionary carrying the obsolete /Name entry (e.g.
<< /BaseFont /Symbol /Name /F5 >>, as ReportLab emits) reports the label
as its font name. get_correct_character() matched only that name against
the base-14 metrics, so Symbol/ZapfDingbats text missed the built-in
encoding tables and fell back to StandardEncoding: Symbol's arrowright
(0256) decoded as the 'fi' ligature and ZapfDingbats' a35 star as 'H'.
The bogus ligature then merged into the neighbouring line under the
sanitator's cross-font ligature exemption, which is why issue docling-project#317
presented as a merging bug.

Mirror the font_name-or-base_font fallback that every width/bbox lookup
in page_font.h already uses.

Fixes docling-project#317.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

✅ DCO Check Passed

Thanks @wittjeff, all your commits are properly signed off. 🎉

@mergify

mergify Bot commented Aug 21, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Enforce conventional commit

Make sure that we follow https://www.conventionalcommits.org/en/v1.0.0/

  • title ~= ^(fix|feat|docs|style|refactor|perf|test|build|ci|chore|revert)(?:\(.+\))?(!)?:

@PeterStaar-IBM PeterStaar-IBM left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please run the pre-commit

Comment thread tests/test_unit_font_name_resolution.py Resolved
Signed-off-by: Jeff Witt <1848307+wittjeff@users.noreply.github.com>
@wittjeff

Copy link
Copy Markdown
Contributor Author

Done — renamed to tests/test_unit_font_name_resolution.py and ran pre-commit (Ruff format/lint, MyPy, uv-lock all pass) in 11283dc.

@PeterStaar-IBM

Copy link
Copy Markdown
Member

can you merge with main (such that we have the latest regression tests) and resolve the merge conflicts?

@wittjeff

Copy link
Copy Markdown
Contributor Author

Done — merged main (post-#325) into the branch. The merge resolved cleanly: git's rename detection handled the test_font_name_resolution.py → test_unit_font_name_resolution.py rename (which I had already made per review, matching #325's convention), so relative to main the branch still carries only the one hunk in get_correct_character() plus the regression test file.

Full suite locally after the merge: 227 passed, 1 skipped, 3 failed — the three failures are the known annots_01.pdf groundtruth mismatch described in the PR body: that document's /F3 is << /BaseFont /ZapfDingbats /Name /F3 >> with no /Encoding, i.e. exactly the bug pattern this PR fixes, so the stored groundtruth records the misdecode (l where ZapfDingbats code 108, a71, is the black-circle bullet ●).

I've opened a companion PR on the regression dataset with the four regenerated annots_01 parser artifacts: https://huggingface.co/datasets/docling-project/regression-dataset-for-docling-parse/discussions/3 — the diff there is exactly the four l → ● cells (the .json.gz also re-encodes one embedded page bitmap; verified pixel-identical, just non-deterministic PNG compression on rewrite). With those artifacts in place, the three tests pass locally (59 passed across test_regression_parse / test_regression_threaded_parse / test_regression_threaded_render).

Once the dataset PR is merged I'll bump HF_DATASET_REVISION in tests/constants.py here — which also plays nicely with the corpus cache keying introduced in #329. Until then, CI on this PR will show those three known failures.

@PeterStaar-IBM

Copy link
Copy Markdown
Member

@wittjeff Seems we caught something in the regression testing,

=========================== short test summary info ============================
FAILED tests/test_regression_parse.py::test_reference_documents_from_filenames - AssertionError: 3 page(s) failed: annots_01.pdf@1[char], annots_01.pdf@1[word], annots_01.pdf@1[line]
assert 3 == 0
FAILED tests/test_regression_threaded_parse.py::test_threaded_reference_documents_from_filenames - AssertionError: true_cell.text == pred_cell.text => l == ● for /home/runner/work/docling-parse/docling-parse/tests/data/groundtruth/parser/annots_01.pdf.page_no_1.py.json
assert 'l' == '●'
  
  - ●
  + l
FAILED tests/test_regression_threaded_render.py::test_render_reference_documents_from_filenames - AssertionError: true_cell.text == pred_cell.text => l == ● for /home/runner/work/docling-parse/docling-parse/tests/data/groundtruth/parser/annots_01.pdf.page_no_1.py.json
assert 'l' == '●'
  
  - ●
  + l
============ 3 failed, 68 passed, 12 skipped in 2928.31s (0:48:48) =============

@wittjeff

wittjeff commented Aug 28, 2026 •

Copy link
Copy Markdown
Contributor Author

@PeterStaar-IBM

I've opened a companion PR on the regression dataset with the four regenerated annots_01 parser artifacts:

Please see https://huggingface.co/datasets/docling-project/regression-dataset-for-docling-parse/discussions/3

Are you intending for the GT dataset to include these issues? My interpretation was that if you incorporate the dataset PR following the HF link above, to fix the dataset first, then re-run the CI here, this docling-parse fix PR should clear.

If I'm actually nulling out an intended error test case in the GT dataset, I'll back out the dataset PR and start over.

Boyukkhan added a commit to Boyukkhan/docling-parse that referenced this pull request Oct 11, 2026
get_correct_character() matched only the name-derived font_name against
the built-in font metrics. A font dictionary that carries the obsolete
/Name entry and no /FontDescriptor reports that label as its font_name
(ReportLab writes `/BaseFont /Symbol /Name /F3`), so Symbol.afm and
ZapfDingbats.afm were missed and the codes fell back to StandardEncoding:
lessequal read as sterling, greaterequal as daggerdbl, Omega as W, and
ZapfDingbats a73 (a black square) as n.

The decoder now selects the built-in font as the width and bbox lookups
already do (matched_base_font()): a successful font_name match is kept,
/BaseFont is consulted only when font_name matches no built-in font, and
the selected match decides the condition, the name and the metrics.
/ToUnicode, /Encoding /Differences, the declared-base-encoding logic and
unresolved-glyph markers are unchanged, and init_font_name() is not
touched, so reported font names stay as they were.

Port of docling-project/docling-parse PR docling-project#327 (head 5d62ee8), whose hunk
targets code before the cached base-font helpers. Its parse tests are
ported, with controls for genuine Latin characters, /ToUnicode,
/Differences, a declared WinAnsi on Symbol, /FontDescriptor /FontName
priority, a labelled Type3 font, and the documented limitation that a
label which itself names a built-in font still selects that font.

Local build version 7.22.1+overprint.3.

Signed-off-by: Boyukkhan <beyukxan12@gmail.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01P8yzQiM732FSNxrxh4YfVK
@cau-git

cau-git commented Oct 28, 2026

Copy link
Copy Markdown
Member

Thanks for this contribution. This PR is on hold until it has been consolidated as described in #428. Please resubmit it as part of the topic PR listed there, with the evidence the issue asks for.

@wittjeff

Copy link
Copy Markdown
Contributor Author

Superseded by #431, which consolidates the font and ToUnicode decoding fixes as described in #428. This fix is commit 3 of 6 there, rebased on current main.

@wittjeff wittjeff closed this Oct 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Merged textline cells decode every run with one font, corrupting Symbol glyphs (→ becomes "fi")

3 participants