Repository navigation
Conversation
…plicate_cells remove_duplicate_cells drops a later cell with the same text and font when all four corners lie within an absolute 0.5pt of an earlier one. On text smaller than about a point the next occurrence of the same letter in a word is itself closer than that, so "Committee" decoded as "Com it e". Also require the corners to lie within half of the earlier cell's width. For any glyph at least 1pt wide the bound stays 0.5pt, so nothing changes; an exact redraw (stroke-then-fill bold) is still removed at any size. Resolves docling-project#371 Signed-off-by: galer1us <66819755+galer1us@users.noreply.github.com>
|
✅ DCO Check Passed Thanks @galer1us, all your commits are properly signed off. 🎉 |
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Enforce conventional commitMake sure that we follow https://www.conventionalcommits.org/en/v1.0.0/
|
|
The I built this branch and its base (
One side effect is worth stating plainly. In both Arabic documents, copies of the same glyph are drawn about half a glyph width apart (offset ≈ 0.46–0.50 pt on glyphs ≈ 0.82–0.86 pt wide). That spacing now counts as a separate letter rather than a redraw: the same rule that keeps "ll" in Could you regenerate the parser ground truth for these five documents when you review, as was done for the |
Resolves #371
Problem
pdf_decoder<PAGE>always callsremove_duplicate_cells(page_cells, 0.5, true)(src/parse/pdf_decoders/page.h). It drops a later cell with the same text and font if all four corners are within an absolute 0.5pt of an earlier one. On text smaller than about 1pt, the next copy of the same letter in a word is itself closer than 0.5pt, so distinct glyphs get dropped.Committeedecodes asCommit eat 1pt andCom it eat 0.5pt. No decode option avoids this, because the call does not checkdo_sanitization.Change
The corners must now also be within half of the earlier cell's width, i.e.
tol = min(eps, 0.5 * width). An exact repeat (distance 0) still counts, so zero-width cells behave as before.tolis still 0.5pt, so ordinary text behaves exactly as before.duplicate_bold_text_01.pdf) overlaps its original almost completely, so it is still removed at any size.What I deliberately did not change:
do_sanitization. That would bring back duplicate fake-bold text for anyone who turns sanitisation off.remove_adjacent_cells, which has nothing calling it.This composes with #367 / #369 (for #366, the opposite failure in the same function). They add an exact-match pass before this loop and leave the tolerant scan alone. Both diffs apply cleanly on top of this branch (
git apply --check).Tests
New file
tests/test_unit_duplicate_cells.py. It builds its own PDFs withtests/pdf_builder.pyand checks the result throughDoclingPdfParser→textline_cells:test_repeated_letters_survive_at_any_text_size[12|2|1|0.8|0.5]: before this change the 1, 0.8 and 0.5 cases fail.test_run_drawn_twice_in_place_is_read_once[12|1]: a run drawn twice (1 Trthen0 Tr) at the same origin decodes once. The 1pt case also failed before, because it lost the same letters.Results:
uv run pytest tests/test_unit_*.py: 211 passed, 1 skipped.pre-commit run --all-files(ruff format, ruff check, mypy, uv-lock): all pass.Groundtruth changes (parser regression)
I ran
test_regression_parse.py,test_regression_threaded_parse.py,test_regression_cell_contraction.py,test_regression_embedded_fonts.pyandtest_regression_locale_safety.py: 81 passed, 2 failed. The two failures are the parse and threaded-parse groundtruth comparisons. Both fail on the same 5 pages and nothing else.On every changed page the patch only adds cells. Each added cell is 0.31–0.50pt from its nearest same-text neighbour, so none of them is an exact overprint.
ligatures_01.pdfp1l, +53 narrow spaces, +1f:interpupil ary→interpupillary,il uminator→illuminator(21 lines)l: 334 (312 → 334)1265837786161076401-1.pdfp1-:R→-R-: 127 (126 → 127)b6855cce7fbfd31b_0004.pdfp1right_to_left_04.pdfp1ة: the elongated runs no longer split mid-word (ةةةةةةةة ةل→ةةةةةةةةةةل)153a16136ed8f5ac_0011.pdfp1ي: one more stretch glyph in 30 elongated Arabic wordsThe two Arabic pages need a maintainer's view. Their stretched glyphs repeat at about 0.43–0.50pt spacing, and stock already decodes more of them than pdfium does, so I can't say whether the extra copy is ink or a redraw.
The groundtruth for those 5 pages lives in the HF dataset, which I can't update from a fork. I can supply the files
--update-groundtruthregenerates: for each page,<doc>.page_no_1.py.json.gzplus its.char.txt,.word.txtand.line.txt; forb6855cce…, only the.json.gzand.char.txtchange.Before / after on real documents
These are SEC filings from FinanceBench. I converted them with docling 2.129.0 (TableFormer accurate, OCR off) and compared against the pdfium text of each page.
WALMART_2019_10K: the 28 pages whose glyphs are all under 1pt tall (p91–p145, odd pages)
The rest of the "missing" count (212) is curly quotes folded to ASCII, the same in both builds.
p139, before → after:
In any mat er elating sole y to a Com it e member's indiv dual rights or ben fits→In any matter relating solely to a Committee member's individual rights or benefits.36 control pages (dense, table and TOC pages from 10-K/10-Q/8-K filings, plus 3 table fixtures)
Disclosure: Found, reproduced and fix drafted with an AI coding agent (Claude Code); reviewed before filing.