Repository navigation
BUG: Arabic-Indic digits are reversed during text extraction - #4077
Merged
stefan6419846 merged 2 commits intoSep 14, 2026
Merged
Conversation
Arabic-Indic (U+0660-0669) and Extended Arabic-Indic (U+06F0-06F9) digits sit inside the U+0590-U+08FF range of RTL_CHARACTER_RANGES, so is_char_rtl() reported them as right-to-left and get_display_str() prepended each one, reversing the run: 1234 came back as 4321. Per the Unicode bidirectional algorithm digits are weak types (AN/EN) and a digit run is never reordered inside a right-to-left paragraph, so treating them as neutral is what the standard prescribes. This is deliberately narrow. The broader question of ordering a digit run relative to the surrounding Arabic text, discussed in py-pdf#1629, is untouched.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4077 +/- ##
=======================================
Coverage 98.11% 98.12%
=======================================
Files 59 59
Lines 11489 11475 -14
Branches 2150 2143 -7
=======================================
- Hits 11273 11260 -13
Misses 122 122
+ Partials 94 93 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
stefan6419846
approved these changes
Sep 14, 2026
stefan6419846
added a commit
that referenced
this pull request
Sep 16, 2026
## What's new ### Security (SEC) - Limit size of alphabetical page labels (#4096) by @stefan6419846 ### Deprecations (DEP) - Replace PdfWriter method add_js (#3979) by @j-t-1 ### Performance Improvements (PI) - Move static value out of loop body for appearance stream data (#4087) by @stefan6419846 - Reduce number of full data lookups for attachment mapping API (#4081) by @stefan6419846 ### Bug Fixes (BUG) - Do not copy unrelated pages when appending pages with non-terminal fields (#4078) by @anandghegde - Use page reference for existing internal link targets (#4076) by @r-kamei - Arabic-Indic digits are reversed during text extraction (#4077) by @Syamjith-NK - Parse a string rect for add_uri into a rectangle (#4074) by @RavSinghChandan [Full Changelog](6.18.1...6.19.0)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AI disclosure, per
CONTRIBUTING.md: this change and this description were written with AI assistance (Claude Opus 5, via Claude Code). Every measurement quoted below was run on my machine and checked before opening this.Problem
extract_text()returns Arabic-Indic digits in reverse order.١٢٣٤(1234) comes back as٤٣٢١(4321), and Extended Arabic-Indic (Persian) digits۱۲۳۴do the same.I checked the files were correct before blaming the extractor. I rendered the same strings to an image and looked at the result: the glyphs are drawn left to right as 1, 2, 3, 4. It reproduces with two unrelated producers (matplotlib and fpdf2), and on a line containing nothing but digits, so no direction heuristic is involved. PyMuPDF extracts the same files correctly.
Cause
U+0660-U+0669andU+06F0-U+06F9fall inside the("", "ࣿ")entry ofRTL_CHARACTER_RANGES, sois_char_rtl()returnsTruefor them. In_text_extraction/__init__.pythe neutral check runs first and keeps insertion order; anything that falls through to the RTL branch is prepended, so a digit run comes out reversed.Fix
Add the two numeric ranges to
NEUTRAL_CHARACTER_RANGES. Under the Unicode bidirectional algorithm digits are weak types (AN/EN) and a digit run is not reordered inside an RTL paragraph, so this is decidable from the standard rather than a matter of taste.is_char_rtl()itself is untouched, so the other caller (generic/_appearance_stream.py) is unaffected.Scope
Deliberately narrow. This does not settle #1629, which is about where a digit run sits relative to the surrounding Arabic text. That question is untouched here. Arabic and Hebrew letters keep reversing exactly as before, and the new test asserts that so it cannot regress silently.
Test
The test builds its page from a Type0/Identity-H font with a
ToUnicodeCMap, so it adds no binary fixture and depends on nothing but pypdf.Full suite on this branch: 1717 passed, 34 skipped, 2 xfailed, 0 failed.
ruff check .clean on the pinned 0.16.0,mypyclean on both changed files, all pre-commit hooks pass. The singlexpassedin that run (test_issue_2336) also xpasses on unmodifiedmain.Before and after on real generated files:
١٢٣٤٤٣٢١١٢٣٤۱۲۳۴۴۳۲۱۱۲۳۴٥٠٪٪٠٥٥٠٪سنة ٢٠٢٦٦٢٠٢ سنة٢٠٢٦ سنة(digits correct, run order still #1629)Happy to rename the constant or narrow the range to
٠-٩if you would rather not include the numeric separators.