Skip to content

BUG: Arabic-Indic digits are reversed during text extraction - #4077

Merged
stefan6419846 merged 2 commits into
py-pdf:mainfrom
Syamjith-NK:fix/arabic-indic-digit-order
Sep 14, 2026
Merged

stefan6419846 merged 2 commits into
py-pdf:mainfrom
Syamjith-NK:fix/arabic-indic-digit-order

Conversation

@Syamjith-NK

Copy link
Copy Markdown
Contributor

AI disclosure, per CONTRIBUTING.md: this change and this description were written with AI assistance (Claude Opus 5, via Claude Code). Every measurement quoted below was run on my machine and checked before opening this.

Problem

extract_text() returns Arabic-Indic digits in reverse order. ١٢٣٤ (1234) comes back as ٤٣٢١ (4321), and Extended Arabic-Indic (Persian) digits ۱۲۳۴ do the same.

I checked the files were correct before blaming the extractor. I rendered the same strings to an image and looked at the result: the glyphs are drawn left to right as 1, 2, 3, 4. It reproduces with two unrelated producers (matplotlib and fpdf2), and on a line containing nothing but digits, so no direction heuristic is involved. PyMuPDF extracts the same files correctly.

Cause

U+0660-U+0669 and U+06F0-U+06F9 fall inside the ("֐", "ࣿ") entry of RTL_CHARACTER_RANGES, so is_char_rtl() returns True for them. In _text_extraction/__init__.py the neutral check runs first and keeps insertion order; anything that falls through to the RTL branch is prepended, so a digit run comes out reversed.

Fix

Add the two numeric ranges to NEUTRAL_CHARACTER_RANGES. Under the Unicode bidirectional algorithm digits are weak types (AN/EN) and a digit run is not reordered inside an RTL paragraph, so this is decidable from the standard rather than a matter of taste.

is_char_rtl() itself is untouched, so the other caller (generic/_appearance_stream.py) is unaffected.

Scope

Deliberately narrow. This does not settle #1629, which is about where a digit run sits relative to the surrounding Arabic text. That question is untouched here. Arabic and Hebrew letters keep reversing exactly as before, and the new test asserts that so it cannot regress silently.

Test

The test builds its page from a Type0/Identity-H font with a ToUnicode CMap, so it adds no binary fixture and depends on nothing but pypdf.

  • With the fix: 5 passed.
  • Without the fix: the 3 digit cases fail, the 2 letter cases pass. So it really tests this change.
  • No existing test contains an Arabic-Indic digit, so nothing was asserting the old behaviour.

Full suite on this branch: 1717 passed, 34 skipped, 2 xfailed, 0 failed. ruff check . clean on the pinned 0.16.0, mypy clean on both changed files, all pre-commit hooks pass. The single xpassed in that run (test_issue_2336) also xpasses on unmodified main.

Before and after on real generated files:

input before after
١٢٣٤ ٤٣٢١ ١٢٣٤
۱۲۳۴ ۴۳۲۱ ۱۲۳۴
٥٠٪ ٪٠٥ ٥٠٪
سنة ٢٠٢٦ ٦٢٠٢ سنة ٢٠٢٦ سنة (digits correct, run order still #1629)
plain Arabic, plain Hebrew unchanged unchanged

Happy to rename the constant or narrow the range to ٠-٩ if you would rather not include the numeric separators.

Arabic-Indic (U+0660-0669) and Extended Arabic-Indic (U+06F0-06F9) digits sit
inside the U+0590-U+08FF range of RTL_CHARACTER_RANGES, so is_char_rtl()
reported them as right-to-left and get_display_str() prepended each one,
reversing the run: 1234 came back as 4321.

Per the Unicode bidirectional algorithm digits are weak types (AN/EN) and a
digit run is never reordered inside a right-to-left paragraph, so treating them
as neutral is what the standard prescribes.

This is deliberately narrow. The broader question of ordering a digit run
relative to the surrounding Arabic text, discussed in py-pdf#1629, is untouched.
@codecov

codecov Bot commented Sep 13, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.12%. Comparing base (d26df36) to head (de0a35d).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #4077   +/-   ##
=======================================
  Coverage   98.11%   98.12%           
=======================================
  Files          59       59           
  Lines       11489    11475   -14     
  Branches     2150     2143    -7     
=======================================
- Hits        11273    11260   -13     
  Misses        122      122           
+ Partials       94       93    -1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread tests/test_text_extraction.py Resolved
@stefan6419846
stefan6419846 merged commit 96d81f0 into py-pdf:main Sep 14, 2026
20 checks passed
stefan6419846 added a commit that referenced this pull request Sep 16, 2026
## What's new

### Security (SEC)
- Limit size of alphabetical page labels (#4096) by @stefan6419846

### Deprecations (DEP)
- Replace PdfWriter method add_js (#3979) by @j-t-1

### Performance Improvements (PI)
- Move static value out of loop body for appearance stream data (#4087) by @stefan6419846
- Reduce number of full data lookups for attachment mapping API (#4081) by @stefan6419846

### Bug Fixes (BUG)
- Do not copy unrelated pages when appending pages with non-terminal fields (#4078) by @anandghegde
- Use page reference for existing internal link targets (#4076) by @r-kamei
- Arabic-Indic digits are reversed during text extraction (#4077) by @Syamjith-NK
- Parse a string rect for add_uri into a rectangle (#4074) by @RavSinghChandan

[Full Changelog](6.18.1...6.19.0)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants