You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(nlp): match single-letter bullets in the enumerated bullet pattern #4635
[a-z][A-Z] is two single-character classes placed next to each other, so it matches exactly two characters (aB) and never a single a or A. The two ordered-bullet helpers in this repo both accept a single letter as a marker:
unstructured/cleaners/core.py:53clean_ordered_bullets — docstring says "accounting numeric and alphanumeric types" and shows a.b This is a very important point -> This is a very important point.
unstructured/cleaners/extract.py:104extract_ordered_bullets — docstring shows a.1 This is a very important point -> ("a", "1", None).
The splitter regex used by the list path cannot, so lettered lists come out as one blob:
This is user visible through partition_image and partition_pdf(strategy="ocr_only"): ocr_data_to_elements -> normalize_layout_element -> layout_list_to_list_items (unstructured/partition/common/common.py:137).
Change
Keep the numeric branch byte for byte, and add a letter branch that only matches at the start of a line, only with a literal dot, and only when whitespace or the end of the text follows. The line anchor and the (?!\s[a-zA-Z]\.) guard are both required: \.? is optional, so an unanchored [a-zA-Z] turns every letter into a separator ("1. foo\n2. bar" would split into []), and without the guard "e. g. spaced example" loses its first word.
Verification
test_unstructured/nlp/test_patterns.py (new) covers both the regex and the converter, including the existing numeric behaviour and the abbreviation cases:
The 10 remaining failures are the pre-existing test_tokenize.py ones (no spaCy model in this offline environment); they are identical in both runs. The new file alone goes from 9 failed, 11 passed to 20 passed.
ruff format --check and ruff check are clean on both files.
Activity