Skip to content

fix(nlp): match single-letter bullets in the enumerated bullet pattern #4635

Open

Description

@Eapp1e

ENUMERATED_BULLETS_RE cannot match a single-letter bullet.

unstructured/nlp/patterns.py:111 compiles

ENUMERATED_BULLETS_RE = re.compile(r"(?:(?:\d{1,3}|[a-z][A-Z])\.?){1,3}")

[a-z][A-Z] is two single-character classes placed next to each other, so it matches exactly two characters (aB) and never a single a or A. The two ordered-bullet helpers in this repo both accept a single letter as a marker:

  • unstructured/cleaners/core.py:53 clean_ordered_bullets — docstring says "accounting numeric and alphanumeric types" and shows a.b This is a very important point -> This is a very important point.
  • unstructured/cleaners/extract.py:104 extract_ordered_bullets — docstring shows a.1 This is a very important point -> ("a", "1", None).

The splitter regex used by the list path cannot, so lettered lists come out as one blob:

from unstructured.partition.common.common import layout_list_to_list_items
[e.text for e in layout_list_to_list_items("a. foo\nb. bar", None, None, None, None)]
# ['a. foo\nb. bar']   expected ['foo', 'bar']

This is user visible through partition_image and partition_pdf(strategy="ocr_only"): ocr_data_to_elements -> normalize_layout_element -> layout_list_to_list_items (unstructured/partition/common/common.py:137).

Change

Keep the numeric branch byte for byte, and add a letter branch that only matches at the start of a line, only with a literal dot, and only when whitespace or the end of the text follows. The line anchor and the (?!\s[a-zA-Z]\.) guard are both required: \.? is optional, so an unanchored [a-zA-Z] turns every letter into a separator ("1. foo\n2. bar" would split into []), and without the guard "e. g. spaced example" loses its first word.

Verification

test_unstructured/nlp/test_patterns.py (new) covers both the regex and the converter, including the existing numeric behaviour and the abbreviation cases:

python -m pytest test_unstructured/nlp/test_patterns.py test_unstructured/nlp/test_tokenize.py test_unstructured/cleaners/test_core.py test_unstructured/cleaners/test_extract.py -q

pristine: 19 failed, 211 passed
patched:  10 failed, 220 passed

The 10 remaining failures are the pre-existing test_tokenize.py ones (no spaCy model in this offline environment); they are identical in both runs. The new file alone goes from 9 failed, 11 passed to 20 passed.

ruff format --check and ruff check are clean on both files.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions