Skip to content

prose_lint.py Decodes Git Output With the Locale Encoding, So a Windows Run Can Crash on a UTF-8 Diff #1538

Description

@ptr727

Problem

prose_lint.py reads git output with text=True and no encoding, so Python decodes it with the locale's preferred encoding. On Windows, where Python's UTF-8 mode is off by default, that is the ANSI code page, commonly cp1252. Some UTF-8 byte sequences have no cp1252 mapping, so a diff carrying one raises UnicodeDecodeError. The except (subprocess.CalledProcessError, FileNotFoundError) around the call does not catch it, and the run ends in a traceback.

.github/actions/prose-gate/prose_lint.py at main (fb42028) has text=True without encoding= at lines 167, 201, 345, 504, 570, 599 and 615. Lines 167 and 201 are the diff-scope reads a --diff run always takes.

Evidence

This is unverified on Windows itself. The decode failure is shown directly:

>>> "Kyōto".encode("utf-8").decode("cp1252")
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8d in position 3: character maps to <undefined>

A diff that adds or removes ō (UTF-8 C5 8D) therefore cannot be decoded under cp1252. CI on Linux runs in a UTF-8 locale and is unaffected.

Reach

A local run from Git Bash or PowerShell with a Windows CPython, which is the route a downstream repository's docs give for running the gate before a push.

Suggested fix

Pass encoding="utf-8" (and errors="replace" where a lossy read is acceptable) alongside text=True on each git call. Setting PYTHONUTF8=1 is a local workaround until then.

Found while documenting the local prose-gate invocation in ptr727/aiopurpleair.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions