[Bug]: Windows non-UTF-8 locale — Path text I/O decodes structured state files with the locale encoding #4941

Activity

1. GZY-SUPER-HACKER commented on Sep 23, 2026

   

   GZY-SUPER-HACKER

   on Sep 23, 2026

   ContributorAuthor

   More actions

   Two updates after rebasing onto eb16c54 — one correction to what I wrote above, and one severity upgrade.

   Correction: the site count is 9, not 10

   Current main replaced the direct read in loopx/capabilities/manager_context/evidence_export.py with load_registry / registry_codec, which reads bytes (path.read_bytes() → json.loads(bytes)) and is therefore already encoding-safe. I verified by running the documented command on a cp936 host with a project in a Chinese-named directory:

   loopx goal-portfolio --manager-view portfolio --format json
   -> {"ok": true, "view": "portfolio", ...}
   

   That already succeeds on eb16c54 without my change, so the registry read is no longer part of this class. The remaining scope is 9 sites in loopx/: 8 reads + 1 write, across 6 modules. An AST scan over the package now reports zero remaining read_text() / write_text() calls without an encoding.

   Severity upgrade: one of the nine is worse than described above

   I described connector_registry/core.py:149 as a write-side site where the loader "silently falls back to built-in defaults". That understates it, and the reason is worth stating precisely.

   save_connector_registry writes with the locale codec; load_connector_registry reads the same file as UTF-8 inside try/except (OSError, ValueError): pass. On a cp936 host:

   1. loopx connector register <id> returns {"ok": true, ...};
   2. the file on disk is not valid UTF-8 — the first Chinese character lands as byte 0xd0;
   3. the next loopx connector list cannot decode it, the UnicodeDecodeError is swallowed, and the registry silently reverts to the built-in catalog. Exit status 0, nothing on stderr;
   4. registering a second connector loads (fails silently), then saves — which overwrites the file and permanently destroys the first entry.

   So on a non-UTF-8 host this registry is effectively write-only: every write is discarded by the next read, and a subsequent write erases what came before.

   The reason this needs no unusual user input: BUILTIN_CONNECTOR_CATALOG itself carries 37 non-ASCII field values (新浪日线行情, ARK 联网搜索, 同花顺财务摘要, …). The write side therefore emits GBK on a cp936 host for any state, including a registration the user typed entirely in ASCII. I confirmed the ASCII-only variant reproduces identically.

   An independent consequence: three tests already in this repository are red on such a host for exactly this reason and go green with the fix — test_register_use_rank_persists, test_builtin_connector_updates_survive_catalog_resync, and test_cli_path_override_reads_the_registry_it_writes (3F → 0F, with the full per-test failure list diffed against the base tree and no other outcome changed). They pass on a UTF-8 locale, so CI on Linux does not currently observe this.

   PR: #4942.
2. BigDataDZ commented on Oct 4, 2026

   

   BigDataDZ

   on Oct 4, 2026

   Contributor

   More actions

   Claiming the GH-A01 closeout slice from the contributor task board: native cp936 verification of the repaired read/write paths, reusing the landed guard.

   Host: zh-CN Windows, ANSI code page 936 (GetACP() = 936), UTF-8 mode off (PYTHONUTF8/PYTHONIOENCODING unset, sys.flags.utf8_mode == 0), Python 3.12, loopx @ fd65e71 (current main).

   Validation suite (board command, native-cp936 environment):

   python -m pytest -q tests/test_loopx_text_io_utf8.py tests/capabilities/test_connector_registry_encoding.py tests/test_runtime_subprocess_utf8.py
   7 passed in 17.44s
   

   Native-host readback (isolated temp state root, current HEAD):

   • connector register with non-ASCII name/purpose (新浪日线行情) -> save on the GBK host: the on-disk file decodes as strict UTF-8 and both fields round-trip through load_connector_registry;
   • second register_connector + save: both entries survive - the "registry is write-only / first entry destroyed" failure mode from the severity upgrade above is gone;
   • collaboration/inbox.py _write -> _read under a Chinese directory path: strict-UTF-8 bytes, read-back text identical (用户任务：处理中文请求);
   • manager-context registry under a Chinese directory with non-ASCII knowledge_root/state_file: load_registry (read_bytes -> json.loads) decodes with paths intact - matches the correction above.

   Package state: an AST scan over loopx/ at this HEAD finds zero Path.read_text() / Path.write_text() / text-mode open() calls without a pinned UTF-8 codec. The single method-name match, loopx/doctor.py:193, is importlib.metadata.Distribution.read_text("INSTALLER") - a metadata-key read, not file I/O - which test_loopx_text_io_utf8.py explicitly excludes.

   The reproduction from this issue now succeeds on a native cp936 host without PYTHONUTF8. With reporter confirmation this looks ready to close.
3. GZY-SUPER-HACKER

   closed this as completedon Oct 4, 2026