Repository navigation
readline.set_history_length corrupts history files when used in a libedit build #121160
Description
Bug report
Bug description:
Since the libedit transition (correction: If an end user switches from a GNU readline build of Python to an editline build of Python), several important functions related to history files are broken.
- It is inconvenient enough that history files written by GNU readline based
readlinemodule are unreadable and result in a rather confusingOSErrorwithexc.errno == errno.EINVAL(without a corresponding operating system routine that returns such an error), but a workaround intercepting thisOSErrorcan be written that converts the old format to the new one. This has been covered before, e.g. in The New REPL Does Not Load My Command History #120766. - More importantly, history files written by libedit based
readlinemodule are unreadable by itself whenset_history_lengthis used, as demonstrated by the code below. - While a workaround for that is also possible (also below), as far as I can tell, until 3.13, so until a year from now (?) (whenever Add
readline.backendfor the backendreadlineuses #112510 becomes available in a release), your only option is"libedit" in readline.__doc__, which I expect will be what many people will leave in the codebase even oncereadline.backendbecomes available.
Reproducer:
import readline
readline.add_history("foo bar")
readline.add_history("nope nope")
readline.write_history_file("history-works")
# this works:
readline.read_history_file("history-works")
readline.set_history_length(2)
readline.write_history_file("history-breaks")
# this breaks:
readline.read_history_file("history-breaks")Workaround:
def _is_using_libedit():
if hasattr(readline, "backend"):
return readline.backend == "editline"
else:
return "libedit" in readline.__doc__
readline.set_history_length(1000)
if _is_using_libedit():
readline.replace_history_item(
max(0, readline.get_current_history_length() - readline.get_history_length()),
"_HiStOrY_V2_")CPython versions tested on:
3.12
Operating systems tested on:
Linux
Linked PRs
- gh-121160: Add a test for readline.set_history_length #121326
- gh-121160: Note that readline libraries using different history formats. #121327
- [3.13] gh-121160: Add some tests for readline.set_history_length (GH-121326) #121856
- [3.12] gh-121160: Add some tests for readline.set_history_length (GH-121326) #121857
- [3.13] gh-121160: Note that readline libraries using different history formats. (GH-121327) #122030
- [3.12] gh-121160: Note that readline libraries using different history formats. (GH-121327) #122031
Activity
pythongh-121160: Add some tests for readline.set_history_length (pyth…
pythongh-121160: Add some tests for readline.set_history_length (pyth…
[3.13] gh-121160: Add some tests for readline.set_history_length (GH-…
[3.12] gh-121160: Add some tests for readline.set_history_length (GH-…
pythongh-121160: Add some tests for readline.set_history_length (pyth…
encukou commentedon Jul 18, 2024
Reported to python-build-standalone: astral-sh/python-build-standalone#281
gh-121160: Note that readline libraries using different history forma…
pythongh-121160: Note that readline libraries using different history…
[3.12] pythongh-121160: Note that readline libraries using different …
[3.13] gh-121160: Note that readline libraries using different histor…
[3.12] gh-121160: Note that readline libraries using different histor…
encukou commentedon Jul 19, 2024
This is about as much as we can do on the CPython side. If anyone has more ideas, let's reopen.
whitequark commentedon Jul 19, 2024
Thanks again for handling this! It wasn't my proposed solution but as a maintainer I recognize that mine wasn't really viable for CPython.
python3: skip readline limited history tests
python3: skip readline limited history tests
python3: skip readline limited history tests
python3: skip readline limited history tests
python3: skip readline limited history tests
python3: skip readline limited history tests
python3: skip readline limited history tests
gpshead commentedon Sep 8, 2026
FWIW I'm tackling this with a libedit specific workaround in #157165 for effective duplicate issue #123018. yes it is gross to carry a workaround. but it is practical. getting NetBSD libedit to adopt a fix and getting that plumbed through to the various other versions of it is frankly not likely to happen in any meaningfully timely manner is unlikely.
encukou commentedon Sep 8, 2026
Yeah, carrying a 200-line workaround for an unsupported system still feels inappropriate.