Skip to content

Crash after error in PickleBuffer #122306

Closed as not planned
Closed as not planned
Crash after error in PickleBuffer#122306
Labels
3.12only security fixes3.13only security fixes3.14bugs and security fixestype-crashA hard crash of the interpreter, possibly with a core dump

Description

@serhiy-storchaka
serhiy-storchaka
opened on Jul 26, 2024

Crash report

The following script crashes at finalization stage:

import unittest
import pickle

class Test(unittest.TestCase):
    def test_non_continuous_buffer(self):
        pb = pickle.PickleBuffer(memoryview(b"foobar")[::2])
        pb.raw()

unittest.main()

Output:

E
======================================================================
ERROR: test_non_continuous_buffer (__main__.Test.test_non_continuous_buffer)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/serhiy/py/cpython/test_non_continuous_buffer.py", line 7, in test_non_continuous_buffer
    pb.raw()
    ~~~~~~^^
BufferError: cannot extract raw buffer from non-contiguous buffer

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)
python: Objects/memoryobject.c:125: mbuf_dealloc: Assertion `self->exports == 0' failed.
Aborted (core dumped)

I cannot yet reproduce the crash without unittest, perhaps it is related to some reference loop.

The crash is only happens in 3.12 an newer. In 3.8-3.11 it just raises an exception (PickleBuffer was added in 3.8).

cc @pitrou

Linked PRs

Activity

  1. serhiy-storchaka
    added
    type-crashA hard crash of the interpreter, possibly with a core dump
    3.12only security fixes
    3.13only security fixes
    3.14bugs and security fixes
    on Jul 26, 2024
  2. serhiy-storchaka
    added this to Pickle and copy issues 🥒on Jul 26, 2024

  3. devdanzin commented on Jul 26, 2024

    @devdanzin
    devdanzin
    on Jul 26, 2024
    Member

    Try:

    import pickle
import atexit

m = memoryview(b"foobar")[::2]
pb = pickle.PickleBuffer(m)
def p(): pb.raw()
atexit.register(m.bit_count)

  4. devdanzin commented on Jul 26, 2024

    @devdanzin
    devdanzin
    on Jul 26, 2024
    Member

    Better:

    import pickle

m = memoryview(b"foobar")[::2]
pb = pickle.PickleBuffer(m)
def p(): m.bit_length()
m.invalid

  5. marko1616 commented on Aug 27, 2024

    @marko1616
    marko1616
    on Aug 27, 2024

    The following analysis report will explain the bug by inserting comments into the log.

    Main Causes

    1. Creating a circular reference through a constructed exception, preventing garbage collection of PickleBuffer and memory objects
    2. Calling gc.collect()
    3. During the collection of the memory object, PyMemoryViewObject->mbuf is cleared for the first time (mbuf is set to a null pointer)
    4. When collecting PickleBuffer, memory_dealloc is called again through picklebuf_clear, at which point an exception occurs because mbuf was set to a null pointer in step 3

    Next

    I'm trying to find a method fix this this days.

    Bug reproduction(tested on mainline)

    import gc, pickle

mv = memoryview(b"test")
error = None

def func():
    try:
        pb = pickle.PickleBuffer(mv)
        raise Exception("test")
    except Exception as e:
        # Ensure `error` references the exception
        error = e
    finally:
        print(f"pb:{pb}\nmv:{mv}")

# Cyclic References Explanation:
# An exception creates a traceback object referencing the frame.
# The frame references local variables (`error`, `pb`).
# The exception object references the traceback.
# The `error` variable references the exception.

func()
del mv
gc.collect() # Crash here

    GDB Debugging Details

    ╭─xxx at xxx in ~/path/to/repos/cpython on main✔
╰─± gdb ./python 
GNU gdb (Debian 13.1-3) 13.1
Copyright (C) 2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./python...
warning: File "/home/marko1616/Documents/repos/cpython/python-gdb.py" auto-loading has been declined by your `auto-load safe-path' set to "$debugdir:$datadir/auto-load".
To enable execution of this file add
        add-auto-load-safe-path /home/marko1616/Documents/repos/cpython/python-gdb.py
line to your configuration file "/home/marko1616/.config/gdb/gdbinit".
To completely disable this security protection add
        set auto-load safe-path /
line to your configuration file "/home/marko1616/.config/gdb/gdbinit".
For more information about this security protection see the
"Auto-loading safe path" section in the GDB manual.  E.g., run from the shell:
        info "(gdb)Auto-loading safe path"
(gdb) b _memory_release
Breakpoint 1 at 0x14a97e: _memory_release. (3 locations)
(gdb) disable 1
(gdb) run
Starting program: /home/marko1616/Documents/repos/cpython/python 
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Python 3.14.0a0 (heads/main:54a05a46002, Aug 27 2024, 11:19:04) [GCC 12.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
[Detaching after vfork from child process 302825]
>>> import gc, pickle
... 
... mv = memoryview(b"test")
... error = None
... 
... def func():
...     try:
...         pb = pickle.PickleBuffer(mv)
...         raise Exception("test")
...     except Exception as e:
...         # Ensure `error` references the exception
...         error = e
...     finally:
...         print(f"pb:{pb}\nmv:{mv}")
... 
... # Cyclic References Explanation:
... # An exception creates a traceback object referencing the frame.
... # The frame references local variables (`error`, `pb`).
... # The exception object references the traceback.
... # The `error` variable references the exception.
... 
... func()
... input(f"del mv {mv}")
... del mv
... input("before gc")
... gc.collect() # Crash here
... 
pb:<pickle.PickleBuffer object at 0x7ffff744f740>
mv:<memory at 0x7ffff76cbf40>
del mv <memory at 0x7ffff76cbf40>
before gc
         Program received signal SIGINT, Interrupt.
                                                   0x00007ffff7de419d in __GI___libc_read (fd=fd@entry=3, buf=buf@entry=0x555555de78e0, nbytes=nbytes@entry=10000) at ../sysdeps/unix/sysv/linux/read.c:26
26      ../sysdeps/unix/sysv/linux/read.c: No such file or directory.
(gdb) watch (*(PyMemoryViewObject *)0x7ffff76cbf40)->mbuf
Hardware watchpoint 2: (*(PyMemoryViewObject *)0x7ffff76cbf40)->mbuf
(gdb) c
Continuing.


Hardware watchpoint 2: (*(PyMemoryViewObject *)0x7ffff76cbf40)->mbuf

Old value = (_PyManagedBufferObject *) 0x7ffff744e3c0
New value = (_PyManagedBufferObject *) 0x0
memory_clear (_self=0x7ffff76cbf40) at Objects/memoryobject.c:1168
1168        Py_CLEAR(self->mbuf);
(gdb) bt
#0  memory_clear (_self=0x7ffff76cbf40) at Objects/memoryobject.c:1168
#1  0x00005555557c5ec4 in delete_garbage (old=0x555555ae2930 <_PyRuntime+183280>, collectable=0x7fffffffd740, gcstate=0x555555ae28e8 <_PyRuntime+183208>, 
    tstate=0x555555b17858 <_PyRuntime+400152>) at Python/gc.c:1119
...
--Type <RET> for more, q to quit, c to continue without paging--q
Quit
(gdb) c
Continuing.
Exception ignored in tp_clear of memoryview:
Traceback (most recent call last):
  File "<python-input-0>", line 26, in <module>
BufferError: memoryview has 1 exported buffer

Program received signal SIGSEGV, Segmentation fault.
0x000055555569e9ad in _memory_release (self=0x7ffff76cbf40) at Objects/memoryobject.c:1111
1111            if (--self->mbuf->exports == 0)
(gdb) bt
#0  0x000055555569e9ad in _memory_release (self=0x7ffff76cbf40) at Objects/memoryobject.c:1111
#1  memory_dealloc (_self=0x7ffff76cbf40) at Objects/memoryobject.c:1148
#2  0x00005555556c999d in picklebuf_clear (self=<optimized out>) at Objects/picklebufobject.c:103
#3  0x00005555557c5ec4 in delete_garbage (old=0x555555ae2930 <_PyRuntime+183280>, collectable=0x7fffffffd740, gcstate=0x555555ae28e8 <_PyRuntime+183208>, 
...
(gdb)

  6. marko1616 commented on Aug 31, 2024

    @marko1616
    marko1616
    on Aug 31, 2024

    cc @serhiy-storchaka

  7. serhiy-storchaka commented on Sep 10, 2024

    @serhiy-storchaka
    serhiy-storchaka
    on Sep 10, 2024
    MemberAuthor

    This looks like an example of the hypothetical issue discussed in #77894. The solution was proposed there, but we did not have any reliable way to test it. Now we can, using pickle.PickleBuffer. And other way -- using memoryview.__getbuffer__(). Both ways were not available when the original issue was opened.

    I'm closing this issue as a duplicate of #77894.

  8. serhiy-storchaka
    closed this as not plannedon Sep 10, 2024
  9. github-project-automation
    moved this to Done in Pickle and copy issues 🥒on Sep 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    3.12only security fixes3.13only security fixes3.14bugs and security fixestype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions