Skip to content

Excessive QSBR memory usage when delay freeing large blocks. #132519

Open
@tom-pytel

Description

@tom-pytel

Bug report

Bug description:

I ran across this while profiling memory usage in #130771. The results are same for both that and list objects which use QSBR to free memory, so this is a QSBR thing. Note this is an extreme synthetic case, this CAN happen but does not mean it WILL happen in real-world apps.

Memory usage numbers:

             VmHWM
GIL      135104 kB  - normal GIL-enabled baseline
noGIL   6702788 kB  - free-threaded current QSBR behavior

Test script:

import threading
from queue import Queue

def thrdfunc(queue):
    while True:
        l = queue.get()

        l.append(0)  # force resize in non-parent thread which will free using _PyMem_FreeDelayed()

queue = Queue(maxsize=2)

threading.Thread(target=thrdfunc, args=(queue,)).start()

while True:
    l = [None] * int(3840*2160*3/8)  # sys.getsizeof(l) ~= 3840*2160*3 bytes

    queue.put(l)

See PR for proposed fix.

CPython versions tested on:

3.14

Operating systems tested on:

No response

Linked PRs

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions