Skip to content

SIGSEV in datetime.timedelta (possibly from datetime's C delta_new) #132413

Open
@Jacoblightning

Description

@Jacoblightning

Crash report

What happened?

I was testing some AI code with ollama and stumbled across a really weird crash.
The fact that it happens during an IndexError and a specific function has to be there leads me to believe that this is a CPython bug and not an ollama bug.

from ollama import chat

# Has to be here to segfault???
def colorSwitch(color): 
    print(color, end="", flush=True)


stream = chat(
    model="llama3.2", # I think it works with any but this is what I used.
    messages=[{"role": "user", "content": ""}],
    options={"seed":0}, # Does not need this but I figured it would be helpful
    stream=True,
)
# Any iteration works. I just simplified it down to this.
part = next(iter(stream))['message']['content']
temp = part.split("</think>", 1)

# Crash Here
temp[1]

PythonCore.zip

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.13.2 (main, Feb 5 2025, 08:05:21) [GCC 14.2.1 20250128]

Linked PRs

Activity

added
type-crashA hard crash of the interpreter, possibly with a core dump
on Apr 11, 2025
picnixz

picnixz commented on Apr 11, 2025

@picnixz
Member

I was testing some AI code with ollama and stumbled across a really weird crash.

What's the crash? please provide the traceback (shown on the terminal if possible) (not just the core dump).

Jacoblightning

Jacoblightning commented on Apr 11, 2025

@Jacoblightning
Author

Sorry. Here is the python traceback. If you wanted a gdb or other traceback, let me know.

  File "crasher1.py", line 19, in <module>
    temp[1]
    ~~~~^^^
IndexError: list index out of range
Segmentation fault (core dumped)
picnixz

picnixz commented on Apr 11, 2025

@picnixz
Member

If you wanted a gdb or other traceback, let me know

If possible yes, so that we know where the crash exactly happens. If possible, you can use python -X faulthandler crasher.py as well though I'm not sure if we're able to know more.

Also, it might be that ollama is using the Python C API behind the scene (AFAIU, ollama is written in Go and C but there are Python bindings, which may be the ones where the issue arise).

changed the title [-]Segfault on IndexError under specific conditions[/-] [+]SIGSEV following an `IndexError` when using `ollama`[/+] on Apr 11, 2025
Jacoblightning

Jacoblightning commented on Apr 11, 2025

@Jacoblightning
Author

Ah. It appears that it is an ollama issue after all. Ill raise it over there.

Fatal Python error: Segmentation fault

Current thread 0x00007d584222bbc0 (most recent call first):
  Garbage-collecting
  File "/home/jacoblightning3/PycharmProjects/AiFight/.venv/lib/python3.13/site-packages/httpx/_client.py", line 158 in close
  File "/home/jacoblightning3/PycharmProjects/AiFight/.venv/lib/python3.13/site-packages/httpx/_models.py", line 972 in close
  File "/home/jacoblightning3/PycharmProjects/AiFight/.venv/lib/python3.13/site-packages/httpx/_client.py", line 877 in stream
  File "/usr/lib/python3.13/contextlib.py", line 162 in __exit__
  File "/home/jacoblightning3/PycharmProjects/AiFight/.venv/lib/python3.13/site-packages/ollama/_client.py", line 163 in inner
Segmentation fault (core dumped)
Jacoblightning

Jacoblightning commented on Apr 11, 2025

@Jacoblightning
Author

@picnixz So, I just checked and it appears that ollama-python is pure python. (Ofc I realized this after I made the issue.) Would that bring the issue back here? They don't appear to be using CTypes, etc.

JelleZijlstra

JelleZijlstra commented on Apr 11, 2025

@JelleZijlstra
Member

That does look like it could be a CPython issue, though ollama has a few dependencies that include compiled code. I tried to reproduce it (3.13.1, MacOS, latest ollama) but I got httpx.ConnectError: [Errno 61] Connection refused instead (does ollama require a local server or something? probably not something I'm interested in setting up).

Two useful ways forward could be:

  • Get the full C stack trace in gdb or lldb or a similar tool and explore what's happening when we hit the segfault. For example, maybe that will tell you what type it's looking at when the crash happens.
  • Reduce the reproducer to something simpler. For example, you can start by removing more and more parts of ollama that aren't relevant to the crash and then see if it still reproduces.
Jacoblightning

Jacoblightning commented on Apr 11, 2025

@Jacoblightning
Author

I tried to reproduce it (3.13.1, MacOS, latest ollama) but I got httpx.ConnectError: [Errno 61] Connection refused instead (does ollama require a local server or something? probably not something I'm interested in setting up).

Yes. The ollama package requires ollama to be installed and running a local server.

Jacoblightning

Jacoblightning commented on Apr 11, 2025

@Jacoblightning
Author

Running python with the debug build I just compiled produces:

Modules/_datetimemodule.c:2745:13: runtime error: member access within null pointer of type 'struct datetime_state'
AddressSanitizer:DEADLYSIGNAL
=================================================================
==9445==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000010 (pc 0x7c8cc6b93737 bp 0x7ffe5c2d11f0 sp 0x7ffe5c2d1040 T0)
==9445==The signal is caused by a READ memory access.
==9445==Hint: address points to the zero page.
    #0 0x7c8cc6b93737 in delta_new Modules/_datetimemodule.c:2745
    #1 0x5790c86108b1 in type_call Objects/typeobject.c:1987
    #2 0x5790c8410b28 in _PyObject_MakeTpCall Objects/call.c:242
    #3 0x5790c8410fc7 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:166
    #4 0x5790c8411018 in PyObject_Vectorcall Objects/call.c:327
    #5 0x5790c883c457 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1502
    #6 0x5790c8472fda in _PyEval_EvalFrame Include/internal/pycore_ceval.h:119
    #7 0x5790c8473ce6 in gen_send_ex2 Objects/genobject.c:229
    #8 0x5790c84786a3 in gen_send_ex Objects/genobject.c:270
    #9 0x5790c847adb5 in _gen_throw Objects/genobject.c:543
    #10 0x5790c847b17c in gen_throw Objects/genobject.c:580
    #11 0x5790c883ebd2 in _PyEval_EvalFrameDefault Python/generated_cases.c.h:1640
    #12 0x5790c8883c6b in _PyEval_EvalFrame Include/internal/pycore_ceval.h:119
    #13 0x5790c8883f2f in _PyEval_Vector Python/ceval.c:1812
    #14 0x5790c84105c3 in _PyFunction_Vectorcall Objects/call.c:413
    #15 0x5790c8419a0e in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #16 0x5790c841b924 in method_vectorcall Objects/classobject.c:62
    #17 0x5790c8410ed7 in _PyObject_VectorcallTstate Include/internal/pycore_call.h:168
    #18 0x5790c8411018 in PyObject_Vectorcall Objects/call.c:327
    #19 0x5790c888107e in _PyEval_EvalFrameDefault Python/generated_cases.c.h:6205
    #20 0x5790c8472fda in _PyEval_EvalFrame Include/internal/pycore_ceval.h:119
    #21 0x5790c8473ce6 in gen_send_ex2 Objects/genobject.c:229
    #22 0x5790c84786a3 in gen_send_ex Objects/genobject.c:270
    #23 0x5790c8479898 in gen_close Objects/genobject.c:392
    #24 0x5790c8479dcd in _PyGen_Finalize Objects/genobject.c:106
    #25 0x5790c894557a in finalize_garbage Python/gc.c:980
    #26 0x5790c8947b80 in gc_collect_main Python/gc.c:1408
    #27 0x5790c8949657 in _PyGC_CollectNoFail Python/gc.c:1657
    #28 0x5790c89fc3c1 in finalize_modules Python/pylifecycle.c:1757
    #29 0x5790c8a08ab6 in _Py_Finalize Python/pylifecycle.c:2125
    #30 0x5790c8a08fbc in Py_FinalizeEx Python/pylifecycle.c:2252
    #31 0x5790c8ab41fa in Py_RunMain Modules/main.c:778
    #32 0x5790c8ab440a in pymain_main Modules/main.c:806
    #33 0x5790c8ab4787 in Py_BytesMain Modules/main.c:830
    #34 0x5790c813ab41 in main Programs/python.c:15
    #35 0x7c8cc7c35487  (/usr/lib/libc.so.6+0x27487) (BuildId: 0b707b217b15b106c25fe51df3724b25848310c0)
    #36 0x7c8cc7c3554b in __libc_start_main (/usr/lib/libc.so.6+0x2754b) (BuildId: 0b707b217b15b106c25fe51df3724b25848310c0)
    #37 0x5790c813aa64 in _start (/home/jacoblightning3/Documents/python/3.13debug/bin/python3.13+0x11a5a64) (BuildId: c2486f8b3c246fe979c4f6575406636585088b8a)

AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV Modules/_datetimemodule.c:2745 in delta_new
==9445==ABORTING
JelleZijlstra

JelleZijlstra commented on Apr 11, 2025

@JelleZijlstra
Member

On current 3.13 tip that line in datetime is

y = accum("seconds", x, second, CONST_US_PER_SECOND(st), &leftover_us);
which doesn't have an obvious bug. This sort of crash could be the result of memory corruption earlier in the program.

31 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    extension-modulesC modules in the Modules dirtype-crashA hard crash of the interpreter, possibly with a core dump

    Projects

    Status

    No status

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @JelleZijlstra@picnixz@ZeroIntensity@neonene@StanFromIreland

        Issue actions

          SIGSEV in `datetime.timedelta` (possibly from datetime's C `delta_new`) · Issue #132413 · python/cpython