test_interpreters.test_stress Crashes in managed_static_type_state_clear()

[ericsnowcurrently]

Crash report

What happened?

I haven't seen this before, but I don't think it's new. Rather, I suspect it's a very uncommon race in interpreter finalization. It might also be specific to ARM (or more likely there).

The same crash happened in two different tests in test_interpreters.test_stress:

• test_create_many_threaded
• test_many_threads_running_interp_in_other_interp

From https://buildbot.python.org/#/builders/234/builds/6611 run on the "aarch64 Fedora Stable Clang 3.x" buildbot worker:

test_create_many_threaded (test.test_interpreters.test_stress.StressTests.test_create_many_threaded) ... Warning -- Uncaught thread exception: InterpreterError
Exception in thread Thread-45 (task):
RuntimeError: Failed to import encodings module

During handling of the above exception, another exception occurred:

interpreters.InterpreterError: sub-interpreter creation failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang/build/Lib/threading.py", line 1041, in _bootstrap_inner
    self.run()
    ~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang/build/Lib/threading.py", line 992, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang/build/Lib/test/test_interpreters/test_stress.py", line 30, in task
    interp = interpreters.create()
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang/build/Lib/test/support/interpreters/__init__.py", line 76, in create
    id = _interpreters.create(reqrefs=True)
interpreters.InterpreterError: interpreter creation failed
ok
test_many_threads_running_interp_in_other_interp (test.test_interpreters.test_stress.StressTests.test_many_threads_running_interp_in_other_interp) ... ok

----------------------------------------------------------------------
Ran 152 tests in 33.434s
OK (skipped=5)
python: Objects/typeobject.c:296: void managed_static_type_state_clear(PyInterpreterState *, PyTypeObject *, int, int): Assertion `!_PyRuntime.types.managed_static.types[full_index].interp_count' failed.

On that same buildbot run:

test_many_threads_running_interp_in_other_interp (test.test_interpreters.test_stress.StressTests.test_many_threads_running_interp_in_other_interp) ... Warning -- Uncaught thread exception: InterpreterError
Exception in thread Thread-239 (run):
RuntimeError: Failed to import encodings module

During handling of the above exception, another exception occurred:

interpreters.InterpreterError: sub-interpreter creation failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang/build/Lib/threading.py", line 1041, in _bootstrap_inner
    self.run()
    ~~~~~~~~^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang/build/Lib/threading.py", line 992, in run
    self._target(*self._args, **self._kwargs)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang/build/Lib/test/test_interpreters/test_stress.py", line 47, in run
    interp = interpreters.create()
  File "/home/buildbot/buildarea/3.x.cstratak-fedora-stable-aarch64.clang/build/Lib/test/support/interpreters/__init__.py", line 76, in create
    id = _interpreters.create(reqrefs=True)
interpreters.InterpreterError: interpreter creation failed
ok

----------------------------------------------------------------------
Ran 152 tests in 32.930s

OK (skipped=5)
python: Objects/typeobject.c:296: void managed_static_type_state_clear(PyInterpreterState *, PyTypeObject *, int, int): Assertion `!_PyRuntime.types.managed_static.types[full_index].interp_count' failed.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Other

Output from running 'python -VV' on the command line:

No response

[ericsnowcurrently]

This may affect 3.13 too, but we'd need to double-check.

[freakboy3742]

@ericsnowcurrently Some more data for you - this is a semi-regular failure on the iOS buildbot. There have been 6 examples of in the last 2 months:

• https://buildbot.python.org/#/builders/1380/builds/2955
• https://buildbot.python.org/#/builders/1380/builds/2860
• https://buildbot.python.org/#/builders/1380/builds/2813
• https://buildbot.python.org/#/builders/1380/builds/2528
• https://buildbot.python.org/#/builders/1380/builds/2481
• https://buildbot.python.org/#/builders/1380/builds/2425

This may affect 3.13 too, but we'd need to double-check.

I can confirm that I've also seen it on 3.13 (though less frequently, probably because there are less builds):

• https://buildbot.python.org/#/builders/1386/builds/988