Skip to content

seg fault on libtorch_cpu Node destructor after ERR Function returned unexpected number of outputs #422

Open
@filipecosta90

Description

@filipecosta90

RedisAI version: c6c3dd0

spin up standalone redis-server ( force reuse environment for tests )

redis-server  --loadmodule /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./redisai.so

Issue tests while reusing environment

RedisAI/test$ RLTest --env existing-env -t tests_pytorch.py

catch with gdb

$ sudo gdb $(which redis-server) $(pgrep redis-server)
[sudo] password for filipe: 
GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/redis-server...done.
Attaching to program: /usr/local/bin/redis-server, process 11021
[New LWP 11022]
[New LWP 11023]
[New LWP 11024]
[New LWP 11025]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fa91d311bb7 in epoll_wait (epfd=5, events=0x55b3d7103180, maxevents=4192, timeout=100) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
30	../sysdeps/unix/sysv/linux/epoll_wait.c: No such file or directory.
(gdb) continue
Continuing.

Thread 1 "redis-server" received signal SIGABRT, Aborted.
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007fa91d230801 in __GI_abort () at abort.c:79
#2  0x00007fa91d279897 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7fa91d3a6b9a "%s\n") at ../sysdeps/posix/libc_fatal.c:181
#3  0x00007fa91d28090a in malloc_printerr (str=str@entry=0x7fa91d3a4d88 "free(): invalid pointer") at malloc.c:5350
#4  0x00007fa91d287e1c in _int_free (have_lock=0, p=0x7fa904003220, av=0x7fa91d5dbc40 <main_arena>) at malloc.c:4157
#5  __GI___libc_free (mem=0x7fa904003230) at malloc.c:3124
#6  0x00007fa90e1db609 in torch::jit::Node::~Node() () from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./backends/redisai_torch/lib/libtorch_cpu.so
#7  0x00007fa90e1c313f in torch::jit::Graph::~Graph() () from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./backends/redisai_torch/lib/libtorch_cpu.so
#8  0x00007fa90e1e7921 in torch::jit::GraphFunction::~GraphFunction() () from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./backends/redisai_torch/lib/libtorch_cpu.so
#9  0x00007fa90e19ef21 in std::_Sp_counted_ptr_inplace<torch::jit::CompilationUnit, std::allocator<torch::jit::CompilationUnit>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() ()
   from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./backends/redisai_torch/lib/libtorch_cpu.so
#10 0x00007fa91a3e5126 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
   from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./backends/redisai_torch/redisai_torch.so
#11 0x00007fa91a3e5db4 in std::_Sp_counted_ptr_inplace<torch::jit::Module, std::allocator<torch::jit::Module>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() ()
   from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./backends/redisai_torch/redisai_torch.so
#12 0x00007fa91a3e5126 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
   from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./backends/redisai_torch/redisai_torch.so
#13 0x00007fa91a3da7a5 in torchDeallocContext () from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./backends/redisai_torch/redisai_torch.so
#14 0x00007fa91c60e5b4 in RAI_ModelFree () from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./redisai.so
#15 0x00007fa91c60e681 in RAI_Model_DTFree () from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./redisai.so
#16 0x000055b3d59a0c3f in freeModuleObject (o=<optimized out>) at object.c:341
#17 0x000055b3d59a0ce5 in decrRefCount (o=0x55b3d7edd470) at object.c:369
#18 0x000055b3d59865f1 in _dictClear (d=d@entry=0x55b3d710fb00, ht=ht@entry=0x55b3d710fb10, callback=callback@entry=0x0) at dict.c:455
#19 0x000055b3d5987068 in dictEmpty (d=0x55b3d710fb00, callback=callback@entry=0x0) at dict.c:1020
#20 0x000055b3d59a6eb9 in emptyDbGeneric (dbarray=<optimized out>, dbnum=dbnum@entry=-1, flags=<optimized out>, callback=callback@entry=0x0) at db.c:417
#21 0x000055b3d59a6ffa in emptyDb (callback=0x0, flags=<optimized out>, dbnum=-1) at db.c:444
#22 flushAllDataAndResetRDB (flags=<optimized out>) at db.c:512
#23 0x000055b3d59a70a0 in flushallCommand (c=0x55b3d7ecf770) at db.c:557
#24 0x000055b3d598abdb in call (c=0x55b3d7ecf770, flags=15) at server.c:3268
#25 0x000055b3d598b60d in processCommand (c=c@entry=0x55b3d7ecf770) at server.c:3652
#26 0x000055b3d5999280 in processCommandAndResetClient (c=c@entry=0x55b3d7ecf770) at networking.c:1785
#27 0x000055b3d599dbdf in processInputBuffer (c=0x55b3d7ecf770) at networking.c:1867
#28 0x000055b3d5a1d5da in callHandler (handler=<optimized out>, conn=0x55b3d7b47140) at connhelpers.h:79
#29 connSocketEventHandler (el=<optimized out>, fd=<optimized out>, clientData=0x55b3d7b47140, mask=<optimized out>) at connection.c:281
#30 0x000055b3d5984063 in aeProcessEvents (eventLoop=eventLoop@entry=0x55b3d7096630, flags=flags@entry=27) at ae.c:479
#31 0x000055b3d598446d in aeMain (eventLoop=0x55b3d7096630) at ae.c:539
#32 0x000055b3d5980abb in main (argc=<optimized out>, argv=0x7ffe9a3dc478) at server.c:5173

redis-server log

(...)
11021:M 28 Jun 2020 18:10:43.287 * <ai> Redis version found by RedisAI: 6.0.5 - oss
11021:M 28 Jun 2020 18:10:43.287 * <ai> RedisAI version 999999, git_sha=c6c3dd03b4b9c2450428e60d863e8b23b25dfd87
11021:M 28 Jun 2020 18:10:43.288 * Module 'ai' loaded from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./redisai.so
11021:M 28 Jun 2020 18:10:43.288 * Ready to accept connections
11021:M 28 Jun 2020 18:11:49.391 # <ai> backend TORCH not loaded, will try loading default backend
11021:M 28 Jun 2020 18:11:50.425 * <ai> TORCH backend loaded from /home/filipe/redislabs/RedisAI/bin/linux-x64-release/install-cpu/./backends/redisai_torch/redisai_torch.so
11021:M 28 Jun 2020 18:11:50.439 * DB saved on disk
11021:M 28 Jun 2020 18:11:50.450 # <ai> ERR Function returned unexpected number of outputs - forward
free(): invalid pointer
Aborted (core dumped)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions