FATAL SIGNAL 6 when estimating fees

[fixone]

seems (from the stack trace) that a fee estimation triggered a crash, the log is below

2018-08-25T16:51:06.636Z lightning_connectd(1667): Failed connected out for 02368ff86eb0d15146ea7141acce3455e9a10079d82c33a118936ce147dbb11459: 34.244.171.203:9735: Connection establishment: Connection refused. 34.244.171.203:9735: Connection establishment: Connection refused.
2018-08-25T16:51:12.509Z lightningd(1658): FATAL SIGNAL 6 (version v0.6-337-ga3178b8)
2018-08-25T16:51:12.509Z lightningd(1658): backtrace: common/daemon.c:42 (crashdump) 0x430b19
2018-08-25T16:51:12.509Z lightningd(1658): backtrace: (null):0 ((null)) 0x7f7207b9d4af
2018-08-25T16:51:12.509Z lightningd(1658): backtrace: (null):0 ((null)) 0x7f7207b9d428
2018-08-25T16:51:12.509Z lightningd(1658): backtrace: (null):0 ((null)) 0x7f7207b9f029
2018-08-25T16:51:12.509Z lightningd(1658): backtrace: (null):0 ((null)) 0x7f7207bdf7e9
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: (null):0 ((null)) 0x7f7207bea650
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: (null):0 ((null)) 0x7f7207beb2af
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: (null):0 ((null)) 0x7f7207bec838
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: ccan/ccan/tal/tal.c:692 (tal_resize_) 0x478878
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: lightningd/bitcoind.c:28 (add_arg) 0x403a3c
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: lightningd/bitcoind.c:64 (gather_args) 0x403c81
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: lightningd/bitcoind.c:274 (start_bitcoin_cli) 0x404644
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: lightningd/bitcoind.c:354 (do_one_estimatefee) 0x4049b4
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: lightningd/bitcoind.c:343 (process_estimatefee) 0x404901
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: lightningd/bitcoind.c:189 (bcli_finished) 0x404284
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: ccan/ccan/io/poll.c:183 (destroy_conn) 0x46c4f1
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: ccan/ccan/io/poll.c:189 (destroy_conn_close_fd) 0x46c511
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: ccan/ccan/tal/tal.c:235 (notify) 0x4778fe
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: ccan/ccan/tal/tal.c:395 (del_tree) 0x477daf
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: ccan/ccan/tal/tal.c:479 (tal_free) 0x478135
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: ccan/ccan/io/io.c:443 (io_close) 0x46aee8
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: ccan/ccan/io/poll.c:314 (io_loop) 0x46c900
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: lightningd/lightningd.c:455 (main) 0x41512f
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: (null):0 ((null)) 0x7f7207b8882f
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: (null):0 ((null)) 0x403618
2018-08-25T16:51:12.510Z lightningd(1658): backtrace: (null):0 ((null)) 0xffffffffffffffff

[rustyrussell]

That's deeply weird; what version, and is this bitcoind or spruned?

[fixone]

version v0.6-337-ga3178b8 with bitcoind

[iemwill]

had this one too today, in the end I just let my pi a few hours offline and booted again, started c-lightning, had the same error but lightning didn't shut down and started normally after 2-4 minutes...
If needed I can check the logs.

[SimonVrouwe]

@iemwill

had the same error but lightning didn't shut down

Are you sure it was the same error? FATAL usually means that it kills the program.

[rustyrussell]

OK, I think this is unrelated memory corruption, probably not related to estimatefee at all :(

It's dying inside realloc(), which implies we've overrun a buffer somewhere. There have been no obvious candidates recently, unfortunately. I will keep this open, with an eye to finding it.

[cdecker]

Would running with valgrind help here? It's a long shot, since it's not sure to be reproducible, but it might just work.

[iemwill]

@SimonVrouwe sorry for answering so late!

@iemwill

had the same error but lightning didn't shut down

Are you sure it was the same error? FATAL usually means that it kills the program.

I'm not really sure that it was again a fatal error after the offline time and reboot.

But before that the "Fatal Signal 6" error and following shutdown/kill of the program i am 100% sure.

Won't get to the log again and am learning right now for the future to just post the log!