Fatal signal 11 received on Mac OS

[bitstein]

Issue and Steps to Reproduce

I am running Mac OS X 10.13.2. With the new libwally update, I was able to build c-lightning just fine this morning. However, after I got my bitcoind testnet daemon synced and booted lightningd, upon making a command, I received this output in the lightningd log:

lightningd(68725): Connected json input
lightningd(68725): FATAL SIGNAL 11 RECEIVED

getinfo output

getinfo hangs, no output.

[cdecker]

Can you run with a debugger? gdb lightningd might work, and once it breaks type in bt to see a backtrace. The copy paste it here :-)

[yashbhutwala]

@cdecker I'm having the same issues as @bitstein on a Mac. I tried to run debug using lightningd, but for me both the daemon and the client hang upon any request, so I'm not getting a backtrace.

[adam-hudspith]

@bitstein , @cdecker @yashbhutwala I'm getting the same problem. I managed to get this with lightning-cli help :

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
    frame #0: 0x00007fff73cf041a libsystem_kernel.dylib`read + 10
  * frame #1: 0x0000000104730b1a lightningd`check_with_child(batch=0x00007ffeeb540608, p=0x00000001046be000, size=8, is_write=false) at ptr_valid.c:258
    frame #2: 0x000000010473087b lightningd`ptr_valid_batch(batch=0x00007ffeeb540608, p=0x00000001046be000, alignment=1, size=8, write=false) at ptr_valid.c:289
    frame #3: 0x000000010471cf8f lightningd`autodata_make_table(example=0x00000001047d0d70, name="json_command", nump=0x00000001047d1b20) at autodata.c:33
    frame #4: 0x00000001046caedf lightningd`get_cmdlist at jsonrpc.c:281
    frame #5: 0x00000001046cc335 lightningd`find_cmd(buffer="{ \"method\" : \"help\", \"id\" : \"lightning-cli-68421\", \"params\" : [ ] }", tok=0x00007fc51d009978) at jsonrpc.c:312
    frame #6: 0x00000001046cc0de lightningd`parse_request(jcon=0x00007fc51d000760, tok=0x00007fc51d009950) at jsonrpc.c:574
    frame #7: 0x00000001046cbb3c lightningd`read_json(conn=0x00007fc51d0084c0, jcon=0x00007fc51d000760) at jsonrpc.c:667
    frame #8: 0x0000000104726e83 lightningd`next_plan(conn=0x00007fc51d0084c0, plan=0x00007fc51d0084f0) at io.c:59
    frame #9: 0x0000000104727b6c lightningd`do_plan(conn=0x00007fc51d0084c0, plan=0x00007fc51d0084f0, idle_on_epipe=false) at io.c:387
    frame #10: 0x00000001047279c4 lightningd`io_ready(conn=0x00007fc51d0084c0, pollflags=1) at io.c:397
    frame #11: 0x0000000104729397 lightningd`io_loop(timers=0x00007fc51bc028d0, expired=0x00007ffeeb540a00) at poll.c:305
    frame #12: 0x00000001046ccc56 lightningd`main(argc=4, argv=0x00007ffeeb540a68) at lightningd.c:347
    frame #13: 0x00007fff73b9f115 libdyld.dylib`start + 1
    frame #14: 0x00007fff73b9f115 libdyld.dylib`start + 1

[yashbhutwala]

Is it possible to run a lightning node in a container but bitcoind on the host? I've tried but to no avail.

[cdecker]

@yashbhutwala yes, you just need to make sure that bitcoin-cli from the docker container can talk to bitcoind on the host. Something like the following should do it:

docker run -v $HOME/.bitcoin:/root/.bitcoin --net=host docker/image

That'll mount the hosts .bitcoin directory in the docker image so it can read the RPC config and it'll allow bitcoin-cli to connect to bitcoind over the loopback interface.

[yashbhutwala]

@cdecker I've already tried that but container is still not able to talk to the bitcoind server on my machine. --net=host (docker host) seems to have a different ip than my machine.

[yashbhutwala]

Upon some research, it turns out Docker on Mac works such that Docker host is not the same as localhost, there is no docker0 bridge on macOS.

[Sjors]

I get the same reader FATAL SIGNAL 11 RECEIVED on MacOS 10.13.3

gdb --args lightningd/lightningd --network=testnet --port=9000 --log-level=debug
start
During startup program terminated with signal ?, Unknown signal.

I'm not familiar with gdb. How do I make it spit out more useful information?

[Sjors]

Note that lightningd/lightningd --help doesn't crash.

Also, if bitcoind isn't running, I get FATAL SIGNAL 6 RECEIVED and the lightningd process stays active. I then have to kill it with killall -9 lightningd as cli/lightning-cli stop will complain Connecting to 'lightning-rpc': No such file or directory.

[Sjors]

@benharold have you encountered this error during your work on Voltage? Or do you always run c-lightning on a remote linux server?

[benharold]

@Sjors I have always run on a remote linux server.

[laanwj]

I get similar backtraces in autodata_make_table→run_child on FreeBSD, though surprisingly it seems the process keeps running, and working. It does create core files every time though (maybe no more after #922).

While running getinfo, in lightningd:

#5  <signal handler called>
#6  run_child (infd=17, outfd=20) at ccan/ccan/ptr_valid/ptr_valid.c:190
#7  0x00000000004760fc in create_child (batch=0x7fffffffe618) at ccan/ccan/ptr_valid/ptr_valid.c:216
#8  0x0000000000475ce5 in check_with_child (batch=0x7fffffffe618, p=0x733000, size=8, is_write=false) at ccan/ccan/ptr_valid/ptr_valid.c:245
#9  0x0000000000475afb in ptr_valid_batch (batch=0x7fffffffe618, p=0x733000, alignment=1, size=8, write=false) at ccan/ccan/ptr_valid/ptr_valid.c:289
#10 0x000000000046283b in autodata_make_table (example=0x733950, name=0x4f6a6a "json_command", nump=0x735830) at ccan/ccan/autodata/autodata.c:38
#11 0x000000000040e949 in get_cmdlist () at lightningd/jsonrpc.c:286
#12 0x000000000040fdd5 in find_cmd (buffer=0x801a367a0 "{ \"method\" : \"getinfo\", \"id\" : \"lightning-cli-82041\", \"params\" : [ ] }\034", tok=0x801d408c8) at lightningd/jsonrpc.c:346
#13 0x000000000040fbbb in parse_request (jcon=0x801b7d060, tok=0x801d408a0) at lightningd/jsonrpc.c:608
#14 0x000000000040f5ef in read_json (conn=0x801d40600, jcon=0x801b7d060) at lightningd/jsonrpc.c:794
#15 0x000000000046c576 in next_plan (conn=0x801d40600, plan=0x801d40630) at ccan/ccan/io/io.c:59
#16 0x000000000046d219 in do_plan (conn=0x801d40600, plan=0x801d40630, idle_on_epipe=false) at ccan/ccan/io/io.c:387
#17 0x000000000046d084 in io_ready (conn=0x801d40600, pollflags=1) at ccan/ccan/io/io.c:397
#18 0x000000000046ea3b in io_loop (timers=0x801a20108, expired=0x7fffffffe9d0) at ccan/ccan/io/poll.c:305
#19 0x0000000000410717 in main (argc=5, argv=0x7fffffffea78) at lightningd/lightningd.c:35

While running connect, in gossipd:

#5  <signal handler called>
#6  run_child (infd=7, outfd=10) at ccan/ccan/ptr_valid/ptr_valid.c:190
#7  0x000000000043da0c in create_child (batch=0x7fffffffe7a8) at ccan/ccan/ptr_valid/ptr_valid.c:216
#8  0x000000000043d5f5 in check_with_child (batch=0x7fffffffe7a8, p=0x6dd000, size=8, is_write=false) at ccan/ccan/ptr_valid/ptr_valid.c:245
#9  0x000000000043d40b in ptr_valid_batch (batch=0x7fffffffe7a8, p=0x6dd000, alignment=1, size=8, write=false) at ccan/ccan/ptr_valid/ptr_valid.c:289
#10 0x000000000042a14b in autodata_make_table (example=0x6dd6c0, name=0x4b8e01 "type_to_string", nump=0x6decc8) at ccan/ccan/autodata/autodata.c:38
#11 0x0000000000416c01 in type_to_string_ (ctx=0x801a15120, typename=0x4b56ad "struct pubkey", u={}) at common/type_to_string.c:24
#12 0x00000000004069da in connection_out (conn=0x801a1c1e0, reach=0x801be6020) at gossipd/gossip.c:1565
#13 0x0000000000433e86 in next_plan (conn=0x801a1c1e0, plan=0x801a1c240) at ccan/ccan/io/io.c:59
#14 0x0000000000434b29 in do_plan (conn=0x801a1c1e0, plan=0x801a1c240, idle_on_epipe=false) at ccan/ccan/io/io.c:387
#15 0x0000000000434a03 in io_ready (conn=0x801a1c1e0, pollflags=4) at ccan/ccan/io/io.c:403
#16 0x000000000043634b in io_loop (timers=0x801bc60d0, expired=0x7fffffffea08) at ccan/ccan/io/poll.c:305
#17 0x0000000000402c21 in main (argc=1, argv=0x7fffffffeaa8) at gossipd/gossip.c:2029

I'll try to debug, but I couldn't figure out quickly what this code was supposed to do in the first place.