Repository navigation
deadlock in channeld caused by gossip traffic #2286
Description
Issue and Steps to Reproduce
Was testing my own PR #2274 on main net, opening a channel with satoshis.place. But one day after funding broadcast, the channel was still in state AWAITING_LOCKIN even though tx had many confirmations. Doing a ping with the peer hanged ... which reminded of #1928 and #1943
My guess is that this is similar to #1943 only this time it is channeld that is blocking on writing out gossip msg to peer (see below).
The socket buffers are not empty.
ss -4 -t -p | egrep "lightning|State"
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 268428 33176 192.168.1.166:33074 88.98.213.235:9735 users:(("lightning_chann",pid=21151,fd=3))
Attaching gdb to channeld while hanging in the `ping` command.
...
Attaching to process 21151
Reading symbols from /usr/local/libexec/c-lightning/lightning_channeld...done.
...
(gdb) backtrace
#0 0x00007fe50b682730 in __write_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000555de42722ab in write_all (fd=3, data=0x555de4f6b538, size=466) at ccan/ccan/read_write_all/read_write_all.c:11
#2 0x0000555de424093b in sync_crypto_write (cs=0x555de4f691d8, fd=3, msg=0x555de4f6b358) at common/crypto_sync.c:44
#3 0x0000555de424860f in handle_gossip_msg (peer_fd=3, cs=0x555de4f691d8, msg=0x555de4f69c78 "\r\256\001\260\001") at common/read_peer_msg.c:95
#4 0x0000555de4236ff7 in main (argc=1, argv=0x7fff1cadd798) at channeld/channeld.c:2909
kill and restart ... locks it right away
ss -4 -t -p | egrep "lightning|State"
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 255465 27808 192.168.1.166:57568 88.98.213.235:9735 users:(("lightning_chann",pid=30032,fd=3))
(gdb) where
#0 0x00007f4321a04730 in __write_nocancel () at ../sysdeps/unix/syscall-template.S:84
#1 0x0000563a3afb12ab in write_all (fd=3, data=0x563a3c742598, size=466) at ccan/ccan/read_write_all/read_write_all.c:11
#2 0x0000563a3af7f93b in sync_crypto_write (cs=0x563a3c7401d8, fd=3, msg=0x563a3c742358) at common/crypto_sync.c:44
#3 0x0000563a3af8760f in handle_gossip_msg (peer_fd=3, cs=0x563a3c7401d8, msg=0x563a3c740c78 "\r\256\001\260\001") at common/read_peer_msg.c:95
#4 0x0000563a3af75ff7 in main (argc=1, argv=0x7ffc79d565c8) at channeld/channeld.c:2909
getinfo output
{
"id": "02682020a37a3b5876b16bffbf0ac56d7f379695a5d273cf16ed0f35f18ab2dc98",
"alias": "LookBusy4",
"color": "026820",
"num_peers": 1,
"num_pending_channels": 0,
"num_active_channels": 1,
"num_inactive_channels": 0,
"address": [
{
"type": "ipv4",
"address": "x.x.x.x",
"port": 9735
}
],
"binding": [
{
"type": "ipv4",
"address": "192.168.1.166",
"port": 9735
}
],
~"version": this one: https://github.com/ElementsProject/lightning/pull/2274/commits/8bc75a5d4893175d01cec590dfda5276a043f41f
"blockheight": 559480,
"network": "bitcoin",
"msatoshi_fees_collected": 0
}
When I modify channeld's while loop
Line 2872 in 018a3f1
peer->from_gossipd queue ?
Will likely make PR.
Activity
channeld: prioritize read from peer over (read from gossipd and) writ…