Repository navigation
No channel or funds after node crash #2227
Description
Issue and Steps to Reproduce
Tried to open a 2 million satoshi channel to Opennode when my node crashed.
lightning-cli fundchannel 03abf6f44c355dec0d5aa155bdbdd6e0c8fefe318eff402de65c6eb2e1be55dc3e all
I now have no channel with Opennode but the 2M satoshi are no longer visible as well (with listfunds)
I was able to trace the transaction id and the bech32 address that has the unspent 2M satoshi:
9f6e8589d5756182a57c2b4d5a6f4777c97188e9fde92e48eeebf9137a9fb597
bc1q53jdv0mf964tey9pz52mmk002x77uujnut4uac80x40njwyqzn7q8uue9r
Later I also found the txid in the crash log.
The relevant part of the crash log is here:
+805.324522623 lightningd(26404):DEBUG: 035607df8a5e6293524a60aa4b1231b735fff71fc7bd837a12bcc2db1c417322aa (tx 9f6e8589d5756182a57c2b4d5a6f4777c97188e9fde92e48eeebf9137a9fb597)
+805.325991429 lightningd(26404):DEBUG: sendrawtransaction: 020000000001019d65c32a7a6c59cc1ea36b8ec0d6af24c667706be7a36aa9ce889cb529c29c960100000000ffffffff010bc31e0000000000220020a464d63f692eaabc90a11515bdd9ef51bdee7253e2ebcee0ef355f39388014fc0247304402201fd382e4ec0ffcd33c074fd281afbf6d5232a745989a624da02c4b9ccdfcee03022014a02a8e8ec9cc057bb5104a30dee490c9ffe1eeb951517e1bb93e6c4def649901210364b5e27e97f80c14583d40a893e8d30e99bc7f738d063af8ddcb756d4a631f8300000000
+805.330887361 lightningd(26404):BROKEN: Unable to mark output as spent
+805.482003996 lightningd(26404):BROKEN: FATAL SIGNAL 6 (version v0.6.2-295-g6f027a2)
+805.482021440 lightningd(26404):BROKEN: backtrace: common/daemon.c:45 (crashdump) 0x437cd7
+805.482030570 lightningd(26404):BROKEN: backtrace: (null):0 ((null)) 0x7f798ef6b4af
+805.482036467 lightningd(26404):BROKEN: backtrace: (null):0 ((null)) 0x7f798ef6b428
+805.482041395 lightningd(26404):BROKEN: backtrace: (null):0 ((null)) 0x7f798ef6d029
+805.482049072 lightningd(26404):BROKEN: backtrace: lightningd/log.c:624 (fatal) 0x419ecf
+805.482054666 lightningd(26404):BROKEN: backtrace: wallet/wallet.c:246 (wallet_confirm_utxos) 0x4634f4
+805.482063697 lightningd(26404):BROKEN: backtrace: lightningd/opening_control.c:383 (opening_funder_finished) 0x41cacf
+805.482070598 lightningd(26404):BROKEN: backtrace: lightningd/opening_control.c:660 (openingd_msg) 0x41d600
+805.482079229 lightningd(26404):BROKEN: backtrace: lightningd/subd.c:474 (sd_msg_read) 0x43252d
+805.482085478 lightningd(26404):BROKEN: backtrace: lightningd/subd.c:302 (read_fds) 0x431ec3
+805.482091612 lightningd(26404):BROKEN: backtrace: ccan/ccan/io/io.c:59 (next_plan) 0x474cbe
+805.482097354 lightningd(26404):BROKEN: backtrace: ccan/ccan/io/io.c:395 (do_plan) 0x4757b3
+805.482102889 lightningd(26404):BROKEN: backtrace: ccan/ccan/io/io.c:405 (io_ready) 0x4757f1
+805.482110145 lightningd(26404):BROKEN: backtrace: ccan/ccan/io/poll.c:310 (io_loop) 0x477351
+805.482116004 lightningd(26404):BROKEN: backtrace: lightningd/lightningd.c:823 (main) 0x417ff3
+805.482123163 lightningd(26404):BROKEN: backtrace: (null):0 ((null)) 0x7f798ef5682f
+805.482128206 lightningd(26404):BROKEN: backtrace: (null):0 ((null)) 0x4037f8
+805.482132688 lightningd(26404):BROKEN: backtrace: (null):0 ((null)) 0xffffffffffffffff
getinfo output
{
"id": "02c5371591b640da03f9a645e2ddfa5620d8537388b03ddd524d36766c0f550db0",
"alias": "Bitkoins.nl",
"color": "23354d",
"num_peers": 48,
"num_pending_channels": 3,
"num_active_channels": 45,
"num_inactive_channels": 0,
"address": [
{
"type": "ipv4",
"address": "82.217.214.215",
"port": 9735
}
],
"binding": [
{
"type": "ipv4",
"address": "192.168.178.20",
"port": 9735
}
],
"version": "v0.6.2-295-g6f027a2",
"blockheight": 557446,
"network": "bitcoin",
"msatoshi_fees_collected": 11645
}
Activity
cdecker commentedon Jan 7, 2019
That looks a lot like a database error, or an inconsistency. You should be able to fix it using
lightning-cli dev-rescan-outputs. If the funds are still not there restart the node withlightningd --rescan=5000 [other options]to rescan the blockchain.cdecker commentedon Jan 7, 2019
As for root cause, the state of the output must have changed while we were negotiating with the peer. Did you call
dev-rescan-outputsor change the DB manually in any way?cdecker commentedon Jan 23, 2019
I think it might actually have been a funding that went wrong, and the next funding attempting to use the change output from the failed one, which then results in the funds being displayed twice. This will get a lot better once #2274 is merged, with that we can handle failing fundings better.
I'll close this for now, but feel free to ping us should it reappear.