Repository navigation
Channel "AWAITING_UNILATERAL" since days. Attempted force closing, CPU 100% #2171
Description
I found one of my channel was in "AWAITING_UNILATERAL" so I waited but days after it was still in that state.
Relevant log with grep peerID:
2018-11-25T13:41:40.283Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: 0: 10648 satoshi (SEGWIT) 695254bb08add33af6f907960913869c25903c967541b2b8765be96431d2cb59? [100/260]
2018-11-25T13:41:40.283Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: 1: 2005802 satoshi (SEGWIT) bd729c73e2cd2647f5ce6a2ebd24419720486f720124d9e0c66649b0b3d2835d?
2018-11-25T13:41:40.283Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: 2: 252848 satoshi (SEGWIT) bbe3783d4c7fde8eb6433403fc1c34fa0098e4c738cedeac087243a1655a0b81?
2018-11-25T13:41:40.300Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: Getting HSM to sign funding tx
2018-11-25T13:41:40.397Z lightningd(17827): lightning_channeld-03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: pid 31244, msgfd 33
2018-11-25T13:41:40.397Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: Waiting for funding confirmations
2018-11-25T13:41:41.001Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: Peer internal error CHANNELD_AWAITING_LOCKIN: Funding broadcast exited with 25: error code: -25?error message:?Missing inputs?
2018-11-25T13:41:41.001Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: Peer permanent failure in CHANNELD_AWAITING_LOCKIN: Internal error: Funding broadcast exited with 25: error code: -25?error message:?Missing inputs?
2018-11-25T13:41:41.001Z lightningd(17827): lightning_channeld-03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: Status closed, but not exited. Killing
2018-11-25T13:41:41.013Z lightningd(17827): 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10: State changed from CHANNELD_AWAITING_LOCKIN to AWAITING_UNILATERAL
2018-11-25T13:42:58.779Z lightning_gossipd(17983): Trying to find a route from 03b49e64e9bce0de005d774c3fbe39dcafd9ce96b6c8ffaaf3a1e111d4f9554e2e to 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 for 150096 msatoshi
2018-11-25T13:42:58.779Z lightningd(17827): pay(0x5599e8874d28): sendpay via route: us -> 551183:2675:1 (151109msat, 590blk) -> 036ac04a69f2ab587fd83c4f2aca255a81f138276fbca344ac3ecfae28d733be56 -> 532217:2079:0 (150109msat, 446blk) -> 02529db69fd2ebd3126fb66fafa234fc3544477a23d509fe93ed229bb0e92e4f
b8 -> 548838:1988:1 (150098msat, 302blk) -> 031d2bbc75802689312220a017c6b51fa246efc59c7aa9355f6f7395038ffb4d6a -> 549057:2281:0 (150097msat, 158blk) -> 02c5371591b640da03f9a645e2ddfa5620d8537388b03ddd524d36766c0f550db0 -> 548930:191:0 (150096msat, 144blk) -> 03e50492eab4107a773141bb419e107bda3de3d55
652e6e1a41225f06a0bbf2d56
2018-11-25T13:43:04.954Z lightning_gossipd(17983): Trying to find a route from 03b49e64e9bce0de005d774c3fbe39dcafd9ce96b6c8ffaaf3a1e111d4f9554e2e to 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 for 150218 msatoshi
2018-11-25T13:43:04.955Z lightningd(17827): pay(0x5599e8874d28): sendpay via route: us -> 551183:2675:1 (152234msat, 446blk) -> 036ac04a69f2ab587fd83c4f2aca255a81f138276fbca344ac3ecfae28d733be56 -> 550459:2332:1 (151234msat, 302blk) -> 0390b5d4492dc2f5318e5233ab2cebf6d48914881a33ef6a9c6bcdbb433ad986
d0 -> 550102:1165:0 (150219msat, 158blk) -> 02c5371591b640da03f9a645e2ddfa5620d8537388b03ddd524d36766c0f550db0 -> 548930:191:0 (150218msat, 144blk) -> 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56
2018-11-25T13:43:11.852Z lightning_gossipd(17983): Trying to find a route from 03b49e64e9bce0de005d774c3fbe39dcafd9ce96b6c8ffaaf3a1e111d4f9554e2e to 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 for 150170 msatoshi
2018-11-25T13:43:11.852Z lightningd(17827): pay(0x5599e8874d28): sendpay via route: us -> 551183:2675:1 (152271msat, 864blk) -> 036ac04a69f2ab587fd83c4f2aca255a81f138276fbca344ac3ecfae28d733be56 -> 544611:1240:1 (151271msat, 720blk) -> 020e56a13babec99abdc2c4afbe34e1e44230d79b234c059fd4ff1e367765fdb
1b -> 541282:1481:0 (150271msat, 576blk) -> 03434a39cd9a537c852fc8fb72454086d726f9111e9f730cef4985c39c11fae944 -> 535271:1925:0 (150171msat, 432blk) -> 03bb88ccc444534da7b5b64b4f7b15e1eccb18e102db0e400d4b9cfe93763aa26d -> 549187:1932:1 (150170msat, 288blk) -> 03295d2e292565743a40bd44da227a820f873087
7bc3dfadebade8785bcf355258 -> 544098:2501:1 (150170msat, 144blk) -> 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56
2018-11-25T13:43:18.520Z lightning_gossipd(17983): Trying to find a route from 03b49e64e9bce0de005d774c3fbe39dcafd9ce96b6c8ffaaf3a1e111d4f9554e2e to 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 for 150426 msatoshi
2018-11-25T13:43:18.520Z lightningd(17827): pay(0x5599e8874d28): sendpay via route: us -> 551183:2675:1 (152615msat, 474blk) -> 036ac04a69f2ab587fd83c4f2aca255a81f138276fbca344ac3ecfae28d733be56 -> 550180:1827:0 (151615msat, 330blk) -> 03a5927b64b1ea8657d5b770d61a3e2d0554fdb5d568773e0d6a92907913ca21
f6 -> 551184:469:0 (150628msat, 186blk) -> 0217090c397c3e8ced66abb7b8bf6e05cd49adde55766a7d14e2e4a541164b0935 -> 551318:1032:0 (150428msat, 172blk) -> 03f18da23535adee245903c81750745197e3079d3b88b3898981c90a5f2411008c -> 544620:1321:1 (150427msat, 158blk) -> 02c88a0df5e9ea7df582647f79b70b8e441047de3
08fd6b24408b8bd775f2234bd -> 549696:815:1 (150426msat, 144blk) -> 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56
Today I have tried to force close the channel but the command is stucked and the CPU is 100%.
lightning-cli close 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 true
I have interrupted the command but it seems like every command (getinfo included) is causing 100% CPU and the freezing of the command itself.
Restarting the service, I was able to collect some more infos:
$ lightning-cli listpeers 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 debug
{
"peers": [
{
"id": "03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56",
"connected": true,
"netaddr": [
"35.172.33.197:9735"
],
"global_features": "",
"local_features": "82",
"globalfeatures": "",
"localfeatures": "82",
"channels": [
{
"state": "AWAITING_UNILATERAL",
"scratch_txid": "458cd396b558361c2a616d0e2a0c56a50f5c8eacd9ca71e536d016bf3ae6233d",
"channel_id": "84649e5d4c01a17a4f1612c722df7be59e31cdf39f969a8794d2adcd54b1b731",
"funding_txid": "31b7b154cdadd294879a969ff3cd319ee57bdf22c712164f7aa1014c5d9e6484",
"msatoshi_to_us": 1000000000,
"msatoshi_to_us_min": 1000000000,
"msatoshi_to_us_max": 1000000000,
"msatoshi_total": 1000000000,
"dust_limit_satoshis": 546,
"max_htlc_value_in_flight_msat": 18446744073709551615,
"their_channel_reserve_satoshis": 10000,
"our_channel_reserve_satoshis": 10000,
"spendable_msatoshi": 990000000,
"htlc_minimum_msat": 0,
"their_to_self_delay": 144,
"our_to_self_delay": 144,
"max_accepted_htlcs": 483,
"status": [
"AWAITING_UNILATERAL:Attempting to reconnect"
],
"in_payments_offered": 0,
"in_msatoshi_offered": 0,
"in_payments_fulfilled": 0,
"in_msatoshi_fulfilled": 0,
"out_payments_offered": 0,
"out_msatoshi_offered": 0,
"out_payments_fulfilled": 0,
"out_msatoshi_fulfilled": 0,
"htlcs": [
]
}
],
"log": [
{
"type": "DEBUG",
"time": "5.221303026",
"source": "03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #10:",
"log": "Peer has reconnected, state AWAITING_UNILATERAL"
},
{
"type": "DEBUG",
"time": "5.225104641",
"source": "lightning_openingd-03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #11:",
"log": "pid 15044, msgfd 25"
},
{
"type": "DEBUG",
"time": "5.293310285",
"source": "lightning_openingd-03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #11:",
"log": "peer_out WIRE_ERROR"
},
{
"type": "DEBUG",
"time": "5.294715801",
"source": "lightning_openingd-03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56 chan #11:",
"log": "Handed peer, entering loop"
}
]
}
]
}
getinfo output
not possible (due to 100%CPU issue) but running tags/v0.6.2
EDIT: restarting the service:
lightning-cli getinfo
{
"id": "03b49e64e9bce0de005d774c3fbe39dcafd9ce96b6c8ffaaf3a1e111d4f9554e2e",
"address": [
{
"type": "torv2",
"address": "byy4lqhaaley7e7k.onion",
"port": 9735
}
],
"binding": [
{
"type": "ipv4",
"address": "127.0.0.1",
"port": 9731
},
{
"type": "ipv4",
"address": "10.2.206.180",
"port": 9731
}
],
"version": "v0.6.2-modded",
"blockheight": 553513,
"network": "bitcoin",
"msatoshi_fees_collected": 0
}
Activity
gabridome commentedon Dec 16, 2018
Yeah. Exactly. I will look into the log files and report.
gabridome commentedon Dec 17, 2018
These are some lines of the log in which also appears the opening tx with some error.
gabridome commentedon Dec 18, 2018
mmh. Honestly I don't know.
What I normally do is:
git checkout -b v0.6.2 tags/v0.6.2These steps could lead to a modded version?
Months ago I built a version to use
dev-forget-channel#1996 and was v0.6.1-modded(I don't know why that was "modded" either).
This is the last line of
git log;gabridome commentedon Dec 24, 2018
mmvphil commentedon Dec 25, 2018
I have similar issues awaiting forever.. looks like i lost a couple hundred usd on my LN node.. server was down for 8 days when i got it repaired and put it up.. i cant close channels. and have no outputs :(
gabridome commentedon Dec 25, 2018
Also:
Sorry to be a noob about sqlite3 but it seems like I can't see the database.
The file is 12M and I assume is not empty.
gabridome commentedon Dec 27, 2018
gabridome commentedon Dec 28, 2018
Bingo!
gabridome commentedon Dec 29, 2018
OTOH if you guys think it doesn't worth to investigate further due to the "modded" nature of my tree
(I have no idea why I'm in a modded version) I could forget the transactions.
gabridome commentedon Jan 4, 2019
"chan #10" doesn't seem to me the channel ID. Do I have to
grepit?cdecker commentedon Jan 6, 2019
I was able to get some of my channels unstuck by re-broadcasting the funding transaction the other day, and then using
dev-rescan-outputs.That'll give you a list of transactions that we broadcast but where not yet confirmed. We can then take those transactions and identify the funding transaction (complicated a bit by the fact that we store the
txidin non-reversed order). So if the fundingtxidends inABCDEFwe need to go look for a transaction with IDEFCDABin the database:You can then take the rawtx and re-broadcast it either with one of the
pushtxgateways, or your ownbitcoind:You might also want to first check that it is the transaction you were expecting using
decoderawtransaction, but it should always be safe to broadcast these transactions since they were broadcast in the past. If all goes well you should see the channels slowly progressing through the stages of confirming the channel on-chain and then closing them.gabridome commentedon Jan 8, 2019
Thank you Christian:
The first command gave me a series of transactions. Among them I found two corresponding with my two stuck channels (I didn't mention but there's an other one).
The transaction id of the two are:
If I look for them with the second query you gave me they give me bak the raw funding transaction which have no inputs:
I'm probably missing something but if the raw funding transaction had no input (see above), if I rebroadcast it still haven't.
I haven't a clue why two transactions have been prepared lacking inputs.
Decoding the first:
EDIT: it turns out that 1 input of the first transaction and 1 of the second funding transaction belonged to two NON broadcast transactions. I did broadcast them (NOW). Waiting... Stay tuned. :)
gabridome commentedon Jan 8, 2019
Ok. The txs are all confirmed.
The situation has evolved a lot but I don't know what to do now:
gabridome commentedon Jan 9, 2019
Update: I have tried to rebroadcast all the transaction I had in result of the query and now the situation is:
I have still three transactions not broadcasted, using the first query. The first and the second ones use inputs already spent by other transactions and the third is already on the blockchain.
What I should do?
Do I need to investigate further or this is simply dirt left after an old accident?
gabridome commentedon Jan 10, 2019
The situation seems to be resolved:
I would like thank you @SimonVrouwe and @Snyke for the precious help.
Two question remain unsolved:
apart from that I would agree in closing the issue if we don't need further investigate the issue.
Lessons learned so far:
.lightning/lightningd.sqlite3to investigate or query the database directly but be very careful not to update it@SimonVrouwe
gabridome commentedon Jan 11, 2019
So, if I understand correctly, if c-lightning was aware of the failure would consider all the inputs as unspent output for subsequent transactions.
Instead it considers the change_output of the failed transaction as available and this is why subsequent transactions, using those change_outputs fail as well.
It seems like something like a "rollback" function is needed in this case, given the negative outcome of
sendrawtransactionis available of course.gabridome commentedon Jan 11, 2019
Or until confirmation? before confirmation a new output shouldn't be considered spendable IMVHO.
It seems a problem also normal wallets have: when change output are to be considered spendable and what to do in the meanwhile?
openingd/json_fund_channel:
sd4096 commentedon Feb 27, 2020
Hi, I'm a noob but wanted to note that I have needed to reference this thread a couple times. Once with a channel stuck at "AWAITING_UNILATERAL" for days and again this morning with a channel stuck at "CHANNELD_AWAITING_LOCKIN" for days.
In both cases, rebroadcasting the transaction through bitcoin_cli with the rawtx did the trick. My bitcoin node connects via tor proxy and I also use NordVPN. Maybe I missed something and this config is not recommended with c-lightning? Otherwise, I just wanted to state things for the record...