[bug]: lnd unable to sweep funds from FCed channels, because tries to spend anchor it itself already spent

[xmrk-btc]

Background

When sweeping FCed channels, lnd includes an anchor which it already spent. So the transaction is not published and funds remain in limbo.

The problematic anchor TXO is 2452fa96d28bc14b88f17f1773f3fa2f56bcc42f8d58ebd613b4e9d53f660e1e:1 .

Lnd spends it by 801d3793ee0a7900a2b7aa58990b68d9f60a315a10e935be1555a772e1fd663a

2025-05-25 16:48:00.594 [INF] SWPR: Created initial sweep tx=801d3793ee0a7900a2b7aa58990b68d9f60a315a10e935be1555a772e1fd663a for 2 inputs: feerate=253 sat/kw, fee=0.00000194 BTC, inputs:
2452fa96d28bc14b88f17f1773f3fa2f56bcc42f8d58ebd613b4e9d53f660e1e:1 (CommitmentAnchor) 08e8ddad7e15b808c4da8b1c4e8775a4f92ffa66afa426beb18a612c75bb9eff:0 (WitnessKeyHash)
2025-05-25 16:48:00.600 [INF] BTWL: Inserting unconfirmed transaction 801d3793ee0a7900a2b7aa58990b68d9f60a315a10e935be1555a772e1fd663a
2025-05-25 16:48:00.803 [ERR] RPCS: [/lnrpc.Lightning/QueryRoutes]: unable to find a path to destination
2025-05-25 16:48:00.996 [WRN] BTWL: Transaction 801d3793ee0a7900a2b7aa58990b68d9f60a315a10e935be1555a772e1fd663a not accepted by mempool: txn-already-in-mempool

learns that spending tx is confirmed:

2025-05-25 17:30:45.540 [INF] BTWL: Marking unconfirmed transaction 801d3793ee0a7900a2b7aa58990b68d9f60a315a10e935be1555a772e1fd663a mined in block 898327

even spends the only output of that 801d3 tx:

2025-05-25 20:51:54.206 [INF] SWPR: Created initial sweep tx=04c7c9db24025e705c8786b0e9d4a23790e4b26143e37654fff0f23ecf5c54b1 for 3 inputs: feerate=253 sat/kw, fee=0.0000042
8 BTC, inputs:
2452fa96d28bc14b88f17f1773f3fa2f56bcc42f8d58ebd613b4e9d53f660e1e:3 (HtlcOfferedTimeoutSecondLevelInputConfirmed)
2452fa96d28bc14b88f17f1773f3fa2f56bcc42f8d58ebd613b4e9d53f660e1e:2 (HtlcOfferedTimeoutSecondLevelInputConfirmed)
801d3793ee0a7900a2b7aa58990b68d9f60a315a10e935be1555a772e1fd663a:0 (TaprootPubKeySpend)

Yet it tries to sweep the anchor again after restarting lnd:

2025-05-25 18:44:08.017 [INF] LTND: Version Info rev=5ea723 version=0.19.0-beta.rc5 commit=v0.19.0-beta.rc5 debuglevel=production logging=info
...
2025-05-25 20:22:47.127 [INF] CNCT: ChannelArbitrator(2321eb4e61157e9fff63e4e3269fab3f5ea1ccf5285744ac0f243f585f50bc06:0): offering anchor from local commitment 2452fa96d28bc14b88f17f1773f3fa2f56bcc42f8d58ebd613b4e9d53f660e1e:1 to sweeper with deadline=64, budget=0.00013159 BTC

and doesn't find spending tx because it only searches block 898336, while tx 801d3 was confirmed in block 898327. (But it shouldn't even need to search if lnd itself spent it.)

2025-05-25 20:22:48.693 [INF] SWPR: Registered sweep request at block 898336: out_point=2452fa96d28bc14b88f17f1773f3fa2f56bcc42f8d58ebd613b4e9d53f660e1e:1, witness_type=CommitmentAnchor, amount=0.00000330 BTC, deadline=898400, state=Init, params=(startingFeeRate={false 0}, immediate=false, exclusive_group=926934481797316608, budget=0.00013159 BTC, deadline=898400)
2025-05-25 20:22:49.009 [INF] NTFN: Historical spend dispatch finished for request outpoint=2452fa96d28bc14b88f17f1773f3fa2f56bcc42f8d58ebd613b4e9d53f660e1e:1, script=0 db9e993732080a1a1fb1989e7f91fb53495e7e8ca17eecdba1bed1af509705be (start=898336 end=898336) with details: <nil>

I had used chantools sweepremoteclosed before, I published the resulting tx probably 2025-05-24. This could have confused lnd, but I don't see how.

lncli pendingchannels for the channel with that problematic anchor:

        {
            "channel": {
                "remote_node_pub": "028268dcb4c68311613dd3bbb0164f7685b6710022bfa6dcce639acd44695049a2",
                "channel_point": "2321eb4e61157e9fff63e4e3269fab3f5ea1ccf5285744ac0f243f585f50bc06:0",
                "capacity": "8368180",
                "local_balance": "7422261",
                "remote_balance": "732983",
                "local_chan_reserve_sat": "0",
                "remote_chan_reserve_sat": "0",
                "initiator": "INITIATOR_LOCAL",
                "commitment_type": "ANCHORS",
                "num_forwarding_packages": "0",
                "chan_status_flags": "",
                "private": false,
                "memo": "bos open",
                "custom_channel_data": ""
            },
            "closing_txid": "2452fa96d28bc14b88f17f1773f3fa2f56bcc42f8d58ebd613b4e9d53f660e1e",
            "limbo_balance": "7473576",
            "maturity_height": 899327,
            "blocks_til_maturity": -838,
            "recovered_balance": "0",
            "pending_htlcs": [
                {
                    "incoming": false,
                    "amount": "17105",
                    "outpoint": "cc0e457da6e95014de3a507f21ce66a9553a0ee43b2b61532509f75add04f256:0",
                    "maturity_height": 899348,
                    "blocks_til_maturity": -817,
                    "stage": 2
                },
                {
                    "incoming": false,
                    "amount": "34210",
                    "outpoint": "2bd12c02df47db6ecc4541930f7be265fa0963035312468376094ca7a9c9b0f0:0",
                    "maturity_height": 899348,
                    "blocks_til_maturity": -817,
                    "stage": 2
                }
            ],
            "anchor": "LOST"
        },

Difference with #9471 : now lnd itself tried to spend the anchor, it was not spent by external tool.

Your environment

• version of lnd: 0.19.1-rc1, but was running 0.19.0-rc5 or rc4 when the anchor was spent
• which operating system: Linux 6.1.0-18-amd64, Debian 12.5
• version of btcd, bitcoind, or other backend: bitcoind 0.28.0

Steps to reproduce

don't know

[xmrk-btc]

The immediate problem was solved by setting height-hint-cache-query-disable=true, lnd found the spending tx of the problematic anchor and didn't try to spend it again, and did sweep the funds from FCed channels (well, at least the sweep tx got into mempool).

But this means a bug in height hint cache, arguably a less urgent one.

[yyforyongyu]

2025-05-25 20:22:47.127 [INF] CNCT: ChannelArbitrator(2321eb4e61157e9fff63e4e3269fab3f5ea1ccf5285744ac0f243f585f50bc06:0): offering anchor from local commitment 2452fa96d28bc14b88f17f1773f3fa2f56bcc42f8d58ebd613b4e9d53f660e1e:1 to sweeper with deadline=64, budget=0.00013159 BTC

Could you share more logs around this time range? Logs from CNCT, SWPR and NTFN would be very helpful.

I had used chantools sweepremoteclosed before

What's the reason for using this command instead of letting it resolve by itself? Not sure if it's gonna mess up with the internal state of the contract court.

2025-05-25 20:22:49.009 [INF] NTFN: Historical spend dispatch finished for request outpoint=2452fa96d28bc14b88f17f1773f3fa2f56bcc42f8d58ebd613b4e9d53f660e1e:1, script=0 db9e993732080a1a1fb1989e7f91fb53495e7e8ca17eecdba1bed1af509705be (start=898336 end=898336) with details:

This line is weird as the contract court will use the closing tx height as the height hint, which should be 898322 instead of 898336.

The immediate problem was solved by setting height-hint-cache-query-disable=true, lnd found the spending tx of the problematic anchor and didn't try to spend it again

I doubt it solves it - the flag only tells lnd to look for the height hint db or not, but since it's using the wrong height as a starting point, it won't bother checking the db. It's likely another restart fixed it.

Since the first restart happened at 2025-05-25 18:44:08.017,

2025-05-25 18:44:08.017 [INF] LTND: Version Info rev=5ea723 version=0.19.0-beta.rc5 commit=v0.19.0-beta.rc5 debuglevel=production logging=info

The contract court should have known the FC tx was confirmed. Even if it tried to sweep anchors, there should be more logs preceding the following one, which is a few more blocks later,

2025-05-25 20:22:47.127 [INF] CNCT: ChannelArbitrator(2321eb4e61157e9fff63e4e3269fab3f5ea1ccf5285744ac0f243f585f50bc06:0): offering anchor from local commitment 2452fa96d28bc14b88f17f1773f3fa2f56bcc42f8d58ebd613b4e9d53f660e1e:1 to sweeper with deadline=64, budget=0.00013159 BTC

Also based on this log, the contract court still thought the channel FC tx was not confirmed since it's not using the default non-urgent deadline.

[xmrk-btc]

Sorry for the delay, I don't have earlier logs, I already checked last Monday. I increased my maxlogfiles to 200.

I have 3 pending FCs now, with maturity of 2, 4 and 6 days, respectively, and I set height-hint-cache-query-disable=false again, so perhaps I will replicate the problem. But none of those channels has pending HTLCs, and anchors are all spent by 3rd party (lncli pendingchannels reports my anchors as lost).

[xmrk-btc]

The immediate problem was solved by setting height-hint-cache-query-disable=true, lnd found the spending tx of the problematic anchor and didn't try to spend it again

I doubt it solves it - the flag only tells lnd to look for the height hint db or not, but since it's using the wrong height as a starting point, it won't bother checking the db. It's likely another restart fixed it.

By "solved it" I mean: after setting height-hint-cache-query-disable=true (and restart, obviously), lnd successfully swept all funds from FCed channels which it wasn't able to sweep before. I did several restarts before and it didn't help, only restart + hint-disable=true helped.

BTW, that sweeping tx - 0f5ea0c14be063033cc741d9aeebb2e3ffa921e442b90a9ce620fb405555b51f - also swept 1st stage HTLC (probably 2c166bfb84085efe3cd40e5cd5143f0e0560a93cdb47567a21c62543997e11a6:2) with apparent urgency (8.37 sats/vB feerate while the median feerate of the block 900225 is 2 sats/vB). So the bug is more serious than I originally thought, because I guess lnd was unable to sweep that HTLC earlier but wanted to do it urgently.

[xmrk-btc]

Regarding the restarts, the rough timeline is:
2025-06-04 restart without changing lnd.conf
2025-06-05 restart without changing lnd.conf
2025-06-06 restart after changing sweeper.nodeadlineconftarget=144, it was default before
2025-06-07 restart after changing height-hint-cache-query-disable=true, kept sweeper.nodeadlineconftarget=144

And as I said, the funds were only swept with the last restart.