Planning for SPV backend

[ZmnSCPxj]

First off: SPV is worse security than fullnodes. Please use fullnodes if you can.

On the other hand, if you trust a particular cloud service provider, you can run a fullnode on a cloud service, and thus not spend a lot of bandwidth in your own local connection. You might then think to run lightningd locally and point the --bitcoin parameters at the cloud service fullnode, but this actually downloads every block in full and gets you as much bandwidth use as a locally-run bitcoind with blocksonly option. You could instead run an SPV server of some kind on your remote cloud service fullnode, and as long as the cloud service does not secretly replace your bitcoind with minercoin2xd-unlimitedvision, you should be fine. Then we would want support for SPV in a locally-run lightningd to limit the bandwidth needed, and point it only at your trusted cloud service fullnode.

Here is my current plan:

When lightningd/bitcoind object is initialized, it checks for the following two sets of APIs:

• getrawblockbyheight, getutxout, estimatefees, getchaininfo, sendrawtransaction - current 0.9.0 set.
• gettxesbyheight, getutxobyscid, estimatefees, getchaininfo, sendpsbttomempool, checkspent - next version of the backend plugin API.

gettxesbyheight is given a height to fetch, plus a receive_scriptpubkeys an array of hex string scriptPubKeys to check, and a spend_utxos an array of objects with txid, vout, and scriptPubKey fields. It returns a field found as false if not yet found, or if the block is found it returns found field as true and in addition returns the blockid the hash of the block header, header the hex string dump of the header, and txes an array of hex string transactions in the block that match the given receive_scriptpubkeys or spend_utxos.

getutxobyscid is given an scid of BOLT-standard form BBBxTTxOO. It returns a field found, which is false if the given SCID does not match an unspent SegWit v0 32-byte witness (P2WSH), or a SegWit v1 32-byte witness (Taproot). If true it returns the scriptPubKey of the output as well as the amount.

sendpsbttomempool is given a Base64 PSBT that contains a completely signed transaction, and a allowhighfees parameter, and broadcasts it. We have to switch bitcoind_sendrawtx to use struct bitcoin_tx which conveniently contains a PSBT.

checkspent is given an array of objects. Each object represents a txo with at least txid and vout and scriptPubKey fields, and optional blockheight, which is the known confirmation height of the txo. For each txo, it checks if the txo is spent, and if so, adds it to the spent result, which is an array of objects with txid, vout, and spendheight fields, with 'spendheight': 0 for TXOs that are spent in unconfirmed transactions.

Internally, we also have these changes to lightningd/bitcoind functions:

• bitcoind_can_getutxobyscid. This accepts a const struct bitcoind * and returns true if the bitcoin backend implements getutxobyscid.
  • If it returns true, it is safe to call a new bitcoind_getutxobyscid and bitcoind_checkspent, but not safe to call bitcoind_getfilteredblock or bitcoind_getutxout.
  • If it returns false, it is not safe to call bitcoind_getutxobyscid or bitcoind_checkspent but it is safe to call bitcoind_getfilteredblock and bitcoind_getutxout.
• bitcoind_gettxesbyheight. This is always callable. chaintopology should use this API instead of bitcoind_getrawblockbyheight, since the latter is removed.
  • The only change in interface relative to bitcoind_getrawblockheight is that bitcoind_gettxesbyheight requires filter arguments. It still returns a struct block albeit one with a possibly-incomplete set of tx.
  • If the backend plugin is using the 0.9.0 interface this just calls into getrawblockbyheight and ignores the filter arguments.
• As mentioned, bitcoind_sendrawtx should accept struct bitcoin_tx instead of const char *hextx.

Finally, we also add a new lightningd command, checknewblock, which triggers chaintopology into checking for a new block. The backend plugin should call this when the block tip height changes. Normal bitcoin-cli and BIP158 plugins will poll, though a BIP158 plugin could have a peer push a new block at it (not likely unless the peer is a miner). An Electrum plugin would subscribe to blockchain.headers.subscribe. For now we also do polling ourselves in lightningd to support older plugins that do not perform this call.

After we change lightningd, we can then update bcli to use the new interface. In order to check the new interface, gettxesbyheight will also apply filtering to transactions in the block, and will not provide txes that do not match the given filters. getutxobyscid could have a simple on-disk cache: a mapping of SCID to either a tuple of txid and vout, or Nothing. If an SCID exists in the on-disk cache, and it maps to nothing, it is known-invalid SCID, else if it maps to a txid vout the bcli will go query bitcoin-cli getutxout to see if it is still unspent. If the SCID does not exist in the on-disk cache, then we scan the block and update the cache, and we also iterate over all possible SCIDs (e.g. even absurd ones like BBBx16777215x65535) and map those that do not match a P2WSH or Taproot output to Nothing.

---

Original text:

First off: SPV is worse security than fullnodes. Please use fullnodes if you can.

Currently, the interface to bcli inherently assumes the backend is a fullnode of some kind. This fullnode is always fully trusted to not lie, and we always download every block from the backend.

When doing block synchronization, what we can do is to have a txfilter containing:

• UTXOs which, if spent, we want to know about.
• Addresses which, if spent from or received into, we want to know about.

Now, it is important that we keep track of UTXOs that we want to track spendedness of channels. However, each such UTXO has its own address as well.

Most SPV protocols (eg BIP158, Electrum) are address-centric. So, we should provide just addresses to the backend. The backend should inform us

So I propose for the bcli getrawblockbyheight:

• Add a new required parameter, txfilter, which is an array of scriptPubKey hexdumps.
• It may return a result without a block field (but must still return blockhash). This is used to inform lightningd that the block at the given height does not involve any of the addresses in the txfilter.
• It has to return a separate blockheader field regardless. This is needed so that chaintopology can recognize reorgs.

Then:

• For Electrum, we do blockchain.scripthash.get_history on every listed scriptPubKey, and if any of them has a transaction whose confirmed height equals the height being queried, then we should download that block (elsewhere) and provide it to the lightningd.
  • But we should be wary of reorgs. Thus, we should do a blockchain.block.header and save the header, then blockchain.scripthash.get_history for a single scriptpubkey, then a blockchain.block.header and compare if it is the same as previous; if not the same, we should restart the querying again, since the alternate block might now contain the previous scriptpubkey. This is not resilient against very fast ABA problems but hopefully those are rare.
• For BIP158, the plugin maintains a header chain, and we download the version 0x00 Neutrino filter for the block header hash at the requested height. Then we check if anything in our txfilter matches the Neutrino filter.

---

Ideally, our fullnode-trusting bcli should also apply the txfilter to blocks it returns. This is to ensure that any bugs in maintaining the lightningd-side txfilter can be caught by our test suite without having to add SPV plugins of our own.

Unfortunately, to determine if an address is spent from, we would require a txindex, or have the plugin maintain its own transaction index. Both are obviously undesirable.

However, in practice:

• We only care about addresses we receive from.
• We only care about UTXOs that somebody spends from.

So let me propose instead passing in two arguments to getrawblockbyheight of bcli:

• txfilter_receive_addr, an array of scriptpubkeys, which, if any of them receive funds in the given block, we should return the block.
• txfilter_spend_utxo, an array of {'txid': 'xx', outnum: 42, scriptpubkey: 'xx'}, which, if any of the given UTXOs are spent, we should return the block.

Then:

• The default fullnode-trusting bcli only needs to scan the requested block and every transaction:
  • If a transaction input has a prevout matching a txfilter_spend_utxo entry, return the block.
  • If a transaction output has a scriptPubKey matching a txfilter_receive_addr entry, return the block.
  • Otherwise do not return the block, just its header.
• The SPV interfaces, which are all address-based, just get the scriptpubkey from the txfilter_spend_utxo and append them to the txfilter_receive_addr.

This allows us to check in our test suite that lightningd is maintaining its txfilter correctly, using only the bcli plugin, while allowing address-based SPV interfaces to be used.

[ZmnSCPxj]

For now, a minimal scan of the chaintopology component suggests we keep track of the below:

• Addresses we own, stored in a struct txfilter owned in struct lightningd.
• TXOs we are tracking, stored in a struct txowatch_hash owned in struct chain_topology.
• Transactions we are tracking, stored in a struct txwatch_hash owned in struct chain_topology.

Unfortunately, BIP158 does not include entire transactions: it only includes spends from and spends to specific scriptPubKey.

However, transactions-being-tracked is only used in two places:

• lightningd/peer_control.c uses it to check for the funding transaction to confirm. It knows only the txid, but the funding TXO is known to have a specific scriptPubKey.
• lightningd/onchain_control.c uses it to check for transactions being confirmed or deconfirmed (based on reorgs) and restart onchaind. It knows the entire transaction being watched.

So I think we can modify the watch_txid function and the struct txwatch structure to include an address that the transaction is funding.

• We know the funding transaction has a funding TXO with a known scriptPubKey, so we can provide this.
• Over in onchain_control we know the entire transaction, we can just extract any scriptPubKey from any output.

That way, our only interface absolutely necesary is a scriptPubKey that is spent to.

Of note as well, is that watch_txo is only used in the same places as well. We also need to modify watch_txo to add the scriptPubKey the UTXO has, which is known in both places as well.

[ZmnSCPxj]

Okay, I think getrawblockbyheight is just a bit too low-level.

Instead, I propose a new bcli method.

---

getfilteredblockbyheight height receive_scriptpubkeys spent_utxos

DESCRIPTION

getfilteredblockbyheight returns the block header, and any appropriate transactions, of the block at the given height.

receive_scriptpubkeys is an array of Bitcoin scriptPubKeys; each is a hexadecimal string.

spent_utxos is an array of objects.
Each object contains the fields txid and outnum which indicate the UTXO being monitored, as well as a field scriptpubkey which is the scriptPubKey of that UTXO.

RETURN VALUE

This returns a result object with the following fields:

• blockheader, a hexadecimal string representation of the block header.
• txes, an array of hexadecimal string representations of transactions in the block.

This command returns null for the above fields if the backend has not reached that block height yet.

The txes array reliably contains only those transactions which spend from one of the given UTXOs in spent_utxos, or transactions which pay to one of the given scriptPubKeys in receive_scriptpubkeys.
However, it may report more transactions than that (i.e. it may add transactions that false-positive match the given constraints).
Clients of this method should not rely on anything other than the transactions that match being returned, and should be prepared to handle transactions that do not match the constraints.

---

By using the above, we can use just an Electrum Stratum connection. Electrum Stratum does not look like it can support downloading entire blocks, which is what getrawblockbyheight should return. But it can provide the history of a scriptPubKey i.e. all transactions that spend from and pay to that address, and that is why we should switch to the above interface instead.

Thoughts? @darosior in particular?

[ZmnSCPxj]

Hmmm so I was looking at our code as well, and it seems we have an existing bitcoind_getfilteredblock API as well.

As best as I can determine, this bitcoind_getfilteredblocks reports the TXOs created in the block which are P2WSH outputs and are still unspent now. It is used when querying the information from an SCID.

Electrum has a specific blockchain.transaction.id_from_pos which is perfectly designed for SCID queries. However, plain Bitcoin does not have a simple way of getting a transaction from a specific block at a specific index in that block. And neither can BIP158 Neutrino --- how the heck does Neutrino validate SCIDs??

Hmmm.

Now, in order to prevent too much activity from interacting with bitcoind, lightningd caches blocks and the P2WSH TXOs created in the block. This caching is of course done in the database. So if some SCID claims to have been created in a block lightningd has not downloaded, we go download it, checking only the P2WSH unspent outpoints. Then we store the block in the database, plus all P2WSH unspent outpoints, in the db, and then we check if the SCID is actually valid.

However, the above will not work well with Electrum, since we cannot simply download a block.

Further, the spentness of P2WSH outpoints depends on us processing every transaction in a block. We cannot just depend on data that involves only our own UTXOs and addresses!

Hmmmm. I suppose this needs more design thinking.

[ZmnSCPxj]

All right, let us add a new getutxobyscid. It returns the exact txid, amount, scriptPubKey, and the height was spent at (or 0 if it is not spent) and a found field that is true. If the SCID given does not match to any transaction outpoint, it just returns a found field that is false.

A backend plugin may choose not to implement getutxobyscid. It signals this by providing a getutxobyscid that always fails using a specific error code. If so, it must commit to implementing getrawblockbyheight.

The backend plugin getfilteredblockbyheight should also return a flag, all. If this flag is present and set, then the backend gave all the transactions in the block. If absent, or false, then the backend did not give all transactions in the block.

We also need to create a new db table, utxoset_blocks, with the same schema as blocks. When upgrading db, it just copies the blocks table.

The utxoset_blocks is those blocks which we know we have seen in full, i.e. not filtered as above.

Then, in gossip_control, if a gossip_get_txout message is received:

• First, check if it is listed in the utxoset table.
  • Need to modify wallet_outpoint_for_scid to report spendedness -- the current version returns failure if it is listed but marked as spent.
  • If so listed and marked as spent, return failure to gossipd.
  • If so listed and not marked as spent, check the utxoset_blocks table.
    • If utxoset_blocks is filled in for every height from the SCID height to the max height, we know that we are tracking spendedness correctly, and we can return a success to gossipd.
    • Otherwise, we should go ask getutxoset from bcli. If it returns success, we know that it is still unspent, so we can return a success to gossipd. If it fails, record the current tip height as the spendheight in the utxoset table (so future queries for the same SCID return quickly).
• If the SCID is not listed in the utxoset table, check if the block is in the utxoset_blocks table, in which case we know the SCID is invalid and was never created anyway, so we can report failure to gossipd.
• Try getutxobyscid bcli.
  • If it succeeds and returns found as true, store the data into utxoset.
    • Report success to gossipd if it is not spent yet, else report failure.
  • If it succeds and returns found as false, report failure to gossipd.
• If getutxobyscid fails with the "not implemented" error code, fall back to the existing bitcoind_getfilteredblock, which uses getrawblockbyheight.
  • wallet_filteredblock_add should add blocks to the utxoset_blocks table.

Then, on normal synching at the block tip:

• Use getfilteredblockbyheight.
  • Add the returned block to blocks, just like now.
  • If it returns with all as true, also add it to the utxoset_blocks table, and check for transactions that spend items in the utxoset.
  • If it returns with all as false, it requeries getutxoset on all utxoset entries that are unspent, and if they are now spent, records the current block as the spending height.

Thoughts? @cdecker in particular since this is a massive change in chaintopology and wallet db. Sigh. The complication is in short-channel-ids.

[darosior]

Thoughts? @darosior in particular?

Supporting protocol that expose a less natural (scriptPubkey based instead of txid / txout based) interface would be great as it would allow to use more different backends than just bitcoind or esplora, such as for example the low-storage-cost electrs.
That said, I'm not sure it makes sense for us to support Stratum as we fetch a lot of outputs and such backends would be quite inefficient or don't work at all (EPS for example) as they are designed for a single onchain wallet (so there would just be the Python Electrum supported in addition to the Blockstream fork of electrs).

This comes down to the reason I did not go for this initially: efficiency. We moved from a getoutput-based (3 calls by output to bitcoind by channel ann) polling to get_filteredblock (1 call to bitcoind each reasonable amount of time) for more efficiency with gossip.
So moving back to a one-call-per-channel-ann would be kind of a regression unless the plugin does some sort of caching, and in which case it needs to cache the state to instantly answer to lightningd's getoutputbyscid.
This means more complexity, and what would have been small logic changes to lightningd (eg support a new Element hotness, by changing the parsing of the raw blocks received) needs to be replicated to all plugins.

There is still BIP158 which would be nice to have though. So this is trading (a lot of) complexity for BIP158 support: I have no strong opinion on this.

[ZmnSCPxj]

BIP158 does not have any specific support we can get for the short-channel-id-lookup case.

What we can do with BIP158 would be to continue with the current system, which is that lightningd requests for the raw block, and the BIP158 backend connects to a random fullnode and downloads the entire block. We store the transaction outputs that are P2WSH, and determine their spendedness. That means adding an advisory scriptPubKey argument to the getutxout command, which is ignored in bitcoin-cli plugin, but informs BIP158 backend to scan all saved filters for the scriptPubKey being spent.

But caching at the plugin level is probably needed indeed, especially for Electrum and BIP158 backends. Sigh.

Then a backend plugin could support a getutxobyscid. If this method is found we use it exclusively in gossip_get_txout requests and ignore the lightningd db, on the assumption the backend has implemented all the needed optimizations, in particular the Electrum backend has a specific command for such queries.

This means more complexity, and what would have been small logic changes to lightningd (eg support a new Element hotness, by changing the parsing of the raw blocks received) needs to be replicated to all plugins.

Maybe the block-synching command can be allowed to return multiple options. It gets a set of UTXOs and addresses the lightningd wants to monitor. Then it could return ONE of the following:

• The non-existence of the block.
• The exiistence of the block, but not any block data --- the block exists, here is the header, but no information other than the fact that none of the given UTXOs were spent and none of the given addresses received.
• The raw block data, as with current getrawblockbyheight.
• An array of transactions, some of which spent the given UTXOs and/or paid money to one of the addresses.

Thus, a plugin need not parse block data. However, the array of transactions return would be used by an Electrum backend, and presumably the Electrum server is the one that parsed the block for us, so the new Elements parsing thing need not be understood by the plugin either. BIP158 backends would just match the addresses to the BIP158 filters, and download the entire block as well too.

If the block-synch command returns:

• raw block data

Then lightningd knows it can acquire accurate SCID-create and SCID-delete information for that block.

But if the block-synch command returns:

• existence of block but not any block data
• array of matched transactions

Then lightningd knows that it cannot get SCID-create and `SCID-delete information for that block.

So I think a new table of blocks that we have full knowledge of SCID events in that block should be maintained by us (i.e. the utxoset_blocks I proposed above).

If lightningd gets a gossip_get_txout, if a plugin implements getutxobyscid then we just depend on that. Else:

• Check if the blockheight in the SCID is in the utxoset_blocks table. If it is, go to the next step.
  • If it does not, we make it exist by doing getrawblockbyheight, which must exist. A Stratum backend can get away with not implementing this (or providing one that always errors) by simply implementing getutxobyscid.
• Check if the SCID exists in the utxoset table.
  • If it does not exist, or is spent, then we know that the SCID is no longer valid.
  • If it exists and looks like it is unspent, we check if every block from that height to the tip is in the utxoset_blocks table. If so, that is sufficient for us to know that it is still unspent now.
  • If some block is missing, we call another command (which might not be implemented), isutxospent. This includes the UTXO itself, but also includes an advisory array of blockheights that are missing in the utxoset_blocks table. from the known creation height of the UTXO. The BIP158 backend plugin can use that to limit its filtering to only the specified blocks. We can then update the entry in utxoset if it turns out that the UTXO is spent at one of those heights. There is a small false-positive rate here, in that the UTXO could end up false-matching the filter at that height, in which case we would reject an SCID needlesly, but hopefully that is a small enough effect on the network (the alternative would be for the plugin to download the block and parse it to check for UTXOs being spent in inputs, which is exactly what we want to avoid).

The upshot of this is that with a BIP158 backend, we can "spontaneously catch" some of the short-channel-id updates if it happens that a block matches the filter we request.

Aargh. The hard problem is the SCID lookup!

[darosior]

I think it's cleaner for us to make a (backward compatible) breaking change at this point.

Bitcoin backend, revived

• Bitcoin logic in the Bitcoin backend plugins
  • Allows for a lightningd agnostic of its Bitcoin backend, and thus a Bitcoin backend extensible without complexifying lightningd.
• No more getrawblockbyheight.
  • Use only the hash part for the topology update
  • Use getoutput for requesting an output (no matter its nature, scid - wallet rescan - channel funding)
• A "backend_version" field in the manifest sent by the plugin for compatibility.
  • We can treat the current plugin as version 0, and specify "version": 1. If lightningd sees version 1, it uses the new protocol.
    This also means we can upgrade the bakend version later for e.g. notification support.
  • It's a nasty hack (as it makes the backend plugins even more different, but they already arguably are) but it works (I think, needs to double check but sending garbage in the getmanifest does intentionally not make lightningd cringe).
• For now, we just move the block parsing in bitcoind and cache blocks in RAM (we can have a, say, 10-blocks cache). We can optimize the behaviour afterwards, and in the meantime we can do some nice smart thing in alternative Backend plugins. We can also have a --neutrino startup option which translates in --disable-plugin bcli --plugin bip158_backend later.

[AmkG]

Use getblockcount for the topology update

We need to be aware of reorgs, especially those that take, say, 3 or 4 blocks, and which might have conflicting transactions affecting our wallet or channels, so I think this is mildly undesirable to just depend on a count rather than a header chain.

[darosior]

Yes, sure (I meant getrawblockbyheight but without the getrawblock part). Edited to be explicit.

Also, an even simpler upgrade mechanism: if a plugin registers getoutputwhatever then lightningd switch to the new behaviour, otherwise it fallbacks to parsing the raw blocks.

[ZmnSCPxj]

We also expose a getutxout method. This is used in the utxoset / scid tracking, but also used in the dev-forget-channel implementation to check the UTXO of the channel being forgotten is still unspent.

Now, for scid tracking, I believe what we are now planning is to let the plugin handle this by implementing a getutxobyscid and doing all the block parsing in the plugin.

However, for dev-forget-channel this is not appropriate. One use-case for dev-forget-channel is a funding transaction never confirming because weird blockchain reasons, so there is no scid. We can implement a getutxoutv2 that accepts an additional scriptpubkey argument.

Then getutxoutv2 is invoked only in dev-forget-channel (the only other current use of getutxout is in scid tracking, which we want to give over to the plugin side completely)..This requires matching the scriptpubkey to downloaded Neutrino filters, and (re-)download the appropriate blocks that match it to see creation and spend events. This is potentially an expensive operation, but if it is triggered only in dev-forget-channel then it should be acceptable.

Alternately we can just remove the dev-forget-channel checks, it could just check if the channel has an scid and if so it uses getutxobyscid, or if it does not have an scid, dev-forget-channel does not do any further checks. dev-forget-channel is a "big boy developer" command, so maybe acceptable?

[ZmnSCPxj]

@darosior @cdecker @niftynei @rustyrussell I have updated the original post with the current plan I have.