So you want to setup a Lightning Network node and you want to do it somewhat safely. More than likely the node software that you've chosen is lnd by Lightning Labs. It's a reasonable choice. It provides many tools and interfaces including CLI, JSON-RPC, REST, several backup and restore options, watchtowers, and more.
There are other node implementations for the Lightning Network:
You can investigate those other implementations to decide which one is best for your use-case. The practical guidance in this article will assume you're using lnd.
The Risks Are Real
Lightning Network software is still very much a work-in-progress. And in case you had ideas about putting a significant amount of money into Lightning Network channels, you should read this comment thread on GitHub: "Still cannot force close inactive channels". The comments contain such gems as:
Aah... looks like you restored from an outdated backup? If you're unlucky, you force-closed the channels (which is a breach when your backup is outdated), and the remote took the funds. If you're lucky, the remote force closed and has data-loss protection enabled. If that's the case, you can try to connect to the peer and see what happens. Please post debug logs!
.. you lost data. Therefore your node is now waiting on the remote peer to connect and give you data you need to recover the channel state. Depending on which version they're running (or even implementation), they may never give you this special data.
I have the same problem with more than 3 BTC missing and "If you restored from an old backup (which you should never do!)" sounds like the most irrational thing...
Ouch! But lucky for the last commenter they were able to recover their funds in the end - after a long and difficult debugging process with the help of one of the lnd project contributors.
Still interested in being your own bank?
The first step towards being your own sovereign individual (at least with regards to finance) is to run your own full bitcoin node. Since lnd can interface with bitcoind and btcd, I recommend to use one of those as your full node software. Some notes regarding full node setup:
For both bitcoind and btcd:
- Set the
txindexflag to build the transaction index
- Configure RPC access for your lnd
From lnd's installation documentation:
We don't require --txindex when running with bitcoind or btcd but activating the txindex will generally make lnd run faster.
Additional notes for bitcoind:
- Do not enable pruning
It is also possible to run lnd with neutrino:
Neutrino is an experimental Bitcoin light client written in Go and designed with mobile Lightning Network clients in mind. It uses a new proposal for compact block filters to minimize bandwidth and storage use on the client side, while attempting to preserve privacy and minimize processor load on full nodes serving light clients.
I've found neutrino to be pretty unreliable. But maybe the situation has improved since the last time I used it. Give it a try if you want to avoid setting up a full bitcoin node.
Node Software Installation
There already exist detailed installation instructions provided by the lnd project itself. So if you don't already have lnd installed, follow those instructions and then return here to continue.
Every Lightning Network node needs a bitcoin wallet. The lnd node software provides an easy-to-follow setup process via its CLI:
Follow the instructions to generate a new seed, set a wallet password, and set your cipher seed passphrase.
Before it can be used, you need to unlock the wallet as follows:
Provide the wallet password when prompted. The node will begin to download block headers and other information needed to build its internal database.
Watchtowers act as a second line of defense in responding to malicious or accidental breach scenarios in the event that the client’s node is offline or unable to respond at the time of a breach, offering greater degree of safety to channel funds.
At present there are only "altruistic" watchtowers, meaning node operators are running these watchtower nodes not for profit but to help the network. You might be able to find an altruistic watchtower to watch over your node's channels here:
You can configure a watchtower via the CLI with the
lncli wtclient add command. And it is important to note that your lnd node can use multiple watchtowers in case one of them disappears or doesn't fulfill its function.
Or you can configure a watchtower in your
Setup Your Own Watchtower
You don't need to use someone else's watchtower. You can run your own. But you should note that it is highly recommended to run a watchtower on its own physical machine with its own infrastructure (electricity, internet access, etc) or in a separate data center from your primary lnd node. You could even run multiple watchtower nodes all in separate data centers around the world.
A watchtower node is an lnd node that has been configured to act as a watchtower. You will need to go thru the usual wallet setup process as if you were preparing a normal lnd node - e.g
lncli create and so on.
Here's an example
lnd.conf for a watchtower node:
bitcoin.active=1 bitcoin.mainnet=1 bitcoin.node=bitcoind bitcoind.rpchost=localhost:8332 bitcoind.rpcuser=XXX bitcoind.rpcpass=XXX bitcoind.zmqpubrawblock=tcp://localhost:28332 bitcoind.zmqpubrawtx=tcp://localhost:28333 watchtower.active=true watchtower.listen=localhost:9911
This sample configuration uses bitcoind. The important lines are the last two which are prefixed
watchtower. These tell the lnd node to enable its watchtower and to listen at
localhost:9911. This works for my setup because I use port-forwarding via SSH reverse proxies to connect my servers.
For more details:
After you've finished your node setup, there are a few things you can do to manage your new Lightning Network node and to help save yourself some pain later.
Use a Mobile App
These days there are several solid mobile apps that can help you manage your node:
Using a mobile wallet to access your lnd node remotely will allow you to:
- Use bitcoin as a currency in the real-world
- Check your remote/local channel balances
- Monitor for offline channels
Follow each project's configuration instructions to connect them to your lnd. In general you need three things: Your node's "hostname" (IP Address plus port number), TLS certificate, and macaroon (with admin privileges). The TLS certificate allows your mobile app to establish encrypted communications with your node directly. The macaroon is an authorization token that has some privileges associated with it. In this case you will need the admin macaroon.
Prevent Zombie Channels
This is not an official term, but "Zombie Channel" could be used to refer to an open channel where both nodes are offline. Since neither node involved in the channel is online anymore, the channel will stick around, holding the funds hostage inside.
This could happen in the event that you have an open channel with an offline peer and then your node also goes offline due to some kind of failure. This will make recovering those funds difficult. To prevent this scenario, you can force-close a channel if your peer has been offline for some time.
To protect your funds, you should ensure that you are backing up important files to help with a possible recovery process in the event of node failure. The most important file to backup is
channel.backup - the so called "Static Channel Backups" file.
After version v0.6-beta of lnd, the daemon now ships with a new feature called Static Channel Backups (SCBs). We call these static as they only need to be obtained once: when the channel is created. From there on, a backup is good until the channel is closed. The backup contains all the information we need to initiate the Data Loss Protection (DLP) feature in the protocol, which ultimately leads to us recovering the funds from the channel on-chain. This is a foolproof safe backup mechanism.
On Linux systems, the default location of this file is
~/.lnd/data/chain/bitcoin/mainnet/channel.backup. Have a look at LND backup script for channel.backup using inotify for an example how to backup your
channel.backup file is encrypted using your wallet's seed phrase, so it is safe to copy it to your cloud file storage service provider. If you are already using an S3-compatible cloud storage provider then you might find this article useful for configuring your automated remote backup.
If you're here that means something has gone wrong. But, there is hope! You will very likely be able to recover most if not all of the bitcoin from your crashed lnd node. Stop for a moment. Breathe. Feeling ok? Good. Now is the time to go slow and make a plan before doing anything.
What should you not do? The absolute worst thing you can do is copy an old
channel.db file into a new instance of lnd and run it. That will almost certainly result in permanent loss of funds.
Here is a nice flow chart that can help you figure out your next steps in the recovery process:
The above flow chart was taken from the very helpful chantools project. It is an open-source set of tools developed by one of the contributors to lnd. The chantools program will (hopefully) help you recover your funds from an old copy of your lnd's
channel.db. But before we get to that, let's first recover any on-chain funds and attempt a recovery using your static channel backup file.
Recovering On-Chain Funds
Before you do anything with an old
channel.db file, let's follow steps 2 and 3 from the above flow chart. With a fresh lnd/lncli instance, let's create a new wallet using your old seed:
When prompted for your seed and cipher seed phrase, be sure to provide the same exact values that were used for the crashed lnd node which you are recovering. This will allow the wallet to check for and recover its existing on-chain balance. Wait for your lnd to finish syncing to the chain backend (bitcoind or btcd). You can check its progress as follows:
Recovering Off-Chain Funds
Once your lnd has finished syncing, you can move on to the next step: Restoring from channel backup. Check the integrity of your static channel backup file (SCB):
lncli verifychanbackup --multi_file=./channel.backup
Expected output if your backup is OK:
Initiate the recovery process with the following command:
lncli restorechanbackup --multi_file=./channel.backup
You should see a prompt similar to the following:
WARNING: You are attempting to restore from a static channel backup (SCB) file. This action will CLOSE all currently open channels, and you will pay on-chain fees.
Are you sure you want to recover funds from a static channel backup? (Enter y/n):
Enter "y" to initiate the restore process.
This command can take a long time to execute. Do not stop it prematurely. For more information about these steps, see recovering using SCBs.
And now... you wait. It may take a week or more to recover your funds from channel force-closures.
Further Recovery Steps
After following all the previous recovery steps, if you still have open or pending channels, you will need to use chantools. Carefully follow the steps described in the project's readme file to recover your remaining funds.