There are many potential applications that need a reliable, fast stream of bitcoin transactions: payment processing, gathering transaction data for statistics and network analysis, fee estimation, and many more that I can't think of at the moment. You might think about using some existing third-party services (or APIs), but you will quickly realize that most of those services are not reliable, too slow, inefficient, or too expensive to license. And let's not forget the biggest problem of all: They are a third-party. In the world of cryptocurrencies, it's always better to verify yourself and cut out the middleman.

So in this guide, we will walk through the steps needed to setup a bitcoin node that will stream transactions over TCP using the ZeroMQ protocol. For the sake of simplicity, it is assumed that both your bitcoin node and the subscriber application will be on the same machine.

Setup bitcoind

Download bitcoind:

wget -O bitcoind.tar.gz https://bitcoin.org/bin/bitcoin-core-0.16.2/bitcoin-0.16.2-x86_64-linux-gnu.tar.gz;

Check bitcoin.org for the latest version number.

Extract the compressed tar file:

tar -xvzf bitcoin.tar.gz

And move the bitcoind and bitcoin-cli binaries to the current directory:

mv bitcoin-0.16.2/bin/bitcoind ./
mv bitcoin-0.16.2/bin/bitcoin-cli ./

Create a new folder to be used as the data directory:

mkdir -p .bitcoin

Create a bitcoin configuration file:

cat > ./.bitcoin/bitcoin.conf << EOL
# Use the regtest network, because we can generate blocks as needed.
regtest=1

# In this example, we will keep bitcoind running in one terminal window.
# So we don't need it to run as a daemon.
daemon=0

# RPC is required for bitcoin-cli.
server=1
rpcuser=test
rpcpassword=test

# In this example we are only interested in receiving raw transactions.
# The address here is the URL where bitcoind will listen for new ZeroMQ connection requests.
zmqpubrawtx=tcp://127.0.0.1:3000
EOL

For more details on how ZeroMQ is implemented and configured in bitcoind, see bitcoin/zmq.md.

Start bitcoind:

./bitcoind -datadir=./.bitcoin

Open a new terminal window (or tab), leaving bitcoind running in the first one.

Use bitcoin-cli to generate new blocks:

./bitcoin-cli -datadir=./.bitcoin generate 101

This will give your local wallet some regtest bitcoin to play with.

Get a new address:

./bitcoin-cli -datadir=./.bitcoin getnewaddress

Send some bitcoin to the new address:

./bitcoin-cli -datadir=./.bitcoin sendtoaddress "ADDRESS" 1.000

You will need to repeat this last step when testing the example node.js application later.

Example Subscriber Application

In the following example node.js script, we will use zeromq.js and bitcoinjs-lib to subscribe to a TCP socket using the ZeroMQ protocol and parse the received raw transaction data. The code is commented to help understand how it works.

// Library for working with the bitcoin protocol.
// For working with transactions, hd wallets, etc.
var bitcoin = require('bitcoinjs-lib');

// Implementation of ZeroMQ in node.js.
// From the maintainers of the ZeroMQ protocol.
var zmq = require('zeromq');

// Create a subscriber socket.
var sock = zmq.socket('sub');
var addr = 'tcp://127.0.0.1:3000';

// Initiate connection to TCP socket.
sock.connect(addr);

// Subscribe to receive messages for a specific topic.
// This can be "rawblock", "hashblock", "rawtx", or "hashtx".
sock.subscribe('rawtx');

sock.on('message', function(topic, message) {

    if (topic.toString() === 'rawtx') {

        // Message is a buffer. But we want it as a hex string.
        var rawTx = message.toString('hex');

        // Use bitcoinjs-lib to decode the raw transaction.
        var tx = bitcoin.Transaction.fromHex(rawTx);

        // Get the txid as a reference.
        var txid = tx.getId();

        console.log('received transaction', txid, tx);

        // To go further you can get the address for a specific output as follows:
        // var address = bitcoin.address.fromOutputScript(tx.outs[0].script, bitcoin.networks.testnet);
    }
});

Copy/paste the above code into a new file named example.js.

In this example, we are listening for raw transactions. But there are other "topics" that you can subscribe to via zeromq:

  • "rawblock" - Receive raw block data for new blocks.
  • "hashblock" - Receive only the block hash for new blocks.
  • "rawtx" - Receive raw transaction data for new transactions.
  • "hashtx" - Receive only the transaction hash for new transactions.

Initialize a new npm project:

npm init

Hold down the <enter> key to accept all the defaults.

Install bitcoinjs-lib and zeromq modules via npm:

npm install bitcoinjs-lib zeromq

Run the example node.js application:

node example.js

Leave this running and open another terminal window (or tab).

To test whether everything is working properly, you will need to send a test transaction. Send some bitcoin using the command you tried earlier:

./bitcoin-cli -datadir=./.bitcoin sendtoaddress "ADDRESS" 1.000

If all is well, you should see something like the following printed in the terminal window of the example node application:

received transaction ffe5030e79e3092d6781095e61bf19996297fc5650bf8329ccb880e9c6141015 Transaction {
  version: 2,
  locktime: 101,
  ins: 
   [ { hash: <Buffer b4 0a 3f 90 c4 ab b5 f3 37 3a 22 ac d0 47 1a cd 5b 61 5a ea dd 50 7c 87 9d b5 7e e6 82 d6 da 46>,
       index: 1,
       script: <Buffer 16 00 14 d6 08 78 ca 69 0b 3e b9 3e f1 c8 8d 67 8b ea 8f 86 59 0d 59>,
       sequence: 4294967294,
       witness: [Array] } ],
  outs: 
   [ { value: 100000000,
       script: <Buffer a9 14 6c d8 77 4d e7 1c 89 91 f9 f3 c3 3b d2 de 65 bf 21 da 2c 64 87> },
     { value: 4799992920,
       script: <Buffer a9 14 2f 0f b0 dc ab 89 06 e8 e0 5e ef 7e 53 94 82 22 cb 78 82 29 87> } ] }

If you don't see the above output, then something is wrong. Double check that your bitcoind instance is still running. You can also check the debug log of bitcoind:

tail .bitcoin/regtest/debug.log

Remote Setup and Security Considerations

In a production environment, the bitcoin node will run on a separate server from your node.js application. Before you go and configure your ZeroMQ end-point to listen on 0.0.0.0, it's important to understand that the ZeroMQ protocol uses TCP so it lacks any encryption or authentication mechanism.

To ensure that the bitcoin transactions being streamed to your node.js application are authentic, it's a good idea to use port forwarding via an SSH tunnel. This will provide a strong layer of encryption and authentication. For a detailed guide to configuring such a setup see Secure Cloud Services via SSH Tunneling.