For most web-based business applications these days, it is necessary to run secondary services such as databases, search indexes, shared caches, and so on. Typically these services will be running on their own dedicated box (VPS or dedicated hardware). As a security best practice, these services should not be listening for incoming requests and also inbound requests should be blocked by a firewall.

This presents a problem: How to allow the applications that depend on these services to reach them? Enter port forwarding via SSH tunneling. This guide will walk you through the process of configuring and debugging such a setup so that you can have the best of both worlds.

Configuring the Service Server

First we will need to allow SSH tunneling on the service server. To do this we will create a dedicated user that is restricted to a sub-set of commands. We will further restrict the incoming SSH tunnels to only allow port-forwarding.

Create a new user ("remote_access") with the following command:

useradd --shell /bin/rbash --home-dir /home/remote_access --create-home remote_access;
  • Use rbash (restricted bash) to limit the user to a small sub-set of commands.

Create an authorized keys file for SSH authorization:

mkdir -p /home/remote_access/.ssh;
touch /home/remote_access/.ssh/authorized_keys;
chown -R remote_access:remote_access /home/remote_access/.ssh;

Prepend each public key in the authorized keys file with the following:

no-pty,no-X11-forwarding,permitopen="localhost:3306",command="/bin/echo do-not-send-commands" 

This will restrict the remote user to port-forwarding on a specific port ("3306" in this example).

You can allow more than one port by specifying the permitopen option once for each port. For example, to allow ports 3306 and 8080:

no-pty,no-X11-forwarding,permitopen="localhost:3306",permitopen="localhost:8080",command="/bin/echo do-not-send-commands" 

Configuring the Application Server

Create a new RSA public/private key pair:

mkdir -p ~/.ssh;
ssh-keygen -f ~/.ssh/id_rsa -t rsa;

Print the new RSA public key with the prepended SSH options:

echo "no-pty,no-X11-forwarding,permitopen=\"localhost:3306\",command=\"/bin/echo do-not-send-commands\" $(cat ~/.ssh/id_rsa.pub)"

Copy/paste the output from the above command to the "remote_access" user's authorized keys file on the service server.

Now you can try connecting from the application server to the service server:

ssh -vNTL 3307:localhost:3306 remote_access@REMOTE_HOST
  • -v Print verbose log messages.
  • -N Do not execute a remote command.
  • -T Disable pseudo-terminal allocation.
  • -L Specifies that the given port (3307) on the local host is to be forwarded to the given host and port (localhost:3306) on the remote side.
    • The local and remote ports must be different otherwise the tunnel won't work. There are some work-arounds to this, but that is outside the scope of this article.

Have a look at the output. You should see something like the following:

debug1: Authentication succeeded (publickey).
Authenticated to REMOTE_HOST ([192.168.0.1]:22).
debug1: Local connections to LOCALHOST:3307 forwarded to remote address localhost:3306
debug1: Local forwarding listening on ::1 port 3307.
debug1: channel 0: new [port listener]
debug1: Local forwarding listening on 127.0.0.1 port 3307.
debug1: channel 1: new [port listener]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: Remote: PTY allocation disabled.
debug1: Remote: X11 forwarding disabled.
debug1: Remote: Forced command.

If you see the following message:

debug1: No more authentication methods to try.
remote_access@bitcoind-1.cryptoterminal.eu: Permission denied (publickey).

This means that the server couldn't authenticate you as the "remote_access" user. Most likely you forgot to add or copied incorrectly your public key. Double check the "remote_access" user's authorized keys file.

If you see the following message while attempting to use the SSH tunnel:

channel 2: open failed: administratively prohibited: open failed

This means that the permitopen option is not set correctly. Remember that the second port number is the remote port that should be allowed given the following value for -L: 3307:localhost:3306.

If you are still having trouble, maybe take a break and come back to the problem later ;)

Automate the SSH Tunnel

Once you've got the SSH tunnel running, you can move on to configuring your application server to connect automatically on server start.

For managing the SSH tunnel (in case of network problems), it's a good idea to use autossh:

sudo apt-get install autossh

This will automatically handle reconnects in case the remote server is rebooted, or if either server temporarily loses connectivity.

To start the SSH tunnel whenever the application server starts, add a crontask:

crontab -e

And then add the following on a new line:

@reboot autossh -fnNTL 3307:localhost:3306 remote_access@REMOTE_HOST
  • -f Requests ssh to go to background just before command execution.
  • -n Redirects stdin from /dev/null (actually, prevents reading from stdin). This must be used when ssh is run in the background.

Conclusion

Now you have the best of both worlds: Your services are safe and secure behind a firewall while still permitting your dependent applications to reach them. This limits any potential damage that can be caused by a compromised application server. And as a bonus, communications between the two servers are encrypted via the SSH tunnel. Yay!