Skewed Reality

Following a previous article on tunneling over dynamic IP links I extended the concept to monitoring my online servers via munin from a single webpage, rather than having to keep a host of bookmarks to munin on each node. This has the added benefit of disabling the graphing code and vhost on the clients. Effectively, this means you don’t need apache to run munin on the nodes, a good thing if your node is a tiny VPS. Finally, you get your rrd’s on the master server for each node, making backups easier.

After doing everything described in the previous article above, you are only missing an automatic way of creating tunnels for each node. The idea here is to get the configuration directly from munin.conf, parse the hosts’ address and port and set up the tunnels accordingly. So:

1. Your munin.conf should be like this (/etc/munin/munin.conf in Debian/Ubuntu):
   [hostname]
address 127.0.0.1
port 5050
use_node_name yes

   [hostname2]
   address 127.0.0.1
   port 5051
   use_node_name yes

   and so on.

2. Then, use the following snippet in a cron job to parse munin.conf. It removes comments, grabs the two lines after each hostname in brackets, removes the brackets and passes the whole thing to an awk script just cause I like awk:
   server# sudo -H -u munin /bin/bash
server$ cd ~
server$ crontab -e
*/5 * * * * grep -v "#" /etc/munin/munin.conf | grep "^\[" --after-context=2 | tr -d "[]" | awk -f ~/t.awk
3. This is the awk script. It takes grep’s output and uses awk’s awesome record/field parsing to call the script checktunnel from the previous post:
   server$ nano -w ~t.awk
BEGIN { RS="--" }
{
#print "Host:" $1 " Addr:" $3 " Port:" $5;
print "~/checktunnel \"" $1 "\" \""$3"\" \""$5"\"" | "/bin/sh";
close("/bin/sh");
}

The whole thing takes about 5 minutes to set up and from then on setting up a munin node is as easy as executing a single command (Debian/Ubuntu), provided of course you have sshd installed:
client# apt-get install munin

Now, your http://server/munin page will have an entry for each host with all the data you need. If you want to you can remove all the cron jobs for munin ( /etc/cron.d/munin ) from the node, so it doesn’t graph at all.

I have often needed to create tunnels between machines. In this case I wanted to set up a munin server to monitor various nodes on the Net. The problem was that my server is on a dynamic IP (yes, I’m mazochistic like that). So my tunnels were problematic because I could not set “allow connection” permissions on the client nodes properly (no static IP).

Solution: SSH tunelling to an unprivileged user account via public-key authentication and a cronjob to keep the tunnels alive:

1. You need an unprivileged user on the server (originator of the connection), in this example I use munin. Make sure the user has a workable home directory and create a public-private key pair.
   server# sudo -H -u munin /bin/bash
server$ cd ~
server$ mkdir .ssh
server$ ssh-keygen -b 1024 -C munin@`hostname -f` -t rsa
server$ cat .ssh/id_rsa.pub
2. Copy the key to your clipboard
3. Connect to client (Receiver of connection). Use another unprivileged user with a workable home directory, I’ll use munin on this side, too.
   client# sudo -H -u munin /bin/bash
client$ cd ~
client$ mkdir .ssh
client$ nano -w .ssh/authorized_keys
4. Set up access restrictions, allow only connections to the local port we need (I use 4949 in this case). So paste: command="/bin/false",no-pty,no-X11-forwarding,no-agent-forwarding,no-port-forwarding,permitopen="localhost:4949"
5. Leave a space and then paste the KEY from step 2. It should all be in one line.
6. Save your file, log out of client
7. On the server, initiate the tunnel. You must do this manually at least once to check it works and answer ‘yes’ to permanently accept the client’s public key. In this case I forward local port 5050 to remote port 4949:
   server$ ssh -L 5050:localhost:4949 -f -N clientuser@client

Done!

And now, on to the interesting bit, how to keep this tunnel alive. I needed some scripts to do this for me via a cron job. Ideally, you want the unprivileged user on the server to handle this job, since we can benefit from economies of scale: better to create a script that handles all connections to multiple clients on one machine than having to install a copy of the script onto each and every client. So:

1. Become the unprivileged user:
   server# sudo -H -u munin /bin/bash
2. Create script file you need (here I put it in the home directory of the user). Note the $4 parameter can be something like “-oPort=19222” for example, if you run ssh on a different port on the client.
   server$ nano -w checktunnel
   #!/bin/bash
   #Usage: ./checktunnel "host name" "ip address" "port" ["ssh options"]
   #======================================================================
   PORT=$3
   ADDRESS=$2
   HOST=$1

   #Looks like: ssh -NfL 5050:localhost:4949 -f -N munin@client
if [ "`ps -eaf | grep "ssh -NfL $PORT" | grep -v grep`" ] ; then
echo "tunnel to $HOST ($ADDRESS:$PORT) is up"
else
echo "SSH tunnel ${HOST} NOT alive ... Restarting ..."
logger -p daemon.notice "SSH tunnel ${HOST} NOT alive ... Restarting ..."
echo "/usr/bin/ssh -NfL $PORT:localhost:4949 $4 munin@$HOST"
/usr/bin/ssh -NfL $PORT:localhost:4949 $4 munin@$HOST
sleep 1
fi

3. Add the appropriate cronjob entry:
   server$ crontab -e
*/5 * * * * ~/checktunnel client 127.0.0.1 5050

Now you have a tunnel that will be checked every 5 minutes and re-upped if disconnected. You should also make sure you can read mail from the unprivileged user on the server, just to see if something went wrong! Then you might want to comment out the “echo … is UP” line since it will cause mail for you every 5 minutes, even if everything works!

Obviously there is room from improvement here, for example, making 4949 another parameter and  grepping for the $ADDRESS specifically, but these I leave as an exercise to you genius readers.