I popped out of the office to cycle to meet some folks arriving at a London railway station on Friday when my phone started going wild with text messages from my custom intrusion detection system. Stopping by the side of the road my heart leaped to see text messages informing me that someone had logged in as root on one of my servers:
complex password scheme and trip wires to catch intruders. I like to keep software up to date and restrict access to only the ports necessary, etc. etc.
But intrusion detection helps to answer the "what if someone gets in?" question.
On Friday, a total 65 messages were sent over a space of a few minutes. My intrusion detection system uses a combination of iptables rules and constant inspection of log files to signal odd behaviour at the packet level, Apache and other service level.
And there I was in the middle of the street without access to the server while it appeared that someone was inside the machine as root. What I needed right then was an SSH buddy: someone I could call and give credentials to so they could log in and shutdown the machine.
There was one family member who I knew would be capable of this, but when I called I found that they were in a car driving somewhere. Finally, I went with a colleague and the machine was shutdown 8 minutes after the first text message was sent.
Ultimately this turned out to be a false alarm. Although the machine was under attack (on many levels: there was activity hitting the packet filter, trying all sorts of injection at the Apache level and having a go at SSH) the actual alert (based on looking in auth.log) was a false alarm based on a bad regexp.
Despite the heart attack I'm still glad I had my out-of-band mechanism (in this case, SMS) for getting machine alerts. But it made me realize that I need to tighten up my SSH buddy plans for the next time.
Who's your SSH buddy?
PS A number of people have mentioned having an SSH client on my iPhone. I do now. But this still doesn't mitigate the need for an SSH buddy: if I'm abroad or in an area without data access I still need someone I can call upon.
PPS Other people have suggested that I make the SMS system two-way so I can SMS in some standard commands (such as shutting the machine down). This is a good idea.