One interesting thing I learnt from Phil Haack and Jeff Atwood ‘s VPS being ruined, is that you should test your backups, not just perform them.
According to my current backup plan, I’ve tested the TimeMachine backup, by formatting my macbook (clean install) a couple of times, and getting all my files back. When ma.gnolia went offline I was also satisfied with the backup I had at the time.
But when it comes to services, it is more complicated. My VPS was having some issues with the RAM reaching the maximum (and I only had a couple of django, sinatra and rails websites). I decided to reformat it in order to clean all the trash and try to identify the problem if not solved.
Starting to download my daily backups, I found there was no space on disk on the machine, and I had to delete my backups in order to backup it up (weird I know). I ended up using rsync to copy the machine and to restore the ~/sites folder back again. In between I managed to delete my ~/admin folder in which all the configurations to apache, nginx and a couple more utilities lived. Thankfully I had a pretty good convention for VirtualHosts and managed to bring all of them back up.
Right now everything seems to be running fine, and I only lost 1 day of uptime, which is not a big deal for the websites I host1. The major headache I came across was to install all the requirements for each of the projects I’m hosting. For python-based projects, I maintain a requirements.txt that pip reads and installs automatically, but I feel like there should be something like that in the rubygems world. Or maybe I’m missing something.
1 Looking back, I regret having done the format&restore in Portugal, since in Sweden I would have transferred the files much, much faster.