August 6th, 2012

For the Ubuntu crowd: updating packages via cron... including cron itself

Tom Boutell
Chief Software Architect

"We have nothing to fear but cron itself." - Franklin "DevOps" Roosevelt

We heart Ubuntu Linux, and we prefer it on servers. When we order a VPS from servergrove or linode, we get a choice of Linux flavors, and always pick Ubuntu.

One of our major reasons for that is that the package manager, apt, works so well. It's even possible to upgrade a server to a new major version of Ubuntu. Not exactly fun, but possible. In CentOS or Red Hat, that's not even a thing.

One catch that we run into with Ubuntu (and Red Hat and CentOS, for that matter): system packages don't update automatically to get the latest security fixes. You can automate package updates, but the system isn't really set up to encourage you to do that without manual intervention. The underlying idea seems to be that, even though package updates in a stable version of Ubuntu are supposed to just hit security issues and other critical fixes, they just might break something important.

This is what we have to say to that: getting your server totally hacked six ways to sunday will also "break something important." And it won't do it in a way you can easily clean up by logging in via the rescue shell of your hosting company (something all decent VPS hosting companies provide). So unless you have time to log into every server and muck about manually on at least a weekly basis, or don't care that your clients' sites have been turned into malware distribution points, an automated package updater is a good idea. 

In short: be realistic in your time management as a system administrator. 

But how can we update system packages automatically? Well, we could just schedule a command like this as a cron job (cron lets you schedule commands to run at particular times):

apt-get update && apt-get upgrade

That would work until the command wanted you to confirm something. And then fail completely. You can work around that by asking apt-get to be a little more forceful. This command asks apt-get to always trust its defaults:

apt-get update -y --force-yes -o DPkg::options::=--force-confdef

That's progress. But what if cron itself gets updated? Guess what: the cron daemon shuts down... and your job is killed in mid-run... and you're stuck. 

A few months ago I wrote a simple shell script and pushed it to all of our servers. This script, when scheduled as a cron job that kicks off in the background (with &), can update all packages including cron itself. And since it's effectively not running via cron anymore once it goes into the background, the script also takes care of sending our admin team the output via email (normally, cron emails all output automatically). The script also starts off by using the dpkg --configure command to clean up the mess if a previous package update task somehow failed:


# Update packages. Email the above user if there is any output, otherwise
# no email is sent. Start this as a cron job and PUT IT IN THE BACKGROUND
# so it doesn't die if cron itself is updated

# Update at 4am every Sunday with this crontab line:
#0 4 * * 0 /bin/sh /opt/punk-ubuntu/ &

export DEBIAN_FRONTEND=noninteractive
PATH=/usr/local/sbin:/usr/sbin:/sbin:/usr/bin:/bin; export PATH

(dpkg --configure -a &&
apt-get -y -qq update &&
apt-get -y --force-yes -qq -o DPkg::options::=--force-confdef upgrade
) >> /tmp/$$ 2>&1

if [ -s "/tmp/$$" ]
  /usr/bin/mail -s "Ubuntu Update" $MAILTO < /tmp/$$
  rm /tmp/$$

Members of our team did find even weekly emails from numerous servers to be a bit much. Unfortunately, one thing we haven't been able to do is convince apt-get to refrain from outputting any messages when nothing exciting happens (although we did our best to quiet it down with the -qq option). But as it turns out, these emails are very easy to collect with a gmail filter. So I've successfully convinced my backup system administrators to filter them for reading at times when they are "Tom for a week." 

Hopefully you'll find this script as useful as we do.

Tom Boutell
Chief Software Architect