Outbound SMTP Management March 8, 2008

Just about all Linux server distributions install a sendmail daemon that listens on 127.0.0.1 and relays email for the localhost. This is all fine and dandy if you have a handful of Linux servers, but can turn in to chaos once you get 30+ Linux servers.

Certain system tasks rely on getting emails to someone important. By default there are crons that run that when they need attention they email root@localhost. If all of these emails go to a local mbox for the root user then you’re probably missing some important emails.

If all systems are sending email to the internet, then you have 30+ email queues to monitor with bounces to yahoo and aol (Why are there so many emails in every email queue that are @yahoo or @aol?) filling them all up. Also, if you want to scan all email generated from your networks for spam/viruses (just to be sure you’re not spreading something) then you have to set that up on all 30+ servers.

To make all this easier, use sendmail’s smarthost option. But first you need an outgoing mail hub. I much prefer qmail when it comes to administering and running an SMTP queue. So I suggest you find a good candidate to be your outgoing mail hub and follow the instructions on Life with qmail. After you get that up and running you’ll have to do a couple extra things.

First, set up your /etc/tcp.smtp appropriately for relaying from your network. I’ve got it set here to allow connections, allow relays, don’t do RBL checks, and do scan for spam/viruses for the internal network. But, straight up deny all access from other networks:

127.:allow,RELAYCLIENT=”",RBLSMTPD=”",QMAILQUEUE=”/var/qmail/bin/qmail-queue”
10.10.:allow,RELAYCLIENT=”",RBLSMTPD=”",QMAILQUEUE=”/var/qmail/bin/qmail-scanner-queue.pl”
:deny

Once you have tcp.smtp set right, run “qmailctl cdb”. Now one more thing, if your domain’s internal mail server is on the local network, make sure that this outbound mail hub gets a proper internal IP for your own domains mail server. I had an environment once that had a screwy dns design and I’d sometimes get an internal IP for it and other times an external IP. If you can’t get the dns set right, add an entry to /var/qmail/control/smtproutes:

yourdomain.com:10.10.0.10
.yourdomain.com:10.10.0.10

Where yourdomain.com is the domain after the @ for local deliveries and the 10.10.0.10 ip is the internal IP of your internal email server.

Now, for all your other servers.

Edit your /etc/mail/sendmail.mc file and look for a section like this:

dnl # Uncomment and edit the following line if your outgoing mail needs to
dnl # be sent out through an external mail server:
dnl #
dnl define(`SMART_HOST’,`smtp.your.provider’)

Uncomment that line so it looks like:

define(`SMART_HOST’, `outboundsmtp.yourdomain.com’);

Obviously you’ll set outboundsmtp.yourdomain.com to the host name of your “outgoing mail hub”. Once you have that set, compile your new sendmail config with “make -C /etc/mail”.

And finally, edit the /etc/mail/aliases file to make sure that all system emails go to someone real. By default it all gets funneled to the “root” user. So set an alias for “root” to go to “it@yourdomain.com” or whatever is appropriate. After editing aliases you’ll need to run “newaliases”.

Got some windows servers in the mix? No problem, IIS can create an SMTP service on localhost. Set relay restrictions to only allow email from 127.0.0.1 and set the smarthost to your new mail hub.

Now you need to get monitoring in place. Nagios is pretty common, so I’ll go that route. There’s already a nagios plugin for sendmail. On each and every one of your 30+ Linux servers configure nagios to monitor for the sendmail process and that the email queue is small. I suggest warn at 5 emails in the queue, alert at 10. The queue should actually almost always be empty.

For your qmail outgoing mail hub you’ll want some extra monitoring. I suggest you monitor for the qmail procs, spamassassin, your virus scanner, disk space for your qmail queue (you do NOT want the headache involved with a corrupted qmail queue because you didn’t realize disk space was tight), and of course the size of the queue. There are already written nagios scripts for this. You’ll have to experiment with the warn/alert levels. You want to accommodate for surges in email activity but still get alerted if something is actually wrong. Don’t set it too low and get numb to the noise from email surges. Don’t set it so high that you won’t be notified of an issue until it’s been far too long. There is no magic number so you’ll have to experiment.

Next is documentation. Update your documentation for creating a new server so that you and whoever else may create one remembers to properly configure the email system.

Also, make sure all software developers are aware that if they want to send email from their apps that they should use localhost as an SMTP server or use the “/usr/sbin/sendmail” on the localhost. Applications almost never have full mail queue features, meaning a temporary delivery failure would result in a lost email. If the app is able to attempt to send an email then more than likely the sendmail daemon on localhost will be able to accept delivery of that message. Especially since you have it properly monitored. And sendmail is a fully functioning mail queue with retries and proper logging.

So hopefully all of this will reduce the number of smtp queues you need to worry about. And also keep the system chatter emails off the localhosts and get them to your inbox. Some distributions install very noisy crons that send useless email noise though, don’t be afraid to tell them to shut it. It’s good though, reduce the noise and increase the signal. Even if it may be more work up front, it’s worth it in the long run.