Click on our Sponsors to help Support SunWorld
Wizard's Guide to Security by Carole Fennelly

Setting up sendmail on a firewall

New features in sendmail 8.9.3 and how to build the binary

April  1999
[Next story]
[Table of Contents]
Subscribe to SunWorld, it's free!

A tremendous amount of information is available about sendmail. But if you're in a hurry to just get it working, you don't have the time to go through and digest it all. In this first installment in a three-part series on sendmail, Carole discusses its implementation on the firewall and focuses on features new to sendmail 8.9.3. Next month she'll cover the particulars of the config file. (3,000 words)

Mail this
article to
a friend
One of the greatest features of sendmail is the extreme flexibility it provides the administrator. This is also one of its worst features, the reason being that it is so often misconfigured. Usually, a system administrator inherits a sendmail configuration file that he or she is afraid to touch, lest it should break. Over time, this file becomes hopelessly confusing, contradictory, and redundant.

I began writing this column with the intention of providing a simple, straightforward, "sendmail for dummies" approach. I ended up appreciating how Bryan Costales must have felt when he wrote his handbook, Sendmail, for O'Reilly & Associates (see Resources): Nothing is simple in sendmail. I also learned to appreciate just how powerful and flexible sendmail really is. It's worth learning. It's also worth a few columns. I'll devote this month's column to discussing some of the background and new features in sendmail 8.9.3 as well as how to build the source code. Next month we'll look into the configuration file. If there is enough interest, I will add a third column for special situations and testing techniques.

Sendmail on the firewall?
Like most security admins, I've always been told it's a bad idea to run sendmail on the firewall. It's generally considered better to run something like smap instead. Indeed, I ran smap myself until I discovered a problem I couldn't fix with smap. The problem was that company e-mail was forwarded from an internal mail gateway system to the firewall. The firewall could only rewrite the external header to say the mail came from "" There was still an internal header that showed the mail was routed through "," complete with the mailgate internal IP address. I couldn't have the inside machine masquerade as because there were other internal mail gateways it had to communicate with. (They couldn't all be But because sendmail is extremely flexible, open source software, it's possible to add functionality to strip out the inside header. Also, many of the security risks with sendmail have to do with the fact that it's generally run setuid to root. It isn't necessary to run it this way on a firewall because there are no direct mail users on the firewall. For added security, I run sendmail in a chrooted cell with the program mailer using the sendmail restricted shell, smrsh.

There are certain resources you should have available before you start working with sendmail. This column isn't going to attempt to cover all aspects of sendmail. For that, I direct you to Bryan Costales's Sendmail, second edition. Aside from being an excellent resource, at more than 1,000 pages this book can also double as a child's booster seat!

You'll also need the latest copy of the sendmail source code (sendmail 8.9.3 as of this writing), available from Verify the PGP signature! Some very useful links are included with the latest information.

Although it should go without saying, you need to be able to build the code using make and gcc or cc and have your environment variables set appropriately. It is strongly recommended that you use a system other than the firewall to compile and test the code. Remember that any system you use has to be running the same release of the operating system and have a compatible architecture. The firewall shouldn't have a development environment installed. If for some reason you have to use the firewall to compile, at least keep the development environment on a spare drive that you can unmount when you don't need it. Life with sendmail is much easier if you have the GNU utilities already installed because it is assumed in the makefile that groff is installed. You can get the GNU utilities ( For the purposes of verifying the source (and not just for sendmail), you should have a copy of PGP (Pretty Good Privacy) installed (version 2.6.3 or higher). You can download a personal copy or buy one from Network Associates.

You can find links to these and other sendmail resources in the Resources section at the bottom of this article.

New features in sendmail 8.9.3
Sendmail 8.9x contains several new features that will probably cause your mail to break unless you configure them properly. These features mostly help filter spam and prevent your site from being used to relay someone else's mail. You have to decide how to configure them based on your site's policy. For my own network, I can rather tightly define from whom I'll accept mail. However, an ISP cannot be so selective. Please note that I haven't personally configured mail for an ISP, so I can only speculate on the special concerns of ISPs. Because ISPs have to service the general public, I would imagine they have legal issues to consider before blocking mail from a particular site.

Relaying is denied by default in sendmail 8.9.x. While this might sound desirable, this feature will probably cause your mail to stop working because it will refuse to relay mail from your own internal systems. There are several ways to fix this. The easiest is to use the feature, relay_entire_domain. Refer to for a full description of these new features.

Another major change to sendmail is that it is now picky about the permissions on the directories it uses. You can turn this off by using the DontBlameSendmail option. If you do, don't blame sendmail if you have security problems. I recommend fixing the directory permissions.

Using the default options, you can simply create a file called /etc/mail/relay-domains that contains the names of the systems for which you want to accept relaying. For example, a domain known as "" and also known as "" would have the following:

This feature allows relaying from anywhere (not recommended).

This feature allows any host within your domain as defined by the m class ($=m) to relay through your system. By default, this would be *

This allows relaying based on the individual host name, not the domain name.

Mail blocking
Here are the basics of sendmail's antispam features.

By default, if the sender's domain cannot be resolved in DNS, the mail is rejected. For example:

MAIL FROM: <wkeys@nonexistent> 501 <wkeys@nonexistent>... Sender
domain must exist

Using accept_unresolvable_domains overrides this and accepts mail from any domain or IP address.

To use the access database feature, your system must support at least one map type such as NDBM (standard on most commercial systems such as Solaris) or the Berkeley database (Berkeley DB) 2.X. If you want to use Berkeley DB, you can obtain it from Sleepycat Software. If you install the Berkeley version, make sure you build sendmail with the NEWDB flag (and include the Berkeley DB libraries and include files).

The access_db feature causes sendmail to look in a database map file (by default /etc/mail/access.db) and decide whether to accept or reject mail from a particular user or site. You can even send a custom error message. This feature can also be used to control relaying permissions.

By default, if the sender's address isn't fully qualified, sendmail will refuse the connection. For example:

mail from: 
553 ... Domain name required

Use of this feature overrides the default so the connection will be accepted.

This allows you to block incoming mail to accounts you don't want to receive mail by listing the account in the access database. The mail appears to be accepted but is actually dumped to /dev/null.

If spam is a major problem for your site, you'll be interested in the Realtime Blackhole List. A list of known spam hosts is maintained at The rbl feature causes sendmail to check with (or another RBL server if you specify) and blocks mail if the host is on that list.

If a host has an MX record that points to your site, this feature accepts and relays mail for them.

This allows your users to relay mail though you if the domain portion of the sender address is a local host.

Building the sendmail binary
The first step of course is to download sendmail.8.9.3.tar.gz and sendmail.8.9.3.tar.sig from

cp sendmail.8.9.3.tar.sig /usr/local/src [or wherever you keep your source]
cp  sendmail.8.9.3.tar.gz   /usr/local/src  	
cd   /usr/local/src
gunzip    sendmail.8.9.3.tar.gz    

You'll want to download the sig file so you can verify the validity of the tar file. Downlaod the PGPKEYS file from in directory /pub/sendmail. This file contains the signature keys for Greg Shapiro and Eric Allman of as well as the sendmail signing keys for 1997, 1998, and 1999. You must have the 1999 key for sendmail 8.9.3. You may want to include the older keys anyway if you're installing older versions of sendmail. PGP doesn't behave the same way in all versions. I used PGP Business 4.0.1 for Unix and discovered it doesn't like to have all the keys in the same file. It only wanted to install Greg Shapiro's key. I put each PGP key block into a separate file and then added them one at a time. You will want to certify the keys and check the fingerprints in the PGPKEYS file for each key against what the PGP software thinks the fingerprint is. If you're unfamiliar with PGP, you should refer to the documentation provided with your version of PGP. For Business 4.0.1, you would type the following command to add the key:

	pgp -ka /tmp/key1

/tmp/key1 is the name of the first key to add. Once you add the keys to your public key ring, you're ready to check sendmail's key. Type:

	pgp sendmail.8.9.3.tar.sig

You must have sendmail.8.9.3.tar in the same directory as the signature. PGP will verify the signature. If it's okay, it will tell you that the key used to sign the software is Sendmail Signing Key/1999. Once you have completed this step, you're ready to untar the software and begin installation.

	tar xvf sendmail.8.9.3.tar

This will unload the tar file and create a top-level directory, /usr/local/src/sendmail-8.9.3, which has everything you need to build and configure sendmail. You should review the README files for more information. In most cases, you can just run the build command without any configuration changes. Note that, by default, sendmail is compiled with DNS support. If you're not using DNS, sendmail will run very slowly until the DNS lookup times out. See the README file in the src directory for configuring compile-time options. To use the defaults, type:

cd  /usr/local/src/sendmail-8.9.3/src
./Build  -c

The -c option to build ensures a clean compile from scratch. Alternatively, you can run make from the top-level directory (/usr/local/src/sendmail-8.9.3) to build all the utilities provided with the release. It isn't necessary if you just want the sendmail binary, but you may want to use some of the included utilities such as makemap (for building an access database). Once the build has completed successfully, the binary will be in /usr/local/src/sendmail-8.9.3/src/obj.[ostype].[release].[arch], (obj.SunOS.5.6.sun4).

Next month
This concludes the first part of this series. At this point, you should go through the sendmail source directories and review the README files. Decide which features support your site's policy. Next month, we'll continue building the sendmail config file.

Many thanks to Greg Shapiro at for his prompt and patient explanations. I'm beginning to wonder if he ever sleeps. Also thanks to my partner at Wizard's Keys, Jonathan Klein, for technical input and for running out to get wine when I got frustrated!

Disclaimer: The information and software in this article are provided as-is and should be used with caution. Each environment is unique and the reader is cautioned to investigate with his or her company as to the feasibility of using the information and software in the article. No warranties, implied or actual, are granted for any use of the information and software in this article and neither author nor publisher is responsible for any damages, either consequential or incidental, with respect to use of the information and software contained herein.

Click on our Sponsors to help Support SunWorld


Related SunWorld articles Additional sendmail resources Additional SunWorld resources

About the author
Carole Fennelly is a partner in Wizard's Keys Corporation, a company specializing in computer security consulting. She has been a Unix system administrator for more than 15 years on various platforms and has particularly focused on sendmail configurations of late. Carole provides security consultation to several financial institutions in the New York City area.

What did you think of this article?
-Very worth reading
-Worth reading
-Not worth reading
-Too long
-Just right
-Too short
-Too technical
-Just right
-Not technical enough

[Table of Contents]
Subscribe to SunWorld, it's free!
[Next story]
Sun's Site

[(c) Copyright  Web Publishing Inc., and IDG Communication company]

If you have technical problems with this magazine, contact

Last modified: