Setting up sendmail on a firewall New features in sendmail 8.9.3 and how to build the binary

April  1999

A tremendous amount of information is available about sendmail. But if you're in a hurry to just get it working, you don't have the time to go through and digest it all. In this first installment in a three-part series on sendmail, Carole discusses its implementation on the firewall and focuses on features new to sendmail 8.9.3. Next month she'll cover the particulars of the config file. (3,000 words)

Mail this article to a friend

ne of the greatest features of sendmail is the extreme flexibility it provides the administrator. This is also one of its worst features, the reason being that it is so often misconfigured. Usually, a system administrator inherits a sendmail configuration file that he or she is afraid to touch, lest it should break. Over time, this file becomes hopelessly confusing, contradictory, and redundant.

I began writing this column with the intention of providing a simple, straightforward, "sendmail for dummies" approach. I ended up appreciating how Bryan Costales must have felt when he wrote his handbook, Sendmail, for O'Reilly & Associates (see Resources): Nothing is simple in sendmail. I also learned to appreciate just how powerful and flexible sendmail really is. It's worth learning. It's also worth a few columns. I'll devote this month's column to discussing some of the background and new features in sendmail 8.9.3 as well as how to build the source code. Next month we'll look into the configuration file. If there is enough interest, I will add a third column for special situations and testing techniques.

Sendmail on the firewall?
Like most security admins, I've always been told it's a bad idea to run sendmail on the firewall. It's generally considered better to run something like smap instead. Indeed, I ran smap myself until I discovered a problem I couldn't fix with smap. The problem was that company e-mail was forwarded from an internal mail gateway system to the firewall. The firewall could only rewrite the external header to say the mail came from "company.com." There was still an internal header that showed the mail was routed through "mailgate.company.com," complete with the mailgate internal IP address. I couldn't have the inside machine masquerade as company.com because there were other internal mail gateways it had to communicate with. (They couldn't all be company.com.) But because sendmail is extremely flexible, open source software, it's possible to add functionality to strip out the inside header. Also, many of the security risks with sendmail have to do with the fact that it's generally run setuid to root. It isn't necessary to run it this way on a firewall because there are no direct mail users on the firewall. For added security, I run sendmail in a chrooted cell with the program mailer using the sendmail restricted shell, smrsh.

Preliminaries
There are certain resources you should have available before you start working with sendmail. This column isn't going to attempt to cover all aspects of sendmail. For that, I direct you to Bryan Costales's Sendmail, second edition. Aside from being an excellent resource, at more than 1,000 pages this book can also double as a child's booster seat!

You'll also need the latest copy of the sendmail source code (sendmail 8.9.3 as of this writing), available from sendmail.org. Verify the PGP signature! Some very useful links are included with the latest information.

Although it should go without saying, you need to be able to build the code using make and gcc or cc and have your environment variables set appropriately. It is strongly recommended that you use a system other than the firewall to compile and test the code. Remember that any system you use has to be running the same release of the operating system and have a compatible architecture. The firewall shouldn't have a development environment installed. If for some reason you have to use the firewall to compile, at least keep the development environment on a spare drive that you can unmount when you don't need it. Life with sendmail is much easier if you have the GNU utilities already installed because it is assumed in the makefile that groff is installed. You can get the GNU utilities ( For the purposes of verifying the source (and not just for sendmail), you should have a copy of PGP (Pretty Good Privacy) installed (version 2.6.3 or higher). You can download a personal copy or buy one from Network Associates.

You can find links to these and other sendmail resources in the Resources section at the bottom of this article.

New features in sendmail 8.9.3
Sendmail 8.9x contains several new features that will probably cause your mail to break unless you configure them properly. These features mostly help filter spam and prevent your site from being used to relay someone else's mail. You have to decide how to configure them based on your site's policy. For my own network, I can rather tightly define from whom I'll accept mail. However, an ISP cannot be so selective. Please note that I haven't personally configured mail for an ISP, so I can only speculate on the special concerns of ISPs. Because ISPs have to service the general public, I would imagine they have legal issues to consider before blocking mail from a particular site.

Relaying is denied by default in sendmail 8.9.x. While this might sound desirable, this feature will probably cause your mail to stop working because it will refuse to relay mail from your own internal systems. There are several ways to fix this. The easiest is to use the feature, relay_entire_domain. Refer to sendmail.org for a full description of these new features.

Another major change to sendmail is that it is now picky about the permissions on the directories it uses. You can turn this off by using the DontBlameSendmail option. If you do, don't blame sendmail if you have security problems. I recommend fixing the directory permissions.

Default
Using the default options, you can simply create a file called /etc/mail/relay-domains that contains the names of the systems for which you want to accept relaying. For example, a domain known as "company.com" and also known as "othername.com" would have the following:

company.com
othername.com

• Advantage: If you only need to relay mail from a few systems, this is probably the simplest solution and the one I recommend for most installations. It prevents you from acting as a relayer, while allowing your mail to go through.

• Disadvantage: You have to keep the file updated, and you may have other requirements. If you're an ISP, you may have to allow relaying from quite a large number of domains. Also, every time you update this file you have to restart sendmail (kill -1 [sendmail_pid]).

promiscuous_relay
This feature allows relaying from anywhere (not recommended).

• Advantage: You don't have to worry about mail being denied.

• Disadvantage: Anyone can use your system as a mail relay. Use of this feature defeats the antirelay features of sendmail and is definitely not recommended.

relay_entire_domain
This feature allows any host within your domain as defined by the m class ($=m) to relay through your system. By default, this would be *.company.com.

• Advantage: You don't have to worry about mail within your domain being denied. Systems outside your domain can't use your system to relay mail unless they're defined in the relay-domains file or in the access database. If you only want to relay hosts in your own domain, you can use this instead of the file /etc/mail/relay-domains.

• Disadvantage: You may not want to allow other organizations in your company to route mail through your system. Also, if you're known by several domain names (popular with companies seeking "product recognition"), this won't buy you much.

relay_hosts_only
This allows relaying based on the individual host name, not the domain name.

• Advantage: Fine-tunes the permission for relaying by requiring a fully qualified host name rather than just the domain name.

• Disadvantage: Requires you to specify in either the file /etc/mail/relay-domains or the access database the host name of the system you're permitting to relay. For example, if I use this, my /etc/mail/relay-domains file would have:

  company.com
  
  mailgate.company.com
  
  othername.com
  
  mailgate.othername.com 
  

Mail blocking
Here are the basics of sendmail's antispam features.

accept_unresolvable_domains
By default, if the sender's domain cannot be resolved in DNS, the mail is rejected. For example:

MAIL FROM: <wkeys@nonexistent> 501 <wkeys@nonexistent>... Sender
domain must exist

Using accept_unresolvable_domains overrides this and accepts mail from any domain or IP address.

• Advantage: You may have to accept mail from IP addresses if the senders don't have their headers rewritten to come from a registered domain.

• Disadvantage: You can be spammed from anywhere. You may be better off listing known sites with this problem in the access database.

access_db
To use the access database feature, your system must support at least one map type such as NDBM (standard on most commercial systems such as Solaris) or the Berkeley database (Berkeley DB) 2.X. If you want to use Berkeley DB, you can obtain it from Sleepycat Software. If you install the Berkeley version, make sure you build sendmail with the NEWDB flag (and include the Berkeley DB libraries and include files).

The access_db feature causes sendmail to look in a database map file (by default /etc/mail/access.db) and decide whether to accept or reject mail from a particular user or site. You can even send a custom error message. This feature can also be used to control relaying permissions.

• Advantage: Really allows you to fine-tune who you will accept mail from. For example, I may not want to accept mail from domains that can't be resolved, but I want to make an exception for a particular domain or address. If yours is a large site, you may want the greater flexibility this can give you -- it's rather like a firewall rule base.

• Disadvantage: You have to build the database file and keep it updated. It can become rather complex -- it's rather like a firewall rule base. The more complex you make it, the harder it is to maintain.

accept_unqualified_senders
By default, if the sender's address isn't fully qualified, sendmail will refuse the connection. For example:

mail from: 
553 ... Domain name required

Use of this feature overrides the default so the connection will be accepted.

• Advantage: I recommend always using fully qualified addresses. However, on an internal mail gateway, you may not be able to control how the other local systems send you mail, and this will allow you to accept mail with unqualified sender addresses.

• Disadvantage: You lose some ability to track where mail is coming from. Don't use this on a firewall.

blacklist_recipients
This allows you to block incoming mail to accounts you don't want to receive mail by listing the account in the access database. The mail appears to be accepted but is actually dumped to /dev/null.

• Advantage: You may want to use this for "nobody" or "guest" accounts.

• Disadvantage: You have to set up the access database. This isn't much of a disadvantage, but it is another step.

rbl
If spam is a major problem for your site, you'll be interested in the Realtime Blackhole List. A list of known spam hosts is maintained at rbl.maps.vix.com. The rbl feature causes sendmail to check with rbl.maps.vix.com (or another RBL server if you specify) and blocks mail if the host is on that list.

• Advantage: You don't have to worry about maintaining your own list of spam hosts.

• Disadvantage: You're trusting that the RBL server is accurate. This may also cause a delay in accepting a connection while the RBL server is contacted.

relay_based_on_MX
If a host has an MX record that points to your site, this feature accepts and relays mail for them.

• Advantage: You don't have to add hosts to the access database if they have an MX record that points to you.

• Disadvantage: This seems like a lazy way out. It's probably quicker to look up systems in the access database than check for MX records. This will allow a third party to relay through you without your permission.

relay_local_from
This allows your users to relay mail though you if the domain portion of the sender address is a local host.

• Advantage: If you have users that belong in your domain but are routed through another domain, this will allow their mail to be relayed.

• Disadvantage: It's very easy to fake the sender address, and using this opens you up to spammers. I don't recommend it.

Building the sendmail binary
The first step of course is to download sendmail.8.9.3.tar.gz and sendmail.8.9.3.tar.sig from sendmail.org.

cp sendmail.8.9.3.tar.sig /usr/local/src [or wherever you keep your source]
cp  sendmail.8.9.3.tar.gz   /usr/local/src  	
cd   /usr/local/src
gunzip    sendmail.8.9.3.tar.gz    

You'll want to download the sig file so you can verify the validity of the tar file. Downlaod the PGPKEYS file from sendmail.org in directory /pub/sendmail. This file contains the signature keys for Greg Shapiro and Eric Allman of sendmail.org as well as the sendmail signing keys for 1997, 1998, and 1999. You must have the 1999 key for sendmail 8.9.3. You may want to include the older keys anyway if you're installing older versions of sendmail. PGP doesn't behave the same way in all versions. I used PGP Business 4.0.1 for Unix and discovered it doesn't like to have all the keys in the same file. It only wanted to install Greg Shapiro's key. I put each PGP key block into a separate file and then added them one at a time. You will want to certify the keys and check the fingerprints in the PGPKEYS file for each key against what the PGP software thinks the fingerprint is. If you're unfamiliar with PGP, you should refer to the documentation provided with your version of PGP. For Business 4.0.1, you would type the following command to add the key:

	pgp -ka /tmp/key1

/tmp/key1 is the name of the first key to add. Once you add the keys to your public key ring, you're ready to check sendmail's key. Type:

	pgp sendmail.8.9.3.tar.sig

You must have sendmail.8.9.3.tar in the same directory as the signature. PGP will verify the signature. If it's okay, it will tell you that the key used to sign the software is Sendmail Signing Key/1999. Once you have completed this step, you're ready to untar the software and begin installation.

	tar xvf sendmail.8.9.3.tar

This will unload the tar file and create a top-level directory, /usr/local/src/sendmail-8.9.3, which has everything you need to build and configure sendmail. You should review the README files for more information. In most cases, you can just run the build command without any configuration changes. Note that, by default, sendmail is compiled with DNS support. If you're not using DNS, sendmail will run very slowly until the DNS lookup times out. See the README file in the src directory for configuring compile-time options. To use the defaults, type:

cd  /usr/local/src/sendmail-8.9.3/src
./Build  -c

The -c option to build ensures a clean compile from scratch. Alternatively, you can run make from the top-level directory (/usr/local/src/sendmail-8.9.3) to build all the utilities provided with the release. It isn't necessary if you just want the sendmail binary, but you may want to use some of the included utilities such as makemap (for building an access database). Once the build has completed successfully, the binary will be in /usr/local/src/sendmail-8.9.3/src/obj.[ostype].[release].[arch], (obj.SunOS.5.6.sun4).

Next month
This concludes the first part of this series. At this point, you should go through the sendmail source directories and review the README files. Decide which features support your site's policy. Next month, we'll continue building the sendmail config file.

Acknowledgments
Many thanks to Greg Shapiro at sendmail.org for his prompt and patient explanations. I'm beginning to wonder if he ever sleeps. Also thanks to my partner at Wizard's Keys, Jonathan Klein, for technical input and for running out to get wine when I got frustrated!

Disclaimer: The information and software in this article are provided as-is and should be used with caution. Each environment is unique and the reader is cautioned to investigate with his or her company as to the feasibility of using the information and software in the article. No warranties, implied or actual, are granted for any use of the information and software in this article and neither author nor publisher is responsible for any damages, either consequential or incidental, with respect to use of the information and software contained herein.

Resources

• Sendmail.org
  http://www.sendmail.org
• Download sendmail 8.9.3
  ftp://ftp.sendmail.org/pub/sendmail
• Sendmail Inc.
  http://www.sendmail.com
• Description of new relaying features
  http://www.sendmail.org/tips/relaying.html
• Description of new and antispam features
  http://www.sendmail.org/m4/anti-spam.html
• Summary of rulesets and reserved MACROS
  http://www.sendmail.org/m4/admin.html
• GNU utilities
  ftp://sunsite.unc.edu/pub/gnu
• PGP program
  http://www.pgp.com
• Berkeley DB
  http://www.sleepycat.com
• Realtime Blackhole List
  http://maps.vix.com/rbl/

• "Postfix and Mailman deliver enhanced e-mail security and performance" by Cameron Laird and Kathryn Soraiz (Cover story, March 1999)
  http://www.sunworld.com/swol-03-1999/swol-03-mailtools.html
• "Creating a basic padded cell" by Carole Fennelly (SunWorld, January 1999)
  http://www.sunworld.com/swol-01-1999/swol-01-security.html
• "Web server wiles '98, Part 1: Securing your site with a padded cell environment" by Peter Galvin (SunWorld, May 1998)
  http://www.sunworld.com/swol-05-1998/swol-05-security.html
• "Web server wiles '98, Part 2" by Peter Galvin (SunWorld, June 1998)
  http://www.sunworld.com/swol-06-1998/swol-06-security.html
• Network security-related articles listed in the SunWorld topical index
  http://www.sunworld.com/common/swol-siteindex.html#netsec
• Web server security-related articles listed in the SunWorld topical index
  http://www.sunworld.com/common/swol-siteindex.html#websec

• "Rule, set, match" by Hal Stern (Advanced Systems magazine, December 1993)
  http://www.sunworld.com/sunworldonline/asm-12-1993/asm-12-sysadmin.html
• "The Good, The Bad, and The Impressive" (Hacker News Network)
  http://www.hackernews.com/orig/whoever.html#critic
• Costales, Bryan, et al., Sendmail (O'Reilly & Associates, 1997)
  http://www.amazon.com/exec/obidos/ASIN/1565922220/o/qid=922906891/sr=2-1/002-0274177-5869440
• Costales, Bryan, et al., Sendmail Desktop Reference (O'Reilly & Associates, 1997)
  http://www.amazon.com/exec/obidos/ASIN/1565922786/ref=sim_books/002-0274177-5869440
• The Hacker News Network
  http://www.hackernews.com

• Full listing of previous Security columns in SunWorld
  http://www.sunworld.com/common/swol-backissues-columns.html#security
• Peter Galvin's Solaris Security FAQ (recently updated!)
  http://www.sunworld.com/sunworldonline/common/security-faq.html
• Peter Galvin's Unix Secure Programming FAQ
  http://www.sunworld.com/swol-08-1998/swol-08-security.html
• The SunWorld Topical Index -- a comprehensive listing of all SunWorld articles by subject
  http://www.sunworld.com/common/swol-siteindex.html
• Take a look at sunWHERE, our launchpad to hundreds of online resources for Sun users
  http://www.sunworld.com/sunworldonline/sunwhere.html
• Check out SunWorld's back issues
  http://www.sunworld.com/common/swol-backissues.html
• IDG.net, your one-stop IT resource
  http://www.idg.net

About the author
Carole Fennelly is a partner in Wizard's Keys Corporation, a company specializing in computer security consulting. She has been a Unix system administrator for more than 15 years on various platforms and has particularly focused on sendmail configurations of late. Carole provides security consultation to several financial institutions in the New York City area.

If you have technical problems with this magazine, contact webmaster@sunworld.com

URL: http://www.sunworld.com/swol-04-1999/swol-04-security.html
Last modified: