Web server wiles '98, part two
Flipping the switch on your secure Web site
June 1998
![[Next story]](/sunworldonline/icons/tb-next.gif){kind=icon}
![[Table of Contents]](/sunworldonline/icons/tb-toc.gif){kind=icon}
![[Search]](/sunworldonline/icons/tb-search.gif){kind=icon}
|
|
Web server wiles '98, part twoFlipping the switch on your secure Web site |
June 1998
|
Peter and co-author Carole Fennelly continue their discussion of how to set up a "padded cell" environment for your Web site. This month they walk you through setting up sendmail in the padded cell, and provide a real-life example of the steps required to go live with your secure Web site.We've also got Bugs, Break-ins, and more. (3,000 words)
 Mail this article to a friend |
ast month, we discussed the
configuration of a chroot padded cell for the Netscape Web
server. This month we show you how to create a similar cell for sendmail
(and boy does it need one), including the relevant logging and authentication
details.
Sendmail
If you need mail processing, be sure to install the latest version of
sendmail (presently 8.9.0) and use it only to process the queue. Other mail
processors could be used, but we've chosen sendmail for our example.
Sendmail does not have to listen on port 25 unless you want to receive incoming mail on your Web server. For security reasons, most sites do not want to do this. You don't want to be using your Web server as a mail gateway. For added paranoia, sendmail lives in the padded cell (/usr/local/netscape/usr/lib/sendmail). You will need to copy the files that sendmail needs into the padded cell:
/usr/local/netscape/etc/mail/sendmail.cf
/usr/local/netscape/etc/mail/aliases
/usr/local/netscape/var/spool/mqueue
Remember, sendmail is not running as a daemon. So we're not worrying about forwarding mail anywhere and we don't need anything in /etc/aliases. If mail from this machine is directed to a user in our domain, DNS (Domain Name System) will find the mail exchanger for the domain and direct the mail there. Don't make the sendmail configuration more complicated than it has to be.
To test sendmail in the padded cell, use:
/usr/sbin/chroot /usr/local/netscape /usr/lib/sendmail -bt -d35.9
Or use your favorite debugging options.
Make sure you either remove or turn off the permissions on the real /usr/lib/sendmail and /var/spool/mqueue.
When you're satisfied that it works, set up a cron job in the
main environment to process the queue
(/var/spool/cron/crontabs/root):
0,15,30,45 * * * * /usr/sbin/chroot /usr/local/netscape-server /usr/lib/sendmail -q
Note that if you completely disable sendmail (disable the daemon and don't
run it from cron) then mail will neither be received nor
delivered.
Logging
You can use the syslog facility to forward system log messages
out of the padded cell by defining the syslog device
(/usr/local/netscape/devices/pseudo/log@0:conslog) using
mknod with the proper major and minor device numbers. Next,
create a symbolic link to it in /usr/local/netscape/dev:
cd /usr/local/netscape/dev ln -s ../devices/pseudo/log@0:conslog conslog
Sendmail, like any application that supports syslog, can write
to conslog -- the device that syslogd reads.
Syslogd will then write the data to whatever file or loghost is
specified in /etc/syslogd.conf. This way, if an intruder breaks into
the padded cell she has no access to the padded cell logs or system logs.
Netscape does not support syslog, so the http logs have to be
copied out of the padded cell nightly via a cron job, as in:
cp -p /usr/local/netscape/etc/httpd/https-[system]/logs/* /var/web-log
Authentication
In the sample installation, we'd gone through quite a bit of effort to set up
a secure Web server. At this point in the system build, we discovered, to our
horror, that the client intended to have a dial-in modem installed to perform
updates and administration. This is an important object lesson because
experiences like this are all too common: People without security knowledge
undo all the system-securing efforts. Don't let this be you. Now we needed an
alternative that would meet their requirements for remote access and would
not compromise the system.
The virtual private network
Since bandwidth was not an issue, we decided to solve this problem with a
virtual private network (VPN). Here, we went with VTCP/Secure from
InfoExpress. The product has Windows NT, Windows 95, and Solaris clients,
and it supports a Solaris daemon. The package lets the administrator create
an initial key that is distributed to the clients. InfoExpress's VPN supports
several different encryption algorithms. (For security reasons, we cannot
disclose which algorithm we used).
The administrator chooses the algorithm. This initial key is used to negotiate the authentication session with the user. Each user gets the same initial key. After the user authenticates, the initial key is used to generate the session key. This session key is changed periodically (in our implementation, every 15 minutes). The InfoExpress VPN supports several different authentication and encryption methods. We chose to use the Security Dynamics SecurID card through TACACS+ for authentication.
In addition to creating the initial key, the administrator may also choose which addresses the VPN should pass through the tunnel. The InfoExpress client intercepts all TCP/IP traffic but will only pass designated traffic through its tunnel. It also validates traffic, in case the user decides to change to to an IP address that does not conform to the address lists defined by the administrator.
We allowed only Telnet, FTP and Netscape administration. When the user starts a TCP application, the tunnel software intercepts traffic that is destined for authorized target machines, encrypts it, and passes it through the tunnel. For an added benefit, we used a compressed TCP session. This cuts down on the data stream size and makes up for the time lost in encryption and decryption.
InfoExpress can usually manage 50-70% compression on a Telnet or FTP session. Once the traffic reaches the end of the tunnel (the Web server in this case), the InfoExpress daemon determines which port/address the protocol wishes to connect to, and establishes a client connection to that service. From an audit standpoint, the log files show all FTP and Telnet connections originating from the Web server, even though they were started on the client machine.
We used TCP-wrappers to ensure that the Telnets and FTPs originated from the Web server. Netscape security verifies that the Web browser that configures it was started on the Web server. We also used access lists on the Cisco router (described later) to enforce that policy.
For added security, we did not keep any security information on the server (e.g., userids and passwords). We implemented VTCP/Secure's support for TACACS+ authentication. We ran a TACACS+ server inside the corporate network and plugged the TACACS+ authentication session to the TACACS+ server through the firewall. The TACACS+ daemon we use has the Security Dynamics SecurID library built in. It interfaces with the Security Dynamics AceServer to provide the one-time authentication mechanism. Some of our VPN sessions originate over the Internet, and some originate within the corporate network. Any administrators on the inside network have their VPN sessions plugged through the firewall to the VPN daemon on the outside Web Server.
The end result is that all remote administration connections, whether they come from inside the company or from the Internet, come through VPN tunnels that are authenticated with SecurID cards and are strongly encrypted.
Cisco router issues
In front of the Web server, we configured a Cisco router with IP and port
access lists that restrict machine access to port 80 and the VPN port.
All other ports are blocked. This helps ensure that all FTP, Telnet and
Netscape admin sessions come through the VPN. Additionally, the Cisco router
is protected by the same TACACS+/SecurID mechanism that protects the VPN.
This limits the exposure to unauthorized access.
Sudo
To further protect the machines, all non-essential setuid programs were
either removed or had their setuid bits taken off. The public domain package
sudo controls the remaining programs. This package allows for a selective use
of setuid programs, including su. We limited the administrators who can
become root.
The surprise pre-production audit
At this point in the build-out, our client's internal audit department
brought in a company to check out the system before it went into production.
The system was not yet ready when they came in, since the Web administrators
hadn't copied their files into the cell. Also, we had not yet checked and
changed all the suid programs, since we were going to have to re-do it anyway
as part of the production rollout plan. Further, we didn't have the latest
patch cluster installed, because we wanted to have the latest just before
production. Up to this point, we had concentrated on the hard outer shell.
We really weren't worried about the audit, though. After all, it was yet
another Big Six firm that probably didn't know jack about Unix. Yeah, right.
The audit team did not look like Big Six types, especially the auditor, who had shoulder length hair and a Walter (mascot of hacker magazine 2600) T-shirt. They'd brought in a ringer, and we thought we were in trouble. As it turned out, the audit was revealing -- but the results were not unexpected.
We gave the auditor a login on a desktop within the company, and one of our SecureID cards. This was to determine what could happen in our worst-case scenario: a hacker gaining access to the server. Thus, he was already past the Cisco router access lists, TCP-wrappers, VPN and SecurID security measures. We monitored all typing over the tunnel, because it is best to trust no one, even (or especially) external auditors.
Our auditor broke in immediately, demonstrating the necessity of keeping up
with the patches. He also found some suid programs that we had missed. The
auditor admitted that this system would be hard to break but said that we
should consider the improbable. Needless to say, we applied all the patches
and cleaned up the suid programs.
Next, our Web administrators copied their files into the padded cell for
testing -- along with suid programs, shells, compilers and other goodies. We
helped them get things running without all the compromising programs
and set up some cron jobs to monitor the cell. We also set up a
backup schedule (nightly online to read-only filesystems, weekly to tape) and
ran a Day One backup.
Flipping the switch
The cutover to the new system was surprisingly smooth. We planned a whole
night, and were done in about twenty minutes. Pizza and beer time!
In the next week, the Web administrators were grumbling about the inconvenience of administration and the slowness of the Web site. The performance issue turned out to involve tuning problems with Netscape. Netscape recommended additions to /etc/init.d/inetinit to support connections from users who are dialed in to a service provider:
ndd -set /dev/tcp tcp_keepalive_interval 900000 ndd -set /dev/tcp tcp_rexmit_interval_min 2000 ndd -set /dev/tcp tcp_rexmit_interval_initial 2000 ndd -set /dev/tcp tcp_rexmit_interval_max 5000 ndd -set /dev/tcp tcp_close_wait_interval 60000 ndd -set /dev/tcp tcp_ip_abort_interval 60000 ndd -set /dev/tcp tcp_conn_req_max_q 512
This tuned the retransmit times, allowing slower speed connections the time to
acknowledge packets, and increased the number of concurrent connections. We
also had to turn off DNS lookups in the Netscape logging programs, and log raw
addresses only. We added a cron job to process the log at night,
translating addresses into names. The system performed well after that.
The final audit
The next audit came four months into production, without prior
notification. The entire company was targeted and it was set up to look like
real hackers. It was actually too convincing. Our monitoring and alerting
mechanisms worked. We were about to call the FBI and report that the client
was under attack when the technical contact broke down and admitted that it
was a sanctioned audit. They didn't find any weaknesses in the Web facility,
vindicating all the steps taken to make the facility as secure as possible.
Conclusions
This may seem like an excessive effort just to protect a Web server.
Remember, however, that if your site is hacked, the hacked page will be
displayed on hacker Web sites long after you've cleaned things up. The
example site has been running with this configuration for about 7 months now,
getting about 400,000 hits per day. There had been some grumbling about the
site being inconvenient to administer, but this stopped when the unannounced
audit could not come up with a suggestion for improvement. The administrators
were quite happy that they did not have to spend weeks writing memos
responding to the audit. They did, however, have a few suggestions for
internal audit....
The resulting system can be categorized as highly secure. The router limits
traffic to the VPN and the Web. The VPN traffic is logged and strongly
authenticated, and only a few protocols are allowed. The Web server and
sendmail live in a chrooted environment where only the necessary
operating system and application components reside. Finally, the servers
themselves are hardened using standard techniques. If all facilities on the
Internet were this well protected, there'd be a lot less to write about in
this column.
Break-ins
The U.S. Government's General Accounting Office has issued a report about the
vulnerability of the State Department's computers -- indeed, many
government-run computers. The results are not encouraging, except to
crackers. The full information is to be released any day now, during U.S.
Senate panel testimony. For news on this, check out Resources below.
The recent login security alert (also in Resources) from 3Com is frightening. It's frightening not just because it affects a lot of sites (all users of CoreBuilder and SuperStack II), but because it reveals a back door built into those products, ostensibly to be used by their service people. One would hope that companies no longer included back doors in their products, but unfortunately, that's not the case.
Here's something to try: Call tech support for any significant hardware products you use, tell them you've lost the password, and see if they can help you. If they say that you'll need to send the unit in for replacement, you can breathe easy. Otherwise...
Conferences
The 8th Annual Network Security conference (NetSec '98) looks like an
interesting collection of tutorials and talks. It takes place in San Antonio,
TX, from June 15 to 17 and is produced by the Computer Security Institute.
The annual Usenix Technical Conference is going on from June 15 to 19 in New Orleans, LA.
Further down the road, SANS is having its annual Network Security Conference from October 25 to 30 in Orlando, FLA. The call for papers has just gone out. This is a great opportunity to learn and share your knowledge with others -- give a talk, attend tutorials, and take a well deserved break for some sightseeing.
Bugs
Sun has released new kernel jumbo patches for Solaris 2.5.1 (103640-20) and
Solaris 2.6 (105181-05). The patches include security patches. Recent
experience shows that the jumbo patches are well tested and integrated, and
so can be safely installed. Unfortunately, there are rumors that this patch
is unstable, causing OpenWindows to hang. In this instance, unless your need
is urgent, you may want to wait for the next revision of these patches.
Sun also released Solaris 2.6 patch 105755-04, which fixes security holes in DNS. CERT just released a summary warning that BIND and DNS bugs are being actively exploited by hackers. The details are found at CERT.
As we write this, a major satellite (Galaxy IV) has gone out of whack. The news is all over the press. Pager and news feeds across the country have been brought to a halt due to the simultaneous failure of primary and secondary systems, which caused Galaxy IV to tilt. The reason for this tilt is not yet known, though apparently it wasn't caused by some intergalactic pinball. Why write about this here? It makes a good cautionary tale: primary and backup systems can fail, causes are hard to determine, and over-dependence on any one technology can lead to major problems when it fails. Full resumption of service, provided by other satellites, is expected to take up to six days.
Errata
One correction to last month's column. The mknod sequence of
commands for making the arp device entry was incorrect. The correct
sequence is
ls -l /dev/arp lrwxrwxrwx 1 root root 29 Jun 12 1997 /dev/arp - ../devices/pseudo/clone@0:arp # ls -l /devices/pseudo/clone@0:arp crw-rw-rw- 1 root sys 11, 44 Jun 12 1997 devices/pseudo/clone@0:arp # cd /usr/local/netscape/devices/pseudo # mknod clone@0:arp c 11 44 # chown root clone@0:arp # chgrp sys clone@0:arp # chmod 666 clone@0:arp # cd /usr/local/netscape/dev # ln -s ../devices/pseudo/clone@0:arp arp
The correction has been made to the May column, but some readers may have seen it before the corrected version was made available. Thanks to Sam Sexton for pointing this out.
Next month
Next month's column will cover the happenings at the recent SANS '98
conference. If you couldn't attend, this should be the next best thing to
being there. If you did attend, this is your chance to find out if you
snagged all the good trinkets.
|
|
Resources
About the author
![[Peter Galvin's photo]](/sunworldonline/icons/galvin.gif){kind=icon}
Peter Galvin is chief technologist for Corporate Technologies Inc., a systems
integrator and VAR. He is also adjunct system planner for the Computer
Science Department at Brown University, and has been program chair for the
past four SUG/SunWorld conferences. As a consultant and trainer, he has given
talks and tutorials worldwide on the topics of system administration and
security. He has written articles for Byte and Advanced Systems
(SunWorld) magazines, and the Superuser newsletter. Peter is
co-author of the best-selling Operating Systems Concepts textbook.
Reach Peter at peter.galvin@sunworld.com.
Carole Fennelly is a partner in Wizard's Keys Corporation, a company specializing in computer security consulting. She has been a Unix systems administrator for over 15 years on various platforms. Of late, she has particularly focused on sendmail configurations. Carole provides security consulting to several financial institutions in the New York City area. Reach Carole at carole.fennelly@sunworld.com.
|
|
|
|
![[(c) Copyright Web Publishing Inc., and IDG Communication company]](/sunworldonline/icons/b-copyright98.gif){kind=icon}
If you have technical problems with this magazine, contact webmaster@sunworld.com
URL: http://www.sunworld.com/swol-06-1998/swol-06-security.html
Last modified: