Web server wiles '98, part one
Securing your site with a padded cell environment
Peter picks up where his previous Web server security columns ("Web server wiles," parts one, and two) left off. In this first of two parts, Peter and Carole cover the recommended steps to create a tight "padded cell" execution environment on your Web server using the
Also this month, more Ministry of Defense break-ins, BIND bugs you should know about, as well as a couple of interesting reader letters on last month's column, "Designing secure software."
Peter's co-author is security consultant and sendmail expert Carole Fennelly. (3,000 words)
In fact, it's much less painful to install a secure Web server from day one. This way you get to shake your head in disbelief rather than hide in fear when you hear of a site being hacked.
Securing your Web server by using a
chrooted area greatly
enhances the security of your Web site and is not as difficult to set
up as you may think.
Setting up a
chrooted Web server requires careful
attention to many technical details. To make the processes clearer, we
will use a running example of a successful installation set up in this
The Web server resides on a Sun UltraSPARC running Solaris 2.5.1. All unnecessary services such as NFS, NIS, and
uucp have been
removed from the system, regardless of reported (or unreported)
security holes. See
"Web server wiles, part one"
Network access to the Web server is controlled through tcp_wrappers, a software package written by Wietse Venema (see Resources below for more information). This package provides a basic level of authentication for machine addresses attempting to initiate a session. Two configuration files manage permitted and denied services for specific addresses. Any addresses and services not specifically permitted are explicitly denied. Preventive measures against IP spoofing are handled at the router. No modems are connected to the system.
Although IP address-based authentication is far from secure, it is certainly better than no authentication. Remember we're implementing an onion here, not an M&M.
The system used in this example has its own T1 connection to the Internet and is protected by a router, not a firewall. The techniques described here would work just as well for a Web server that operates in the DMZ -- a third network connected to the firewall in order to separate it from the Internet and from the company's internal networks. Beyond the work described here, security could be increased with a network monitoring mechanism such as NetRanger by Wheelgroup or RealSecure by ISS.
Web server software
The Netscape Enterprise Server was chosen for the Web server due to its features and support. While consideration was certainly given regarding the security of the server software, it was decided not to trust any Web server software, whether from Netscape, Microsoft, or Apache. Instead, a Unix utility called
chroot was implemented to
run the Web server in a restricted environment known as a padded cell.
This restricts the Web server process (HTTPd) to a specific directory
tree. Only files and programs specifically placed in that tree are
available to the Web server. In this way, even if there is a
programming error in a CGI script, for instance, there is no way for a
cracker to get to any significant part of the system. Some Web server
software includes support for
chroot as an option. We
preferred not to use the vendor implementation and instead to set it up
manually. The sendmail process, used for the site's Comments feature,
is also run in the
As far as security is concerned, the Achilles heel of every computer system is convenient administration. For this Web server, we have deployed a virtual private network (VPN) package to allow administrators to safely administer the machine. The VPN package chosen, VTCPSecure from InfoExpress, allows us to create encrypted sessions between our client machines and the Web server system and forces the administrators to authenticate with a SecurID token card from Security Dynamics. The advantage of this scenario is that only valid administrators will be able to gain administrative access to the machine, and the sessions themselves will be secured using one-time passwords and triple DES (Data Encryption Standard) encryption.
The procedure outlined in our previous "Web server wiles" columns was followed for the installation of the OS, with the following specific options chosen.
Installing the OS
Solaris 2.5.1 was installed with the latest patches. It was decided to make /tmp a UFS filesystem and mount it "no-suid". While Sun did patch the tmpfs problems, we didn't want to worry about them resurfacing. Besides, we had plenty of disk space and /tmp file performance was not a concern in this instance.
Our machine only has one Ethernet interface so we did not have to worry about IP forwarding. We created the file /etc/defaultrouter to keep in.routed from starting.
Additionally, we removed all unnecessary startup files from /etc/rc2.d. These included S71rpc, S74autofs, S73nfs.client, S88sendmail, S76nscd, and S80lp. We also chopped all services except FTP and telnet from /etc/inetd.conf.
Installing the padded cell
Most Web servers are exploited through system configuration problems, such as a lack of current patches, or through programming errors in CGI, Perl, or Java scripts. Often, the Web site administrator, while knowledgeable in Web design, is inexperienced with system administration. If the server is managed by an experienced system administrator, the administrator may not have experience in Web programs. We decided to separate the Web pages in a padded cell so that even if there was a programming error, the cracker would only be able to see the limited part of the system that is duplicated in the
The padded cell is set up using the
chroot. This is not
new technology and has often been used for software testing or to
establish a restricted environment (Bill Cheswick's An Evening With
Berford describes setting up a "jail" using
chroot causes a command to be executed
believing an alternate directory is root. In the case of the Web
server, it is invoked in the startup script
/usr/sbin/chroot /usr/local/netscape /bin/sh /usr/local/etc/httpd/start-admin
Of course, if you are telling the program that an alternate directory is root, it will not be able to see any of the libraries, configuration files, or programs it may need to run. These have to be carefully added to the alternate directory so that the Web server will run securely. This can be tricky, which is probably why more people are not using padded cells.
The tedious portion of a padded cell is that a complex application that uses networking resources requires certain components to be available. Among these components are device files, libraries, the loader, and utilities. An administrator creating a padded cell will have to go through many iterations to determine what he will need to put in the padded cell. Avoid the temptation to blindly copy device files or programs without confirming that you really need them. Otherwise, you could defeat the purpose of the padded cell.
Let's say we want to install our padded cell making the root directory /usr/local/netscape. We need to create a directory structure for our "root," with the appropriate permissions:
Next, we need to copy only the programs, libraries, and devices required for our application. For example, if we want to run /bin/sh, we need:
/dev/null (Null device)
/dev/zero (Zero special file)
/lib/ld.so (Unix loader)
/lib/libc.so (C library)
To determine the requirements for the application (in this case the
Netscape Enterprise Server), run the application with
truss and save the output to a file:
truss /usr/local/etc/httpd/admserv/bin/ns_admin 2>&1 | tee /tmp/out
This prints out the system call trace for the application to both the
console and to the file /tmp/out to be reviewed. What you are
looking for are all the libraries, files, and devices that the
application opens when it is loading and running. You will then put
these libraries, files, and devices in the appropriate directories in
the padded cell (Remember to use
mknod to make the devices
-- you can't just copy them.)
For devices you may need:
clone@0:arp clone@0:zsh sy@0:tty cn@0:console tl@0:ticlts cn@0:syscon tl@0:ticots cn@0:systty tl@0:ticotsord log@0:conslog mm@0:null clone@0:ip clone@0:tcp clone@0:udp mm@0:zero
Remember, with devices you need to define them in /usr/local/netscape/devices/pseudo and then create symbolic links from /usr/local/netscape/dev to them. For example, to recreate the entry for /dev/arp:
# ls -l /dev/arp lrwxrwxrwx 1 root root 29 Jun 12 1997 /dev/arp -> ../devices/pseudo/clone@0:arp # ls -l /devices/pseudo/clone@0:arp crw-rw-rw- 1 root sys 11, 44 Jun 12 1997 devices/pseudo/clone@0:arp # cd /usr/local/netscape/devices/pseudo # mknod clone@0:arp c 11 44 # chown root clone@0:arp # chgrp sys clone@0:arp # chmod 666 clone@0:arp # cd /usr/local/netscape/dev # ln -s ../devices/pseudo/clone@0:arp arp
This will maintain the same standard that Solaris uses in its main
boot -r in Solaris automatically does the
mknods and creates the links for you, but Solaris boot
does not know about the
chroot environment and therefore
cannot perform this "automagically." You will also have to copy the
libraries in the same way that Solaris does. For example,
/usr/lib/libc.so is symbolically linked to
/usr/lib/libc.so.4. If you need this library (and you probably
will), you must copy /usr/lib/libc.so.4 to
/usr/local/netscape/usr/lib/libc.so.4 and then create a
symbolic link, libc.so in that directory pointing to it. This
way, if you upgrade libraries through patches, you can easily upgrade
the padded cell. Also, copy any configuration files (/etc
files) or make directories that the application will need in the padded
cell. Make sure you remove any entries that are not required by the
application software. Do not put a valid /etc/passwd
file in here. A common configuration file would be /etc/services (for
network based applications) or /etc/TIMEZONE. You may need to copy
terminfo information. The output from truss will help with this immensely.
Next, put /bin/sh into the padded cell. Enter the padded cell using the following command:
/usr/sbin/chroot /usr/local/netscape /bin/sh
Now you are at a shell prompt within the padded cell. This is exactly what a cracker would see if she broke out of your Web software. Run the application from here and see if you missed anything that you require.
Once the application is running, clean out any unnecessary tools that may have been left behind. You'll want to give the cracker as little information as possible and provide as few tools as possible. Define a limited set of ports in /usr/local/netscape/etc/services.
Next month's column will conclude the description of creating a secure padded cell to contain your Web and mail server. The topics include setting up sendmail, configuring logging, secure administrative access, the use of sudo, and the results of an audit of the completed system.
MOD, The Masters of Downloading, claims to have broken into NASA and stolen key software. MOD is the same group that broke into the U.S. military telecommunications backbone. Quite a flurry of claims, accusations, and counter-claims are flying between the agencies and the MOD members. MOD says it will release, on May 1 or 2, software that it stole from the U.S. military for tracking and communicating with submarines. See Resources below for links to this story.
It was a banner month for BIND, as three separate bugs were uncovered. These bugs can lead to remote users gaining root access. The bugs will be fixed in BIND versions 4.9.7 and 8.1.2, available from ISC. Sun has not released patches for these bugs yet but should shortly. Details are contained in CIAC Bulletin I-044.
Also of interest, Sun has released three new sets of patches for bugs in
Details are in
Sun Security Bulletin #165,
Sun Security Bulletin #166, and
Sun Security Bulletin #167 respectively.
The 8th Annual Network Security conference (NetSec '98) looks like an interesting collection of tutorials and talks. It takes place in San Antonio, TX, from June 15 to 17 and is produced by the Computer Security Institute.
And of course don't forget SANS '98 this month. Check out last month's column for more details. The SANS (System Administration, Networking and Security) Institute is also helping to review and improve a potentially valuable advance in network traffic sensors and traffic analysis methods, created by the U.S. Navy and the U.S. Energy Department laboratories. The results of this work will be announced at SANS '98. Hope to see you there.
Last month's column about a programming methodology for security programs drew some interesting reader responses.
Daniel Barlow sent the following lengthy reply about the commonality of the problem and applicability of the methodology:
I read the article and laughed in that "been there, seen that" fashion common to anyone with system administration pretensions. The problem that you describe and the methodology you advocate has far wider applicability than just set[gu]id programs in operating systems, though. Guidelines like that ought to be applied to every CGI program in existence.
It's unfortunate that CGI programmers are generally lower on the food chain than anyone else except HTML coders and possibly janitors. They often tend to construct their programs by cargo cult methodologies, and are probably under pressure to get the job done fast, to hack it about at great speed afterwards (because the customer has a slightly different conception of what he wanted), then to have it go into production use on a machine outside the firewall. Running your Web server as "nobody" is the knee-jerk fix which doesn't actually help -- for any script complex enough to be useful, you tend to end up with half the disk owned by "nobody" (or worse, world-writable) so it can store appropriate state.
I don't know exactly what the answer is, but part of it is surely to admit that raw CGI is not an appropriate base on which to build anything much larger than "mail me your fill-out form" or "search our Web site." We need something with a process model that lasts longer than an HTTP request, and a permissions model that allows different viewers different access rights, or we'll continue to see Web sites with online ordering CGIs that trust the "Total Cost" hidden field that the browser sent them, and file viewers that happily show any file on the disk.
Alex Cozzi points out:
I would like to add another tip about how to write secure software.
Consider writing it in a language other than C/C++. Probably the safest option is Eiffel: You will not have to worry about buffer overflow, string handling, wrong pointers, or memory leaks, and you will be able to concentrate on better design and testing. SmallEiffel is a very efficient Eiffel compiler that runs just fine on Solaris, and it's free.
In further conversation Alex indicates that SmallEiffel can generate C. Optionally, the resulting code can have embedded require and ensure routines surrounding all system calls for runtime assertion checking. SmallEiffel code, therefore, is likely to be error-checked and less buggy. Besides, Bill Joy is a big Eiffel fan!
There are at least two other useful resources that address the issues of writing secure code. They are Security Code Review Guidelines, by Adam Shostack and Writing Safe Setuid Programs, by Matt Bishop. It makes sense to merge these efforts into a FAQ. Maybe in a future column...
Sean Boran has released an extensive document about IT Security, including information on securing Unix, NT, and Win/95. The first release could use some editing and improved formatting, but it looks to be useful nonetheless. He says the site is "virtually unknown at the moment," and has asked us to pass on the URL to others.
In June we'll conclude the coverage of how to build a secure sandbox for your Web and mail server. Topics will include setting up the VPN tunnels, using sudo, and the results of an audit of the finished product.
About the author
Peter Galvin is chief technologist for Corporate Technologies Inc., a systems integrator and VAR. He is also adjunct system planner for the Computer Science Department at Brown University, and has been program chair for the past four SUG/SunWorld conferences. As a consultant and trainer, he has given talks and tutorials worldwide on the topics of system administration and security. He has written articles for Byte and Advanced Systems (SunWorld) magazines, and the Superuser newsletter. Peter is co-author of the best-selling Operating Systems Concepts textbook. Reach Peter at firstname.lastname@example.org.
Carole Fennelly is a partner in Wizard's Keys Corporation, a company specializing in computer security consulting. She has been a Unix systems administrator for over 15 years on various platforms and has particularly focused on sendmail configurations of late. Carole provides security consulting to several financial institutions in the New York City area. Reach Carole at email@example.com.
If you have technical problems with this magazine, contact firstname.lastname@example.org