Click on our Sponsors to help Support SunWorld
Security: Pete's Wicked World by Peter Galvin

Web server wiles '98, part one

Securing your site with a padded cell environment

May  1998
[Next story]
[Table of Contents]
Subscribe to SunWorld, it's free!

Peter picks up where his previous Web server security columns ("Web server wiles," parts one, and two) left off. In this first of two parts, Peter and Carole cover the recommended steps to create a tight "padded cell" execution environment on your Web server using the chroot command.

Also this month, more Ministry of Defense break-ins, BIND bugs you should know about, as well as a couple of interesting reader letters on last month's column, "Designing secure software."

Peter's co-author is security consultant and sendmail expert Carole Fennelly. (3,000 words)

Mail this
article to
a friend
Hacking Web servers is becoming a cottage industry these days. Although the hacked site may be more amusing than the original, things are not so funny for the site administrator. No, as site administrator you may find yourself roped into meetings with upper management, where you are served up as the proverbial special of the day. Next, you'll have to implement a rush repair and protection solution formulated by Someone Less Technical Than You.

In fact, it's much less painful to install a secure Web server from day one. This way you get to shake your head in disbelief rather than hide in fear when you hear of a site being hacked.

Securing your Web server by using a chrooted area greatly enhances the security of your Web site and is not as difficult to set up as you may think.

Setting up a chrooted Web server requires careful attention to many technical details. To make the processes clearer, we will use a running example of a successful installation set up in this manner.

The Web server resides on a Sun UltraSPARC running Solaris 2.5.1. All unnecessary services such as NFS, NIS, and uucp have been removed from the system, regardless of reported (or unreported) security holes. See "Web server wiles, part one" for details.

Network services
Network access to the Web server is controlled through tcp_wrappers, a software package written by Wietse Venema (see Resources below for more information). This package provides a basic level of authentication for machine addresses attempting to initiate a session. Two configuration files manage permitted and denied services for specific addresses. Any addresses and services not specifically permitted are explicitly denied. Preventive measures against IP spoofing are handled at the router. No modems are connected to the system.

Although IP address-based authentication is far from secure, it is certainly better than no authentication. Remember we're implementing an onion here, not an M&M.

Network topology
The system used in this example has its own T1 connection to the Internet and is protected by a router, not a firewall. The techniques described here would work just as well for a Web server that operates in the DMZ -- a third network connected to the firewall in order to separate it from the Internet and from the company's internal networks. Beyond the work described here, security could be increased with a network monitoring mechanism such as NetRanger by Wheelgroup or RealSecure by ISS.

Web server software
The Netscape Enterprise Server was chosen for the Web server due to its features and support. While consideration was certainly given regarding the security of the server software, it was decided not to trust any Web server software, whether from Netscape, Microsoft, or Apache. Instead, a Unix utility called chroot was implemented to run the Web server in a restricted environment known as a padded cell. This restricts the Web server process (HTTPd) to a specific directory tree. Only files and programs specifically placed in that tree are available to the Web server. In this way, even if there is a programming error in a CGI script, for instance, there is no way for a cracker to get to any significant part of the system. Some Web server software includes support for chroot as an option. We preferred not to use the vendor implementation and instead to set it up manually. The sendmail process, used for the site's Comments feature, is also run in the chrooted environment.

As far as security is concerned, the Achilles heel of every computer system is convenient administration. For this Web server, we have deployed a virtual private network (VPN) package to allow administrators to safely administer the machine. The VPN package chosen, VTCPSecure from InfoExpress, allows us to create encrypted sessions between our client machines and the Web server system and forces the administrators to authenticate with a SecurID token card from Security Dynamics. The advantage of this scenario is that only valid administrators will be able to gain administrative access to the machine, and the sessions themselves will be secured using one-time passwords and triple DES (Data Encryption Standard) encryption.

The procedure outlined in our previous "Web server wiles" columns was followed for the installation of the OS, with the following specific options chosen.

Installing the OS
Solaris 2.5.1 was installed with the latest patches. It was decided to make /tmp a UFS filesystem and mount it "no-suid". While Sun did patch the tmpfs problems, we didn't want to worry about them resurfacing. Besides, we had plenty of disk space and /tmp file performance was not a concern in this instance.

Our machine only has one Ethernet interface so we did not have to worry about IP forwarding. We created the file /etc/defaultrouter to keep in.routed from starting.

Additionally, we removed all unnecessary startup files from /etc/rc2.d. These included S71rpc, S74autofs, S73nfs.client, S88sendmail, S76nscd, and S80lp. We also chopped all services except FTP and telnet from /etc/inetd.conf.


Installing the padded cell
Most Web servers are exploited through system configuration problems, such as a lack of current patches, or through programming errors in CGI, Perl, or Java scripts. Often, the Web site administrator, while knowledgeable in Web design, is inexperienced with system administration. If the server is managed by an experienced system administrator, the administrator may not have experience in Web programs. We decided to separate the Web pages in a padded cell so that even if there was a programming error, the cracker would only be able to see the limited part of the system that is duplicated in the chrooted environment.

The padded cell is set up using the chroot. This is not new technology and has often been used for software testing or to establish a restricted environment (Bill Cheswick's An Evening With Berford describes setting up a "jail" using chroot). Quite simply, chroot causes a command to be executed believing an alternate directory is root. In the case of the Web server, it is invoked in the startup script (/etc/init.d/netscape) as:

/usr/sbin/chroot  /usr/local/netscape  /bin/sh /usr/local/etc/httpd/start-admin

Of course, if you are telling the program that an alternate directory is root, it will not be able to see any of the libraries, configuration files, or programs it may need to run. These have to be carefully added to the alternate directory so that the Web server will run securely. This can be tricky, which is probably why more people are not using padded cells.

The tedious portion of a padded cell is that a complex application that uses networking resources requires certain components to be available. Among these components are device files, libraries, the loader, and utilities. An administrator creating a padded cell will have to go through many iterations to determine what he will need to put in the padded cell. Avoid the temptation to blindly copy device files or programs without confirming that you really need them. Otherwise, you could defeat the purpose of the padded cell.

Let's say we want to install our padded cell making the root directory /usr/local/netscape. We need to create a directory structure for our "root," with the appropriate permissions:


Next, we need to copy only the programs, libraries, and devices required for our application. For example, if we want to run /bin/sh, we need:

/dev/null (Null device)
/dev/zero (Zero special file)
/lib/ (Unix loader)
/lib/ (C library)

To determine the requirements for the application (in this case the Netscape Enterprise Server), run the application with truss and save the output to a file:

truss /usr/local/etc/httpd/admserv/bin/ns_admin  2>&1 | tee /tmp/out

This prints out the system call trace for the application to both the console and to the file /tmp/out to be reviewed. What you are looking for are all the libraries, files, and devices that the application opens when it is loading and running. You will then put these libraries, files, and devices in the appropriate directories in the padded cell (Remember to use mknod to make the devices -- you can't just copy them.) For devices you may need:

clone@0:arp             clone@0:zsh        sy@0:tty
cn@0:console            tl@0:ticlts
cn@0:syscon             tl@0:ticots
cn@0:systty             tl@0:ticotsord
log@0:conslog   	mm@0:null
clone@0:ip              clone@0:tcp        
clone@0:udp             mm@0:zero

Remember, with devices you need to define them in /usr/local/netscape/devices/pseudo and then create symbolic links from /usr/local/netscape/dev to them. For example, to recreate the entry for /dev/arp:

# ls -l /dev/arp
lrwxrwxrwx   1 root     root          29 Jun 12  1997 /dev/arp -> ../devices/pseudo/clone@0:arp 
# ls -l /devices/pseudo/clone@0:arp
crw-rw-rw-   1 root     sys       11, 44 Jun 12  1997 devices/pseudo/clone@0:arp 
# cd /usr/local/netscape/devices/pseudo
# mknod clone@0:arp c 11 44
# chown root clone@0:arp
# chgrp sys clone@0:arp
# chmod 666 clone@0:arp
# cd  /usr/local/netscape/dev 
# ln -s  ../devices/pseudo/clone@0:arp  arp

This will maintain the same standard that Solaris uses in its main environment. A boot -r in Solaris automatically does the mknods and creates the links for you, but Solaris boot does not know about the chroot environment and therefore cannot perform this "automagically." You will also have to copy the libraries in the same way that Solaris does. For example, /usr/lib/ is symbolically linked to /usr/lib/ If you need this library (and you probably will), you must copy /usr/lib/ to /usr/local/netscape/usr/lib/ and then create a symbolic link, in that directory pointing to it. This way, if you upgrade libraries through patches, you can easily upgrade the padded cell. Also, copy any configuration files (/etc files) or make directories that the application will need in the padded cell. Make sure you remove any entries that are not required by the application software. Do not put a valid /etc/passwd file in here. A common configuration file would be /etc/services (for network based applications) or /etc/TIMEZONE. You may need to copy terminfo information. The output from truss will help with this immensely.

Next, put /bin/sh into the padded cell. Enter the padded cell using the following command:

/usr/sbin/chroot  /usr/local/netscape  /bin/sh

Now you are at a shell prompt within the padded cell. This is exactly what a cracker would see if she broke out of your Web software. Run the application from here and see if you missed anything that you require.

Once the application is running, clean out any unnecessary tools that may have been left behind. You'll want to give the cracker as little information as possible and provide as few tools as possible. Define a limited set of ports in /usr/local/netscape/etc/services.

Wrap up
Next month's column will conclude the description of creating a secure padded cell to contain your Web and mail server. The topics include setting up sendmail, configuring logging, secure administrative access, the use of sudo, and the results of an audit of the completed system.

MOD, The Masters of Downloading, claims to have broken into NASA and stolen key software. MOD is the same group that broke into the U.S. military telecommunications backbone. Quite a flurry of claims, accusations, and counter-claims are flying between the agencies and the MOD members. MOD says it will release, on May 1 or 2, software that it stole from the U.S. military for tracking and communicating with submarines. See Resources below for links to this story.

It was a banner month for BIND, as three separate bugs were uncovered. These bugs can lead to remote users gaining root access. The bugs will be fixed in BIND versions 4.9.7 and 8.1.2, available from ISC. Sun has not released patches for these bugs yet but should shortly. Details are contained in CIAC Bulletin I-044.

Also of interest, Sun has released three new sets of patches for bugs in ndd, rpc.cmsd, and rpcbind. Details are in Sun Security Bulletin #165, Sun Security Bulletin #166, and Sun Security Bulletin #167 respectively.

The 8th Annual Network Security conference (NetSec '98) looks like an interesting collection of tutorials and talks. It takes place in San Antonio, TX, from June 15 to 17 and is produced by the Computer Security Institute.

And of course don't forget SANS '98 this month. Check out last month's column for more details. The SANS (System Administration, Networking and Security) Institute is also helping to review and improve a potentially valuable advance in network traffic sensors and traffic analysis methods, created by the U.S. Navy and the U.S. Energy Department laboratories. The results of this work will be announced at SANS '98. Hope to see you there.

Last month's column about a programming methodology for security programs drew some interesting reader responses.

Daniel Barlow sent the following lengthy reply about the commonality of the problem and applicability of the methodology:

I read the article and laughed in that "been there, seen that" fashion common to anyone with system administration pretensions. The problem that you describe and the methodology you advocate has far wider applicability than just set[gu]id programs in operating systems, though. Guidelines like that ought to be applied to every CGI program in existence.

It's unfortunate that CGI programmers are generally lower on the food chain than anyone else except HTML coders and possibly janitors. They often tend to construct their programs by cargo cult methodologies, and are probably under pressure to get the job done fast, to hack it about at great speed afterwards (because the customer has a slightly different conception of what he wanted), then to have it go into production use on a machine outside the firewall. Running your Web server as "nobody" is the knee-jerk fix which doesn't actually help -- for any script complex enough to be useful, you tend to end up with half the disk owned by "nobody" (or worse, world-writable) so it can store appropriate state.

I don't know exactly what the answer is, but part of it is surely to admit that raw CGI is not an appropriate base on which to build anything much larger than "mail me your fill-out form" or "search our Web site." We need something with a process model that lasts longer than an HTTP request, and a permissions model that allows different viewers different access rights, or we'll continue to see Web sites with online ordering CGIs that trust the "Total Cost" hidden field that the browser sent them, and file viewers that happily show any file on the disk.

Alex Cozzi points out:

I would like to add another tip about how to write secure software.

Consider writing it in a language other than C/C++. Probably the safest option is Eiffel: You will not have to worry about buffer overflow, string handling, wrong pointers, or memory leaks, and you will be able to concentrate on better design and testing. SmallEiffel is a very efficient Eiffel compiler that runs just fine on Solaris, and it's free.

In further conversation Alex indicates that SmallEiffel can generate C. Optionally, the resulting code can have embedded require and ensure routines surrounding all system calls for runtime assertion checking. SmallEiffel code, therefore, is likely to be error-checked and less buggy. Besides, Bill Joy is a big Eiffel fan!

There are at least two other useful resources that address the issues of writing secure code. They are Security Code Review Guidelines, by Adam Shostack and Writing Safe Setuid Programs, by Matt Bishop. It makes sense to merge these efforts into a FAQ. Maybe in a future column...

The Bookstore
Sean Boran has released an extensive document about IT Security, including information on securing Unix, NT, and Win/95. The first release could use some editing and improved formatting, but it looks to be useful nonetheless. He says the site is "virtually unknown at the moment," and has asked us to pass on the URL to others.

Next month
In June we'll conclude the coverage of how to build a secure sandbox for your Web and mail server. Topics will include setting up the VPN tunnels, using sudo, and the results of an audit of the finished product.

Click on our Sponsors to help Support SunWorld


Bugs Conferences Products Secure software design Reading Other SunWorld security resources

About the author
[Peter Galvin's photo] Peter Galvin is chief technologist for Corporate Technologies Inc., a systems integrator and VAR. He is also adjunct system planner for the Computer Science Department at Brown University, and has been program chair for the past four SUG/SunWorld conferences. As a consultant and trainer, he has given talks and tutorials worldwide on the topics of system administration and security. He has written articles for Byte and Advanced Systems (SunWorld) magazines, and the Superuser newsletter. Peter is co-author of the best-selling Operating Systems Concepts textbook. Reach Peter at

Carole Fennelly is a partner in Wizard's Keys Corporation, a company specializing in computer security consulting. She has been a Unix systems administrator for over 15 years on various platforms and has particularly focused on sendmail configurations of late. Carole provides security consulting to several financial institutions in the New York City area. Reach Carole at

What did you think of this article?
-Very worth reading
-Worth reading
-Not worth reading
-Too long
-Just right
-Too short
-Too technical
-Just right
-Not technical enough

[Table of Contents]
Subscribe to SunWorld, it's free!
[Next story]
Sun's Site

[(c) Copyright  Web Publishing Inc., and IDG Communication company]

If you have technical problems with this magazine, contact

Last modified: