Me and my SHADOW
Using the new SANS intrusion-detection software with Solaris
The System Administration Networking and Security (SANS) Institute made a big splash in July by announcing a freely-available network-monitoring tool: the SANS Heuristic Analysis system for Defensive Online Warfare (SHADOW). What can SHADOW do for you? What should you let it do? This month James Triplett joins Peter in a discussion of the pros and cons of network monitoring. James provides a hands-on review of the tool.
Peter also has news of a commercial release of SSH and some interesting feedback on previous columns. (3,100 words)
The layers you can employ include:
The newest addition to this list is Intrusion Detection Systems (IDS). All the recent improvements in system performance and disk capacity have done more than just improve Quake performance. They also enable thorough network data collection and analysis. There are several commercial and semi-commercial IDS products available, including ISS RealSecure, Axent Intruder Alert, and Network Flight Recorder (see Resources for links to these products).
And now there's a new player in town. The SANS Institute is distributing a freely-available contributor-based facility called SHADOW. It's part of the Cooperative Intrusion Detection Evaluation and Response (CIDER) project, which is a joint effort by the Dahlgren Naval Surface Warfare Center, Network Flight Recorder (NFR), the SANS community and other interested parties to locate, document, and improve freely-available security software. CIDER actually has two toolsets for IDS, one based on NFR and the other based on the free tcpdump package. Because tcpdump is free and easy to install and use, we'll focus on it this month.
According to the SANS announcement, "A SHADOW system can be built using freely-available software and existing hardware or hardware that can be purchased for less than $10,000. All CIDER components come with source code and tutorials on what they do and how to set them up and what the results mean. Because of this, an organization will find this approach complements and enhances the impact of commercially-available intrusion-detection systems."
The basic idea of an IDS is that there is a tale to be told by the packets traveling on your networks. Within the standard and expected data flows, there could be internal or external hacker and cracker packets. For instance, it could contain some of the standard attacks like the SYN flood -- or it could be more innocuous, like a telnet from or to a forbidden place. Without an IDS, it would be difficult or impossible to detect these security issues and to take the next step to determine their origin, watch for patterns, or set up alarms when similar events are detected.
The SHADOW package is designed to have systems at key locations, say just outside or just inside a firewall, collecting data. The data is then moved to the analysis system for dissection.
I enlisted the help of security expert James Triplett to implement SHADOW and comment on its utility.
SHADOW blow by blow --- through the eyes of James Triplett
Peter and I were chatting one day. He said, "Have you heard about the latest buzz in the NetSec world: SHADOW?"
And I answered, "Sure, the newest thing is to go from monitoring important gateways and servers to actively examining all the data on the network, looking for unfriendly traffic. The SHADOW system, from the Navy's Dahlgren laboratory, has got some good ideas, but isn't the software mostly set up for BSD systems? Of course there's no reason it couldn't be done on a Solaris system -- since it's based on generic Unix tools -- and the supposedly hot new networking code in Solaris 2.6 might make it reasonably efficient."
"Right," says Peter, knowing when to keep quiet and let me talk myself into a hole.
"It probably wouldn't take but a couple minutes to put one of these things on one of my spare Suns, and try it out!"
"Great idea!" he exclaimed, and went on to convince me I should not only do the job, but write about it in time for his SunWorld deadline.
The basic SHADOW setup is two systems: a data-collection machine known as a sensor, and an analysis station. The sensor sits on the network of interest and captures all the traffic. The raw data files are moved by a secure channel to the analysis station for inspection.
The SHADOW work is implemented with some tools (tcpdump and libpcap) from the Lawrence Berkeley Lab (LBL) -- long one of the mainstays of traditional Unix wisdom and software. Their ftp site (ftp.ee.lbl.gov) is a shrine of the Internet: one of the earliest free-software sites. No fancy www.blahblah banner-decorated Web site here: just code, raw code, with only the slightest hint of documentation.
Tcpdump was originally a port of etherfind, which was found on SunOS but didn't make the transition to Solaris 2. It's SystemV-based I/O code. For licensing reasons, the code was completely rewritten at Berkeley and is now free of any legal restrictions. Trouble is, these guys are pretty attached to the old standby of Berkeley Unix -- which extends to operating systems like SunOS, FreeBSD and BSDI Unix. When Solaris first arrived, the old masters were quite rude about it, muttering "feature bloat," "software by committee," and other tribal epithets. So they held out for a long time against porting tcpdump to the new architecture.
* For those who enjoy working
with source-level tools, the
tcpdump tools can be setup
pretty easily. Using a
Solaris 2.6 system (with the
developer packages), and a
copy of GCC (2.7.2 from
opcom.sun.ca), it was
possible to compile both
libpcap and tcpdump. They
work fine, collecting data
with similar performance to
snoop and storing it a little
After saving a unit of network traffic (the SHADOW authors suggest an hour's worth), you can compress and transfer the file to the analysis machine. On that machine, you can analyze the raw data with snoop. Here we get to the meat of the issue: defining filters that can be used to find evidence of malicious activity.
In many cases, packet data analysis with snoop is more convenient than tcpdump. Rather than having to calculate all those nasty packet offsets, there is simple syntax to access the most common tests: packet types and destination ports. So, for example, if you are concerned about the use of broadcasts, in tcpdump you search for an IP packet with the 19th byte set to all ones, as in:
"ip and ip = 0xff " or use snoop with the following argument: -broadcast
ip[12:4] = ip[16:4]This expression looks for a packet with the same source address (the four bytes starting at offset 12) and destination address (offset 16). This is a common denial-of-service attack because it can crash systems that haven't been patched against this particular trick.
As you learn more about the kind of traffic to look out for, you can refine these filters until they can become quite sophisticated. Though I can't find any documentation to confirm it, it seems to me that any of the packet filters written for tcpdump can be used with snoop.
The SHADOW distribution contains example filters for a variety of attack types, including:
(The actual list goes on for pages.)
Click for full-size image
You'll note that a lot of the filters check for the SYN bit (tcp&2); most of the interesting characteristics of a TCP connection are set in this first packet. This allows us to ignore all the subsequent data packets. The whole point of these filters is data reduction: it's necessary to cut down the data by a factor of about 1,000 to 1 in order to make it useful.
SHADOW adds a layer of Perl for report formatting and volume reduction and it puts the resulting reports on a Web server. This software has the reassuring feel of tools that are actually used, not some marketeer's grand scheme that tries to solve every problem. In fact it is already in use on thousands of machines.
My advice is that you get this data collection system in place and check the Web site regularly to get a good feel for what's going on in the woolly world on the outside of your firewall. It could also be used on the inside of a firewall: Instead of watching all the arrows being shot at your castle, you can just watch the ones that make it through your defenses. SHADOW can also be used to watch for internal security problems, which by all accounts are more common than external ones anyway.
Don't have a busy enough network to get meaningful results from your data? Take a look at the Internet Traffic Archive, where some large chunks of data taken at various places in the Internet are stored. These data files are in tcpdump format, so it will eventually become necessary to go out and get the code from Berkeley.
For full details of SHADOW, check out CIDER, or get the SHADOW documentation by emailing firstname.lastname@example.org with subject "SHADOW description."
SSH communications has released SSH version 2, which uses the SSH2 protocol to encrypt all ftp-transmitted data as well as the usual login information. The new version purports to be faster and more secure than the previous one, and adds hooks for integration with the public key infrastructure. Fortunately, SSH V2 is backward-compatible with good ol' SSH. SSH2 has been submitted to the Internet Engineering Task Force (IETF) as part of the protocol-standardization process. The result should be SECSH, an open, secure communication standard. For more information, visit the SSH Web site. Datafellows, a company that sells a commercial version of SSH, is expected to release an SSH2-based version of their F-Secure products shortly.
There was quite a lot of mail on July's report from the SANS conference. Here are the highlights:
Dave Sill points out that "Dan Bernstein's qmail is another secure, high performance sendmail replacement. It's been available for about a year and a half. See www.qmail.org."
And Dan Grady was brave enough to point out a serious factual error. Consider this a retraction:
Hi, Peter. Thanks for writing about this year's SANS '98. One of the reasons I was able to attend this year was because of your excellent writeup on last year's show. Maybe someone else will get the benefit next year. By the way, as one of those "few brave souls" who "made fools of themselves riding around a circle on motorized garbage cans," please note: They were beer kegs not garbage cans! I point this out only because I know that once you realize what we were riding on, you'll agree that the endeavor was really quite respectable. Yes?
Um, Dan, uh, sure...that does make all the difference!
Finally, readers continue to contribute ideas and information about using chroot to secure Web and mail servers.
About the author
Peter Galvin is chief technologist for Corporate Technologies Inc., a systems integrator and value-added reseller (VAR). He is also adjunct system planner for the Computer Science Department at Brown University, and has been program chair for the past four SUG/SunWorld conferences. As a consultant and trainer, he has given talks and tutorials worldwide on the topics of system administration and security. He has written articles for Byte and Advanced Systems (SunWorld) magazines, and the newsletter Superuser. Peter is co-author of the best-selling Operating Systems Concepts textbook. Reach Peter at email@example.com.
If you have technical problems with this magazine, contact firstname.lastname@example.org
I read the note from Gene Spafford in the August column with interest; I've been groping towards a loopback-mount solution for running Apache 1.3.1 in a chroot'ed environment for a cgi server. It works, but there are some gotchas:
It's doable (and, I hope, worthwhile -- people keep clamoring for cgi and the more I learn about it, the more afraid I get) but still kind of a pain.
- Sometimes the loopback mounts don't happen. I have a lot of these mounts listed in /etc/vfstab, and every once and a while, after a reboot, one or more just doesn't take hold. This is on a 2.6 system with the latest recommended patches. I'll probably add some logic to the Apache startup script to check the mounts, just in case.
- Sometimes loopback mounts do bad things -- loopback-mounting /net on, say, /r was a pretty reliable way to panic the system up through at least 2.5.1. Not something I need to do on the Apache machine, but I suspect you can get into thin ice with loopback mounts pretty easily.
- You have to become really used to running truss. Just what to put in the chroot'ed environment is a little mysterious; the in.ftpd(1m) man page is the obvious place to start, but it takes a number of iterations with truss before chasing down things. It wasn't clear at first that Apache needed libresolv.so for instance.
And this from Simon Burr:
I've just read your latest SunWorld column and I've got some comments about the use of loopback mounts within chroot environments.
I'm currently in the process of building a new Web service where customers are allowed to put any CGI scripts they wish into their space and have it work. Naturally, the security side of this worried me a little, so I spent some time playing with ways around this. My first thought was to use chroot to hid the real system, along with other customer data, from customers. Then there was the problem of providing perl et al to customers without having to maintain thousands of individual copies. It would be expensive in disk and maintainence to do this. I too found that loopback mounts were the best way to provide directory structures within a chroot environment. Since each machine in our service will support thousands of users, to have every loopback mount pre-mounted seemed wrong -- probably due to scalablity concerns (although I did test a box with 4000 loopback mounts and it had no real problems). I found that you could use the automounter to do the mounting instead.
While testing I also found that the read-only option on loopback file systems in Solaris 2.5.1 did not work -- it took the options of the actual file system it was mounted from. This has changed with Solaris 2.6 and loopback file systems mounted read-only really are read only, even if the underlying file system is mounted read-write.
Finally, it should be noted that you have to be extremely careful with the files you place within a chroot environment. This is especially true if you're using loopback mounts, as a naive approach would be to do something like:
% mount -r -F lofs /usr /home/www/customer/usr
The problem with this is that any security within the /usr tree would give an attacker root access. Once you have root access it is fairly trival to break out of a chroot environment. I've written a C program to do exactly this; it would be quite easy to write the same thing in Perl.