Click on our Sponsors to help Support SunWorld

Me and my SHADOW

Using the new SANS intrusion-detection software with Solaris

September 1998

Abstract

The System Administration Networking and Security (SANS) Institute made a big splash in July by announcing a freely-available network-monitoring tool: the SANS Heuristic Analysis system for Defensive Online Warfare (SHADOW). What can SHADOW do for you? What should you let it do? This month James Triplett joins Peter in a discussion of the pros and cons of network monitoring. James provides a hands-on review of the tool.
Peter also has news of a commercial release of SSH and some interesting feedback on previous columns. (3,100 words)

Mail this
article to
a friend

he SANS Heuristic Analysis system for Defensive Online Warfare, called SHADOW by most people, is one of that breed of tools known as intrusion detection systems (IDSs). Previously, I've discussed other tools that can be used to implement a secure facility. These tools form the layers of your security onion. The complete onion should match your security needs and should provide your site with the appropriate level of security.

The layers you can employ include:

Security policy -- the roadmap showing what security you need and how you're going to get there
Firewall or filtering router -- restricting the flow between your network and the rest of the world
Encryption -- protecting data as it flows between security domains
Authentication -- identifying who is trying to access a resource
Authorization -- determining whether access requests should be granted
Individual system security -- protecting each machine from security holes
System/Firewall monitoring -- monitoring machines for security incidents
Intrusion detection system -- monitoring networks and systems for security incidents
Obscurity -- hiding information to make it more difficult for a hacker to attack
Legal threat -- using fear of legal action to deter hackers
User education -- teaching your users to be a security weapon rather than a security hole

The newest addition to this list is Intrusion Detection Systems (IDS). All the recent improvements in system performance and disk capacity have done more than just improve Quake performance. They also enable thorough network data collection and analysis. There are several commercial and semi-commercial IDS products available, including ISS RealSecure, Axent Intruder Alert, and Network Flight Recorder (see Resources for links to these products).

And now there's a new player in town. The SANS Institute is distributing a freely-available contributor-based facility called SHADOW. It's part of the Cooperative Intrusion Detection Evaluation and Response (CIDER) project, which is a joint effort by the Dahlgren Naval Surface Warfare Center, Network Flight Recorder (NFR), the SANS community and other interested parties to locate, document, and improve freely-available security software. CIDER actually has two toolsets for IDS, one based on NFR and the other based on the free tcpdump package. Because tcpdump is free and easy to install and use, we'll focus on it this month.

According to the SANS announcement, "A SHADOW system can be built using freely-available software and existing hardware or hardware that can be purchased for less than $10,000. All CIDER components come with source code and tutorials on what they do and how to set them up and what the results mean. Because of this, an organization will find this approach complements and enhances the impact of commercially-available intrusion-detection systems."

The basic idea of an IDS is that there is a tale to be told by the packets traveling on your networks. Within the standard and expected data flows, there could be internal or external hacker and cracker packets. For instance, it could contain some of the standard attacks like the SYN flood -- or it could be more innocuous, like a telnet from or to a forbidden place. Without an IDS, it would be difficult or impossible to detect these security issues and to take the next step to determine their origin, watch for patterns, or set up alarms when similar events are detected.

The SHADOW package is designed to have systems at key locations, say just outside or just inside a firewall, collecting data. The data is then moved to the analysis system for dissection.

I enlisted the help of security expert James Triplett to implement SHADOW and comment on its utility.

SHADOW blow by blow --- through the eyes of James Triplett
Peter and I were chatting one day. He said, "Have you heard about the latest buzz in the NetSec world: SHADOW?"

And I answered, "Sure, the newest thing is to go from monitoring important gateways and servers to actively examining all the data on the network, looking for unfriendly traffic. The SHADOW system, from the Navy's Dahlgren laboratory, has got some good ideas, but isn't the software mostly set up for BSD systems? Of course there's no reason it couldn't be done on a Solaris system -- since it's based on generic Unix tools -- and the supposedly hot new networking code in Solaris 2.6 might make it reasonably efficient."

"Right," says Peter, knowing when to keep quiet and let me talk myself into a hole.

"It probably wouldn't take but a couple minutes to put one of these things on one of my spare Suns, and try it out!"

"Great idea!" he exclaimed, and went on to convince me I should not only do the job, but write about it in time for his SunWorld deadline.

The basic SHADOW setup is two systems: a data-collection machine known as a sensor, and an analysis station. The sensor sits on the network of interest and captures all the traffic. The raw data files are moved by a secure channel to the analysis station for inspection.

The SHADOW work is implemented with some tools (tcpdump and libpcap) from the Lawrence Berkeley Lab (LBL) -- long one of the mainstays of traditional Unix wisdom and software. Their ftp site (ftp.ee.lbl.gov) is a shrine of the Internet: one of the earliest free-software sites. No fancy www.blahblah banner-decorated Web site here: just code, raw code, with only the slightest hint of documentation.

Tcpdump was originally a port of etherfind, which was found on SunOS but didn't make the transition to Solaris 2. It's SystemV-based I/O code. For licensing reasons, the code was completely rewritten at Berkeley and is now free of any legal restrictions. Trouble is, these guys are pretty attached to the old standby of Berkeley Unix -- which extends to operating systems like SunOS, FreeBSD and BSDI Unix. When Solaris first arrived, the old masters were quite rude about it, muttering "feature bloat," "software by committee," and other tribal epithets. So they held out for a long time against porting tcpdump to the new architecture.

^* For those who enjoy working
with source-level tools, the
tcpdump tools can be setup
pretty easily. Using a
Solaris 2.6 system (with the
developer packages), and a
copy of GCC (2.7.2 from
opcom.sun.ca), it was
possible to compile both
libpcap and tcpdump. They
work fine, collecting data
with similar performance to
snoop and storing it a little
more compactly.

Their tools are distributed in source, and to use them you have to have a complete set of development tools at your disposal. Up until recently, you'd have had to do the port to SysV/Solaris as well. So the bad news is you can't use the LBL code on a Solaris system without a bunch of work, including procuring a C compiler, and installing the developer support packages off the Solaris distribution^*. Of course there is good news, or this project would have been brief indeed: Solaris has functional equivalents to tcpdump and the Berkeley Packet Filter. The Solaris team provides a kernel packet filter implementation and a program to exercise it called snoop. You can use the snoop program just like the Navy guys used tcpdump to perform both the data collection and the analysis phase. Using snoop will save you the effort of downloading and compiling tcpdump. Although these two programs come from separate sources, their functionality is nearly identical. No doubt the respective authors shared a pizza or two on occasion.

After saving a unit of network traffic (the SHADOW authors suggest an hour's worth), you can compress and transfer the file to the analysis machine. On that machine, you can analyze the raw data with snoop. Here we get to the meat of the issue: defining filters that can be used to find evidence of malicious activity.

In many cases, packet data analysis with snoop is more convenient than tcpdump. Rather than having to calculate all those nasty packet offsets, there is simple syntax to access the most common tests: packet types and destination ports. So, for example, if you are concerned about the use of broadcasts, in tcpdump you search for an IP packet with the 19th byte set to all ones, as in:

        "ip and ip[19] = 0xff "
or use snoop with the following argument:
        -broadcast

But wait -- there's still an opportunity to use all the dirty details of packet headers they taught you in Networking 101. Snoop has a full bit and byte syntax. Many of the trickiest network security exploits involve building unusual TCP/IP packets. For these, you will need to explore the gory innards of TCP/IP. For instance, you can write an expression like:

        ip[12:4] = ip[16:4]

This expression looks for a packet with the same source address (the four bytes starting at offset 12) and destination address (offset 16). This is a common denial-of-service attack because it can crash systems that haven't been patched against this particular trick.

As you learn more about the kind of traffic to look out for, you can refine these filters until they can become quite sophisticated. Though I can't find any documentation to confirm it, it seems to me that any of the packet filters written for tcpdump can be used with snoop.

The SHADOW distribution contains example filters for a variety of attack types, including:

IMAP service exploit tcp and (tcp[13]&2!=0) and (dst port 143)
Land Attack ip[12:4] = ip[16:4]
ICMP explorations (other than pings) proto ICMP -and "icmp[0]!=8 and icmp[0]!=0"
Directed broadcasts
SNMP explorations
Portmapper and NFS exploits (snoop has handy options to pick these directly)
NetBIOS probes
Xwindows usage (very easy to abuse) tcp and (port 6000 or port 6001 or port 6002)
IRC traffic (often used by attackers once they gain control of a machine) tcp and port 6667

(The actual list goes on for pages.)

Click for full-size image

You'll note that a lot of the filters check for the SYN bit (tcp[13]&2); most of the interesting characteristics of a TCP connection are set in this first packet. This allows us to ignore all the subsequent data packets. The whole point of these filters is data reduction: it's necessary to cut down the data by a factor of about 1,000 to 1 in order to make it useful.

SHADOW adds a layer of Perl for report formatting and volume reduction and it puts the resulting reports on a Web server. This software has the reassuring feel of tools that are actually used, not some marketeer's grand scheme that tries to solve every problem. In fact it is already in use on thousands of machines.

My advice is that you get this data collection system in place and check the Web site regularly to get a good feel for what's going on in the woolly world on the outside of your firewall. It could also be used on the inside of a firewall: Instead of watching all the arrows being shot at your castle, you can just watch the ones that make it through your defenses. SHADOW can also be used to watch for internal security problems, which by all accounts are more common than external ones anyway.

Don't have a busy enough network to get meaningful results from your data? Take a look at the Internet Traffic Archive, where some large chunks of data taken at various places in the Internet are stored. These data files are in tcpdump format, so it will eventually become necessary to go out and get the code from Berkeley.

For full details of SHADOW, check out CIDER, or get the SHADOW documentation by emailing info@sans.org with subject "SHADOW description."

Reader comments: using chroot to secure Web and mail servers

This from Jim Davis:

I read the note from Gene Spafford in the August column with interest; I've been groping towards a loopback-mount solution for running Apache 1.3.1 in a chroot'ed environment for a cgi server. It works, but there are some gotchas:

Sometimes the loopback mounts don't happen. I have a lot of these mounts listed in /etc/vfstab, and every once and a while, after a reboot, one or more just doesn't take hold. This is on a 2.6 system with the latest recommended patches. I'll probably add some logic to the Apache startup script to check the mounts, just in case.

Sometimes loopback mounts do bad things -- loopback-mounting /net on, say, /r was a pretty reliable way to panic the system up through at least 2.5.1. Not something I need to do on the Apache machine, but I suspect you can get into thin ice with loopback mounts pretty easily.

You have to become really used to running truss. Just what to put in the chroot'ed environment is a little mysterious; the in.ftpd(1m) man page is the obvious place to start, but it takes a number of iterations with truss before chasing down things. It wasn't clear at first that Apache needed libresolv.so for instance.
It's doable (and, I hope, worthwhile -- people keep clamoring for cgi and the more I learn about it, the more afraid I get) but still kind of a pain.

And this from Simon Burr:

I've just read your latest SunWorld column and I've got some comments about the use of loopback mounts within chroot environments.
I'm currently in the process of building a new Web service where customers are allowed to put any CGI scripts they wish into their space and have it work. Naturally, the security side of this worried me a little, so I spent some time playing with ways around this. My first thought was to use chroot to hid the real system, along with other customer data, from customers. Then there was the problem of providing perl et al to customers without having to maintain thousands of individual copies. It would be expensive in disk and maintainence to do this. I too found that loopback mounts were the best way to provide directory structures within a chroot environment. Since each machine in our service will support thousands of users, to have every loopback mount pre-mounted seemed wrong -- probably due to scalablity concerns (although I did test a box with 4000 loopback mounts and it had no real problems). I found that you could use the automounter to do the mounting instead.
While testing I also found that the read-only option on loopback file systems in Solaris 2.5.1 did not work -- it took the options of the actual file system it was mounted from. This has changed with Solaris 2.6 and loopback file systems mounted read-only really are read only, even if the underlying file system is mounted read-write.
Finally, it should be noted that you have to be extremely careful with the files you place within a chroot environment. This is especially true if you're using loopback mounts, as a naive approach would be to do something like:

% mount -r -F lofs /usr /home/www/customer/usr

The problem with this is that any security within the /usr tree would give an attacker root access. Once you have root access it is fairly trival to break out of a chroot environment. I've written a C program to do exactly this; it would be quite easy to write the same thing in Perl.

Comments:
Name:
Email:
Company Name: