How to detect a break-in
Why worry? Because your site may be among the 98 out of 100 sites that fail to detect break-ins when they occur. Here is an exhaustive list of methods for break-in detection.
Unfortunately, detecting a break-in is much more difficult than detecting a foul in a Holyfield-Tyson fight. Studies vary in absolute terms but are relatively clear: only two to five percent of sites detect a break-in. This month, we present an exhaustive list of methods you can use to detect a break-in. We also describe ways in which each detection method can be circumvented. It is important to realize what detection limitations there are as well.
Also in Pete's Wicked World this month: YABOB (Yet Another Buffer Overflow Bug) strikes Solaris in the buglist -- and a list of interesting upcoming security conferences.
There are several sources of information regarding break-ins and their detection. Some good ones include Computer Crime and Practical Unix & Internet Security. Unfortunately, there has not been one source of information on how to detect intrusions on Solaris systems. This month's column tries to rectify that situation. The information here will be added to the Solaris Security FAQ after you've had a chance to comment on it and send in your corrections and enhancements. (2,000 words)
The backup is necessary for two reasons. If you are interested in prosecuting whomever is tampering with your systems, then an accurate picture of the system at the time of problem detection is essential. Also, as you explore the system you can affect its state (applying the Heisenberg Principle to computer security), which can make accurate assessment of the system's security difficult.
What follows is a list of the system areas to explore when trying to detect a break-in. Here, we try to avoid duplicating information that's already in the Security FAQ. However, where the FAQ information is directly applicable and needed for completeness, it is included.
6) Break-in detection
While snooping around the system, avoid leaping to conclusions. Try to perform all the main tests and roughly examine all aspects of the system. Only then should you start drawing conclusions and using the fine-toothed comb in likely areas.
Remember that there are many ways for a cracker to compromise your site's security, with the main non-technical means being physical penetration and social engineering. Evaluate these potentials as well as the operating system and applications.
Also remember that the cracker could be anyone. If you find crack tools in a trusted employee's directory, then you must consider the possibility that you've found the cause of your problems. In cases such as this, the next step could be setting traps for the suspects. For instance, replace these tools with instrumented sets that will trace what they are being used for and alert you during their use. This is the type of evidence you will need when confronting a company employee.
If there is evidence of a break-in, you need to determine a proper course of action. Do you "clean" the system and then further secure it? Do you try to detect and trap the cracker? These decision can involve corporate policies that may dictate security incident response. A lot will depend on the role of the machine in question. For instance, if the machine is not a production server, or does not contain sensitive information, then there may be leeway for long-term break-in detection instrumentation.
While exploring your site, remember that crackers can try to gain access as legitimate users and then as root, or directly as root. Protecting user accounts can be effective at keeping crackers off the systems and preventing them from growing their access rights.
6.2) Other resources
The CERT intruder detection checklist is an invaluable guide to system evaluation. It could be used in conjunction with the information here for a thorough system examination.
6.3) Determine current system state
Check current logins and watch for suspicious activity, including long logins and logins at odd times. Other signs that a cracker is using your system include uncharacteristic tool usage (compiler use by an administrative assistant), heavy system use, dial-out modem use, and odd network connections. If the cracker is currently on the system, your activities could be monitored and detection prevented. Depending on what you determine below, the tools you normally use to detect problems may have been compromised and may not be reliable. It would be best to transfer a known good copy of the programs that you use to the system under suspicion and only use those tools. Useful tools include
ps, ifconfig, find, df, and
Other interesting system state information includes what systems have
nfs-mounted your disks. Solaris provides the
command to show current remote mounts. For instance, this output is
problematic because l0pht is a known cracker site:
# dfmounts RESOURCE SERVER PATHNAME CLIENTS - server /cdrom/ alliant.l0pht.com
Also, check the network status of your systems. Perhaps run
snoop to watch network traffic and try to detect
unexpected connections or odd packets.
netstat -a shows
all network daemons running on the machine. Unfortunately,
general-purpose servers provide so many facilities that it is
difficult to separate the wheat from the chaff via
Determine if your machine is in "promiscuous" mode via the
Other system state is recorded in the system's configuration
files. Careful examination of these files, including their access
rights and the last modification date, can be revealing. Files to
/etc/inetd.conf, /.rhosts, /etc/hosts.equiv,
/home/*/.rhosts, /etc/dfs/dfstab, and
Check the e-mail of root and other system directories. Some tools automatically send mail there in case of problems, and those problems can be a sign of an attack or break-in.
Scan the system for problem files. These files can include programs
that are set-uid or set-gid and should not be. They can also include
"trick" file and directory names such as "...", ".. ", and names that
include odd characters. A trustworthy
find is the best
bet for these scans. Bad directory permissions, such as group or
/etc can give away the shop, so all
system directory permissions should be checked.
at queues to make sure
that all your work is not being undone by automated scripts that
re-open security holes.
6.4) Examine system logs
Logs can be useful not only for detecting penetration but also for determining the scope and timeframe of the break-in. However, logs can be (and are) modified by crackers to hide their tracks, so a lack of activity in the logs is not an indication that the system is secure. Useful Solaris logs include:
/var/adm/messages-- general system activity including repeated login failures
sus that have occurred since the operating system was installed
last root-- all "root" logins that occurred since the operating system was installed.
lastis also useful for evaluating the logins of suspect users.
syslogis redirecting logs to another host, be sure to check that host's logs as well.
The logs can indicate events that occur normally but can also be a sign of problems. For instance, a system reboot can be a bad sign. Errors messages from daemons like sendmail and finger can be caused by break-in attempts. Crackers can change the system clock to hide their trails.
6.5) Patch failure
Are the operating system patches up-to-date? If not, determine which patches should have been applied and how the associated security hole could have been used. For instance, if
sendmail is not up-to-date,
then check the cracker sites for exploit scripts. By examining the
scripts, you may be able to find tell-tales that would be left behind if
that script had been used against your site.
6.6) Binary modification
Once crackers penetrate a system, they can leave tools to help gather information from your site, or tools to allow them to regain access to the machine. Searching for these tools can be time consuming, but generally worth the effort. Several specific tools are described below, but a useful approach is to execute
ls -ltr in important
directories. The last files listed are the most recently
changed. Either they were changed by administrators at your site (via
installations or patches) or they were changed by a cracker. Typical
system binaries that crackers attempt to replace include
ifconfig, find, login, df, and
6.7) Enlist the users
Users can be either friend or foe. They can ignore your password selection recommendations, give account access to friends, login remotely with clear-text connections, and generally make the life of a security administrator a living hell. Or your users can help secure your systems. User education is one of the best tools in your arsenal. Teach users proper security methodology, and why it is important to secure the machines. If they do not feel the need for security, they will not help provide it. Users are your first and last defense against social engineering attacks: they can either grant a pushy cracker access, or they can alert you to the attack. Users can help keep their accounts secure by understanding permissions and paying attention to "last login" date information. Really helpful users can examine their
last output to verify that no one
else is using their accounts.
6.8) What about the applications?
Many security holes are introduced by application programs, such as Web and database servers. These applications need to be checked for configuration problems, interesting event logs, and access rights. Also, patches should be installed -- the lack of up-to-date patches can give you a place to start in your cracker hunt.
6.9) Use the Web, Luke
Because crackers use the Web to communicate vulnerability information and to brag about system conquests, it can also be used by security administrators and auditors to check for this information. For instance, search on the known cracker sites, or the Internet as a whole, for your site and machine names and the words "crack break secure" and "hack." The results can be alarming.
Useful cracker sites include:
Bug of the Month Club
In the past month Sun has released a variety of security patches. Some
fix the usual buffer overflow problems, but some are more worrisome.
For instance, there's now a patch to Adminsuite 2.1 and 2.2 to stop it
from allowing any NIS+ user from becoming root. These are relatively
new facilities, and both were written with security in mind. But still
there are security holes. Likewise, there's a patch for the
chkey program, which is used to change NIS+ secure
keys. Unfortunately it can also be used to gain root access by
overflowing its argument buffer. YABOB (Yet Another Buffer Overflow
Bug) allows users of
at to become root. Finally, there's
YABOB in the
eeprom program that allows users to become
"bin," which could lead to root access. Patches are available from
Sun for all of these bugs, as well as for an
for most flavors of Solaris and SunOS.
Fortunately, most companies provide travel money for security professionals to attend conferences. Unfortunately, there's usually a limit on the number of trips per year. This is especially a problem this fall, when there are several excellent conferences that are worth attending. Start buttering up the boss to let you go to:
Peter Galvin is chief technologist for Corporate Technologies, Inc., a systems integrator and VAR. He is also adjunct system planner for the Computer Science Department at Brown University, a member of the board of directors of the Sun User Group, and has been program chair for the past four SUG/SunWorld conferences. As a consultant and trainer, he has given talks and tutorials world-wide on the topics of system administration and security. He has written articles for Byte and Advanced Systems (SunWorld) magazines, and the Superuser newsletter. Peter is coauthor of the best-selling Operating Systems Concepts textbook. email@example.com
If you have technical problems with this magazine, contact firstname.lastname@example.org