Click on our Sponsors to help Support SunWorld
Who ya gonna call?
Do you have an approved incident response procedure in place? Getting your IRP set up before something happens could save you a lot of headaches -- not to mention your job
You're sitting at your desk, browsing through the system and notice something isn't right. Further investigation shows that there may be an intruder in your
system. This wakes you up faster than a can of Jolt Cola, and you're all primed
to take action. The question is: what kind of action? A system administrator's
perception of an appropriate incident response may be very different from
corporate management's. An approved incident response procedure can save you from major troubles -- and embarrassment. Here's how to do it right. (3,600 words)
et's consider how you would react to the following scenarios:
- A user calls complaining about sexually explicit images that have appeared on her computer.
- An administrator at another site contacts you regarding the
actions of one of your users. This could involve anything from nasty
e-mail to attacks on the site. The administrator may be:
- Friendly and knowledgeable -- he asks for your log files to help investigate the use.
- Downright rude -- he threatens a lawsuit unless you cooperate and send
him your log files.
- You are experiencing repeated attacks from the same address. You've contacted the registered technical admin who:
- Never gets back to you
- Pleads ignorance and doesn't do anything
- Claims to have investigated and can't trace it
- Is rude and obnoxious
- An authoritative agency such as the FBI or CERT contacts you
directly and informs you (sounding very official) that it is
conducting an investigation of one of your users and expect full
- You notice that an executive in the company is browsing child
porn sites, and you suspect he may be saving some images on company
- A new employee brags to you about a back door she left in her
- You think your system may have been compromised.
In each of these cases, your first response should be to inform
management. Unfortunately, this is usually the last thing a system
administrator actually does. The administrator will often make an
assumption about the appropriate response. The problem is that there
are political and legal issues to consider that the administrator is
not qualified to address. In the precommercial days of the
Internet, system administrators in research organizations freely
exchanged information without informing management. Most, if not
all, of the decision making was left up to the administrator.
Today, corporations have strict policies regarding the release of
proprietary information. In a commercial environment, even releasing
information to CERT could get an administrator in a lot of trouble
System administrators often complain that management doesn't
understand technical issues. If management can't understand the
technical issues, perhaps they need to be explained in clearer terms.
The movie Armageddon has a scene in which the scientist is attempting
to explain to the president the meteor's trajectory, mass, and velocity. The
president is confused until the explanation is reworded to "a rock the size of
Texas is going to hit Earth and destroy everything." Management just
needs to know the issues so they can make a decision. The
administrator's job is to understand the technology and produce the
results management asks for. Don't give management data
paralysis -- when they start giving you that deer-in-the-headlights
look, you know you've gone too far.
The system administrator can make the process easier by interpreting
the technical issues into business terms and developing an incident
response procedure that has management's approval. This way, the
administrator knows what actions are preapproved. For example, you
may be authorized to call the technical contact for an
address that is conducting repeated probes of your site.
Incident response procedures are nothing new in the industry. Many
of the guidelines for developing an IRP were established during the
precommercial days of the Internet and focus on how to proceed if
your site has been compromised. While you should certainly have such
a procedure in place, hopefully you'll be prepared to respond to
an incident before it gets that far. Many sites are installing
intrusion detection systems -- the technical equivalent of a burglar
alarm. But what good is such a system if you're unprepared to react
to the alarms?
The sidebar at the end of this column
includes an outline I use for preparing an incident response procedure. Often,
the issues that must be considered are the same across sites, but
the details may vary, depending on the site's security policy and
configuration. Hopefully, this outline will help you develop your
Examples of response to above scenarios
While the appropriate response to a situation depends on your
site's policies and procedures, it's very important to keep
accurate chronological logs of events with details such as file
names and the names and phone numbers of all people involved in any
security incident. Check with your legal department for guidelines
on evidence gathering. This is how I might respond to the
hypothetical situations I mentioned earlier:
- A user calls complaining about sexually explicit images
that have appeared on her computer.
If there is an incident response team, contact them and let them
handle it. Otherwise, search the log files for her IP address and
gather technical evidence. Usually, I would
grep for her IP
address in the log files and save the output in another file to try
to establish a pattern. Save the original versions of all log files
as well as all working copies on protected backup media. Prepare a
file for management that summarizes the conclusion and attach
signed printouts of the log files as proof. For example:
Investigation has shown that the user's computer accessed
pornographic sites on the Internet every night between the hours of
11:00 p.m. and 1:00 a.m. At no other time were these sites accessed
from this computer. It is recommended that building access logs be
checked to see who may have had physical access.
- An admin at another site contacts you regarding the actions
of one of your users. This could involve anything from nasty e-mail to
attacks on his site.
Regardless of the administrator's attitude, you should politely
inform him that you are not authorized to release any information
without management approval, but you will pass along all the
information. Ask for technical details such as his log data or a
copy of the e-mail. Make sure you get his name, title, phone number,
company, and e-mail address. Inform your management or appropriate
- You're experiencing repeated attacks from the same address.
You've contacted the registered technical admin and you don't seem to
be getting anywhere.
Inform your management and appropriate internal security
organizations. Search all your log files for the offending addresses
and save to a file to help with the analysis. Prepare a summary of
incident times to assist in investigation. The real evidence will be
the original log file, so you must save an untouched copy on backup
media that is write-protected. If you have any intention of legal
recourse, print out a copy of the original log, sign and date it, and
also have a witness sign and date it. Make sure you have logs or
copies of any e-mail to show all attempted correspondence with the
technical contact. Again, check with your legal department for
appropriate evidence gathering procedure.
- An authoritative agency such as the FBI or CERT
contacts you directly and informs you (sounding very official)
that it is conducting an investigation of one of your users
and expects your full cooperation.
Even the FBI has to have a search warrant to get company data and,
unless you're an officer of the company, you're not the
appropriate person to be served a warrant. Use the same guidelines
as you would for an obnoxious administrator demanding
information: be polite and gather information to be forwarded to
management. The point is, you can't be certain who you're
really speaking to and only management may authorize the release of
company information. If it really is an FBI investigation, it must
be escalated to senior management.
- You notice that an executive in the company is browsing
child-porn sites, and you suspect he may be saving images
on company systems.
This is difficult because it's such a sensitive and emotional
issue. However, if it isn't your company policy to monitor Internet
usage, you shouldn't be. If you happened to come across it, you may
have grounds to make a hostile work environment complaint to
Human Resources. It isn't your place to audit the manager's
system without prior approval, no matter what you suspect. Go to a
manager you know and trust and seek his or her
advice. Don't discuss it with anyone else in the company. For all
you know, someone else may be using the executive's system, and he may
not even be aware of it. Reputations are easily damaged in this kind of
situation, and the damage is almost impossible to fix.
It's also quite easy to lose your job in this kind of situation.
- A new employee brags to you about a back door she left
in her last company.
Because there's really no way to confirm whether
or not it's true, this scenario is difficult to act on.
I wouldn't want such a person working for me,
because she could do the same thing to me, but that's a management
decision. Tell your management and let them decide how to
- You think your system may have been compromised.
Obviously, there is quite a lot involved in responding to this, and
it requires a special response procedure. It's very easy to panic in
this situation and very important that you do not. Strictly follow
your response procedures and document everything. Things can happen
very quickly, and it's easy to get confused later about exactly what
happened. If it's at all serious you will inevitably be asked to
produce a report. An accurate log is invaluable in such a situation.
If there's a possibility that this may be needed in a legal case,
make sure the original log is dated (with timestamps) and signed or
initialed. Don't worry about the log being presentable;
it just needs to be consistent and reasonably understandable.
I have logs that are mostly scribbled, but they're at least semilegible.
You may disagree with my suggested responses to these security
scenarios. If so, that's all the more reason to develop an
incident response procedure that has the approval of your
management. If getting started is your problem, use the attached IRP
and edit it for your site. Come up with some of your own scenarios
and discuss with management how they should be handled -- then put
it in writing. Once you have a response procedure, it should be
tested by the audit department as part of the regular security
assessment. Whatever you do, don't wait for a major incident to
occur before you write one.
Just as I was about to write about installing Sendmail 8.9.2 on a
firewall, an announcement was made about the release of
Sendmail 8.9.3. Oh, well -- that saves me from explaining how to
install the patch. Next month, I'll describe the installation steps for
a firewall and internal mail gateway and include sample
Disclaimer: The information and software
in this article are provided as-is and should be used with caution. Each environment is unique and the reader is cautioned to investigate with his or her company as to the feasibility of using the information and software in the article. No warranties, implied or actual, are granted for any use of the information and software in this article and neither author nor publisher is responsible for any damages, either consequential or incidental, with respect to use of the information and software contained herein.
Click on our Sponsors to help Support SunWorld
Other SunWorld resources
About the author
Carole Fennelly is a partner in Wizard's Keys Corporation, a company
specializing in computer security consulting. She has been a Unix
system administrator for more than 15 years on various platforms and has
particularly focused on Sendmail configurations of late. Carole
provides security consultation to several financial institutions in
the New York City area.
If you have technical problems with this magazine, contact
Sample Computer Security Response Procedure
The purpose of this document is to provide a procedure to recognize
a security incident and take appropriate action. This document also
defines the chain of command for reporting security incidents and
details the responsibilities of each level. A computer security
incident is defined as use of company computing facilities in an
unauthorized manner. See your Company Security Policy for further
Chain of command
The company incident response team must be notified of all security
incidents. The incident response team is responsible for
coordinating the response and is preauthorized to take specified
actions. The incident response team will include two or more of the
representatives of the following organizations to ensure redundancy:
System Administration, Security Administration, Management, Audit,
and Legal. The technical staff should focus on the technical issues
while the other members direct policy issues. All members of the IRT
must be reachable at all hours or designate a backup. Attached will
be the contact information for the IRT members, including pager and
home phone numbers. All IRT members will have an emergency contact
list for senior management in the company, including home phone
numbers. This list is confidential, but must be accessible at all
Categorizing a security incident
The first priority for a security incident is to determine the type and status of the incident.
- Is the incident a penetration attempt or a violation of the security policy?
- Is the incident presently active?
- What is the source of the incident?
- Has the system been compromised?
- Which organizations are involved?
The most obvious indicator of a penetration attempt is your
Intrusion Detection System (IDS) informing you that you have one.
You should be familiar enough with your IDS to recognize whether or
not the alert is important. Remember -- an IDS can only tell you
about the traffic it monitors. Internal users are still considered
the biggest security threat. You also must have a mechanism to
receive alerts by pager since you will not be in front of the
monitor 24 hours a day. The best way to be able to recognize a
security incident is to be observant and familiar with normal
conditions. System administrators should be alert to:
While these signs may not conclusively prove a security incident,
they warrant further investigation. The incident response team
should be informed even if an incident is not confirmed, unless it
can be dismissed outright. While attempting to confirm whether an
incident has occurred, the system administrator should capture the
output of commands run and log files. It is also helpful if
information that appears on the screen can be printed and saved. All
this data should be kept off of the system in question and
write-protected. No discussion of a possible incident should occur
via e-mail. Even if it turns out not to be an incident, it doesn't
hurt to save the data.
- Anomalies in log files, such as a user logging in at odd hours or frequent failed login attempts
- The "last" command showing a user who doesn't travel logging in from an unusual location
- System programs disappearing or behaving strangely
- Unusual processes running
- Entries added to the password file
- Security monitoring software complaining (obviously)
Violation of security policy
Because security policy violations are committed by internal users,
some delicacy is required to handle the situation since it may
result in legal action. It is very important to involve senior
management, Legal, and Human Resources to limit the company's
exposure to liability. Violation of the security policy may be
determined either by routine examination of system log files, or by
a reported incident.
Assigning priority and responding to a security incident
How you respond to a security incident is dependent on the severity
and type of incident. In all cases, details of the incident should
be discussed only on a need-to-know basis. A system administrator
may find it tempting to tell everyone about the porno sites that an
executive was caught browsing, but confidentiality is crucial. A
chronological log must be maintained of all investigative measures.
If this log might be used as legal evidence, it is to be signed by
all involved members of the IRT. Also, a backup should be made of
the data, preferably on write-once media. At the very least, the
media should be write-protected.
An active penetration attempt that is currently in progress. This
may either be directed at the company internal network from outside,
or originating from the internal network and directed either inside
or outside the external network.
This list is a bare minimum and should be modified to reflect your
site's configuration. The point is that you have to act fast, and you
might forget important details. The incident response team should not
be constrained by waiting for management approval to take action.
- Call incident response team, if not already involved.
- Technical staff should get as much information from screens and
log files as quickly as possible and store offline. Files to be saved include:
Also save the log files from any application or security software
you may be running, such as tcpd.log and http.log. Make a list of
all your firewall log files and include them here.
You may not have time to analyze whether the intruder is currently in the
system. Get as much data as you can and analyze it offline later. It
may be helpful to have a script written that you can run. Just don't
make it too complicated:
/usr/bin/ls -laR /tmp > /archive/tmp.save
/usr/bin/ps -aux > /archive/ps.save
Add to this list any other commands that would be useful. Run the
commands in order of priority. Get the quick commands early in the
list in case you run out of time. It would be very useful if
/archive is a write-once CD, but failing that, get the files off the
system and on backup media as soon as possible.
- In the meantime, the management members of the incident response team should be informing management and relevant organizations.
- If there is no management representative immediately available, all members of the IRT are preauthorized to take the following actions:
- Contact the involved Internet service providers and provide appropriate log
data to trace the attack.
- Disconnect the company network from the Internet.
- Request that the ISP for the originating attacker disconnect the attacker at its end.
- Under no circumstances will any administrator or IRT member be permitted to take retaliatory action against the originating network with an intention to cause harm. No matter how you are provoked, don't open the company up to criminal or civil liability.
- Only senior management may authorize informing outside agencies such as law
enforcement, the media and security organizations.
- No attempt will be made to communicate directly with the attacker.
- A complete backup of the system must be performed to preserve the data. Media must be clearly labeled, dated, and signed. You may need to demonstrate the integrity of the backup by showing comparative checksums against the original file. A WORM drive is preferred as a backup device.
- If the system has been successfully penetrated and compromised, the following steps must be taken:
- Disconnect from the Internet, backup the system and begin analysis
- Reinstall system binaries from distribution media
- Install all recommended patches and all security patches
- Change all passwords, including users
- Have the system audited by an external security consultant
An active violation of the company security policy that could expose the company to criminal or civil liability. An example may be that a user is currently sending or receiving harassing images or messages.
- The incident response team managers will inform senior management and follow Human Resources guidelines.
- The technical members of the IRT will discreetly gather all log information and save to backup media.
- There will be no discussion of the incident with anyone who is not on a need-to-know basis, as approved by senior management.
(The following describes incidents that are not currently active and do not have the same degree of urgency.)
There is evidence that a serious penetration attempt has been made, but the attack is not currently active.
Follow the same procedure as in Priority 1, with the following exception:
The IRT must get senior management approval to disconnect the company from the Internet or before sending log data to the ISP.
There is evidence or a report of a security policy violation that could expose the company to legal liability. Follow the same procedures as in Priority 2.
There is evidence of minor penetration attempts or policy violations. The technical members of the IRT may follow up by contacting the originating ISP's technical contact and monitor the log files. The management members of the IRT will inform appropriate management of policy violations.