Incident Handling in
Academia
What to do when you have been
hacked!
The Presenters
 Scott Fendley
– BS Comp Science – U of AR 1999
– MS Comp Science – U of AR 2004
– Security Analyst, Dept of Computing Services
– Volunteer Incident Handler, SANS Institute
 David Merrifield
– Associate Director of Computing Services
Session Description
 Explores how to handle the attacks on your
Internet infrastructure.
 Discusses a time-tested 6 step procedure for
Incident Handling.
 Touches on the legal issues relevant to all
Academic Institutions (K12 or Higher Ed)
 Dealing with Law Enforcement and handling
Evidence
 Employee Monitoring vs Student Monitoring
Disclaimers, Disclaimers, Disclaimers
 I am not a lawyer. Consult your nearest
legal counsel if you choose to handle
incidents on your campus or have questions.
 The majority of this information is the basis
of my procedures at the University of
Arkansas, but your mileage may vary.
Foundation of Incident Handling
 An Action Plan for dealing with intrusions,
cyber-theft, denial of service and other
security-related events
 Events can be of a electronic nature or of a
physical nature.
Definitions
 Incident – an adverse event in an information
system, and/or network, or the threat of the
occurrence of such event.
– Ex: unauthorized use of another user’s account
– Execution of malicious code
– Unauthorized use of system privileges
 Event – Any observable occurrence in a system
and or/network.
– Ex: Packet Traces
– System Boot Sequences
– Anything that you can record in your IH notebook
Incident Handling Metaphor
 Incident Handling is like First Aid.
 The Handler is under pressure and mistakes
can be costly
 Practice is a key. Skills degrade without use.
 Use pre-designed forms and procedures,
and call on others for help.
Emergency Action Plan
 Remain Calm.
 Communicate with your management, and
coordinate with your co-workers to keep
things focused.
 Use formalized language.
– EX: Whiskey Five Yankee Mic, We have a
bogey on your nine.
– Explicit meaning, no room for interpretation is
less likely to cause mistakes.
Emergency Action Plan
 REMAIN CALM (still!) Do not hurry. Mistakes
can be costly.
 Notes, logs and other evidence are crucial
– If the perpetrator is ever found and arraigned, how can
you testify if your notes are not organized and detailed?
 Failure to take notes is the most common mistake.
 Consult your legal counsel for how long you
should keep your logs.
 Quality not Quantity
Emergency Action Plan
 Take good notes.
– Remember what your English teacher taught you.
– The 4 W’s
•
•
•
•
Who?
What?
When?
Where?
– Extra Credit for the 5th W and the H
• Why?
• How?
Emergency Action Plan (1)
 Notify your manager of your progress
 Do you have easy access to your School’s phone
directory? Pager numbers? Home numbers?
 If you are over your head, do not hesitate to ask
for help
–
–
–
–
FBI Field Office
handlers@incidents.org
Local Law Enforcement
Trained Computer Forensic Investigators
Emergency Action Plan (2)
 Enforce a “need to know” policy.
 Do not tip your hand to potential insider
threats.
 Use out of band communications. (Don’t
email people about IH discussions.)
– Telephones
– Faxes
– Personal Visits
 PGP Keys
Emergency Action Plan (3)
 Contain the problem. (stop the bleeding)
– Pull the network plug?
– Pull the power plug?
– Forensic Evidence Quandary.
Containment Micro Example
 Call the user and say “Take your hands off
the keyboard and move away from the
computer.”
 Stand up go to the back of the computer and
unplug the network (and/or modem).
 Don’t touch anything, we’ll be right there.
 Fax instructions/forms for them to fill out.
Emergency Action Plan (4)
 Make a backup of the affected system(s) as
soon as is practical. Use new, unused
media.
 Make a binary, or bit-by-bit backup.
 Failure to make a backup is the second most
common error.
 Chain of custody of the evidence.
Emergency Action Plan (5)
 Get rid of the problem. Identify what went
wrong if you can. Take steps to correct the
deficiencies that allowed the problem to
occur.
 Nuke the computer or just scrub it?
 Get back in business using clean backups
and monitor the system to make sure it can
resume functioning.
Emergency Action Plan (6)
 Learn from this experience.
 Share your experience with others.
– Sys-admin List for K12
– Arktech List for Universities and Colleges
– Another useful list is Unisog@sans.org for all
Educational entities.
 Review the incident from start to completion.
 Identify areas of improvement
 Engineers versus Mathematicians
Seven Deadly Sins of IH
 Failure to report or ask for help
 Incomplete/non-existent notes
 (Accidental) Mishandling/destroying
evidence
 Failure to create working backups.
 Failure to contain or eradicate
 Failure to prevent re-infection
 Failure to apply lessons learned
Emergency Action Plan
Summary
 Remain calm, don’t hurry.
 Notify your oranizations’s management, apply
need to know, use out of band communications.
 Take good notes (even if you aren’t/can’t
prosecute).
 Contain the problem
 Back up the system(s), collect evidence
 Eradicate the problem and get back to business
 Lessons Learned
Six Steps of Incident Handling
 Preparation
 Identification
 Containment
 Eradication
 Recovery
 Lessons Learned
Preparation
 Update your organization’s disaster
recovery plan to include Incident Handling
 Establish visibility and a compensation plan
for the team. (Slush fund for food and
caffeine for long weekends or evenings of
mitigating an emergency.)
 Checklists!
 Emergency Communications Plan
Preparation Key Points
 Password Access
 Conduct training for incident handlers
(War Games)
 Establish guidelines for inter-departmental
cooperation.
 Build relationships with techies and sys admins
 Develop interfaces with law enforcement agencies
in your area.
Preparation - Jump Bag
 Small tape recorder
– Blank Tapes
 Binary Backup Utils
– Safe Back
– Ghost
– Encase
 Forensic Software
– TCT
– Autopsy
– Encase
 Small Hub and cables
 Laptop (extra batteries)
 CD’s with clean binaries
– Sysinternals
– Foundstone
– Windows Resource Kit
 Call List, Phone book
 Cell Phone (batteries)
 Fresh Blank Media
(CD-Rs Floppys, Zip, etc)
Preparation in a nutshell
 Policy
 Transportation
 People
 Space
 Data
 Power and
 Software/Hardware
environmental
controls
 Documentation
 Communications
 Supplies
Identification
 Fire Alarm Analogy
– Who can pull a fire alarm?
– Who authorizes re-entry?
 Maintain situation awareness
 Provide current “intelligence”
 Correlate information (mailing lists are
great sources for newest worms/viruses or
attacks)
Signs of an incident
 Intrusion Detection system alarm
 Suspicious entries in system or networking
accounting
 Discrepancies in logs
 (Un) successful logon attempts
 Unexplained, new user accounts
 Unexplained processes or services running
 Notification via abuse@ address or phone call
 Poor system performance
 Unusual time of usage.
Identification
 Initial Assessment
 “Efficient handling of errors is part of the process”
 Be careful to maintain a provable chain of custody.
 Use the tape record if at all possible to keep notes
for you on what commands you run and actions
you do.
 Make law enforcement sign for any evidence you
hand off to them. Assign a value to it.
Containment
 This is where we cross the threshold in
which we begin to actively modify the
system.
 Keep the system pristine
 Pull the system off the network (or perhaps
the subnet off the network).
 Load your binaries, set the path
 Backup the system
Containment
 Safely store any backup disks/tapes so that
they will not be lost and/or stolen. Multiple
copies are best with volatile media types.
 Keep a low profile.
 Analyze a copy of the backup
 Report to management on progress
 Are you sure you backed up the media in
question?
Containment
 Acquire logs and other sources of
information.
 Firewalls, IDS Logs
 Logs from other systems nearby
Containment
 Consult with system owners (departmental
technical staff)
 Change passwords
 Determine possible other systems that have
potentially had passwords breached.
 Packet sniffers are easy to install.
Eradication
 Is your schools policy to nuke the computer
and reinstall with a secured OS, or just
clean and secure?
 Improve your defenses
 Perform vulnerability analysis and system
audits.
 Locate the most clean backup and carefully
install it.
Recovery
 Restore from backups if required
 Be sure you do not restore the malware
 Secured system?
 Validate the system and create baselines
 Test that everything on the system is working as
expected with the owner.
 Place the final decision on the system owner of
when to restore operations.
 Monitor the systems
Follow-up / Lessons Learned
 Develop a follow-up report
– Start as soon as possible
– Include any forms you used in identification
step
– Details, details, details!
 Lessons Learned Meeting
 Executive Summary Report
 Recommended Changes to procedures?
 Additions to jump kit
Legal Issues to Academia
 HIPAA
– Privacy Rule (2002)
– Security Rule (2005)
 FERPA (Buckley Amendment)
 DMCA
 Patriot Act
Monitoring
 Monitoring employees
 Student Privacy
 Student-employees?
Law Enforcement Contacts
 University Police
 City Police or County Sheriff
 FBI (Field office in LR)
 Secret Service
 Department of Homeland Security
 Infraguard Arkansas
More Information
 http://www.sans.org/
 http://www.securityfocus.com/
 http://www.foundstone.com/
 http://www.sysinternals.com/
 http://www.incidents.org/
 http://ists.dartmouth.edu/
Questions?
 Contact me at scottf@uark.edu or call me at
479-575-2022.
 Also, talk to those in the state and across the
nation for specific questions.
– Sys_admin@lists.state.ar.us
– Unisog@sans.org