Incident Response and Digital Forensics –
Denver Chapter of ISACA
Outline




Introduction
The Incident Response Process






Preparation
Identification
Containment
Eradication
Recovery
Lessons Learned





Reconnaissance
Scanning
Exploitation
Keeping Access
Covering Tracks
The Attacker Process
Conclusion
Introduction
 FishNet Security
 Started in 1996, largest InfoSec VAR
 Support, Consulting, Training, Products
 Locations – Boston, Omaha, Kansas City, St.
Louis, Dallas, Minnesota
 Assessment Team: Policy, Network, WebApp
and DB, Wireless, IR and Digital Forensics
Introduction
 Trey Keifer - (trey.keifer@fishnetsecurity.com)
 Level II Security Engineer
 Sr. Incident Response Analyst
 Coordinator of Incident Response Program
 SANS Certified Incident Handler (GCIH)
Incident Response and Digital Forensics
 One of the least practiced, most stressful, highly
scrutinized areas of Information Security.
 Every incident is unique and can incorporate many
different areas of the affected organization.
 Incident analysts must be able to think quickly,
remain calm and consider all possibilities.
Common Incident Types
 Economic Espionage
 Intellectual Property Theft
 Unauthorized Access
 Stolen Passwords and Data
 Unauthorized Use
 Inappropriate E-Mail and Web Habits
 Malicious Code
 Worms with Backdoors (Sasser)
 Insider Threats
6 Steps of the Incident Handler Methodology
 Preparation
 Identification




Containment
Eradication
Recovery
Lessons Learned
Preparation:
 The key to a successful response is
preparation.




Form a strategy.
Design a procedure.
Gather Resources.
Practice, practice, practice.
Preparation:
 Identify the “Core Team”








Technical (IT, InfoSec and System Owners)
Management
Legal Department
Forensics
Public Relations
Human Resources
Physical Security and Maintenance
Telecommunications
Preparation:
 Organizing Individuals
 All members of the CSIRT team should know their role
and how they will interact with the other members.
 Outsourced or “third party” members should have
contracts in place.
 Contacts for Law Enforcement should be known and
situations for their involvement discussed.
Preparation:
 Develop a Procedure
 Incident response can be a high-stress time. A well
documented procedure, that is easy to follow, can greatly
reduce the anxiety.
 Develop a call tree and notification procedures
 Brainstorm likely scenarios.
 Identify general information needed in most scenarios
ahead of time.
 Make checklists and forms for as much as possible.
Preparation:
 Communication
 Communication is incredibly important during an incident.
Not only the people involved, but the method which it is
done.
 Updates should be frequent.
 Out-of-Band Communications are very important.
 Faxes
 Cell Phones
 Be careful with the Blackberry’s
Preparation:
 Access Rights
 The incident response team must have access to systems
without the administrators authorization.
 Controversial Issue
 User Accounts, Passwords and Encryption keys
 Third-party storage methods are available
Preparation:
 Policies
 Protect the organization from legal liability and allow
investigators to do their job.
Warning Banners are readily displayed.
Search policy is detailed in employee manual.
Human Resources and Legal have signed off.
Employees have acknowledged knowing their
expectations on privacy.
 Beware of international laws (European Privacy Directive)




Preparation:
 Gathering Resources
 Incident analysts should have all information ready and
be able to respond to the incident.
 Procedures, Checklists and Forms are ready.
 Access credentials are available or individuals with them
are known.
 System information, network diagrams, software and
intellectual property are documented thoroughly.
Preparation:
 Training
 SANS Institute and GIAC Certifications
 Track 4: Incident Response and Hacker Techniques
 Track ??: Digital Forensics
 Vendor Training
 Guidance Software
 Access Data
 Partners
 Incident Response Scenarios
Identification:
“Incidents can’t always be prevented, but must
always be detected.”
Incident: Intentional or Unintentional
 Multiple failed logins to the domain administrator
account.
 Administrator credentials were cached on a users
workstation and they are attempting to login.
 Someone is actively attempting to brute-force the
account.
Identification:
 Goals
 Determine Scope
 Identify what systems, people and informational assets
are involved in the event.
 Preserve Evidence
 Protect the facts of the incident while determining the
scenario.
Identification: Suspicious Events
 Unexplained Occurrences
 New Accounts or Files
 File Modifications
 IDS Triggers
 Firewall Entries
 Accounting Discrepancies
 Poor Performance/Unresponsive services
 System Instability
Identification: Passive Identification
 Sniffers and Traffic Analysis
 Cyclical Buffers allow full recording of events at the
packet level to a point, depending on size and utilization.
 Target machine evidence is still preserved.
 Assist in determining new attacks for which signatures
have not yet been written.
Identification: Passive Identification
 Intrusion Detection Systems
 Least invasive method
 Target machine evidence is preserved
 Logs must still be protected
 Write-Once, Read-Many Media
Identification: Passive Identification
 Tripwire-style File Modification
 A hash of the file is taken and stored in a secure
database. Any modification to that file results in a change
of the hash.
 Very indicative of a successful compromise.
 Can be noisy during patching and must be tuned after
every software upgrade.
Identification: HoneyPots and HoneyTokens
 Specific systems or accounts with additional logging
and notification to alert on suspicious activity.
 Operators must be careful of entrapment.
 Systems have to be secured and heavily monitored.
 Systems cannot invite intruders –
 No “hackme” accounts
 No “Salary Database” systems
Identification: Chain of Custody
 Evidence must be accounted for from the time it is
collected until the time it is submitted to the court.
 Each piece of evidence must be under the control of one,
identifiable person at all times.
 A change in control of the evidence must be recorded.
 Evidence in storage must be protected from
contamination. (ie… sealed and secured)
Containment Now that the events have been identified as an
incident and a chain-of-custody for evidence
has been established, we will take the first step
into system modification by beginning our
containment.
Containment:
 Vendor Coordination
 Work closely with your vendors and know how to open
security-related tickets with high priority.
 ISPs can prevent some Denial of Service situations.
 They are more familiar with attacks because they have seen
them with other clients and are up-to-date on advisories.
 Additional people working towards identification, containment
and recovery.
 We are used to the pressure!
Containment:
 Identifying the Trust Model
 The trust model identifies not only the
technology, but also the people that are involved
in the incident.
 What connectivity does the network or system have to
other areas in the organization?
 What information is contained within it?
 Who needs to be involved and to what extent?
Containment:
 Documentation Strategies
 Documentation should be collected from most
volatile to least volatile and least invasive to
most invasive.
 Volatile evidence includes RAM, running processes and
active connections.
 Be careful of running system commands from anything
but recovery media.
Containment:
 Should we Quarantine?
 Changes to a system may be easily observed by
an active attacker.
 Rootkits may identify a pulled network connection or
extensive system modification and protect the attacker.
 Some exploits are entirely memory resident and will
disappear when the power is pulled.
Containment:
 Initial Analysis






Keep a low profile
Never analyze the original
Make frequent updates to CSIRT
Acquire log files
Stick to the facts and avoid blame
Consider all possibilities but keep it simple
Containment:
 Backups
 Numerous backups allow both investigation and
preservation of evidence.





Different strategies exist and depend on the situation.
Original is kept as evidence
Backup 1 – Placed back in production
Backup 2 – Forensic Analysis
Backup 3, 4, etc… separate copies for analysis
Containment:
 Digital Forensics
 Numerous separate analysis all yield the same
results.




Requires specialty hardware, software and training.
Bit by Bit copying and analysis of data.
Recovery of deleted data.
Identification of altered system files (trojans) and
binaries in a safe environment.
Containment:
 Digital Forensics: Hardware Write Blockers
 No modification to the data itself, we want to
observe and duplicate only.
 Hardware device or driver between acquisition machine
and target system.
 May use NIC, USB, FireWire or IDE/SCSI channels.
 Intercepts write commands and gives logical return
results.
 Allows browsing of the filesystem during acquisition.
Containment:
 Digital Forensics: Forensic Software
 Allows quick and efficient analysis of the
information contained on the device.
 Guidance Software’s EnCase used by law enforcement.
 Linux Forensics CD’s are coming along in maturity.
(still must use write blockers!!!)
 Scripts allow quick searching of keywords in files and
deleted data.
 Hash comparisons verify original files, known dangerous
applications and aid the examiner in avoiding the bad
stuff.
Containment:
 Digital Forensics: What are we looking for?
 Many areas of interesting data are forgotten
about.
Cached web content
Email Files (PST’s)
Recoverable Deleted Files
Specific Incidents: CAD drawings, Engineering diagrams,
Pornography
 Known file signatures of hacking tools, backdoors, etc…




Containment:
 Digital Forensics: Other devices?
 May not be able to submit as evidence in court,
but can assist the Incident Handler in their
investigation.
 Personal Organizers (PIMs): Blackberry, Palm Pilots,
IPAQ’s.
 SIM Cards/Cell phones
 USB Tokens/Flash Drives
Containment:
 Digital Forensics: Not Perfect!
 Some tools have been written specifically to
defeat forensics software.
 DoD: 7-Pass, random-write method for secure deletion of
magnetic media. (Rainbow Method)
 Windows: Eraser
 Unix: Wipe
Containment:
 Slowing the Attack






Change passwords and access rights.
Change hostnames and IPs.
Null Route suspicious traffic.
Block IPs or Networks.
Apply Patches to similar systems.
Shutdown services.
Eradication Once an incident has been contained we attempt
the total removal of malicious applications from
a system or network.
Eradication:
 Remove or Restore
 The decision of whether to remove malicious files or
restore from backups is a difficult task.
 Rootkits almost always demand a rebuild.
 Verification of backups is a must.
 Patches may not be available and a total change of
architectures may be necessary.
Eradication:
 Improve Defenses
 Implement additional detection and protection methods
and strengthen existing technologies and processes.
 Apply firewall and router filters.
 Perform “mini-assessments” using the same tools and
techniques as your attackers.
 Look for the same exploits and backdoors on multiple
machines.
Recovery Once the threat has been removed the
organization must begin the process of
returning the business to normal operation.
Recovery:
 Returning to Operation
 System owners make the final call on returning to
production.
 Owners depend on the systems and know their true
value.
 If a disagreement occurs on whether to return to
production or not it should be documented by the
analysts and the owner should acknowledge
responsibility.
Recovery:
 Monitoring
 At this point in the process you should have enough
information to identify the attack if it occurs again.
 Create custom IDS signatures if possible.
 Verify proper operation to baseline configurations.
 Implement additional logging on network, hosts and
applications.
Lessons Learned The lessons learned meeting provides a method
for the organization to coordinate knowledge of
an incident, suggest changes in procedures and
policies for the future and justify the
implementation of new safeguards.
Lessons Learned:
 Recap Meeting
 Should occur promptly after eradication of an
incident while details are fresh in the team
members heads.
 Create a timeline of events.
 Provide a consensus of notes and documentation.
 Finalize facts for a final report.
7 Deadly Sins







Failure to report/ask for help
Incomplete/Non-Existent Notes
Mishandling/Damaging Evidence
Failure to create backups
Failure to eradicate or contain
Failure to prevent re-infection
Failure to apply lessons learned
Attacker Methodology





Reconnaissance
 Profiling the Target
Scanning
 Identifying Weaknesses
Exploitation
 Breaking the Law
Keeping Access
 Backdoors
Covering Tracks
 Staying out of Jail
Reconnaissance:
 The target is profiled –




Employee Information (name, numbers, titles)
Systems Information (usenet postings, job listings)
Process Information (vendors and transactions)
Location Information (external networks, physical locations)
Scanning:
 Port and Vulnerability scanners are run to
identify vulnerable systems.




Open Ports and Services
Vulnerable Applications
Default Usernames and Passwords
Weak Encryption Implementations
Exploitation:
 Execution of attack – usually the first point at
which the law is broken.
 Goals




Gaining Access
Elevating Access
Extracting Information
Denying Service (DoS)
Keeping Access:
 Addition of Admin-level User Accounts
 Enabling of default, insecure services
 Installation of “Backdoor” or “root kit”
applications allowing the attacker to retain
access despite system modifications.
 Application Level
 Traditional Rootkit
 Kernel Level Rootkit
Covering Tracks:
 Modification of system logs, applications and
processes to prevent identification by
administrators.




Hiding files and Directories (… and alt-255 dirs)
Changes in /var/log
Changes in shell history
Removal of events (windows)
Our Example Scenario
 An attacker uses a “0-day” exploit to infiltrate
the target organization, install a backdoor and
retrieve critical intellectual property for a
competitor.
 Normal security procedures alert the
administrators to suspicious activity and the
incident response plan is activated.
Attacker Perspective: Reconnaissance
 Google and the corporate web site are used to
identify the organizational structure of key
personnel including HR managers and
executive management.
 Low-Profile, no data sent directly to
organization.
 Impossible to detect.
Attacker Perspective:
Harvesting
 Freely-available scanning
tools are used to identify
email addresses from the
corporate website.
 Same method as SPAM
groups.
 Many sites do not use
generic web addresses.
Attacker Perspective: Exploitation
 Attacker sends malicious application to email
addresses obtained during scanning.
 Users open emails (possibly through social
engineering) and are immediately infected.
 Attacker can be listening for connections from
infected machines and have immediate control
over systems.
Attacker Perspective: Keeping Access
Incident Timeline
Incident Timeline: Preparation
 IR Team established and roles defined.
 Daily procedures established for log analysis
and identification.
 Containment procedures are outlined in policy.
(Restoration takes priority)
 Roles and Responsibilities are defined
Incident Timeline: Identification
 Bandwidth graphing shows abnormal usage
 Passive sniffing identifies responsible host
Incident Timeline: Containment
 No “watch and learn” policy, power is pulled
from the host.
 System is imaged using forensic tools and
Hardware Write-Blockers which prevent
alteration of data during backup.
 Employee is interviewed to determine method
of infection.
Incident Timeline: Eradication and Recovery
 System is restored from the organizations
hardened base image and patches are applied.
(Analysis can continue through restore)
Incident Timeline: Lessons Learned






Social Engineering Awareness
File attachment blocking
Firewall Rule Revisions
IDS Signature changes
Patch Management
Advisory Alert Services
Questions?