Vulnerability Assessment vs.
Penetration Testing
A guide to needs analysis
Prepared By:
Joel Gridley, CISSP
Security Consultant
DRAFT
February 12, 2016
Modification History
AUTHOR
Joel Gridley
DATE
February 10, 2002
Vulnerability Assessment vs. Penetration Testing
COMMENTS
Initial Draft
Modification History  i
Table of Contents
MODIFICATION HISTORY .......................................................................................... I
TABLE OF CONTENTS ............................................................................................... II
EXECUTIVE OVERVIEW ............................................................................................. 1
INTRODUCTION ........................................................................................................... 2
WHAT IS A VULNERABILITY ASSESSMENT? ......................................................................... 2
WHAT IS A PENETRATION TEST? ....................................................................................... 2
WHAT IS THE DIFFERENCE? .............................................................................................. 3
WHAT ARE THE ADVANTAGES?............................................................................... 4
SUCCESSFUL PENETRATION TESTS PROVIDE CLARITY. ....................................................... 4
VULNERABILITY ASSESSMENTS PROVIDE MANAGEMENT GUIDANCE................................. 4
BOTH REDUCE RISK .......................................................................................................... 4
WHAT ARE THE DISADVANTAGES? ........................................................................ 5
ASSESSMENTS MAY BE DIFFICULT TO ACT UPON. ............................................................... 5
PENETRATION TESTS MAY NOT “SOLVE THE PROBLEM”. .................................................. 5
ASSESSMENTS AND PENETRATION TESTS CAN DESTABILIZE AN ENVIRONMENT. ............... 5
PENETRATION TESTS MAY COMPROMISE DATA. ................................................................ 6
PENETRATION TESTS ARE MORE EXPENSIVE THAN VULNERABILITY ASSESSMENTS. ........... 6
CONCLUSIONS ............................................................................................................... 8
WHICH SERVICE SHOULD I USE? WHEN SHOULD THEY BE DEPLOYED? ............................ 8
Vulnerability Assessment vs. Penetration Testing
Table of Contents  ii
Executive Overview
Vulnerability assessments and penetration testing are security services offered by a wide
variety of vendors. While there is a standard technical terminology, vendors use the
language in different ways in describing their services. Different vendors may deliver
different services under the same service description. This document attempts to clarify
the differences, and help provide guidance on why an enterprise should select one type
of service over the other.
In brief, a vulnerability assessment is a design review conducted within the bounds of a
security environment, and is designed to provide management guidance on the risk
profile associated with a service. A penetration test is an attempt to violate the controls
imposed by a security environment, and is designed to determine errors in design or
configuration.
Both services are valuable. Because of the expense associated with penetration testing, it
is most often performed after assessments have been completed and any guidance
provided by those assessments acted upon, but at times penetration testing can provide
clear evidence of problem, and a powerful justification for action.
Both services have risks. While vulnerability assessments are generally conservative, the
complexity of some IT environments can lead to loss of service. Penetration tests,
because they attempt to contravene the protective or corrective elements of a security
environment, have a much higher rate of loss or compromise of service, and are most
often conducted either in robust environments or before a system goes into production.
In addition, because a penetration test usually involves assessment, and the practitioners
have higher skill sets, penetration tests are generally considerably more expensive than
simple vulnerability assessment services.
Vulnerability Assessment vs. Penetration Testing
Executive Overview  1
Introduction
Technical jargon has a tendency to be used in non-technical parlance in often-imprecise
fashion. This applies to security services, which combines language from the intelligence
community, the military, the computer underground, and popular literature. What
matters is not what a thing is called, but what the thing is and does. It is worthwhile to
take a step back and identify of what features a service is composed, and to identify what
advantage these features provide to the purchaser of a service.
What is a vulnerability assessment?
Vulnerability assessments are a long-standing practice of engineering professions.
Assessment is the practice of analyzing different types of discipline specific scientific
information, combining and evaluating these types of information in an interdisciplinary
framework and presenting the results in a way that is useful to policymakers.
Vulnerability is defined as the degree to which a system is sensitive to, and unable to
cope with, adverse conditions. Thus, a vulnerability assessment is a management
document that provides guidance for a system under risk.
Vulnerability assessments can be applied to physical objects such as buildings; to
complex hierarchies such as ecosystems; and to the physical objects and complex
interdependencies of IT infrastructure. The latter takes the form of a report identifying
known problems that would prevent a service from completing its function, e.g.
susceptibility to “denial of service” type attacks, or would cause a service to perform
unwanted functions, e.g. susceptibility to compromise. The report typically scores the
vulnerabilities to their impact and likelihood of occurrence, and provides
recommendations for risk remediation.
What is a penetration test?
Penetration testing is a term derived from DoD jargon. Originally, it meant war game
exercises in which teams would attempt to enter a controlled area of a building. The
formal definition, which comes from the DoD standard, Trusted Computer System
Evaluation Criteria (Orange Book) is
The portion of security testing in which the penetrators attempt to circumvent the
security features of a system.
This is similar to the task of functional testing, where a vendor’s assertions about a
feature set for a product can be verified. One difference is that functional testing has a
deterministic and quantitative aspect: specific tests can be performed to determine if a
specific feature is present or not. On the other hand, penetration testing is nondeterministic and often relies on qualitative results. Under Assurance, the Orange Book
goes on to note:
Vulnerability Assessment vs. Penetration Testing
Introduction  2
Since no test procedure for something as complex as a computer system can be truly
exhaustive, there is always the possibility that a subsequent penetration attempt could
succeed.
What is the difference?
The key difference here is that in a vulnerability analysis, systemic properties are analyzed
and management guidance is provided. In a penetration test, tactical analysis is
performed and a technical observation is provided. A vulnerability analysis looks at the
system as a whole, and attempts to make statements about how to improve the security
posture. A penetration test looks at the system flaws locally, and attempts to contravene
the system to demonstrate its weakness. Another way to describe this is from the
language of search techniques: a vulnerability assessment is a “breadth first search”,
enumerating the immediately visible flaws. A penetration test is a “depth first search”,
focusing on one local aspect of a system, and exploiting the flaw as deeply as possible.
Vulnerability Assessment vs. Penetration Testing
Introduction  3
What are the respective
advantages?
Successful penetration tests provide clarity.
One problem with vulnerability assessments is that a clear understanding of the risk may
not be presented, and guidance may not be followed. It is common for a vulnerability
assessment to flag problems in system maintenance consistently over time, but that
maintenance is not performed until after a significant security incident.
When a penetration test is successful, it is a clear demonstration of an inadequacy of a
system. If a penetration test is run against a banking application, and the testers are able
to transfer ten million dollars to a private account, then the owners of the application
know that this vulnerability is real. The penetration test provides the ‘significant security
incident’ that motivates management to take action.
Vulnerability Assessments provide management guidance.
Vulnerability assessments address comprehensive elements, painting a broad picture of
the security posture. Instead of focusing on the individual trees that are systemic flaws,
it speaks as well to the forest of problems that is the real management challenge.
Vulnerability assessments often speak to overall problems, organization, or security
administration, and can be helpful in making top-down design changes in an
environment.
While penetration tests solve problems that exist today, vulnerability assessments
provide a basis for secure architectures into the future.
Both reduce risk
Both vulnerability assessments and penetration tests are part of a design review process.
Independent review by professional evaluators provides insight into the functioning of a
system. Here, design review of security controls allows for identification of elements
that would increase the trust in a system, the extent to which someone who relies on a
system can have confidence that the system meets its specifications, i.e., that the system
does what it claims to do and does not perform unwanted functions. This reduces the
overall risk in the system – the expectation of loss expressed as the probability that a
particular threat will exploit a particular vulnerability with a particular harmful result –
because both are reducing the probability of exploit. Vulnerability assessments are a
systemic approach, while penetration tests focus on particular elements.
Vulnerability Assessment vs. Penetration Testing
What are the respective advantages?  4
What are the respective risks?
Assessments may be difficult to act upon.
Assessments are designed to provide guidance on improving a security posture. Such
reports often contain long lists of vulnerabilities identified. When so many problems are
identified, it may be difficult to know which ones to tackle. The scope of the problem
may be so daunting that it is paralyzing. Penetration tests may then be comforting, in
that it identifies one problem, and attention can be paid to that single problem.
A good vulnerability assessment provides not merely insight into the nature of the
problem, but a “next steps” guideline, to help the client move to a more trusted
environment. Sometimes this is not possible, because only limited information is
available to the assessors; putting vulnerabilities into context requires an understanding
of the business drivers and business risks, and that is often outside the scope of an
assessment. But assessments can at least point to problem areas, and provide some
recommendations on addressing those areas.
There may also be a belief that if some of the problems are resolved, then security has
been “improved”, and that is enough. Unfortunately, the security of a system depends
on the security of its weakest element. If an adversary finds one avenue blocked, it
simply seeks out another. Unfortunately, repeated assessments are usually required to
ensure that over time all vulnerabilities are addressed.
Penetration tests may not “solve the problem”.
Penetration tests have two outcomes: either they penetrate the system or they don’t. If
they penetrate the system, they have brought attention to one flaw, but even fixing that
one flaw does not provide assurance on the security of the system. When a system is
penetrated, it usually does have the useful result of ensuring that additional resources are
applied to enhance the security of the environment. However, this is not a complete
resolution, and is really only useful in demonstrating that flaws still remain. If they don’t
penetrate the system, it only means that the approach used by the tester was not
successful; at other points in time, other testers or worse, adversaries, may be successful.
Assessments and penetration tests can destabilize an
environment.
Vulnerability assessments and penetration tests generate unusual data in a system. The
data may be more copious than usual, or have unusual formatting; because digital
systems can react in non-linear fashions, even small changes in traffic from the usual
form can have unforeseen consequences.
Vulnerability Assessment vs. Penetration Testing
What are the respective risks?  5
For example, use of the Telnet protocol to communicate with network infrastructure is
normal. As part of an assessment, one usually looks for Telnet access, and runs some
simple tests. However, opening approximately 98 connections on port 23 will cause
Cisco 760 Series Routers to self reboot (as of March 1999, bug ID CSCdm03231). This
can lead to denial of services associated with that device. Other denials of service with
other devices have been associated with simple port scans (e.g. CVE-2001-0413), or
fragmented packet data (e.g. CVE-199-0052).
Penetration tests have higher risks. Vulnerability assessments stay inside of normal
security boundaries, and this generally affords some protections. Penetration tests
intentionally penetrate boundaries, and this can lead to untested territory. Because the
tester seeks to acquire administrative access, the control is similar to that of authorized
administrators, and can lead to significant loss of system abilities.
Penetration tests may compromise data.
Because a penetration tester seeks to acquire administrative access, they may view data
that is normally only accessible to administrators. In addition, the proof of compromise
that is the end goal of a penetration test may include capture of sensitive data; such data
is usually brought across uncontrolled networks. The rules of engagement for a test
must carefully note the actions expected by the penetration tester that may conflict with
existing constraints, e.g. the handling of third party data.
Penetration Tests require more documentation from purchaser of
service.
A vulnerability assessment, whether cooperative or not, stays within the bounds of a
security control. While services are exercised, there is no attempt to violate controls.
On the other hand, penetration tests intentionally try to break through the constraints of
security controls. This generally requires informed consent from senior management, to
avoid an offense under various jurisdictions’ formulations of Computer Misuse Acts.
Penetration tests are more expensive than vulnerability
assessments.
Penetration tests usually follow a methodology that begins with a reconnaissance phase
very similar to that followed by an assessment team. Consequently, the duration of a
penetration test is usually longer than that of a vulnerability assessment. In addition,
much of the work of a vulnerability assessment can be automated; consequently once the
automation scripts exist, the skill requirement associated with the assessment can be
relatively nominal. Many commercial vulnerability assessment products, such as ISS’
SafeSuite or Cisco’s NetSonar, can be deployed by general IT personnel. On the other
hand, the exploit phase of a penetration test does not lend itself to automation well, and
Vulnerability Assessment vs. Penetration Testing
What are the respective risks?  6
is generally a creative and non-deterministic event. This requires significantly more
experience and higher skill sets than conventional assessment.
Vulnerability Assessment vs. Penetration Testing
What are the respective risks?  7
Conclusions
Which service should I use? When should they be deployed?
Both vulnerability assessments and penetration testing are important services in support
of a trusted environment. Vulnerability assessments are useful early in the process, even
before a system is in place: they can provide guidance on the underlying proposed
architecture of an information service. Vulnerability assessments are useful on a regular,
ongoing basis, as they provide a comprehensive analysis of the risk profile of that
service. They can provide an audit of compliance with specification, and provide
assurance that the implementation is consistent with expressed intent and best security
practices.
Penetration tests are used less frequently, but with no less importance. Perhaps the best
time for a penetration test is just before a service goes online, while it is in “beta”. This
allows for reduced risks associated with the problems of instability or loss of
confidentiality, and can provide a cautionary note before high-value production use is
deployed. Penetration tests should also be performed at regular intervals on a robust
system, to ensure that the security design is sound and that the audit practices are
adequate to verify conformance.
Vulnerability Assessment vs. Penetration Testing
Conclusions  8