::Software Security Quality::
Testing Taxonomy &
Testing Tools Classification
OWASP
AppSec
DC
October 2005
Arian J. Evans, OWASP Tools Project
Random Security Title, FishNet Security
arian.evans@fishnetsecurity.com
913.710.7085 [mobile]
Copyright © 2005 - The OWASP Foundation
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License.
The OWASP Foundation
http://www.owasp.org/
OWASP AppSec DC \NIST 2005
::Software Security Quality::
Testing Taxonomy &
Testing Tools Classification
Subtitle: What is Software Security, how do
we evaluate it, and what do we use to
evaluate it?
OWASP AppSec DC 2005
2
--OR--
::Software Security Quality::
Why does every web-based
Document Management
Solution Suck so Badly?
OWASP AppSec DC 2005
3
Testing Taxonomy & Tools Project
This project, like many other OWASP projects (e.g.-WebScarab, .NET
Tools), is:
Performed mostly nights and weekends
A personal labor of love (or dislike of FUD)
Performed by a team of one
As objective and vendor-agnostic as possible
Owes thanks to FishNet for letting me use their lab and
encouraging vendors to participate
 Results are free and worth every penny
 So don’t complain too much about the results. 
 Emphasis has been on “web app scanners” since that’s where the
predominant market interest seems to lie right now.





OWASP AppSec DC 2005
4
Who
Arian J. Evans
Tester, Teacher, Researcher, Decoder
Application Security Services
FishNet Security
FishNet Security
Security Consulting Company focused only on Infosec
Work in AppSec Group
They pay me to test and help people fix things
(blah, blah)
OWASP AppSec DC 2005
5
Outline:
I.
II.
Introduction (done)
Chapter One: In the Beginning was FUD.
1)
2)
Problems
About the OWASP Testing Taxonomy & Tools Project
III. Chapter Two: What is Application Security?
1)
2)
IV.
V.
VI.
Distinctions
Definition
Chapter Three: Clarification and Goals
Chapter Four: Taxonomy of Testing
Chapter Five: Automation Tools
VII. Chapter Seven: Q&A (Shoot the Messenger)
OWASP AppSec DC 2005
6
Chapter One:
In the Beginning was the FUD
Problems with understanding
application security, vendor FUD,
appsec testing tools, and testing
techniques.
OWASP AppSec DC 2005
7
Webappsec Vendor Marketing Hype
I.
The Problem of Particulars
−
−
−
II.
XSS, XST, XDS, XDF, XBS: why particulars can hurt more
than they help (reference: Issues slide)
Selling managers/auditors bullet-point friendly products
Classification of Threat/Risk/Issue: Lack of distinction
between Category/Class/Particular—Does anyone take
formal logic classes anymore?
Immature Metrics
−
−
−
What is *really* being exploited? (This is not what the
OWASP Top-10 or WASC taxonomy addresses.)
*What* exploits result in real loss? (Not what the threads on
webappsec@sf are talking about.)
No reward in securing without demonstrable risk.
OWASP AppSec DC 2005
8
The Problems:
 Different ways to “test” applications for security quality.
 Different tools to “test” applications for security quality.
 No categorization of differences or metrics to measure
pluses and minuses of testing techniques/tools.
 No single source of information on what tools are
available for testing applications
 No independent source of information on testing tools
that are available.
 Some tool vendors do a terrible job of educating the
consumer on what they do and don’t do, and how they
relate to their competition.
OWASP AppSec DC 2005
9
The Problems:
Unclear and immature terminology
Few Taxonomies, Categorizations, Metrics and
Models on AppSec
No Business Rule/Risk Taxonomy
No mature Threat Models/Taxonomy
No accurate Attack Models/Taxonomy
Other Taxonomies needed
We need a Taxonomy of Taxonomies
OWASP AppSec DC 2005
10
This Presentation is About:
 The OWASP Tools Project Version 1.2 which aims to:
 Categorize the different ways to “test” an application
for security.
 Categorize the variety of tools that may be used “test”
applications for security.
 Provide basic insight into features and functionality of
“software security tools”.
 Offset the active dissemination of misinformation about
what tools can and cannot provide.
 I bit off more than I can chew here (why are there so
many tools out there for this this???)
OWASP AppSec DC 2005
11
This Project is Headed:
 The OWASP Tools Project Version 2.0 which aims to:
 Categorize & Classify tools that may be used “test” applications for
security.
 Work with groups like OWASP, WASC, NIST, and Mitre/CVE
 Evaluate the value of & possibly provide checklist charts of
features and functionality in specific versions of appsec tools.
 I have concerns about above because for example not all XSS
checks are equal, and the quality can and does change overnight
in current tools.
 In non-COTS software metrics tend to lack pragmatic value as
software continues to increase size, complexity, extensibility, and
develop emergent behaviors as disparate technologies are bolted
together and new software evolves.
OWASP AppSec DC 2005
12
Limited Information; Full-on Equivocation
I.
Things Commercial Tool Vendors like to say:
− Sanctum: “Oh, those are *network* scanners”
− Compuware: “the only ones in the world who do this”
− Cenzic: “We’ve automated webapp pen testing”
− Several: “Oh, no, that’s a CVE scanner. It doesn’t find
anything on our sample application which has “real”
vulnerabilities. See here, we’ll just install our sample
application we wrote right here and show you…”
− Tautology: The Scanner and Firewall Vendors
II.
Goings-on in the AppSec Industry:
− No Standards or Standards-bodies
− No Metrics
− No Watchdogs
− Vendors provide the vast bulk of “Factual Literature”
OWASP AppSec DC 2005
13
Chapter II: The Definition
What is Application Security?
OWASP AppSec DC 2005
14
Why and What is Application Security?:
Why do you care about AppSec
Testing Tools?
When will the tools be mature
enough that I just click “Scan”
and get my report?
OWASP AppSec DC 2005
15
Why do you care about AppSec Tools?
Because you lack expertise
Because you lack subject information.
Because you lack resources.
Because you lack time.
Because of the desire for automation.
Because of the need for reliable,
repeatable tests that produce accurate
quantifiable information about software
defects with security implications.
OWASP AppSec DC 2005
16
When can I click ‘Scan’?
Never.
There are things that we can automate and things
that we cannot automate.
There are issues that are easy to automate reliable
detection and issues that are not:
- business logic flaws
- workflow bypasses
- human eyeball parsable errors.
Terminology: Functional versus Technical Issues
OWASP AppSec DC 2005
17
Functional versus Technical Issues
There are two main types of application security issues:
I.
Technical Vulnerabilities
1) Lack of Input Validation, resulting in:
2) XSS
3) SQL Injection
4) Parameter Tampering, etc.
II.
Functional or Logical Vulnerabilities
1) Click two or more buttons at the same time results in
unexpected behavior.
2) Password change form disclosing detailed errors.
3) Session-idle deconstruction not consistent with Policies
4) Spend deposit before deposit funds are validated (B/L)
OWASP AppSec DC 2005
18
Functional versus Technical Issues
These two types of security issues must be tested differently:
I.
Technical Vulnerabilities
− Usually about Data Handling
− Dealing with Strong Typing, Encoding, and Validation
of Data that the application handles
− Can be tested fairly effectively by automated tools (at
least, in theory, as tools mature)
II.
Functional or Logical Vulnerabilities
− Deals with issues allowed by design, but not foreseen
by the designers (or understood to be a risk)
− Deals with eyeball issues—does the error message text
provide a brain-decipherable “secret message”?
OWASP AppSec DC 2005
19
Two Items to Reiterate
Restated:
I.
Automation Tools
− Testing: useful; use daily
− Protection: maybe useful, maybe not (is your issue
XSS? Do you know what your issues are?)
− Anecdote of client who solved 90% of their issues
even though they didn’t know what they were.
− Anecdote of WAF vendor who “successfully deployed in
under an hour” for the java applet communicating on
TCP/5000 with an encrypted binary protocol.
II.
Examples of Vendor Challenges
− Sanctum Support Issues: XSS vs. Phishing debate
− Other vendors “Phishing” “XSS” “blahfoo” feature FUD
OWASP AppSec DC 2005
20
What is “Application Security”?
I.
Things that Application Security is not:
− It is not a product
− It is not a firewall with “AI” (app intelligence)
− It is not a NIDS/NIPS with app signatures
− It is not any “perimeter access control”.
− It is not a “web app firewall” widget.
− For today and tomorrow’s web applications
the concept of a perimeter is dead.
OWASP AppSec DC 2005
21
What is “Application Security”?
II. Things that Application Security is not:
− It is not a process (it is possible, however
unlikely, to have secure applications with
or without a process)
− It is not “application hacking school” or
“secure coding school” for your developers
− It is not the BPPT (Big Production
Penetration Test) you have at the end of
your BUFD app lifecycle.
OWASP AppSec DC 2005
22
What is “Application Security”?
III. Things that Application Security IS:
− It is one possible emergent Quality of a
well designed application.
− It is about predictability, dependability,
reliability. (Business speak !=“security”)
− Rob’s Report must Run Right (bugs are not
difference of kind but difference of degree
and the issue is always the same)
OWASP AppSec DC 2005
23
How do we evaluate applications for Security?
First and most important application
security principle:
The Application Must Defend Itself
OWASP AppSec DC 2005
24
Presentation Outline Updated
I.
Introduction (done)
II. Chapter One: The Problem (done)
III. Chapter Two: The Definition (done)
IV. Chapter Three: The Clarification and Goals
V.
Chapter Four: Taxonomy of Testing
VI. Chapter Five: Tools Tools Tools of Automation
VII. Chapter Six: Q&A (Shoot the Messenger)
OWASP AppSec DC 2005
25
Chapter III
Clarification and Goals
OWASP AppSec DC 2005
26
The Categorization Basics:




Software flaws with Security Implications
Category
Class
Particular
-Example-
1. Category (Coding Error)
2. Class (Output Encoding)
3. Particular (XSS (Cross-Site Scripting))
OWASP AppSec DC 2005
27
Are we trying to Categorize and Classify?
Clarification of Terms
Categorization of Issues
Abstract fixation from particulars
Needs: Category, Class, Particular
Needs Risk, Threat, Attack, Weakness,
Vulnerability
 Needs account for Severity, Attackability
(Ease), Prevalence, Priority





OWASP AppSec DC 2005
28
The Classification Basics:
Risk ((Loss|Impact) : Financial, Legal, Market)
Threat (Likelihood, Severity, Ease, Implication)
Attack (Exploit Vector)
Weakness (Fundamental Issue)
Vulnerability (Particular Issue/Flaw Instance)
OWASP AppSec DC 2005
29
The Classification Breakdown:
-Example One-
 Risk
 Financial Loss, Repudiation, Market Goodwill
 Threat
 Elevation of Privilege; Forged Transaction
 Attack (A: Spoofing of User Identity)
 A1 Session Fixation
 A1-2 Session Token\Cookie set a priori user authentication
 Weakness
 Insecure Authorization Mechanism; Insecure Session Handling
 Vulnerability
 Harvest Session Cookie on Login Form
OWASP AppSec DC 2005
30
The Classification Breakdown:
-Example Two Risk
 Financial Loss(???), Market Goodwill
 Threat
 Elevation of Privilege; Sensitive Information Disclosure
 Attack (A: Arbitrary Script Injection)
 A1 XSS/Cross-Site Scripting
 A1-2 Embedded/Persisted Malicious Javascript
 Weakness
 Insecure Output Encoding
 Vulnerability
 Support forum page allows embedded linking to external
javascript/flash/actionscript/etc.
OWASP AppSec DC 2005
31
Example Problems with Existing Threat Models:
Microsoft: STRIDE and DREAD
Spoofing of User Identity (Threat or Attack?)
Tampering with Data (Attack?)
Repudiation (Risk or Threat?)
Denial of Service (Threat or Attack?)
Elevation of Privilege (Threat)
STRIDE, DREAD and others mix of Threats,
Attacks, Vulnerabilities, disparate elements.
New TRIKE methodology shows promise.
OWASP AppSec DC 2005
32
More on Threat Modeling:
Threat Modeling is very important; I use it
extensively particularly in communicating to
business owners or stakeholders of an
application
For more see the OWASP Testing presentation
The OWASP Testing GUIDE
STRIDE & DREAD in MS Threat Modeling Book
TRIKE is a new Threat Modeling Whitepaper
Visit the Threats and Countermeasures Website
by Foundstone
OWASP AppSec DC 2005
33
How do we evaluate applications for Security?
First and most important application
security principle:
The Application Must Defend Itself
OWASP AppSec DC 2005
34
Conclusion to Chapter III
The previous material is presented to provide a common set of terms
and framework for understanding what we can and can’t do with
automated tools that discover software flaws with security
implications.
 Security is not a “feature” you implement in code
 Security is an emergent Quality of well-designed and well-built
software, much like speed and reliability
 Categorical difference between technical and functional issues and
how you discover them
 Tools tend to flag everything as “vulnerabilities”
 Tools really find a mix of threats, attacks, and vulnerabilities (XSS
is an attack, not a vulnerability)
 Tools tend not to indicate systemic weaknesses, like improper
output encoding techniques, which is what we really need to be
concerned with if our focus is *fixing* the application issues.
OWASP AppSec DC 2005
35
Chapter IV
Taxonomy of Testing
OWASP AppSec DC 2005
36
Taxonomy of Testing Categories
1.
2.
3.
4.
5.
6.
7.
8.
Vulnerability Scanning
Fault-Injection Testing (Blackboxing)
Fault-Injection Testing with Sandboxing
Binary Analysis
Source Code Analysis
Threat Modeling/Architectural Analysis
Rootkit and Trojan Analysis
(What about runtime analysis and
runtime patching?)
OWASP AppSec DC 2005
37
Taxonomy of Testing Categories
1. Vulnerability Scanning
 regex matches on version numbers.
2. Fault-Injection Testing (Blackboxing)
 Inject all possible faults and parse responses.
3. Fault-Injection Testing with Sandboxing
 Inject all possible faults and analyze data
storage, manipulation, and local entry and
exit points, and fuzz/fault-inject those points.
4. Binary Analysis
 Analyze static binary or perform runtime
analysis and look for breakpoints.
OWASP AppSec DC 2005
38
Taxonomy of Testing Categories
5.
Source Code Analysis

Mostly static signature matching, but some tools claim to walk
codepath, follow nested loops and other regression through
objects
6.
Threat Modeling/Architectural Analysis

Detailed, logical modeling. Time consuming, thorough; avoid
wasted man hours of broken feature-implementation up front.
Or find obvious design flaws quickly.
7.
Rootkit and Trojan Analysis

Why not? The NCUA/OCCU mandates it.
8.
What about runtime analysis and runtime patching? Another idea…
OWASP AppSec DC 2005
39
What Follows
The next series of slides is a short summary of common and well-known
applications that fall into these categories.
It is not intended to be comprehensive (later slides will be more
comprehensive).
It is not intended to indicate quality or imply any form of advocacy for
the listed software security tools.
If you are looking for product advocacy there are plenty of sales people
and “sales engineers” that can help you there. 
Most importantly: These tools are constantly evolving. Consider that the
data presented may already be outdated. Use as a guide only and
evaluate any selection you make Yourself, on Your Applications.
OWASP AppSec DC 2005
40
Vulnerability Scanning
1.
2.
3.
4.
5.
6.
7.
ISS Internet Security Scanner (the old basic VA tool)
eEye Retina (CGI Scanning?)
Qualys Qualysguard (XSS, limited SQLi)
NGS Typhon (claims full JS Parsing)
McAfee Foundscan (DNE for webapps)
nCircle IP360 (similar to Qualysguard)
Known-File Scanners e.g.-Whisker, Nikto,
Nessus+Nikto, Nstalker Nstealth
These tools do a regex version match (e.g. 12.8.1.0 diff vuln-database)
or simply match a filename (e.g. “login.asp”…Oh No!, not a Login!)
OWASP AppSec DC 2005
41
Fault Injection Testing
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
SPI Dynamics WebInspect 5.5
Watchfire AppScan 5.5
Acunetix WVS 2.x
Cenzic Hailstorm 2.6
NTO Spider x?
Compuware Devpartner x?
NGS Typhon x?
Parasoft Webking X?
SPIKE (while using binaries with attached debuggers to monitor
breakpoints like any traditional fuzzing)
Peach Fuzzer
These tools inject a fault into the application and in some cases evaluate
the response.
OWASP AppSec DC 2005
42
Fault Injection with Sandboxing
Sandboxing Tools:
• Security Innovation’s Holodeck
• Sysinternals filemon, regmon, procmon
• Any sandboxing malware-analysis tools
OWASP AppSec DC 2005
43
Binary Analysis
1.
2.
3.
4.
IDA Pro
@stake SmartRisk Analyzers
Fortify Application Risk Analyzer
Halvar Flake’s Sabre-Security projects
– BinNavi
– BinDiff
– BinAudit
5. WiSA
OWASP AppSec DC 2005
44
Source Code Analysis
1.
2.
3.
4.
5.
6.
7.
Compuware
Coverity
Fortify Software
Ounce Labs
Parasoft
Secure Software
Lots O’ Free & Open Source Stuff
OWASP AppSec DC 2005
45
Threat Modeling / Architectural Analysis
1.
2.
3.
4.
Microsoft Threat Modeling Tool
SecuriTree
PTA Technologies
TRIKE TM tool (to be released)
OWASP AppSec DC 2005
46
Rootkit/Trojan/Backdoor Analysis
Checking for Rootkits, Trojans, and Backdoors is another interesting issue
that is recommended in the “Secure Coding” pamphlet sold as a
book by O’Reilly, and recommended by the NCUA 2004-03 Guidance
letter “must review source code for backdoors, trojans, etc. etc.”
1. Rootkit and Binary analysis tools for
Backdoors & Trojans
OWASP AppSec DC 2005
47
Runtime Analysis and Patching
Tools that are designed to analyze behavior and code at runtime and
perform runtime limitation/security enforcement and/or “virtual”
patching of userspace coded and kernel code.
1. Immunix kernel & stack protection.
2. Some new virtual patching product for
Linux that I accidentally deleted the
info on and can’t find the emails of the
kind folks who contacted me originally.
OWASP AppSec DC 2005
48
Chapter V
Automation Tools
(or manual human eyeball supplicants)
YMMV
Key:
MA (Mature)
HP (Has Promise)
DNE (Did Not Evaluate)
RE (Refused Evaluation or ignored repeated requests)
TOY (Amusing little widget)
OWASP AppSec DC 2005
49
Vulnerability Scanning
1.
2.
3.
4.
5.
6.
7.
8.
9.
ISS Internet Security Scanner (???)
eEye Retina (CGI Scanning?)
Qualys Qualysguard (XSS, limited SQLI)
McAfee Foundstone Enterprise
nCircle IP360
Rapid 7 Nexus
NGS Typhon (full JS Parsing; XSS & SQLI)
Saint
Known-File Scanners e.g.-Whisker, Nikto,
Nessus+Nikto, Nstalker Nstealth
OWASP AppSec DC 2005
50
Commercial Fault Injection Test Tools
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
SPI Dynamics WebInspect (5.5, MA)
Sanctum now Watchfire AppScan (5.5 MA but may become a ‘dashboard’)
Cenzic Hailstorm (HP…ways to go, version 2.6 DNE)
NT Objectives NTOSpider (HP, DNE new GUI version)
Acunetix Web Vulnerability Scanner 2 (TOY, no scheduling, state, form-fill, etc.)
Compuware DevPartner Fault Simulator (RE)
Whitehats Security (hosted application, ‘dashboard’ approach, think ‘Qualys’)
Fortify Pen Testing Team Tool (RE, unknown; secretive)
Burp Intruder (HP, nice fuzzer)
MaxPatrol 7 (strange phone-home to Russia traffic observed, stopped eval)
Syhunt Sandcat Scanner & Miner (evolving)
TrustSecurityConsulting HTTPExplorer (TOY, limited)
Ecyware BlueGreen Inspector (TOY, limited)
NGS Typhon (DNE, claims full parsing/testing capability)
Parasoft WebKing (DNE, positive references from credible sources)
Kavado Scando (dead, IP purchased, may show up again, CVE-type scanner)
AppSecInc AppDetective for Web Apps (dead, believed permanently)
@stake Web Proxy 2.0 (dead or suffocating at Symantec)
Sandsprite Web Sleuth (HP, but development End-of-Life?)
OWASP AppSec DC 2005
51
Commercial Fault Injection Test Tools
Data on FI Tools:
Data includes version number as many of these tools have new
features/improvements since time of testing:
 Tools tested from March to August of 2005
 Tools tested on sample applications like OWASP WebGoat
 Tools tested on multiple real-world applications
 Tools tested on same versions of applications
 Tools tested on “default” settings and also attempted to
customize tools as best possible to provide optimal use-case
scenario for final result data
 Many tools were configured with help from or onsite support
from the vendors, all installed on clean, new boxes, so should
represent the best the vendor could provide at that time.
 I have been guilty of RTFM errors before, but for this testing this
is why vendor configuration support was solicited.
OWASP AppSec DC 2005
52
Commercial Fault Injection Test Tools
SPI Dynamics WebInspect 5.5
Pros:
 Best manual testing tools of the FI market (Proxy, Cookie Tester)
 Excellent javascript parsing
 Most powerful tools for writing custom checks
 Best API integration with other QA tools
 Have removed a lot of the marketing crap that other tools still
report like detecting “31 Flavors of XSS” and
Cons:
 No ability to look under the hood on adaptive agents
 No ability to customize default checks (to protect “IP”)
 Come on guys, I can run a sniffer and see what you are doing
anyway
 Not the fastest
OWASP AppSec DC 2005
53
Commercial Fault Injection Test Tools
Watchfire AppScan 5.5
Pros:
 Fast on smaller applications
 Best XSS testing
 Most powerful right out of the box with no tweaking
 Excellent reporting
 New free manual testing toolkit; ‘PowerTools’
Cons:
 No ability to look under the hood or customize effectively
 Unnecessary XSS variants, and made-up vuln terminology
 Watchfire is a dashboard company and this is not a effective way
to evaluate/measure application security
 Limited ability to tweak to handle dynamic session affinity and
complex authentication variants
OWASP AppSec DC 2005
54
Commercial Fault Injection Test Tools
Cenzic Hailstorm 2.3
Pros:
 Allow you to get under the hood and customize everything.
 Most potential for automating complex custom checks.
 Nice drag-and-drop configuration in GUI.
 Definitely a contender to keep an eye on.
Cons:
 Custom checks simply didn’t work right
 Slowest tool tested
 No ability to configure operational/engine parameters like
threading, bandwidth, scope, etc.
 At time of testing was long on hype, highest priced, low on
delivery of results
OWASP AppSec DC 2005
55
Commercial Fault Injection Test Tools
NT Objectives NTOSpider
1.something CLI-only beta version
Pros:
 Fast. Wow, this thing is fast.
 Most effective XSS checks behind WI and AS.
 Only tool with effective automated session token testing.
 Auto-state detection and auto-login abilities.
 Small, lightweight, simple configuration.
Cons:
 No CVE-style checks.
 No ability to look under the hood or customize tests easily.
 Roughly half the accuracy of WI or AS on XSS and SQLi.
 No GUI at time of testing.
OWASP AppSec DC 2005
56
Commercial Fault Injection Test Tools
Acunetix WVS
Pros:
 Nice, pretty GUI
 Appears fast, but it doesn’t do much so I guess it’s easy to be
fast at that.
Cons:
 Limited authentication support
 No scripting or scheduling capabilities
 Utter lack of session/state handling
 No manual analysis tools
 False positive centric
 Marketing appears targeted at sub-100 IQ demographic
 Included due to massive advertising blitz recently
OWASP AppSec DC 2005
57
Commercial Fault Injection Test Tools
Compuware DevPartner Fault Simulator
Pros:
 Built with the Developer in mind
 IDE integration
 Integrates with Source-Code analysis
 May be one to keep an eye on but looks like not intended to be
“best of breed” solution.
Cons:
 Vendor has no clue about competitive landscape.
 Vendor appears to have limited security knowledge.
 Did Webex eval only.
 Got tool but not license key; vendor promised follow up
repeatedly but never did.
 Vendor to provide data for the OWASP Tools project but never
followed through in providing this.
OWASP AppSec DC 2005
58
Commercial Fault Injection Test Tools
WhiteHat Security (the Seattle Company)
Hosted FI Tool; think Qualys of WebAppSec
Pros:
 DNE (Did Not Evaluate)
 Some smart people building this
 Hosting approach nice for repeated testing/change management
 Cost effective
Cons:
 No replacement for manual testing
 DNE
OWASP AppSec DC 2005
59
Commercial Fault Injection Test Tools
NGS Typhon
Pros:
 DNE for Web Application Checks
 Experience with NGS tools is that they are very fast, fairly
accurate, lack ability to customize/configure much through the
GUI and have terrible reporting. 
 Will include in next release of this project.
Cons:
 DNE.
OWASP AppSec DC 2005
60
Commercial Fault Injection Test Tools
Parasoft WebKing
Pros:
 DNE
 Have heard has is less mature but some nice features
 Will include in next version of project
Cons:
 DNE
OWASP AppSec DC 2005
61
Commercial Fault Injection Test Tools
SandSprite Web Sleuth
Pros:
 Inexpensive
 Simple easy-to-use Windows GUI
 Nice code markup and views
 Custom scripts for crawling, cookie testing, etc.
Cons:
 Geared towards all manual testing
 No default pre-built checks
 Doesn’t appear under active development
OWASP AppSec DC 2005
62
Commercial Fault Injection Test Tools
OWASP WebScarab v.20051012 
Pros:
 Many advanced testing features geared towards experienced
application security professionals.
 Free & Open Source.
 Session Token entropy analysis.
 Static string fuzzer.
 User-extensible with custom agents.
 Web Services testing interface for complex types
Cons:
 Non-intuitive, complex interface.
 No “scan” or pre-built checks like CVE issues.
 Tends to hang and need restarted.
 No vulnerability reporting mechanism or management style
reporting a’la Watchfire.
OWASP AppSec DC 2005
63
Fault Injection Test Tools Anecdotes
 Similar to diet fads
 My Book “Body of Anecdotal Evidence:
Portion-sized Marketing” out soon
 That was a “Body for Life” joke for you not
from the US
 Again, there is no book for those of you that
asked me after the presentation.
 YMMV
 Anecdotes are subjective
 Anecdotes are not logically valid proofs
 Repeat Disclaimer: YMMV
OWASP AppSec DC 2005
64
Fault Injection Tool Experiences
The following metrics were gathered from realworld assessments performed by myself
along with up to four other testers.
These were commercial assessments on a mix
of COTS software and custom applications
developed by clients ranging from Fortune
100 to smaller ISVs that authorized
benchmarking of these tools during the
course of software security assessment.
OWASP AppSec DC 2005
65
Fault Injection Tool Experiences
Application #1:
 150+ pages, Complex workflow
 Dynamically built javascript menu/functions
 Application contained 1 trivial XSS, 1 Workflow Bypass (WFB),
1 Brute Force (BF)
 The tools in this test were run with default configurations;
essentially clicked ‘scan’ as many people unfortunately use
these tools in RealLife™.
WebInspect 5.0: 147 pages, no XSS, WFB, BF
AppScan 5.5: 38 pages, 1 XSS, no WFB, BF
Acunetix 2: 6 pages, no valid positive results
NTO Spider: 80+ pages, no XSS, WFB, BF
Paros 2.something: 0 (zero) (loop)
Cenzic Hailstorm: Couldn’t get working
(The security defects detected in the applications were discovered during the course of manual analysis)
OWASP AppSec DC 2005
66
Fault Injection Tool Experiences
Application #2:
 30+ templates, Dynamically generated content
 15k+ actual pages, Long forms, simple workflow
 32 XSS from URL parameters to multi-form <--!-->
 Weak Session Handling; Auto-scripting attacks (“Session
Riding” or “Web Trojans”)
 Tests run as ‘scan’ with minimal tweaking
WebInspect 5.0: 31 XSS, no SH or SR issues
AppScan 5.5: 18k+ pages, slowed to 1 rec/6 seconds
Acunetix 2: No Results (couldn’t automate effectively)
NTO Spider: 18 XSS, weak SH, no SR, fast! (CLI version)
Hailstorm 2.2: 3 days, 4 people, Cenzic onsite, nothing
Paros 3.0.2: 18 pages, no results, would hang on audit
OWASP AppSec DC 2005
67
Fault Injection Tool Experiences
Application #3:

100’s of pages/forms

Strong UI, Input Validation, Output Encoding

Web Server packages XML POST string

Passes to routing servlet for authorization

Function Servlet name/route in Hidden FF

Create XML String and POST directly to DST servlet

XML Abstraction Layer has no authentication/authorization

ASMQ does not have crypto enabled; auth is function group code

Point of this example is that there are holes you could drive a Mac Truck through
and every FI Scanner failed to detect them. Some things require humans.
WebInspect 5.5: Nothing
AppScan 5.5: Nothing; WF PowerTools Proxy: failed
Acunetix 2: PHP issue, some other issues, false positives (200 codes)
NTO Spider: Nothing (CLI version); no proxy
Paros 3.0.2: Nothing; Proxy-mode: failed
Proxies: SPI Proxy (only proxy to handle various cert-auth)
OWASP AppSec DC 2005
68
Fault Injection Tool Experiences
Application #4:
 AAWBDMS (Another Awful Web-Based DMS)
 XSS, Parameter Tampering, etc. (everything you can imagine
controlled in client-side hidden form-field parameters)
 URL Parameters passed directly to compiled C code (DLLs, EXEs)
running as Domain Admin with traditional buffer overflows.
 Dual-Factor Authentication
 Authentication Two-step Workflow
 First page UserID & PRNG Token number in HTML Form
 Second Page identical UserID and Password in HTML Form
 All responses 200; everything returns one of two login forms
Every darn tool: NADT
NADT: Not a Darn Thing
Nobody could handle dual-auth with custom error pages identical to
the login pages.
OWASP AppSec DC 2005
69
Fault Injection Tool Experiences
XSS attacks should be fully automated, but as of yet are not. We find XSS in
almost every application we test that is not found by any automated tool;
some are trivially devastating. Examples:






Form
Form
Form
Tools
value expects email address (arian.evans@fishnetsecurity.com)
value validated by sloppy regex; Output not encoded
error returns no data unless you bypass regex filter
submit the following types of checks:





<script>alert(‘somethingclever’);<script>
<script>alert(‘somethingclever’);<script>silly@domain.com
silly@domain.com<script>alert(‘somethingclever’);<script>
some replace ‘silly’ and ‘com’ with <script>alert();<script> check
some add escape characters like ‘>, ‘>>, ‘), and so on



encoding attack strings (reference OWASP Guide 2.0.1 or RSnake’s XSS page)
partial encoding of attack string elements
using specific HTML tags like body elements and background
Actual XSS by silly@<script>alert()<script>.com
Other attack vectors tools fail include
OWASP AppSec DC 2005
70
Open Source or Freeware Fault Injection Test Tools
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
WebScarab (DNE current version, re: HTTPush, Exodus)
Paros Proxy (HP)
Watchfire PowerTools (HP, free, proxy nice GUI, rest so-so)
Burp Spider (HP, free)
Burp Proxy (HP, free)
Snark HTTP Proxy & Tabby Tunnel (DNE)
Peach Fuzzer (HP, free, Python fuzzer)
SPIKE Proxy (Horrible interface, free, limited fuzzing)
SPIKE Fuzzer (dev EOL? Dave pointing people at Peach)
Achilles Proxy (TOY, first Windows proxy, free, old)
Odysseus Proxy (some cool features, GUI needs work)
Webstretch Proxy (DNE)
Absinthe 1.1 (formerly SQLSqueal) (HP, free)
Sensepost E-Or, Wikto (Google hacking utility), other
webappsec and pen testing tools
(everything Sensepost is working on is worth looking at)
OWASP AppSec DC 2005
71
Open Source or Freeware Fault Injection Test Tools
And your Basic Web Browsers…don’t forget them:
15.
16.
17.
Internet Explorer + HTMLBar DLL extension (buggy)
Firefox + LiveHTTP Headers, Tamper Data, Developer Tools,
and many, many, more useful extensions….however,
keep in mind Firefox encoding is often different, more
restrictive, sometimes more secure than IE so if the
application client population includes/is predominantly
IE MAKE SURE you test encoding/format in IE as well.
Foundstone Free Tools:

SiteDigger (Google Hacking Utility)

SSLDigger

Probably others I haven’t looked at yet

SiteGenerator (coming soon, may be useful for testing
Fault-Injection tools for javascript parsing, etc.)
OWASP AppSec DC 2005
72
Fault Injection with Sandboxing
Sandboxing Tools (Commercial & Free):
1. Security Innovation’s Holodeck (HP)
2. Sysinternals filemon, regmon, procmon
can be combined with VMWare
3. Any sandboxing malware-analysis tools
available from various forensics sites
and possibly from the AV vendors.
OWASP AppSec DC 2005
73
Binary Analysis
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
IDA Pro + scripts from Halvar & iDefense
Grammatech CodeSonar
Paradyn Project (claims superiority to IDA Pro when C++ variables present in object)
FX Dumbug
@Stake SmartRisk Analyzers (gone/dead)
Fortify Application Risk Analyzer (marketing freebie)
WiSA (Wisconsin Safety Analyzer)
SABRE BinDiff (IDA Pro extension)
SABRE BinNavi (Graphical flowgraphs, playback of codepaths, Win/Linux debuggers)
SABRE BinAudit (Full Binary Analysis; beta project; Halvar restarted this again)
Secure Software CodeAssure Auditor (Binary Analysis)
Brute Force Binary Checker (http://bfbtester.sourceforge.org)
OWASP AppSec DC 2005
74
Commercial Source Code Analysis
1.
2.
3.
4.
Compuware DevPartner SecurityChecker .NET Languages
Coverity SWAT C/C++
Escher Technologies Eschertech
Fortify Software Suite (analysis, workbench, metrics & trending
console, customization module, won’t show me anything)
C/C++,Java,JSP,HTML,Java bytecode,(PL/SQL),C#,VB.NET,MSIL, …
5. Gimple PC and Flexe-Lint C/C++ (we use, simple, cheap, functional)
6. Grammatech CodeSurfer C/C++
7. Ounce Labs Prexis C/C++, JSP, Java (nice, rough UI, recursion abilities?)
8. Parasoft JTest Java (? NE)
9.
Reasoning SecurityInspect “Service”
10. Secure Software CodeAssure Workbench C/C++, Java
11. Security-Assessments CodeScan ? (April 01 05) won’t give specifics
12.
13.
Note if you are in sales: My name is not Ariel, Arial, Adrian, Aruin, etc
WiSA
C/C++
OWASP AppSec DC 2005
75
Open Source, Free, or Bundled Source Code Analysis
1.
2.
OWASP .NET (Reference Dinis Cruz presentations on OWASP
website about OWASP .NET tools)
Foundstone .NET Tools (same story as above; DNE but
heard these were coming out soon)
The Next Slide has All The Other Stuff
OWASP AppSec DC 2005
76
Open Source, Free, or Bundled Source Code Analysis
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
Bunch http://serg.cs.drexel.edu/bunch
Cigital ITS4 (no longer supported) C/C++
CQual http://www.cs.berkeley.edu/~jfoster/cqual
David Wagner’s BOON C http://www.cs.berkeley.edu/~daw/boon/
David Wheeler’s Flawfinder C/C++ www.dwheeler.com
Reference/Credit David Wheeler’s Page
Microsoft PREFix (full codepath execution, recursion), C families
Microsoft PREFast (stripped down PREfix) only static signatures for
performance reasons, shipping with Visual Team Studio 2005, C families
Microsoft FXCop (even more limited signature tool) C/C++ only
MOPS http://www.cs.berkeley.edu/~daw/mops
PScan http://www.striker.ottawa.on.ca/~aland/pscan
RAT C/C++, Java, Perl, Python, whatever other random signatures
written
ShareFuzz
SPLINT http://splint.org/
OWASP AppSec DC 2005
77
Threat Modeling / Architectural Analysis
1. Microsoft Threat Modeling Tool
(non-intuitive for beginners)
2. SecuriTree
(clunky Java tool, not appsec focused)
3. PTA Technologies
(DNE, have had several references)
4. TRIKE Threat-Modeling Tool
(DNE, in development stage, unsure release
date, but idea has promise)
Need more tools here!
OWASP AppSec DC 2005
78
Rootkit/Trojan/Backdoor Analysis
Checking for Rootkits, Trojans, and Backdoors is another interesting issue
that is recommended in the “Secure Coding” pamphlet sold as a
book by O’Reilly, and recommended by the NCUA 2004-03 Guidance
letter “must review source code for backdoors, trojans, etc. etc.”
1. Rootkit and Binary analysis tools for
Backdoors & Trojans
rootkits.org and rootkit.nl + many Forensic
Distributions have these tools
OWASP AppSec DC 2005
79
Recommended Motto for Vendors
Ne Supra Crepidum
Suter Judicaret
OWASP AppSec DC 2005
80
Chapter 6
Shoot the Messenger
(he may duck and run)
OWASP AppSec DC 2005
81
Arian J. Evans
Does Application Security Stuff
FishNet Security
913.710.7085 [mobile]
arian.evans@fishnetsecurity.com
OWASP Tools Project
OWASP Chapter Leader, Kansas City
Feel free to contact me with questions
Vendor updates, corrections, confusions, or complaints welcome
Email me repeatedly if I miss your first one
OWASP AppSec DC 2005
82