Evaluation of Security Scanners for
Web Application
Presented By:
Sunint Kaur Khalsa (100875000)
Sarabjeet Kaur Saini(6235987)
Outline






Context
Goal and Scope of Study
Methodology
Evaluation Criteria
Evaluation of Candidate Tools
Conclusion and Recommendation
News…

Harvard Website attacked by Syrian Protesters

77 US Law Enforcement Websites hit in mass attack by
“LulzSec” hacking group.

The website of World’s most popular Martial Arts Organisation
“Ultimate Fighting Championship” hacked

…
Solution…
Firewall ?
Blue Crystal Inc.

Web Application Development firm with a Work Force of 15
people

Develop web applications based on .Net Platform

Incepted the idea of giving security services to their clients
after selecting a suitable tool

Wanted a tool with high functionality, low cost, low resource
consumption and high vulnerability detection
Goal and Scope of Study

Goal


Select the most suitable tool for Blue Crystal as per their
given requirements.
Scope

To conduct the evaluation of selected tools on the basis of
High Impact and Low impact criteria.
Methodology Used




Test Cases for the Evaluation
 Test websites provided by the vendors are used
Score given to each tool on the scale of 0-10 for the
corresponding evaluation criteria
Weights have been assigned to the evaluation criteria
Final score =
Where i= Evaluation Criteria
wi = Weight of ith evaluation criteria
si= Score of the tool corresponding to the ith evaluation criteria
High Impact and Low Impact Criteria
Weighing Scheme
Evaluation Criteria
Low Impact
Criteria
Ease of Installation
3
Usability
3
Scan Control Capability
3
Reporting and Documentation
3
Evaluation Criteria
High Impact
Criteria
Crawling and Parsing
5
Vulnerability Identification
5
Performance
4
Cost and License
5
Tools Selected

Rational Appscan




A Product of IBM
Originally developed by Sanctum Ltd.
First released in 1998
HP WebInspect


A Product of HP
Originally developed by SPI Dynamics
Test Websites
Tool
AppScan
Host
http://demo.testfire.net
WebInspect http://zero.webappsecurity
.com
Web
Pages
Operating
System
Web Application
Server Language
34
Win32 –
Windows XP
IIS
ASP.NET
100
Win32 –
Windows XP
IIS
ASP.NET
Ease of Installation




This criterion considered the ease of acquisition and
installation of the tool
Rational Appscan had a file of size 497 MB and took 5 hours
for its installation
HP WebInspect took 2 hours for the installation of 641 MB file
but we had to wait for 6 hours to get the key as that required
domain verification.
WebInspect also required SQL server and there is no such
requirement for Appscan
Appscan = 8 WebInspect = 6
Usability




Usability Criterion is a combination of
 Ease of use
 Efficiency
AppScan takes screenshots of the browser responses
corresponding to the generated attacks
AppScan provides in depth description of the detected
vulnerabilities including possible causes, technical description
and fixing recommendation whereas WebInspect provides only
recommendations
WebInspect creates macros to record testing steps during scan
and automate repeated testing
Appscan = 9 WebInspect = 8
Usability…
Usability…
Usability…
Scan Control Capability



Evaluated the scan control capabilities of both the tools to find
which tool is better for handling the scan.
Both tools provide operator with the ability to
 Pause a scan
 Restart the scan at a later time
Both tools provides the viewing the real-time status of running
scans. This status could include information such as which tests
are currently being run and the scan completion percentage.
Appscan = 9 WebInspect = 9
Reporting and Documentation

This criterion evaluates the tool on the basis of



Generation of reports in different formats
Comprehensiveness of the generated reports
Appscan can generate different types of reports





Security Report
Industry Standard Report
Regulatory Compliance Report
Delta Analysis Report
Template Based Report
Reporting and Documentation

Features of Appscan’s Report



Report was divided into different sections based on the URLs, where
vulnerabilities have been encountered.
Reports consisted of tables, text and graphs and hence more readable
and understandable
The reports by WebInspect comprised of a lot of text with
definitions and explanation and less of graphs, tables.
Appscan = 10 WebInspect = 8
Report Generation in AppScan
Report Generation in WebInspect
Crawling and Parsing




Crawling is an activity by which the scanner browses
various web elements like cookies, forms, parameters,
links etc looking for vulnerabilities
Parsing is defined as crawling for the various types of
contents like HTML, ActiveX objects, Java Applets, Java
Scripts, XML etc
Both the tools have automated crawling
In manual configuration, user is given the option



Specifying a request delay,
Maximum crawl depth
Have concurrent sessions
Crawling and Parsing



WebInspect has a feature which shows the steps the
scanner took to reach a specific vulnerability, pointing to
the specific element.
It is good if we want to retest certain flaws and to see how
the scanner is working on it
WebInspect gives the feature to specify the request delay
which is of interest to Blue Crystal Inc. as it might help
them to use the bandwidth wisely
Appscan = 9 WebInspect = 10
Vulnerability Assessment



This criterion evaluates the total vulnerabilities which have
been found by the web scanners on their respective test cases.
In order to find the vulnerabilities on the test websites the
number of attacks sent by AppScan 18,634 on 34 pages as
compared to 19,968 on 100 pages of WebInspect.
With three times the size of the test website WebInspect
generates less attacks and this results in exposing less
vulnerabilities.
Vulnerability Identification


Appscan exposed 120 vulnerabilities as compared to 272
vulnerabilities exposed by WebInspect. Here it is worth
mentioning that the size of WebInspect’s test case is thrice
as that of Appscan’s test case.
The various types of attacks detected by both the tools are




SQL Injection
Cross Site Scripting
Buffer Overflow
File guessing
Etc…
Appscan = 9 WebInspect = 7
Performance




This criterion covers the time in which the tool completes the scan and
the resources utilized during the scan
Appscan completed the scan of website with 34 pages in 31 minutes
where as WebInspect completed the scan of 100 pages in 15 minutes
showing the better performance of WebInspect
The minimum system requirements of Appscan are
 2.4GHz processor
 2GB RAM
 30GB of free disk space
The minimum system requirements for WebInspect are
 1.5GHz processor
 2GB of available RAM
 10GB of free disk space
Appscan = 7 WebInspect = 8
Cost and License

Cost = Training cost + License Cost
WebInspect Annual Audit License:
This licence type allows
$ 20,000
access to client’s partner portal (They have the ability to scan
unlimited customers on any IP in their environment) + Annual
maintenance + customer support + access to daily updated
vulnerability checks + Additional Overhead for each additional
user
IBM Rational App Scan Standard Edition + SW Subscription & Support
12 Months

$19,700
The Training cost is considered the same for both the tools as
both of them have online tutorials and quick start up kits.
Appscan = 8 WebInspect = 7
Score Earned by each Tool
12
10
8
6
4
2
0
Rational Appscan
HP WebInspect
Total Score of each Tool
Evaluation Criteria(i)
Weight (wi)
AppScan(si)
WebInspect(si)
Ease of Installation
3
8
6
Usability
3
9
8
Scan Control Capability
3
9
9
Reporting and
Documentation
3
10
8
Crawling and Parsing
5
9
10
Vulnerability Identification
5
9
7
Performance
4
7
8
Cost and License
5
8
7
266
245
Total Score
Conclusion and Recommendation



Rational AppScan is a clear winner and hence a better tool to
fulfill the requirements prescribed by Blue Crystal Inc.
Number of attacks sent by AppScan were more as compared to
WebInspect for exposing the vulnerabilities in the test website.
AppScan provides in depth description of the detected
vulnerabilities including possible causes, technical description
and fixing recommendation whereas WebInspect provides only
recommendations, required from development point of view.
References



http://www.ibm.com/software/awdtools/appscan/,
http://welcome.hp.com/country/us/en/prodserv/software.ht
ml
http://en.wikipedia.org