An Industrial Case Study of Bypass
Testing on Web Applications
Jeff Offutt
Software Engineering
George Mason University
Fairfax, VA USA
www.cs.gmu.edu/~offutt/
offutt@gmu.edu
Expanded version of a talk given at the first
International Conference on Software Testing,
Verification and Validation
Joint research Dr. Ye Wu, Xiaochen Du, Hong Huang, Vasileios Papadimitriou,
Qingxiang Wang and Joann J. Ordille of Avaya Labs Research
Based on papers in ISSRE 2004 and ICST 2008
Outline of Talk
• Motivation
• Applying bypass testing – early
results
• Automating bypass testing
• Real-world examples
• Industrial case study
• Conclusions and future work
ICST 2008
© Jeff Offutt
2
Web Application Input Validation
Check data
Check data
Sensitive
Data
Bad Data
• Corrupts data base
• Crashes server
• Security violations
Client
Malicious
Data
Can “bypass”
data checking
Server
Deploying Software
• Bundled : Pre-installed on computer
• Shrink-wrap : Bought and installed by end-users
• Contract : Purchaser pays developer to develop and
install, usually for a fixed price
• Embedded : Installed on a hardware device, usually
with no direct communication with user
• Web : Executed across the Internet through HTTP
© Jeff Offutt, 2004
4
Problem Parameters
•
HTTP is a stateless protocol
–
•
•
Each request is independent of previous request
Servers have little information about where a
request comes from
Web site software is extremely loosely coupled
–
–
–
ICST 2008
Coupled through the Internet – separated by space
Coupled to diverse hardware devices
Written in diverse software languages
© Jeff Offutt
5
Bypass Testing
• “bypass” client-side constraint enforcement
• Bypass testing constructs tests to intentionally
violate constraints :
–
–
–
–
ICST 2008
Eases test automation
Validates input validation
Checks robustness
Evaluates security
© Jeff Offutt
6
Simple Example Web Page
User Name:
Age:
Version to purchase:
ICST 2008
Small
Medium
Large
$150
$250
$500
© Jeff Offutt
7
Proper Behavior
Invalid data, please correct …
User Name: Alan<Turing
Username should be
plain text only.
Age: 500
Age should be
between 18 and 150.
Version to purchase:
ICST 2008
Small
Medium
Large
$150
$250
$500
© Jeff Offutt
8
Abbreviated HTML
<FORM >
<INPUT Type=“text” Name=“username” Size=20>
<INPUT Type=“text” Name=“age” Size=3 Maxlength=3>
<P> Version to purchase:
…
<INPUT Type=“radio” Name=“version” Value=“150” Checked>
<INPUT Type=“radio” Name=“version” Value=“250”>
<INPUT Type=“radio” Name=“version” Value=“500”>
<INPUT Type="submit" onClick="return checkInfo(this.form)">
<INPUT Type=“hidden” isLoggedIn=“no”>
</FORM>
ICST 2008
© Jeff Offutt
9
Bypass Behavior
• Extremely loose coupling …
• combined with the stateless protocol …
• allows users to easily bypass client-side checking :
Users can save and modify the HTML
ICST 2008
© Jeff Offutt
10
Saved & Modified HTML
<FORM >
<INPUT Type=“text” Name=“username” Size=20>
<INPUT Type=“text” Name=“age” Size=3 Maxlength=3>
<P> Version to purchase:
…
Allows an input with arbitrary age,
no checking, cost=$25 …
‘<‘ can crash an XML parser
<INPUT Type=“radio” Name=“version” Value=“150”>
Text fields can have SQL statements
<INPUT Type=“radio” Name=“version” Value=“250”>
<INPUT Type=“radio” Name=“version” Value=“500”
25 Checked>
<INPUT Type="submit" onClick="return checkInfo(this.form)">
yes >
<INPUT Type=“hidden” isLoggedIn= “no”
</FORM>
ICST 2008
© Jeff Offutt
11
SQL Injection
User Name: turing’ OR ‘1’=‘1 Password: enigma’ OR ‘1’=‘1
Original SQL:
SELECT username FROM adminuser WHERE
username='turing' AND password ='enigma'
“injected” SQL:
SELECT username FROM adminuser WHERE username='turing‘
OR ‘1’ = ‘1’ AND password ='enigma‘ OR ‘1’ = ‘1’
© Jeff Offutt, 2004
12
Applying Bypass Testing
Validating input data on the client is like asking
your opponent to hold your shield in a sword fight
• Analyze HTML to extract each form element
• Model constraints imposed by HTML and
JavaScript
• Rules for data generation :
– From client-side constraints
– Typical security violations
– Common input mistakes
ICST 2008
© Jeff Offutt
13
Example Client-Side Constraint
Rules
• Violate size restrictions on strings
• Introduce values not included in static choices
– Radio boxes
– Select (drop-down) lists
•
•
•
•
Violate hard-coded values
Use values that JavaScripts flag as errors
Change “transfer mode” (get, post, …)
Change destination URLs
© Jeff Offutt, 2004
14
Example Server-Side Constraint
Rules
• Data type conversion
• Data format validation
• Inter-field constraint validation
• Inter-request data fields (cookies, hidden)
© Jeff Offutt, 2004
15
Example Security Violation Rules
Potential Illegal Character
Symbol
Empty String
Commas
,
Single and double quotes
’ or ”
Tag symbols
Tag symbols < and >
Directory paths
.. ../
Strings starting with forward slash
/
Strings starting with a period
.
Ampersands
&
Control character
NIL, newline
Characters with high bit set
254 and 255
Script symbols
<javascript> or <vbscript>
© Jeff Offutt, 2004
16
First Example CyberChair
•
CyberChair : Web-based conference management
–
–
•
www.cyberchair.org
ICSE , ICST, ISSRE, ICSM, …
Bypass testing found 5 types of faults
1.
2.
3.
4.
5.
Submission without authentication
Unsafe use of hidden form field
Disclosing information (program crashes)
Lack of validation of file type
Allows papers of negative length
© Jeff Offutt, 2004
17
Automating Bypass Testing
• Autobypass : A web application that accepts a URL
and generates input data for the HTML form fields
– Also accepts any needed login data
– MS thesis by Vasileios Papadimitriou
• Built on top of HttpUnit
–
–
–
–
ICST 2008
Parses HMTL pages
Identifies forms and their fields
Creates bypass test cases
Submits test cases to the application’s server
© Jeff Offutt
18
Types of Client Input Validation
• Client side input validation is performed by HTML form
controls, their attributes, and client side scripts that access
DOM
• Validation types are categorized as HTML and scripting
– HTML supports syntactic validation
– Client scripting can perform both syntactic and semantic validation
ICST 2008
HTML Constraints
Scripting Constraints
• Length (max input characters)
• Value (preset values)
• Transfer Mode (GET or POST)
• Field Element (preset fields)
• Target URL (links with values)
• Data Type (e.g. integer check)
• Data Format (e.g. ZIP code format)
• Data Value (e.g. age value range)
• Inter-Value (e.g. credit # + exp. date)
• Invalid Characters (e.g. <,../,&)
© Jeff Offutt
19
Example Interface:
yahoo registration form
Preset Transfer Mode
in form definition (HTML)
Preset Values (HTML)
Limited Length (HTML)
URL with preset Values
(HTML)
Inter Value
validation (script)
Data Value, Type, & Format
validation (script)
ICST 2008
© Jeff Offutt
Preset No of Fields
(HTML)
20
Test Value Selection
• Challenge:
– How to automatically provide effective test values?
• Semantic Domain Problem (SDP)
– Values within the application domain are needed
– Enumeration of all possible test values is inefficient
• Possible Solutions
–
–
–
–
Random Values (ineffective – lots of junk)
Automatically generated values (very hard)
Taking values from session log files (feasible but incomplete)
Tester input (feasible)
• AutoBypass uses an input domain created by parsing the
interface and tester input
ICST 2008
© Jeff Offutt
21
AutoBypass
• AutoBypass steps (the big picture)
Parse
Interface
Set Default
Values
Generate
Test Cases
& Run Tests
Review
Results
• All HTML violation rules are used to generate test cases
• First version of AutoBypass does NOT automatically violate
scripting validation, but :
– AutoBypass behaves as a browser with scripts disabled
– Tester can provide test inputs that will bypass scripting validation.
ICST 2008
© Jeff Offutt
22
Real-World Examples
atutor.ca
nytimes.com
Atalker
Us-markets
demo.joomla.or
mutex.gmu.edu
Poll, Users
Login form
phpMyAdmin
Main page,
Set Theme,
SQL Query,
DB Stats
yahoo.com
Notepad, Composer,
Search reminder,
Weather Search
barnesandnoble.com
bankofamerica.com
ATM locator, Site search
comcast.com
Service availability
ecost.com
Detail submit,
Shopping cart control
google.com
Froogle, Language tools
Pure Book
black-box
pageflakes.com
search/results testing
Submit Request
Info, New user
Registration
means
amazon.com
myspace.com
wellsfargolife.com
Item dispatch,
no
source
(or
permission)
needed !
Events & Music
brainbench.com
Search
ICST 2008
Cart manager,
Handle buy
© Jeff Offutt
Quote search
23
Classifying Output Responses
• (V) Valid Responses : invalid inputs are adequately processed by the
server
(V1)
Server acknowledges the invalid request and provides an explicit
message regarding the violation
(V2)
Server produces a generic error message
(V3)
Server apparently ignores the invalid request and produces an
appropriate response
(V4)
Server apparently ignores the request completely
• (F) Faults & Failures : invalid inputs that cause abnormal server
behavior (typically caught by web server when application fails to
handle the error)
• (E) Exposure : invalid input is not recognized by the server and
abnormal software behavior is exposed to the users
ICST 2008
© Jeff Offutt
24
Results
v
ICST 2008
© Jeff Offutt
25
Research to Practice
Naw - I already
don’t plow as good
as I know how...
They’re teaching a new way
of plowing over at the Grange
tonight - you going?
“Knowing is not enough, we must apply. Willing is not enough,
we must do.” Goethe
ICST 2008
© Jeff Offutt
26
Industrial Case Study
• Inventions from scientists are slow to move into
industrial practice
• We wanted to investigate whether the obstacles are :
– Technical difficulties of applying to industrial use
– Social barriers
– Business constraints
• Tried to technology transition bypass testing to the
research arm of a software company
ICST 2008
© Jeff Offutt
27
Avaya’s NPP Technology
• Avaya Labs Research creates research prototypes of
software systems, then turns successful prototypes
over to product groups
• NPP : Notification Preference Portal
– Users specify how and when they should be contacted
– Types include phone, email and SMS
– Contacts an be made in parallel or sequentially
• Used to notify users of events
• This study was part of system testing
– NPP is now in production
ICST 2008
© Jeff Offutt
28
NPP Design and Implementation
• NPP is a highly user interactive web application
• Uses many screens
• Javascript is used on the client to :
– Validate inputs
– Dynamically modify screen by manipulating DOM
– Encode input data into XML before sending to the server
• The second two uses necessitated changes to how
bypass testing was applied
ICST 2008
© Jeff Offutt
29
NPP Bypass Tests
• Bypass testing analyzes HTML (statically) and
generates inputs that violate input constraints
• The extensive modification of the HTML DOM
meant HTML could not be analyzed statically
• Instead, the input requirements of the server
software were identified, and tests encoded in XML
– A special-purpose tool was written to convert XML tests
into HtmlUnit tests
ICST 2008
© Jeff Offutt
30
NPP Testing Results
• Six NPP screens were tested
• Tests are invalid inputs – exceptions are expected
• Effects on back-end were not checked
– Failure analysis just based on response screens
Web Screen
Tests
Failing Tests
Unique Failures
Points of Contact
42
23
12
Time Profile
53
23
23
Notification Profile
34
12
6
Notification Filter
26
16
7
5
1
1
24
17
14
184
92
63
Change PIN
Create Account
TOTAL
ICST 2008
© Jeff Offutt
31
Types of Faults
• Invalid data saved into the database
– These lead to more visible failures later
– Example : An invalid password was accepted, but the
account could not subsequently be used
– Example : Subsequent messages could not be sent to
invalid contacts
• No response at all
– Probably a software component failed
– Database or web server sometimes crashed
• Exposure errors
– Internal exception message sent in the response screen
ICST 2008
© Jeff Offutt
32
Conclusions
• Bypass testing worked very well in an industrial
context
– There is no technical obstacle to adoption
– Source is not needed
• Even hand generation of tests was quite cheap in
comparison with other methods
– There is no valid business barrier
• Most problems are unlikely with non-malicious
users and a correct implementation
– But client-side validation is notoriously error-prone
We conclude the primary obstacle is social
ICST 2008
© Jeff Offutt
33
Future Work
• A major observability problem with web application testing
is detecting invalid database values
– A comprehensive valid data model could allow database auditors
to be developed
• Javascript needs to be fully parsed and analyzed
– Implement scripting violation rules
• Widen the scope of testing from a form/ to a site
– Test sequence of events
– Application level Input Domain
• Explore possibilities for automated response evaluation
• Ajax allows client-server messages to be sent
asynchronously through message passing
– This introduces more controllability and observability problems
ICST 2008
© Jeff Offutt
34