Testing Web Applications
Willem Visser
RW334
Overview
• Testing Theory
• Testing Web Apps
• Tools
The Basics
• A test consists of
– Input
– Expected output (also called the Oracle)
• White Box Testing
– Considering the code when coming up with tests
• Black Box Testing
– Only consider the input-output interface
• Automated Testing is the Holy Grail
Types of Testing
•
•
•
•
•
Unit Testing
Integration Testing
Systems Testing
Regression Testing
Acceptance Testing
– Alpha/Beta Testing
– Usability Testing
• Performance Testing
• Robustness Testing
• Security Testing
Lower Level Code Testing
• Unit Testing
– Test individual components of the code
– Can be as little as one function
– White box
• Integration Testing
– Interface between components
– White Box
– Can be as much as the whole system (which case
it becomes more black box)
• Code Coverage as Test Adequacy Measure
Functional Testing
• Testing that the systems meets the
requirements
• Systems Testing
– In the deployed environment
• Unlike Integration testing
– Always black box
Cross Cutting
• Regression Testing
– When code changes (some) old tests must still
pass
– Could be at unit, integration or systems level
• Very high cost involved in regression testing
– Regression failure has high cost to fix
Customer Facing
• Acceptance Testing
– Is the customer happy with the product?
• Alpha/Beta Testing
– Let the system out to a few of your customers and
see how they feel about it
• Usability Testing
– Typically for GUI/Web/Mobile to not just check
that the system is correct but also easy to use
– Harder to create an Oracle
Non-Functional Testing
• Performance
– Load
• See how the system behaves at peak load
– Stress
• Push the system beyond its limits to see how far it will
survive
• Security Testing
– Check whether there are vulnerabilities that might
lead to loss of privacy or other security issues
Web App Testing
• Full Gambit of Features
• Front End GUI
– Usability issues
– Hard to test
– Browser Compatibility issues
• Server Side (including storage)
– Performance issues
• Security issues
Unit Testing
• Utterly important!
• Finding bugs early saves money
• Makes regression testing much more effective
– Write once, run often
• What do you test?
– Business logic!
– No need to test simple code or interactions with
3rd party libraries
• Until they fail of course!
• Use Coverage tools to help you decide if you
have tested enough
Stubs or Mocks
• In unit testing you are interested in local
behavior and assume other things you might
be using behave correctly
• Most unit testing frameworks provides stubs
for these 3rd party components
• Good example is datastore and memcache
stubs provided by GAE
• Only problem is that sometimes these stubs
don’t respect the behavior of the real thing!
Unit Testing Frameworks
• Java – JUnit
– The most famous of them all
• PHP – PHPUnit
• Python – PyUnit
– Actually just “import unittest”
• And many more XUnit tools for language X
– For example GAEUnit, but it looks “dead” now
• Use coverage tools along side, examples…
– Coverage.py for Python
– (Ecl)Emma for (Eclipse) Java
Unit Testing and GUIs
• When doing Unit Testing you try and stay well
clear of the GUI
– Clicking and entering text through a GUI is not
automated, although we will see later that with
Replay Technology it can also be done
• Isolating the GUI to allow more efficient
testing is the reason to use Model-ViewPresenter rather than Model-View-Controller
design pattern
Webapp2 example
import webapp2
Code to Test
class HelloHandler(webapp2.RequestHandler):
def get(self):
self.response.write('Hello, world!')
app = webapp2.WSGIapplication([('/', HelloHandler)])
import unittest
import webapp2
# from the app main.py
import main
Test
class TestHandlers(unittest.TestCase):
def test_hello(self):
#input
request = webapp2.Request.blank('/’)
#output
response = request.get_response(main.app)
#oracle
self.assertEqual(response.status_int, 200)
self.assertEqual(response.body, 'Hello, world!’)
Integration Testing
• Thin line between unit and integration
• Strictly speaking when more than one
component is used you are doing integration
testing
– For example if your web app uses a datastore then
the test on the previous slide that came in via a
GET request could be an integration test not a unit
test
• This is not worth worrying about
– Unit/Integration Testing is fine
System Testing
• Now the full round-trip is being tested,
including the Browser component
• Unit/Integration Testing can be done on a
local environment, but System Testing need to
be in the deployed environment
• State-of-the-Practice
– Record and replay tests
Selenium
http://docs.seleniumhq.org/
• The #1 tool in Web System Testing
• “automates browsers. That’s it!”
– Much like the webapp2 example earlier, just many
more options, including browser specific drivers
• IDE for record and replay
– With scripts that can be edited
• WebDriver that allows one to run without a
browser at all
Selenium Example
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0
# Create a new instance of the Firefox driver
driver = webdriver.Firefox()
# go to the google home page
driver.get("http://www.google.com")
# find the element that's name attribute is q (the google search box)
inputElement = driver.find_element_by_name("q")
# type in the search
inputElement.send_keys("cheese!")
inputElement.submit()
try:
# we have to wait for the page to refresh, the last thing that seems to be updated is the title
WebDriverWait(driver, 10).until(EC.title_contains("cheese!"))
# You should see "cheese! - Google Search"
print driver.title
finally:
driver.quit()
Java Example
import com.thoughtworks.selenium.*;
// This is the driver's import. You'll use this for instantiating a
// browser and making it do what you need.
public class NewTest extends SeleneseTestCase {
public void setUp() throws Exception {
setUp("http://www.google.com/", "*firefox");
}
public void testNew() throws Exception {
selenium.open("/");
selenium.type("q", "selenium rc");
selenium.click("btnG");
selenium.waitForPageToLoad("30000");
assertTrue(selenium.isTextPresent("Results * for selenium rc"));
}
}
Usability Testing
•
•
•
•
Based on user opinions
Manual according to a list of tasks
Observer records behavior
Could use things like eye-tracking for more
precise results
• Automated Usability Testing is still a research
topic
Robustness Testing
• Try anything and see if something breaks
• Most famous of these is Android Monkey
– Sends random keystrokes to your Android app
• gremlins.js does the same thing for web apps
<script src="path/to/gremlins.min.js"></script>
<script>
gremlins.createHorde().unleash();
</script>
Performance Testing
• System performance under specific loads
– Concurrent users doing a certain number of
transactions for a certain duration
• This is a very hard kind of testing to do
– Needs lots of infrastructure
– Often the performance bottleneck is your testing
framework and not the system under test
• In the end we know where the problem is
– THE DATABASE!
– More generally where something has to wait for
something else to finish
Web Performance Testing
• Measure throughput or transaction rate
• Server response time
• Rendering, but that might need additional
scripts on the client side
• You might not know what is the expected
performance, so often you profile the
performance
– You will quickly notice bad performance
Web front-end Performance Tools
•
•
•
•
Google Pagespeed tools
www.webpagetest.org
Google Chrome Developer Tools
Nice summary of tools related to performance
can be found at
http://samsaffron.com/archive/2012/03/23/s
am-s-ultimate-web-performance-tools-andresources
JMeter
• Load and Performance Testing
• Server side
• Not a browser, but can simulate some actions
typically done by a browser
– HTTP, etc.
– No javascript execution
Security Testing
• This is HUGE!
– A whole course or even degree can be devoted to it
• Nice checklist of things to do at
https://www.owasp.org/index.php/Web_Application_Se
curity_Testing_Cheat_Sheet
• Large part of security testing is manual
• Penetration Testing
– Find security vulnerabilities that can be exploited
• Tools to find buffer overflows is one of the most
effective automatic approaches
• Fuzz Testing
– Semi-structured random testing
Zed Attack Proxy (ZAP)
• Automated Penetration Testing
– Point it to URL and it does the rest
• Includes Fuzzing and many more
• Find it here
https://www.owasp.org/index.php/OWASP_Z
ed_Attack_Proxy_Project