Lecture 17
Testing Web Apps, GUIs, and Real-Time Systems
Outline
• Testing Web Apps and Services
• Testing GUIs
• Testing Real-Time Systems
1
July 24, 2016
Web Apps and Services
• Essential difference of native software: deployment
• Web software can be accessed by anyone
• Myriad of issues:
•
•
•
•
•
•
Geographic locations
Demographics
Different languages
Time zones
Disabilities
etc.
2
July 24, 2016
Web Apps and Services
• Very competitive market
• Expected to work every time
• It doesn’t, user will look at competition
• Built in novel ways:
•
•
•
•
Distributed
Small software components
Concurrent operation
Different memory sharing mechanisms
3
July 24, 2016
Web Apps and Services
• HTTP is stateless
• Each request / response interaction between client / server is
independent of each other
• State of web app must be maintained by
• Cookies
• Session objects
• Offline storage (e.g., databases)
4
July 24, 2016
Web Apps and Services
• Use multitude of new technologies
•
•
•
•
•
•
•
•
Servlets
ASPs
C-sharp
JavaBeans
XML
Javascript
Ajax
PHP
5
July 24, 2016
Web Apps and Services
• Testing individual components no different than testing
traditional software
• Unclear how to test interactions
6
July 24, 2016
Web Apps and Services
• Three categories of web app testing
1. Static hypertext websites
2. Dynamic web apps
3. Web services
7
July 24, 2016
Terminology
• Web page- viewed in single browser window
• Website- collection of web pages and associated SW elements
• Static web page- unvarying, same to all users
• Dynamic web page- created by program on demand, based on
user input, server state, or other input
8
July 24, 2016
Terminology
• Web application- full software program deployed across the
web
• Test case- sequence of interactions between components on
clients and servers
• Paths of transitions through web app
9
July 24, 2016
Testing Static Hypertext Websites
• Client-side validation
• Server-side validation of links
• Many test tools out there
•
•
•
•
Link testing
Security testing
Load / performance stress testing
etc.
10
July 24, 2016
Testing Static Hypertext Websites
• Common to model static web page as graph
• Nodes = pages
• Edges = links
• Construct graph by starting at intro page
• Perform Breadth-First Search on all links on page
• Website covered = edge coverage
11
July 24, 2016
Testing Dynamic Web Apps
• Challenge: client has limited access to server artifacts
• Adds complexity to testing
• Cannot directly test all artifacts from single entry point
12
July 24, 2016
Testing Dynamic Web Apps (Client)
• Dynamic content makes link testing undecidable
• How do you create a graph that depends on user input?
• One method: nondeterministically explore action sequences
starting from given URL
• Preselect input
• Another method: generate input data based on previous users
• Based on user session data
• Many web servers capture / store such data
13
July 24, 2016
Testing Dynamic Web Apps (Client)
• Many web apps impose input constraints through HTML forms
• Client-side input validation
• Explicit HTML attributes (e.g., max text width)
• Bypass testing: creates inputs that intentionally violate
validation rules, submit directly to web app
14
July 24, 2016
Testing Dynamic Web Apps (Server)
• Cannot easily make Control Flow Graph of software
• E.g., What if user hits “back” button?
• Users can affect data in unpredictable ways
• E.g., modifying hidden form fields
• Changes in client-side settings can affect web app behavior
• E.g., not storing cookies
15
July 24, 2016
Testing Dynamic Web Apps (Server)
• Many new connections / transitions in software
• Traditional analysis structures (e.g., CFG, data flow graph)
cannot model web apps
• “Possible flow of control” cannot be known statically
• New analysis techniques needed
16
July 24, 2016
Testing Dynamic Web Apps (Server)
• One idea: Split / trace def-use pairs between client / server
• Another idea: define “atomic” sections of HTML
• All or none sent to user
• Use “atomic” sections to define basic blocks for code analysis
• Web app can then be modeled as CFG
17
July 24, 2016
Testing Web Services
• Web service- distributed, modular application, whose
components communicate by exchanging data in structured
form
• E.g., StockQuote web service
• Testing is difficult because:
• Apps are distributed
• Have unusual runtime behaviors
• Testers have no details on design / implementation (can only test client)
18
July 24, 2016
Testing Web Services
• Single business process may use multiple web services
• Run on different servers
• Implemented by different companies
• Difficult to assess behavior on malformed input
• Could possibly use XML grammar-based analysis on sent data
• Area of active research (yawn..)
19
July 24, 2016
Outline
• Testing Web Apps and Services
• Testing GUIs
• Testing Real-Time Systems
20
July 24, 2016
Testing GUIs
• GUIs account for half or more of source code in modern
software systems
• E.g., GUI for medical device
• Problem: very little help for testing this large amount
21
July 24, 2016
Testing GUIs
• GUI testing falls into 2 categories:
1. Usability testing
2. Functional testing
22
July 24, 2016
GUI Usability Testing
• Assesses how usable the GUI is
• Based on principles from user interface design
• Beyond scope of course..
23
July 24, 2016
GUI Functional Testing
• Assess whether user interface works as intended
• Four types of functional testing
1. GUI system testing
2. Regression testing
3. Input validation testing
4. GUI testing
24
July 24, 2016
GUI Functional Testing
1. GUI system testing
a.
System testing through graphical user interface only
2. Regression testing
a.
Test user interface after changes made (often use capture-replay)
i.
ii.
iii.
“Capture” some user input
“Replay” through user interface after changes made
Report differences
3. Input validation testing
a.
Test how well software recognizes / responds to invalid inputs
25
July 24, 2016
GUI Functional Testing
4. GUI testing
a.
Assess how well GUI works
i.
ii.
b.
“Do all UI controls work as intended?”
“Does the software allow user to navigate to all expected screens?”
Can model GUI as Finite State Machine
i.
ii.
iii.
26
July 24, 2016
Every user input causes transition
Path: sequence of edges through transitions
Construct test cases to cover paths and states
Outline
• Testing Web Apps and Services
• Testing GUIs
• Testing Real-Time Systems
27
July 24, 2016
Testing Real-Time Systems
• Real-time software- must respond to externally generated
input within a finite and specified period
• Usually embedded within larger engineering system
• Rea-time Systems (RTS)- typically interact with other subsystems and processes in physical world (environment)
28
July 24, 2016
Testing Real-Time Systems Terminology
• Deadline- time constrain on response time
• Timeliness- ability for software to meet time constraints
• Testers must detection violations of timeliness
• Periodic task- activates at a known frequency
• Aperiodic task- activated at any time
29
July 24, 2016
Testing Real-Time Systems Terminology
• Want to determine longest execution or worst case time
• Difficult to estimate
• Response time- time RTS takes between activated until
execution finished
• Depends on order of execution of tasks (execution order)
30
July 24, 2016
Issues with Testing RTS
• Observability- ability to monitor / log RTS behavior
• Usually increased by inserting probes to reveal state info
• Problem: probes can influence temporal behavior of RTS!
• Known as the probe-effect problem
31
July 24, 2016
Issues with Testing RTS
• Reproducibility- when system exhibits same behavior with
same inputs
• Very desirable for testing
• Difficult to achieve in RTS- behavior of system depends on elements NOT
explicitly defined by user input
32
July 24, 2016
Issues with Testing RTS
• Controllability
• High degree required to test nondeterministic software
• Difficult to achieve in RTS
33
July 24, 2016
RTS Timeliness Faults, Errors, Failures
• Timeliness fault- mistake in implementation of RTS
• May result in incorrect temporal behaviors
• Execution takes longer than expected
• Timeliness error- RTS deviates internally from assumptions
about temporal behavior
• Similar to non-RTS
• Difficult to detect without extensive logging
• Might only be detectable when lead to system-level failures
34
July 24, 2016
RTS Timeliness Faults, Errors, Failures
• Timeliness failures- violation of time constraint, can be
observed externally
• Often “easy” to detect
• May have some consequence for continued operation of overall system
35
July 24, 2016
Testing for Timeliness
• Often use techniques from scheduling analysis
• More general approach: systematically sample execution
orders that can lead to missed deadlines
• Often use formal models of software
• Use a constraint graph
• Use a clock region graph
• Temporal logic (pairs of timed inputs / outputs)
36
July 24, 2016
Testing Timeliness with Mutations
• Potential faults modeled as mutation operators
• Mutants that have potential to violate timeliness identified,
eventually killed
• Several types of mutation operators for RTS testing
37
July 24, 2016
Mutation Operators for RTS Testing
1. Task set- change points in time when resource taken
2. Execution time- increase / decrease exe time of task
3. Hold time- change interval of time for locked resource
4. Lock time- increase / decrease actual resource lock time
38
July 24, 2016
Mutation Operators for RTS Testing
5. Unlock time- change when resource is unlocked
6. Precedence constraint- add / remove relations b/t tasks
7. Inter-arrival time- decrease inter-arrival time b/t tasks
8. Pattern offset- change offset b/t two patterns by time
39
July 24, 2016