USER INTERFACE
TESTING
Lecture 9
What Is?
• graphical user interface testing is the process of testing a product's graphical
user interface to ensure it meets its written specifications
• GUI testing is a process to test application's user interface and to detect if
application is functionally correct
• GUI testing involves carrying set of tasks and comparing the result of same with
the expected output and ability to repeat same set of tasks multiple times with
different data input and same level of accuracy
• GUI testing includes how the application handles keyboard and mouse events,
how different GUI components like menubars, toolbars, dialogs, buttons, edit
fields, list controls, images etc. reacts to user input and whether or not it performs
in the desired manner.
• Implementing GUI testing early in the software development cycle speeds up
development, improves quality and reduces risks towards the end of the cycle
UI Testing
• GUIs make software easy to use, and developers are
dedicating a larger portion of code to implementing them.
• GUIs can constitute as much as 60 percent of an
application’s total code today.
• The use of GUIs in safety-critical systems is also growing,
making their correct operation imperative.
• testing GUIs for correctness can enhance the entire
system’s safety, robustness, and usability
UI Testing
• Current GUI testing techniques are incomplete, ad hoc, and largely manual.
• The most common tools use record-playback techniques. A test designer interacts with the
GUI, generating mouse and keyboard events.
• The tool records the user events, captures the GUI session screens, and then stores the
session—usually as a script.
• The tester later plays back the recorded sessions to re-create the events with different inputs.
• This process is extremely labor intensive, often relying on the test designer’s ability to
generate interesting GUI interactions.
• An automatic test case generator can provide a higher level of support, but a programmer
must code it for all possible decision points in the GUI.
• The record-playback approach misses important GUI decisions. A popular alternative is to
release beta copies of the software and let the users do part of the testing (Microsoft tested
part of its Windows 95 software by releasing almost 400,000 beta copies).
GUI TESTING PITFALLS
• Software testing is already labor and resource intensive—often
accounting for 50 to 60 percent of total software development
costs
• GUI testing - difficulties that traditional software testing
techniques do not adequately address
• Conventional software uses coverage criteria as a guideline for
determining testing adequacy (define sets of rules that test
designers use to determine the type and amount of underlying
code to test)
• a criterion for “event coverage” might require all reachable
events in a GUI to execute at least once during a complete
cycle of test cases.
GUI TESTING PITFALLS
• traditional coverage criteria do not work well for GUIs.
• GUI software differs from the underlying application code in its level of
abstraction, so mapping between GUI events and the underlying code is
not straightforward.
• Code-based coverage criteria do not necessarily address problematic
interactions between the GUI’s user events and the application.
• even when experienced test designers focus on specific parts of a GUI,
they may still find it impractical to generate all possible test cases for
these parts.
• If the designers have to generate a subset of all possible test cases,
they often must select the subset during test case generation. The
difficulty of anticipating a test case’s fault-detection capability makes it
difficult, in turn, to select the most effective subset
Test Oracles
• Verifying whether the GUI executes correctly poses a problem.
• The traditional verification tool is a test oracle— a separate
program that generates expected results for a test case and
compares them with actual results.
• In conventional software testing, the tester invokes the oracle
after the test case executes and compares the final output with
the oracle’s expected output.
• By contrast, a GUI test case requires interleaving the oracle
invocation with the test case execution because an incorrect
GUI state can lead to an unexpected screen, which in turn can
make further test case execution useless (for example, the test
case might involve a button on the GUI screen that no longer
exists).
Test Oracles
• a GUI test case should terminate as soon as the oracle detects
an error.
• if the oracle does not verify the GUI after each execution step
in a test case, pinpointing the error’s actual cause can become
difficult, especially when the final output is correct but the
intermediate outputs are incorrect.
• Consequently, in GUI test case execution, a tester gives the
inputs one step at a time and compares the expected output
with the GUI’s output after each step.
• This interleaving complicates GUI testing.
Regression Testing
• regression testing - ensure that code modifications do not introduce
new errors into already tested code.
• Regressing the testing - special challenges for GUIs
• both inputs and outputs to a GUI depend on the layout of graphical elements.
• Changes in the layout—button placements, menu organization, and so on—
can change the input-output mapping and render older test cases useless.
• the expected outputs used by oracles may become obsolete.
• developers typically use rapid prototyping for GUIs - requires efficient
regression testing mechanisms that can detect the frequent software
modifications and adapt the old test cases to them.
GUI TESTING PROCESS
• Addressing the specific pitfalls of GUI testing requires a clear
methodology that employs tools and techniques integrated to
use a standardized GUI representation.
• This integration ensures that all test results are compatible
with each other.
• Other goals include
• task automation to simplify the test designer’s work;
• efficient overall test cycle to minimize the frustration of an inherently
tedious and expensive process;
• robust testing algorithms to detect each time the GUI enters an
unexpected state and to report all information necessary to debug the
GUI
• portable tools and techniques to allow test information—test cases,
oracle results, coverage reports, and error reports—generated on one
platform to be used on all other execution platforms
GUI Testing Steps
• Determine what to test by defining coverage criteria. A GUI coverage criterion might
require the execution of each user interface event to determine whether it behaves
correctly.
• Generate test case inputs from software specifications and structure. For GUIs, these
inputs consist of events such as mouse clicks, menu selections, and object
manipulations.
• Generate expected output to compare with actual output. In GUIs, the expected output
includes screen snapshots and window positions and titles.
• Execute test cases and verify output. Test cases execute on the software, and the
tester compares the output with the expected output from, for example, an oracle.
• Determine whether the GUI was adequately tested. Once all test cases have executed,
the tester analyzes the software to check which of its parts were actually tested. In
GUIs, the analysis checks events and resulting GUI states.
• The last step is especially important in GUI testing, where coverage criteria may not
What to Check in GUI Testing
• Check all the GUI elements for size, position, width, length and acceptance of characters or
numbers. For instance, you must be able to provide inputs to the input fields.
• Check you can execute the intended functionality of the application using the GUI
• Check Error Messages are displayed correctly
• Check for Clear demarcation of different sections on screen
• Check Font used in application is readable
• Check the alignment of the text is proper
• Check the Color of the font and warning messages is aesthetically pleasing
• Check that the images have good clarity
• Check that the images are properly aligned
• Check the positioning of GUI elements for different screen resolution.
Manual Based Testing
• Under this approach, graphical screens are checked
manually by testers in conformance with the requirements
stated in business requirements document.
•
Record and Replay
• GUI testing can be done using automation tools. This is
done in 2 parts. During Record , test steps are captured
into the automation tool. During playback, the recorded
test steps are executed on the Application Under Test.
Example of such tools - QTP
•
Model based testing
• A model is a graphical description of system’s behavior. It helps us to understand and predict
the system behavior. Models help in a generation of efficient test cases using the system
requirements. Following needs to be considered for this model based testing:
•
•
•
•
•
•
Build the model
Determine Inputs for the model
Calculate expected output for the model
Run the tests
Compare the actual output with the expected output
Decision on further action on the model
• Some of the modeling techniques from which test cases can be derived:
• Charts – Depicts the state of a system and checks the state after some input.
• Decision Tables – Tables used to determine results for each input applied
• Model based testing is an evolving technique for generating the test cases from the
requirements. Its main advantage, compared to above two methods, is that it can
determine undesirable states that your GUI can attain.
GUI Interaction Testing
• GUI behavior is tightly integrated with the context of their usage
• As users invoke sequences of events (e.g., ClickOnCancelButton, TypeInText) on GUI
widgets (e.g., CancelButton, TextBox), the underlying software responds (typically via
the execution of an event handler e.g., an ActionListener) in one of several ways.
• This may include a change to the software state, which may impact the execution of
subsequent events.
• the context, established by the sequence of preceding events, in which an event
executes may have an impact on how it executes
• This context-sensitive and state-based execution behavior of GUI events creates
problems for testing; each event needs to be tested in multiple contexts
• Current model-based GUI testing techniques either test only a subset of event
sequences by restricting the sequence to length two or length three or use a random
method, rather than a systematic one, to test longer sequences
GUI Testing
• means that a GUI-based software application is tested
solely by performing sequences of events on GUI widgets
• the correctness of the software is determined by
examining only the state of the GUI widgets
• the types of faults uncovered are varied
• thousands of real faults detected via GUI testing are in the
underlying business logic of the application, rather than in
the GUI code itself
GUI Testing
• a representation of events within windows (or components) called an
event-flow-graph (EFG)
• Much like how a control-flow graph (CFG) encodes all possible execution
paths in a program
• an EFG represents all possible sequences of events that can be
executed on the GUI.
• Coverage criteria based on the EFG have provided a notion of coverage
of a GUI’s event space for functional correctness
• It is usually only possible to satisfy these criteria for length two event
sequence coverage because the number of sub-paths grows
exponentially with length
• these criteria, as is the case with CFG path coverage, are useful from a
theoretic perspective; they have limited practical significance.
GUI Interaction Testing
• there are a large number of events that do not interact with the
business logic of the GUI application (those responsible for
opening and closing windows)
• a refinement of the EFG was developed called an event-
interaction graph (EIG).
• In an EIG events are modeled that do not pertain to the
structural nature of the GUI (opening, closing windows, etc.)
but that, instead, interact with the underlying application, called
system interaction events.
• We refer to testing only these events as GUI interaction testing
GUI Interaction Testing
• A primary problem for GUI testing is that the length of the event sequence invoked
by the user is often unbounded.
• There are an enormous number of possible permutations of these events which in
turn means the context for testing is very large; testing all possible sequences a
user may invoke is not possible.
• Example: a user can invoke any of the following events on a drawing canvas in
any order: {copy, paste, resize, rotate90, color, erase}.
• The sequence <rotate90, color, copy, paste> may behave differently than the
sequence <rotate90, color, paste, copy> because execution of the event handler
code for copy and paste may differ, e.g., different values of variables may be
read/written, different branches or paths may be executed.
• This relatively small set of events leads to 36 unique length-two sequences, over
7,500 unique length-five sequences, and more than 60 million unique length-ten
sequences
Semi-automated UT GUI Tools
•
•
•
•
JFCUnit
Abbot
Pounder and
Jemmy Module
• used to manually create unit GUI test cases, which are then
automatically executed.
• Assertions are inserted in the test cases to determine whether
the classes/methods in the unit under test function correctly
• The part of the GUI state space explored by these test cases
depends largely on the experience and knowledge of the
testers and the quality of the user sessions.
Model Based UI Testing
• used to automate certain aspects of GUI testing
• 2 approaches:
• State machine models – depends on “state” interpretation in a
GUI
• Graph models – based on EFG and EIG - the nodes in these
graphs represent GUI events and the edges represent different
types of relationships between pairs of events.
EFG Models
• EFG - all possible event sequences that may be executed on a GUI.
• It is a directed graph that contains nodes (one for each event in the GUI) and
edges that represent a relationship between events.
• An edge from node nx to node ny means that the event represented by ny may be
performed immediately after the event represented by nx –this relationship is
called follows
• The EFG is represented by two sets:
• (1) a set of nodes N representing events in the GUI and
• (2) a set E of ordered pairs (ex, ey), where {ex, ey} ⊆ N, representing the directed edges in the
EFG; (ex, ey) ∈ E iff ey follows ex
• EFG can be constructed semi-automatically using a reverse engineering
technique called GUI Ripping
• A GUI Ripper automatically traverses a GUI under test and extracts the
hierarchical structure of the GUI and events that may be performed on the GUIthe result of this process is the EFG
EIG Models
• EIG nodes do not represent events to open or close
menus, or open windows.
• The result is a more compact, and hence more efficient,
GUI model.
• An EFG can be automatically transformed into an EIG by
using graph rewriting rules
Example
• Figure 1(c) shows the corresponding
EIG.
• the EIG does not contain the Edit event.
• a GUI that consists of four events,
Cut, Copy, Paste, and Edit.
• Figure 1(b) shows the GUI’s EFG;
• The four nodes represent the four
events; the edges represent the
follows relationships.
• For example, in this EFG, the event
Copy follows Edit, represented by a
directed edge from the node labeled
Edit to Copy
• the graph-rewriting rule used to obtain
this EIG was to (1) delete Edit because
it is a menu-open event, (2) for all
remaining events ex replace each edge
(ex,Edit) with edge (ex, ey) for each
occurrence of edge (Edit, ey), and (3) for
all ey, delete all edges (Edit, ey) and
store the mapping “ey → (Edit, ey)” for
use during test execution.
• The GUI’s EIG is fully connected with
three nodes representing the three
events.
Generating Test Cases
• A technique to generate test cases, each corresponding to an EIG
edge has been developed; these test cases are called smoke tests
• Two examples of such length two smoke test cases for our example
of Figure 1(c) are <Copy, Cut> and <Cut,Paste>.
• There are a total of nine such tests – one for each EIG edge.
Because EIG nodes do not represent events to open/close menus or
open windows, other events (in this case Edit) needed to reach the
EIG events are automatically generated at execution time using the
mappings {Cut → (Edit, Cut), Paste → (Edit, Paste), Copy → (Edit,
Copy)} stored earlier, yielding an executable test case .
• The two test cases will “expand” to <Edit, Copy,Edit, Cut> and <Edit,
Cut,Edit,Paste>.
UI Testing
• test cases must cover all the functionality of the system and fully
exercises the GUI itself.
• The difficulty in accomplishing this task is twofold:
• to deal with domain size - a GUI has many operations that need to be tested.
A relatively small program such as MicrosoftWordPad has 325 possible GUI
operations. In a large program, the number of operations can easily be
an order of magnitude larger.
• to deal with sequences - Some functionality of the system may only be
accomplished by following some complex sequence of GUI events. For
example, to open a file a user may have to click on the File Menu and then
select the Open operation, and then use a dialog box to specify the file name,
and then focus the application on the newly opened window. This can become
a serious issue when the tester is creating test cases manually.
• %
• In addition, the tester faces more difficulty when they have to
do regression testing.
Validation Testing
• This emphasizes on the Validation aspect of the Application.
• Validation testing mainly depends on the fields set in the dialog box and the
functions it has to perform.
• common rules that can be applied.
1. For text box fields where value entered has to be numeric check following:
• It should accept numbers only.
2. For text box fields where value entered has to be alpha-numeric check following:
• It should accept alphabets and numbers only.
3. If the field is a combo box then it has to be checked for following points:
• Check the combo box has drop down values in it, it is not empty.
• Drop down values should be alphabetically sorted. This might change as per requirement
but as standard practices it should be alphabetically sorted.
• Selection of any drop down value is displayed on closing and opening the same dialog
box.
• Avoid displaying the first default value in the list.
Validation Testing
4. If the field is a list box then it has to be checked for following points:
• Check the list box has values in it, it is not empty.
• List box values should be alphabetically sorted and displayed. This might
change as per requirement but as standard practices it should be
alphabetically sorted.
• Selection of any list box value should put a check before the value and hould
display the correct value(s) selected on closing and opening of the same dialog
box.
• If the list box supports multiple selection then check whether multiple values
can be selected.
5. If the field is a list of radio button then it has to be checked for
following points:
• Check whether as per requirements all the values are listed.
• Same selected value should be displayed on closing and opening of the same
dialog box.
6. Data Controls are to be tested as part of functionality testing.
Coded User Interface Tests (MS)
• Coded UI Tests – automated tests that drive your application
through the interface
• Include functional testing of UI controls
• let you verify that the whole application, including its user
interface, is functioning correctly
• Coded UI Tests are particularly useful where there is validation
or other logic in the user interface, for example in a web page.
• They are also frequently used to automate an existing manual
test.
Coded User Interface Tests (MS)
Coded User Interface Tests (MS)
• Creating a CUIT:
• You perform the test manually while the CUIT Test Builder runs in
the background.
• You can also specify what values should appear in specific fields.
• The CUIT Test Builder records your actions and generates code
from them.
• After the test is created, you can edit it in a specialized editor that
lets you modify the sequence of actions.
Web application UI checklist
COLORS
• Are hyperlink colors standard?
• Are the field backgrounds the correct color?
• Are the field prompts the correct color?
• Are the screen and field colors adjusted correctly for noneditable mode?
• Does the site use (approximately) standard link colors?
• Are all the buttons in standard format and size?
• Is the general screen background the correct color?
• Is the page background (color) distraction free?
Web application UI checklist
CONTENT
• All fonts to be the same
• Are all the screen prompts specified in the correct screen
font?
• Does content remain if you need to go back to a previous
page, or if you move forward to another new page?
• Is all text properly aligned?
• Is the text in all fields specified in the correct screen font?
• Are all the heading left aligned
• Does the first letter of the second word appears in
lowercase?
Web application UI checklist
IMAGES
• Are all graphics properly aligned?
• Are graphics being used the most efficient use of file size?
• Are graphics optimized for quick downloads?
• Assure that command buttons are all of similar size and
shape, and same font & font size.
• Banner style & size & display exact same as existing
windows
• Does text wrap properly around pictures/graphics?
• Is it visually consistent even without graphics?
Web application UI checklist
INSTRUCTIONS
• Is all the error message text spelt correctly on this
screen?
• Is all the micro-help text(i.e tool tip) spelt correctly on this
screen?
• Microhelp text(i.e tool tip) for every enabled field & button
• Progress messages on load of tabbed(active screens)
screens
Web application UI checklist
NAVIGATION
• Are all disabled fields avoided in the TAB sequence?
• Are all read-only fields avoided in the TAB sequence?
• Can all screens accessible via buttons on this screen be
accessed correctly?
• Does a scrollbar appear if required?
• Does the Tab Order specified on the screen go in sequence
from Top Left to bottom right? This is the default unless
otherwise specified.
• Is there a link to home on every single page?
• On open of tab focus will be on first editable field
• When an error message occurs does the focus return to the
field in error when the user cancels it?
Web application UI checklist
USABILITY
• Are all the field prompts spelt correctly?
• Are fonts too large or too small to read?
• Are names in command button & option box names are not
abbreviations.
• Assure that option boxes, option buttons, and command buttons are
logically grouped together in clearly demarcated areas "Group Box"
• Can the typical user run the system without frustration?
• Do pages print legibly without cutting off text?
• Does the site convey a clear sense of its intended audience?
• Does the site have a consistent, clearly recognizable "look-&-feel"?
• Does User cab Login Member Area with both UserName/Email ID ?
• Does the site look good on 640 x 480, 600x800 etc.?
• Does the system provide or facilitate customer service? i.e.
responsive, helpful, accurate?
• Is all terminology understandable for all of the site’s intended users?
Web Applications Testing
• 2 types of web sites:
• Provide services
• Provide content
• need to understand user behavior on the site and gather key
usage pattern - analytic code
• Analytic code - a java script snippet which is included in all the
pages of your website.
• does not have any behavioral impact on the application and is
invisible to the end-user in most of the cases.
• every time a page is loaded, these scripts get executed and
store key information about the users browsing pattern
Analytic code
• Information provided by analytic code:
• From which link users have come to this page
• For how long users are staying on this page
• Where are users going from this page
• From which part of the world users are coming
• Which browser is used
• information is extremely valuable for the organization -
marketing activities.
• useful for people who want to advertise on a the site as it
gives them data about the popularity of site.
Test Automation for Web Applications
• Test automation - using a software tool to run repeatable tests against
the application to be tested
• Test automation supports:
• Frequent regression testing
• Rapid feedback to developers
• Virtually unlimited iterations of test case execution
• Support for Agile and extreme development methodologies
• Disciplined documentation of test cases
• Customized defect reporting
• Finding defects missed by manual testing
• Advantages:
• repeatability of the tests
• the speed at which the tests can be executed.
• Selenium - the most widely-used open source solution
Selenium
• Javascript framework that runs in a web browser
• Works anywhere Javascript is supported
• other supported languages Java, Ruby, Python
• Can simulate a user navigating through pages and then
assert for specific marks on the pages
• HTML knowledge needed to start using it right away
Selenium
• Selenium is a set of different software tools each with a
different approach to supporting test automation
• operations are highly flexible, allowing many options for
locating UI elements and comparing expected test results
against actual application behavior.
• support for executing one’s tests on multiple browser
platforms
Testing Static Content
• The simplest type of test, a content test, is a simple test for the
existence of a static, non-changing, UI element.
• Does each page have its expected page title? This can be used to
verify your test found an expected page after following a link.
• Does the application’s home page contain an image expected to be at
the top of the page?
• Does each page of the website contain a footer area with links to the
company contact page, privacy policy, and trademarks information?
• Does each page begin with heading text using the <h1> tag? Does
each page have the correct text within that header?
• If your page content is not likely to be affected then it may be more
efficient to test page content manually.
Testing links
• A frequent source of errors for web-sites is broken links or
missing pages behind links
• Testing involves clicking each link and verifying the
expected page.
• If static links are infrequently changed then manual testing
may be sufficient.
• if web designers frequently alter links, or if files are
occasionally relocated, link tests should be automated.
Function Tests
• tests of a specific function within your application, requiring some type of user
input, and returning some type of results.
• will involve multiple pages with a form-based input page containing a collection of
input fields, Submit and Cancel operations, and one or more response pages
• User input can be via text-input fields, check boxes, drop-down lists, or any other
browser-supported input.
• Function tests are often the most complex tests you’ll automate, but are usually
the most important.
• Typical tests can be for login, registration to the site, user account operations,
account settings changes, complex data retrieval operations, among others.
• Function tests typically mirror the user-scenarios used to specify the features and
design or your application.