Quality Assurance
IS 101Y/CMSC 101Y
November 12, 2013
Carolyn Seaman
University of Maryland Baltimore County
Data Analysis Assignment
 Stacked bar charts
250
200
150
happiness
#hours self care
#hours studying in groups
100
50
1
8
15
22
29
36
43
50
57
64
71
78
85
92
99
106
113
120
127
134
141
148
0
Data Analysis Assignment
 Aggregate and focus
Avg #hours self care
80
70
60
50
40
Avg #hours self care
30
20
10
0
Quality Assurance
 Refers to any activity whose goal is to make sure that
the system or application being built is free from error
and of high quality
 That it works
 That it solves a problem
 That it’s usable
A process view of QA
Problem Identification
Analysis
CMSC,
CMPE, IS
Design
SDLC:
Systems
Development
Lifecycle
Implementation
QA
Installation
Maintenance
QA includes:
 Testing
 Exercises a system or part of a system to make sure it
produces the correct output or behaves in expected ways
 Cannot happen until some part of the system is
implemented
 Reviews and inspections
 Visual “reading” of some artifact
 Can be done early by reviewing early documents, e.g.
design
Testing Terms
 Coverage
 ideally, testing will exercise the system in all possible ways
 not possible, so we use different criteria to judge how well
our testing strategy “covers” the system
 Test case
 consists of data, procedure, and expected result
 represents just one situation under which the system (or
some part of it) might run
Test Planning
 A test plan includes:




test objectives
schedule and logistics
test strategies
test cases
 procedure
 data
 expected result
 procedures for handling problems
Testing Phases
 Unit testing - does this piece work by itself?
 Integration testing - do these two pieces
In
development/
maintenance
organization
work together?
 System testing - do all the pieces work
together?
 Alpha acceptance testing - try it out with inhouse “users”
 Installation testing - can users install it and
In user
organization
does it work in their environment?
 Beta acceptance testing - try it out with real
users
Testing Techniques
 Structural testing techniques
 “white box” testing
 based on statements in the code or individual
hardware and connections
 coverage criteria related to physical parts of the
system
 tests how a program/system does something
 Functional testing techniques




“black box” testing
based on input and output
coverage criteria based on behavior aspects
tests the behavior of a system or program
System Testing Techniques
 Goal is to evaluate the system as a whole, not its parts
 Techniques can be structural or functional
 Techniques can be used in any stage that tests the
system as a whole (acceptance, installation, etc.)
 Techniques not mutually exclusive
System Testing Techniques
(cont.)
stress testing - test larger-than-normal capacity in
terms of transactions, data, users, speed, etc.
execution testing - test performance in terms of speed,
precision, etc.
recovery testing - test how the system recovers from a
disaster, how it handles corrupted data, etc.
operations testing - test how the system fits in with
existing operations and procedures in the user
organization
compliance testing - test adherence to standards
security testing - test security requirements
System Testing Techniques
(cont.)
requirements testing - fundamental form of testing makes sure the system does what it’s required to do
regression testing - make sure unchanged functionality
remains unchanged
error-handling testing - test required error-handling
functions (usually user error)
manual-support testing - test that the system can be
used properly - includes user documentation
historical test data - tests until the number of defects
found approaches the average number of defects in the
products produced under similar circumstances
Unit Testing
 Goal is to evaluate some piece (file, program, module,
component, etc.) in isolation
 Techniques can be structural or functional
 In practice, it’s usually ad-hoc and looks a lot like
debugging
 More structured approaches exist
Unit Testing Techniques
input domain testing - pick test cases representative of the
range of allowable input, including high, low, and average
values
equivalence partitioning - partition the range of allowable input
so that the program is expected to behave similarly for all
inputs in a given partition, then pick a test case from each
partition
boundary value - choose test cases with input values at the
boundary (both inside and outside) of the allowable range
Unit Testing Techniques
(cont.)
statement testing - ensure the set of test cases exercises every
statement at least once
branch testing - each branch of an if/then statement is
exercised
path testing - every path is exercised (impossible in practice)
fault seeding - put a certain number of known faults into the
code, then test until they are all found
Reviews and Inspection:
Goal and Motivation
By detecting defects early, and preventing their
leakage downstream, the higher cost of later
detection and rework is eliminated.
Basic Steps
 Using a reading technique,
 Perform a visual examination of the document
 Detect and correct:
• Defects
• Violation of design standards
• Other problems
Examples of artifacts that can be
reviewed
Include:







Contracts
Installation plans
Progress reports
Design descriptions
Release notes
Requirements specifications
Source code
What Is a Defect?
 Any occurrence in a work product that is determined to
be incomplete, incorrect, or missing
 Any instance which a requirement is not satisfied
(Fagan, 1986)
 Informal synonyms: bug, fault, issue, problem, anomaly
Common Attributes of Systematic
Reviews and Inspections
Systematic reviews and inspections have these
attributes in common:
• Team participation
• Documented results of the review
• Documented procedures for conducting the review
Management Reviews
 Performed by those directly responsible for the
system
 Monitor progress
 Determine status of plans and schedules
 Confirm requirements and their system allocation
 Or, evaluate management approaches used to
achieve fitness or purpose
Technical Reviews
Confirms that a product




Conforms to specifications
Adheres to regulations, standards, guidelines, plans
Changes are properly implemented
Changes affect only those system areas identified by
the change specification
Inspections (Formal Peer
Reviews)
Confirms that the software product satisfies
 Specifications
 Specified quality attributes
 Regulations, standards, guidelines, plans
 Identifies deviations from standards and specification
 results in logging a defect
Walk-throughs
 Evaluate a product or artifact or document
 Sometimes used for educating an audience
 Major objectives:
•
•
•
•
Find anomalies
Improve the product
Consider alternative implementations
Evaluate performance to standards and specs
Audits
The purpose of an audit is to provide an
independent evaluation of conformance of
products and processes to applicable





Regulations
Standards
Guidelines
Plans
Procedures
Inspection Process

Most popular is the Fagan method

Inspection is separated into 5/6 phases






(Planning)
Overview
Preparation
Inspection Meeting
Rework
Follow-up
Review and Inspection Pitfalls
 Insufficient Preparation
 Moderator Domination
 Incorrect Review Rate
 Ego-involvement and Personality Conflict
 Issue Resolution and Meeting Digression
 Recording Difficulties and Clerical Overhead
Other QA Activities
 Process conformance
 Making sure that quality procedures are followed
 Risk analysis
 Planning for adverse events
 Making the unexpected expected
 Hiring and training
 Developing guidelines and standards
 Identifying new training needs
Project demos
 This Thursday!!!
 No slides
 Everyone must participate and answer questions
 Show your program
 Running
 Code
 Talk about what it does and what it doesn’t do YET
 10 minutes, including questions
 Turn in .pde file and document BEFORE class on Blackboard
Peer Evaluations
 Take a look at the Peer Evaluation form and think about
the following two options:
1. Each student gets the actual sheets filled out by their
teammates
2. Each student gets a summary of the feedback written by
the instructor
 Evaluations due in class on THURSDAY (rewards to
those who bring them to class)