Proactive UAT™ Makes Users
Competent, Confident, Committed
Robin F. Goldsmith, JD
GO PRO MANAGEMENT, INC.
SYSTEM ACQUISITION & DEVELOPMENT
B USI NE
SS ENG
INE
T R A IN
ING
ERING
22 CYNTHIA ROAD
NEEDHAM, MA 02494-1412
INFO@GOPROMANAGEMENT.COM
WWW.GOPROMANAGEMENT.COM
(781) 444-5753 VOICE/FAX
-1
©2005 GO PRO MANAGEMENT, INC.
QUALITY/TESTING
PRODUCTIVITY
Proactive UAT™ Makes Users Competent, Confident, Committed
Most Acceptance Testing Is Reactive
z
z
z
At the end, based on system as written
Technical view dominates with little user
role/awareness of how and what to test
A lot of work, often with little payback
– Doesn’t find many of the errors
– Adversarial, arguments, blame
– Too late to fix errors anyhow
Lacking competence leads to lacking
confidence leads to lacking commitment
Proactive Testing™ provides the basis for more
confident and competent user commitment to UAT
©2005 GO PRO MANAGEMENT, INC.
-2
Proactive UAT™ Makes Users Competent, Confident, Committed
Objectives
Identify reasons users often are reluctant to
participate in User Acceptance Testing (UAT)
z Distinguish conventional testing’s often
ineffective approaches to UAT from more
successful Proactive User Acceptance Testing™
z Describe professional testers’ role in helping
users define Acceptance Criteria that create
competence, confidence, and commitment
z
-3
©2005 GO PRO MANAGEMENT, INC.
Proactive UAT™ Makes Users Competent, Confident, Committed
“V” Model Shows Levels of Testing
Business (User) Requirements
Acceptance Test
High-Level Design
System Test
Low-Level Design
Integration Test
Code
©2005 GO PRO MANAGEMENT, INC.
Unit Test
-4
Proactive UAT™ Makes Users Competent, Confident, Committed
Reasons for User Resistance
Lack of confidence in what and how to test
z Testing against system, not business requirements
z Insufficient time or resources (may signal other
issues)
z “Not my job” syndrome [may be correct]
z Don’t find much anyhow
z Fear of blame for found or unfound defects
z Belief that feedback will be ignored
z Intimidation and misguidance from developerand/or tester-centric dominance
z
©2005 GO PRO MANAGEMENT, INC.
-5
Proactive UAT™ Makes Users Competent, Confident, Committed
Traditional Development and Testing
Gurus Tend to Miss the Boat on UAT
:Consider UAT a test to confirm product’s
system requirements--often expect a
rubber stamp
:Some testing books/courses say
: UAT should be only positive/valid proof-ofconcept tests to demonstrate normal
functionality
: UAT should be a repeated subset of the
System Test, run by users
: UAT simply needs one test for each
functional requirement or use case scenario
Yet, organizations continually rely on UAT to catch many missed problems
©2005 GO PRO MANAGEMENT, INC.
-6
Proactive UAT™ Makes Users Competent, Confident, Committed
Why Do Users Need to Do Thorough
User Acceptance Testing (UAT)?
Self-defense:
Acceptance Testing is the
user’s chance and duty to
confirm the system works
properly, from the user’s
standpoint, before users
and the organization rely
upon it.
Users often are like
goalkeepers defending
alone without protection
from fullbacks and
halfbacks
Can’t we just trust the developers to do their jobs right?
-7
©2005 GO PRO MANAGEMENT, INC.
Proactive UAT™ Makes Users Competent, Confident, Committed
Proactive Testing Life Cycle
TM
FEASIBILITY
ANALYSIS
Feasibility
Report
SYSTEMS
ANALYSIS
SYSTEM
DESIGN
Business
Requirements
See:
www.sdmagazine.com
/articles/2002/0207
/0208 /0209 /0210
High-Level
Low-Level
Design
RequirementsBased Tests
Technical
Test Plans
Acceptance
Criteria
Acceptance
Test Plan
IMPLEMENTATION
OPERATIONS &
MAINT.
[Life Cycle reit.]
Acceptance
Test
©2005 GO PRO MANAGEMENT, INC.
Formal Review
Black/White Box
Unit Tests
Integration Tests
System, Special
Tests
Independent
(QA) Tests
-8
DEVELOPMENT
Code (Debug,
Informal Review)
Proactive UAT™ Makes Users Competent, Confident, Committed
Key Proactive Testing™ Concepts
Find WIIFMs
z Intertwine testing with each development deliverable
z Plan before acting at any point/level, independent paths
– Acceptance testing first; users shouldn’t do technical tests
– Prepare test plans/designs during Design, promote reuse
– Prioritize by level, from full choices, to avoid wasted effort
z
Let testing drive development
– Feedback test plans/designs to guide correct coding
– Selectively structure for early tests that prevent rework
– Measure objectively to guide decisions, more CAT-Scans
©2005 GO PRO MANAGEMENT, INC.
-9
Proactive UAT™ Makes Users Competent, Confident, Committed
Proactive Testing™ Strategy/Roles:
More Thorough Tests of Cleaner Code
¾
¾
¾ Professional testers
Enable developers to code
more correctly by using testing
¾ Specialize in test methods
to improve requirements and
¾ Tend to like testing, detail
designs
¾ Double check, any/all test
levels, technical view, but
Enable developers to catch
with greater user sensitivity
more of their own (fewer)
¾ Users double check cleaner
remaining errors by more
explicit systematic testing from software from business
perspective as they actually
business and technical
will use system
perspectives
©2005 GO PRO MANAGEMENT, INC.
- 10
Proactive UAT™ Makes Users Competent, Confident, Committed
Approaches to Acceptance Testing
Reactive--Run what’s given to you
z
Here it is, try it out
Proactive--Plan what to run
9 Test each requirement
– Assumes understanding of
9 Minimum for acceptability
how system works and is to
9 Assures requirements are
be used
testable (sufficient?
– Reasonable for changes to 9 (Variant) Run Use Cases
known system
z
(Variant) Test that it works
the way it’s written
– Check choices, links, fields
– Match to user instructions
©2005 GO PRO MANAGEMENT, INC.
9 Define acceptance criteria
9 Independently of
requirements, sets priorities
9 A test of the requirements
¾ Execute the plan
- 11
Proactive UAT™ Makes Users Competent, Confident, Committed
ÂTwo Types of Requirements:
Business/User
z
z
z
User view & language,
conceptual; exists within the
business environment
Serves business objectives
What business results must
be delivered to solve a
business need (problem,
opportunity, or challenge) and
provide value when delivered
Many possible ways to accomplish
©2005 GO PRO MANAGEMENT, INC.
System/Software
z
z
z
Technical/product operational
view, language
Human-defined system/
product design of one of the
possible ways, How, to
deliver the business results
What external functions each
piece of the product must
accomplish for it to work as
designed (Functional
Specifications)
- 12
Proactive UAT™ Makes Users Competent, Confident, Committed
Even Requirements “Experts” Think
the Difference is Detail
Business Requirements
(High-Level, Vague)
System/
Software
©2005 GO PRO MANAGEMENT, INC.
- 13
Reqs.
(Detailed)
Proactive UAT™ Makes Users Competent, Confident, Committed
When Business/User Requirements
Are Detailed First, Creep Is Reduced
User Acceptance Test
Technical Tests
Business Requirements
(High-Level)
System/Software Reqs.
(High-Level)
Business
Reqs.
(Detailed)
©2005 GO PRO MANAGEMENT, INC.
System/
Software
- 14
Reqs.
(Detailed)
Proactive UAT™ Makes Users Competent, Confident, Committed
What Tests Are Needed to Be Confident
This Requirement Is Met?
Separately identify sales tax
calculated as 5% of purchase price
and include it in the total amount due
©2005 GO PRO MANAGEMENT, INC.
- 15
Proactive UAT™ Makes Users Competent, Confident, Committed
Define the Use Case Scenario(s)
that Test This Requirement?
©2005 GO PRO MANAGEMENT, INC.
- 16
Proactive UAT™ Makes Users Competent, Confident, Committed
Requirements-Based Test Issues:
Must Be Business Requirements
z
z
Tests are only as good as
the requirements definitionirrelevant for missing/wrong
It can be hard to tell what
the requirements are
– Narratives are confusing
– Itemized format helps
– Needs detail, much more
z
Need at least 2 tests per
requirement, usually more
©2005 GO PRO MANAGEMENT, INC.
z
Use Cases define usage, not
user, requirements of a product
– Users are unlikely to write Use
Cases; they’ll use existing ones,
but then not user’s view
– Don’t address business rules or
quality factors
z
Seldom written from a thorough
testing perspective, happy path
but not the multiple (esp.
negative) test conditions
- 17
Proactive UAT™ Makes Users Competent, Confident, Committed
Acceptance Criteria--Define Without
Referring to the Requirements
What the users/customers/stakeholders (may be
multiple perspectives) must have demonstrated
to be confident the delivered system works—
before they stake their jobs on system working
z Determination will be made whether or not it is
conscious, planned, or explicit
z True empowerment builds cooperation
z
– Ability to accept or reject delivered system
Identifies overlooked and incorrect requirements
©2005 GO PRO MANAGEMENT, INC.
- 18
Proactive UAT™ Makes Users Competent, Confident, Committed
1. What Functions the System Must
Perform
¾ From user’s perspective
¾ Real world business processes, not programs
¾ Original source through end-use
¾ Common and likely to be problematical
¾ May include structure, e.g., checking every
menu choice, every field, all outputs
¾ Technical role is facilitator, gatherer, organizer
(normal, not just new/problems)
©2005 GO PRO MANAGEMENT, INC.
- 19
Proactive UAT™ Makes Users Competent, Confident, Committed
2. How Much Must Be Shown to
Give Confidence It Works
¾ How many (and which)
¾ Transactions, accounts
¾ Users, locations
¾ Days, business cycles
¾ How frequently, how extensively should they be
demonstrated
¾ What environments should they be in
©2005 GO PRO MANAGEMENT, INC.
- 20
Proactive UAT™ Makes Users Competent, Confident, Committed
3. How Well, How System Quality
Will Be Assessed
¾ Performance measures
¾ Response times, turnaround, throughput
¾ Staffing requirements and productivity
¾ Accuracy, errors and responses to them
¾ Subjective judgments
¾ Usability
¾ Suitability
¾ Whose opinion counts, guidelines
©2005 GO PRO MANAGEMENT, INC.
- 21
Proactive UAT™ Makes Users Competent, Confident, Committed
4. By Whom, Who Should Perform
Tests for Confidence We Can Use It
¾ Actual end-users themselves
¾ Typical, randomly selected
¾ Mix of locations, skills, roles
¾ “Super,” “power,” or “good” users
¾ User stand-ins, e.g., business analysts, liaisons,
temporary helpers, full-time acceptance testers
¾ Non-users, e.g., QA or testing group,
documentation writer, developers
©2005 GO PRO MANAGEMENT, INC.
- 22
Proactive UAT™ Makes Users Competent, Confident, Committed
5. In What Manner, Test Format for
Confidence the System Works
¾ Stand-alone--test inputs and outputs
¾ Parallel--production inputs, test outputs
¾ Most complete definition of “right” results
¾ Old, new, reconcile is three times the work
¾ Pilot--subset, production inputs and outputs
¾ No definition of expected results
¾ Bridging adds complexity and risk to test
¾ Big Bang--production inputs and outputs
©2005 GO PRO MANAGEMENT, INC.
- 23
Proactive UAT™ Makes Users Competent, Confident, Committed
Acceptance Criteria Example
1 of 2
FUNCTIONS
1 Create an order for one taxable item.
2 Create an order for one non-taxable item.
3 Create an order that includes two or more taxable items
and two or more nontaxable items.
4 For an order with two or more taxable items and two or more
nontaxable items, cancel one of the taxable items and one of
the nontaxable items.
5 Cancel an entire order that included two or more taxable items
and two or more nontaxable items.
6 Produce credit vouchers for returned taxable and nontaxable items.
7 Modify sales prices of already-entered taxable and nontaxable items.
8 Create orders for items costing less than $1.00 and more than $999.
9 Change the sales tax rate to 5.25 percent.
©2005 GO PRO MANAGEMENT, INC.
- 24
Proactive UAT™ Makes Users Competent, Confident, Committed
Acceptance Criteria Example
2 of 2
HOW MUCH
10 Create orders that cover one page and two pages.
11 Create an order with total sales tax of more than $1000.
HOW WELL
12 Have an untrained user enter orders.
13 Create an order for 25 different types of items.
BY WHOM
14 At least two real users, one with and one without training.
IN WHAT MANNER
15 Stand-alone.
©2005 GO PRO MANAGEMENT, INC.
- 25
Proactive UAT™ Makes Users Competent, Confident, Committed
Summary
¾ Perceived lack of competence and confidence often
makes users reluctant to participate in User
Acceptance Testing (UAT)
¾ Conventional testing’s common testing-centric views
of UAT can render UAT an ineffective
disempowering rubber-stamp time-waster
¾ Proactive User Acceptance Testing™ empowers
users by defining Acceptance Criteria that are
important to the users and create competence,
confidence, and commitment
©2005 GO PRO MANAGEMENT, INC.
- 26
Proactive UAT™ Makes Users Competent, Confident, Committed
Go Pro Management, Inc. Seminars--Relation to Life Cycle
Systems QA
Improving the REAL Software Process
Managing System Projects with Credibility
System Measurement IT ROI Test Process Management
Feasibility
Analysis
Proactive User Acceptance Testing
Proactive Risk-Based Testing
Systems
Analysis
System
Develop-
Design
ment
Defining and Managing
User Requirements
Reusable Test Designs
Re-Engineering:
Test Estimation
Opportunities for IS
Testing Early in the Life Cycle
21 Ways to Test Requirements
Implementation
Operations
Maintenance
Winning Approaches to
Structured Software Testing
Managing Software Acquisition and Outsourcing:
> Purchasing Software and Services
> Controlling an Existing Vendor’s Performance
Making You a Leader
©2005 GO PRO MANAGEMENT, INC.
- 27
Proactive UAT™ Makes Users Competent, Confident, Committed
Robin F. Goldsmith, JD
robin@gopromanagement.com (781) 444-5753
www.gopromanagement.com
• President of Go Pro Management, Inc. consultancy since 1982, working directly with and training
professionals in business engineering, requirements analysis, software acquisition, project
management, quality and testing.
• Previously a developer, systems programmer/DBA/QA, and project leader with the City of
Cleveland, leading financial institutions, and a “Big 5” consulting firm.
• Degrees: Kenyon College, A.B.; Pennsylvania State University, M.S. in Psychology; Suffolk
University, J.D.; Boston University, LL.M. in Tax Law.
• Published author and frequent speaker at leading professional conferences.
• Formerly International Vice President of the Association for Systems Management and
Executive Editor of the Journal of Systems Management.
• Founding Chairman of the New England Center for Organizational Effectiveness.
• Member of the Boston SPIN and SEPG’95 Planning and Program Committees.
• Chair of BOSCON 2000 and 2001, ASQ Boston Section‘s Annual Quality Conferences.
• Member ASQ Software Division Methods Committee.
• Admitted to the Massachusetts Bar and licensed to practice law in Massachusetts.
• Author of book: Discovering REAL Business Requirements for Software Project Success
©2005 GO PRO MANAGEMENT, INC.
- 28
Proactive UAT™ Makes Users Competent, Confident, Committed