Test Effort Estimation
• What is Estimation?
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
1
Test Effort Estimation
• Definition of Estimation
– Estimation is defined as the intelligent anticipation of
the quantum or work that needs to be performed and the
resources (namely, human resources, monetary
resources, equipment resources and time resources)
required to perform the work at a future date in a defined
environment using specified methods.
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
2
Test Effort Estimation
•
Definition of Test Effort Estimation
–
6th Sep, 2011
Test Effort Estimation is that estimation of the testing size,
testing effort, testing cost and testing schedule for a
specified software test project in a specified environment
using defined methods, tools and techniques.
Chemuturi Consultants,
www.chemuturi.com
3
Test Effort Estimation
•
Now the key words are –
–
–
–
–
–
6th Sep, 2011
Estimation – defined earlier
Testing Size – the amount (quantity) of testing that needs
to be carried out
Testing Effort – the amount of effort in either person
days or person hours necessary for conducting the tests
Testing Cost – the expenses necessary for testing,
including the expense towards human effort
Testing Schedule – the duration in calendar days or
months that is necessary for conducting the tests
Chemuturi Consultants,
www.chemuturi.com
4
Test Effort Estimation
• Estimation
Standard Time – is defined as the amount of time taken to accomplish
a unit of work in a defined environment by a qualified worker, after
adjustment, at a pace that can be maintained day after day without
any harmful physical effects
Estimated Time - the time / effort that is expected to be consumed for
accomplishing a defined amount of work in a defined environment
Actual Time - the time / effort that is actually consumed for actual
amount of work in the actual environment
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
5
Test Effort Estimation
• Remember, an “exact/accurate estimate” is an
oxymoron (“credible estimate” is a better phrase)
• Estimate how long will it take you to place to
another place
–
–
–
–
On what basis did you do that?
Experience right?
Likely as an “average” probability
For most software projects there is no such ‘average’
• Most estimations are off by 25-100%
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
6
Test Effort Estimation
• Why testing?
– Uncover all defects
– Assure the customer that the product is built
conforming to his/her specs/requirements
– Certify the product – IV&V
•
•
•
•
6th Sep, 2011
No virus / No spyware / No malware
Works as specified
Safe & secure
Any other objective
Chemuturi Consultants,
www.chemuturi.com
7
Test Effort Estimation
• Test Effort Estimation
– Issues in Test Effort Estimation
– Estimate testing project size as a percentage of
Software size to be tested and derive effort
– Convert Software Size into Testing Project size and then
convert it into testing effort
– Test Case Enumeration
– Task Based Estimation
– STU Estimation
– Delphi
– Analogy
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
8
Test Effort Estimation
•
Issues in Test Effort Estimation
– Who Needs Test Effort Estimation? - There
are four categories of people who would like
to carry out test estimation –
•
6th Sep, 2011
The project Team who obviously have to carry out
testing to ensure that their work is satisfactory
and they can certify it so and pass it on to
professional independent testing team. This set
would carry out all the tests necessary
Chemuturi Consultants,
www.chemuturi.com
9
Test Effort Estimation
•
Issues in Test Effort Estimation
– Who Needs Test Effort Estimation?
The in-house Software Quality Assurance (SQA)
Team that would test and certify the product for
customers. Their objective is to unearth all
defects and make the product as defect proof as
possible while ensuring that the product meets all
the specified functionality.
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
10
Test Effort Estimation
•
Issues in Test Effort Estimation
– Who Needs Test Effort Estimation?
Testing Organizations, whose mainline of business is
to test other’s software and certify the products.
Their objective is to ensure that the product
meets the end-user expectations. This set would
carry out –
a.
b.
c.
d.
e.
6th Sep, 2011
Mainly Black Box testing
Functional Testing
System Testing
Negative Testing
Regression Testing
Chemuturi Consultants,
www.chemuturi.com
11
Test Effort Estimation
•
Issues in Test Effort Estimation
– Who Needs Test Effort Estimation?
Customers who entrusted their software
development to a vendor. Their objective is to
ensure that they are getting what they are paying
for. This set would carry out
a. Acceptance Testing / Positive Testing
b. Regression Testing
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
12
Test Effort Estimation
•
Issues in Test Effort Estimation
–
The type of test effort estimation carried out by
sets would be different.
1.
2.
3.
4.
6th Sep, 2011
Project Team – since they carry out testing as part of
development work – it may be taken as a percentage of overall
development work. Their purpose would be mainly to estimate
resource requirement and meeting the delivery schedule
In-house SQA Team – they need to estimate mainly for the
purpose of resource estimation and meet the delivery schedule
Testing Organizations – they need to estimate so that they can
offer a fixed price to the customers and manage a healthy profit
for their organizations besides resource estimation and meeting
delivery schedule. For them “Size” of testing project would
have significant appeal.
Customers – They really are supported in this activity of
acceptance testing by the vendor organization.
Chemuturi Consultants,
www.chemuturi.com
13
Test Effort Estimation
•
Issues in Test Effort Estimation
– Types of Tests conducted
1. Unit Testing
2. Integration Testing & Incremental Integration
Testing
3. System Testing
4. User Acceptance Testing
5. Volume Testing
6. Load Testing
7. Functional Testing
8. End to end Testing
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
14
Test Effort Estimation
•
Issues in Test Effort Estimation
– Types of Tests conducted
9. Parallel Testing
10. Concurrent Testing
11. Stress Testing
12. Positive Testing
13. Negative Testing
14. Deployment Testing
15. Sanity Testing
16. Regression Testing
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
15
Test Effort Estimation
•
Issues in Test Effort Estimation
– Types of Tests conducted
17. Security Testing
18. Performance Testing
19. Usability Testing
20. Install / Uninstall Testing
21. Compatibility Testing
22. Intuitive Testing
23. Comparison Testing
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
16
Test Effort Estimation
•
Issues in Test Effort Estimation
– Types of Tests conducted
24. Alpha Testing
25. Beta Testing
26. GUI Testing / navigation Testing
•
How of Testing
•
•
•
•
6th Sep, 2011
White Box Testing
Black Box testing
Product Testing
Project Testing
Chemuturi Consultants,
www.chemuturi.com
17
Test Effort Estimation
• Testing Project
– Testing as part of Development projects
• Embedded Continuous Testing
• Testing at the end of the development
– Independent Verification and Validation Testing
• What tests are included in Testing project?
– Often specified
– Often not specified !!
– Quantum of work depends upon the tests included
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
18
Test Effort Estimation
• Test Strategy
– First Step in Test Effort Estimation
– Includes
• Testing Objectives
–
–
–
–
Unearth all defects
Unearth all possible defects within the given time
Obtain Customer acceptance sign-off
Certify the product
» Virus & Spy-ware free
» Function, security, and so on
– So on
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
19
Test Effort Estimation
• Test Strategy
– Includes
• Types of testing to be carried out
• How testing is done
–
–
–
–
Intuitive
Test Plan and Test Case based
White box or Black Box
Manual Testing or Tool based Testing
• Testing Environment
– How many?
• Criteria for completing Testing
• Regression Testing
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
20
Test Effort Estimation
• Test Strategy
– Includes
• Defect Closure and Escalation mechanism
• Progress Reporting
• Defect Analysis
– ABC analysis
– Defect Category Analysis
• Now we are ready for test effort estimation !
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
21
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size
– One way is to compute software size to Test Effort
• Let us say one FP needs 2 hours of testing
– If the project size is 100 FP then Testing Effort is 200 Person Hours
• Let us say one UCP needs 4 hours of testing
– If the project size is 100 UCP then Testing Effort is 400 Person Hours
– Simple to use
– There are no benchmarked data available – therefore,
organizational data needs to be developed and maintained
– Norms to be developed and maintained–
•
•
•
•
6th Sep, 2011
No of FP (or UCP) for one hour of User Acceptance Testing
No of FP (or UCP) for one hour of Integration Testing
No of FP (or UCP) for one hour of System Testing
Etc.
Chemuturi Consultants,
www.chemuturi.com
22
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size
– Demerits
• Too simplistic
• Testing Size is not available in Software Test Units – hence deriving
productivity is based on software size
• Amount of testing differs from project to project – the variance can be very
high
• Organization has to keep meticulous records and strictly adhere to process
to obtain reliable figures
• Time sheet and data collection have to be rigorous
– Merits
• Facilitates arriving testing effort for any individual or combination of tests
• Very quick
• Can be surprisingly accurate – if the norms are well derived and maintained
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
23
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size
– One way is to convert software size to Software Test Units (STU)
• Let us say one FP results in 5 STU
– If the project size is 100 FP then Testing Size is 500 STU
• Let us say one UCP results in 8 STU
– If the project size is 100 UCP then Testing Size is 800 STU
• Now convert the STU into effort
– Let us say each STU consumes 30 minutes then the project consumes » 500 * 0.5 = 250 Person Hours
» 800 * 0.5 = 400 Person Hours
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
24
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size
– We need to define a STU
– Maintain meticulous records to derive and establish relationship
between Project in FP / UCP and TP
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
25
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size
– Demerits
• Too simplistic
• Need to compute the software size first (for IV & V projects) in FP or
UCP
• If the software size is not computed in-house – then the software
size itself may be in question
• Organization has to keep meticulous records and strictly adhere to
process for deriving relationship
• Time sheet and data collection have to be rigorous
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
26
Test Effort Estimation
• Test Effort Estimation Conversion from Software Size
– Merits
• Testing Size is defined in STU – hence facilitates deriving Testing
Productivity
• Very quick
• Can be surprisingly accurate – if the norms are well derived and
maintained
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
27
Test Effort Estimation
• Test Effort Estimation – Test Case Enumeration
– Enumerate the test cases
– Estimate testing effort for each test case
– Use Best Case, Normal Case and Worst Case scenarios for
estimating effort needed for each test case
– Compute Expected Effort for each case using Beta Distribution
• Best Case + Worst Case + (4 * Normal Case) / 6
– Sum up the –
•
•
•
•
6th Sep, 2011
Expected times to get Expected effort estimate for the project
Best Case times to obtain best case effort estimate
Worst Case times to obtain worst case effort estimate
Normal Case times to obtain normal case effort estimate
Chemuturi Consultants,
www.chemuturi.com
28
Test Effort Estimation
• Test Effort Estimation – Test Case Enumeration
– Best Case estimate and the worst case estimate give the range of
effort estimate for the project
– We can use Excel / MS Project to define predecessors and
successors, and build a network diagram – this facilitates arriving
at the project duration and computation of probability
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
29
Test Effort Estimation
• Test Effort Estimation – Test Case Enumeration
– Demerits
• No Testing size – hence productivity can not be derived
• All test cases and attendant overheads need to be enumerated
– Merits
•
•
•
•
6th Sep, 2011
Auditable estimate
Fairly accurate
Probability can be computed
Progress monitoring is facilitated
Chemuturi Consultants,
www.chemuturi.com
30
Test Effort Estimation
• Test Effort Estimation – Test Case Enumeration
– Best Case estimate and the worst case estimate give the range of
effort estimate for the project
• Standard Deviation for a test case is –
– (Worst Case - Best Case)/ 6
• Variance for an a test case is the square of its standard deviation
• Standard deviation (ST) for the entire project is –
– Square Root of (sum of variances of all critical a test case)
– We can use these for computing probability of success of project
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
31
Test Effort Estimation
• Estimation – Activity/Task -based Estimation
– This is concerned with identifying the tasks (Activities) to be
performed to complete a task and estimating effort required to
complete each of them
– Steps are –
•
•
•
•
6th Sep, 2011
Identifying the component tasks in each phase
Estimating the effort for each task
Calculating the effort for each phase
Consolidating the effort for the entire project
Chemuturi Consultants,
www.chemuturi.com
32
Test Effort Estimation
• Estimation – Activity/Task -based Estimation
• Phases in a Testing project
–
–
–
–
–
–
–
–
–
–
6th Sep, 2011
Project Initiation
Project Planning
Specifications Analysis
Test Strategy Design
Test Planning
Test Case Design
Testing and Defect reporting
Regression Testing and defect closure
Documentation
Project closure
Chemuturi Consultants,
www.chemuturi.com
33
Test Effort Estimation
• Estimation – Activity/Task -based Estimation
• List out all the tasks in each of these phases
• Eg – Specifications Analysis
• Study proposal and Order
• Document understanding gaps
• Obtain Clarifications
– Test Strategy Design
•
•
•
•
•
•
•
6th Sep, 2011
Define Testing Objectives
Decide the type and number of tests
Decide the Test Environment
Define Testing Completion Criteria
Define defect reporting and closure mechanism
Define Regression testing mechanism
Define Tools usage
Chemuturi Consultants,
www.chemuturi.com
34
Test Effort Estimation
• Estimation – Activity / Task-based Estimation
–
–
–
–
–
Assign durations for each of the Tasks
Use three time estimates – Best Case, Worst Case and Normal Case
Compute the expected time using formula
(Best Case + Worst Case + 4* Normal Case)/6
Make adjustments for project complexity, familiarity with the platform, skill of
the developers, tools usage
– Make adjustments using heuristics – (e.g. Requirements 10%, Design 25%,
Construction 40%, Testing 25%)
– Use Delphi technique to validate the estimate
– Sum up the total effort estimate of the project
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
35
Test Effort Estimation
• Testing Size Estimation
– Why is size necessary?
•
•
•
•
6th Sep, 2011
Productivity computation
Measure performance
Agree with Client about the quantity of work being performed
Monitor progress
Chemuturi Consultants,
www.chemuturi.com
36
Test Effort Estimation
• Testing Size Estimation
– What are the units of measure available for measuring testing
size?
• STU are being discussed
• But no universally accepted definition is available on what
Constitutes a STU
• The definition is just escaping STU !!
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
37
Test Effort Estimation
• What is my definition of STU?
– STU should cater to all combinations of Tests
– Be it a unit test or be it acceptance test, be it system test – or
any other test – STU should be able to measure its size
– What is common to all types of tests is – the Test Case
• STU is a size measure for measuring the size of a softwaretesting project and that a STU is equivalent to a
normalized test case.
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
38
Test Effort Estimation
• What is my definition of STU?
– What is a test Case?
• Test Case provides an input to the system which would produce a
response from the system. The response could be –
– Store the data
– Produce derived / un-derived data
– A message
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
39
Test Effort Estimation
• STU Estimation approaches
– Derive from the size Estimate
– Bottoms Up approach
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
40
Test Effort Estimation
• STU Estimation approaches – EstimatorPal Approach
– The Method of computing STUs
1. This method uses an existing software size in FP/UCP etc
2. It converts development project size into Unadjusted STUs (UTP)
using a conversion factor which can be modified by the user to
suit his environment
3. It computes a Composite Weight Factor (CWF)
a. Sum up all individual weights of included tests
b. Multiply it by the weight of the application weight
c. Multiply it by the language weight if Unit Testing is selected
d. Multiply it by Tools Weight if Tools Usage is selected
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
41
Test Effort Estimation
•
STU Estimation
4.
5.
6.
6th Sep, 2011
UTP are multiplied by CWF to obtain the testing size in STUs
size
The Productivity Factor indicates the number of person hours
per one STU
Testing Effort in Person hours is computed by multiplying
STU Size by the Productivity Factor.
Chemuturi Consultants,
www.chemuturi.com
42
Common Pitfalls in Estimation
•
•
•
•
•
•
•
Inexperienced people carrying out estimation activity
Lack of training
Lack of historical data
Absence of tools
Over-estimation and under-estimation
Inadequate time for estimation
Lack of review
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
43
Common Pitfalls in Estimation
• Lack of causal analysis for deviations in actual Vs.
estimates
• Productivity Figures - no focus
• Application of one productivity figure for conversion
from size to effort
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
44
Over and Under Estimation
• Over estimation issues
– The project will not be awarded
• Conservative estimates guaranteeing 100% success may mean winning
probability of zero.
– Parkinson’s Law: Work expands to take the time available for its
completion
– Danger of feature and scope creep
– Be aware of “double-padding”: team member + manager
• Under estimation issues
– Quality issues (short changing key phases like testing)
– Inability to meet deadlines
– Morale and other team motivation issues
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
45
Estimation Good Practices
•
•
•
•
Experienced people carry out estimation
Process is defined for estimation
Estimation guidelines are defined and available
Productivity figures are derived and maintained upto-date
• Estimators are trained
• Estimation tools are available
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
46
Estimation Good Practices
• Repository of estimates are maintained and are
made available to estimators when needed
• Expert guidance is available
• All estimates are subjected to peer review as well as
managerial review
• Actual Vs. Estimate analysis (postmortem) is always
carried out and causal analysis is carried out and
norms of estimation updated
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
47
Estimation Guidelines
• Estimate iteratively!
–
–
–
–
–
Review two heads are better than one
Make your best estimates at each planning stage
Refine estimates and adjust plans iteratively
Plans and decisions can be refined in response
Balance: too many revisions vs. too few
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
48
Estimation “Presentation”
• How you present the estimation can have huge
impact
• Techniques
• Plus-or-minus qualifiers
• Ranges
• Risk Quantification
– +/- with added information
– +1 month of new tools not working as expected
– -2 weeks for less delay in hiring new developers
• Cases
– Best / normal / Worst cases
• Confidence Factors
– April 1 – 10% probability, July 1 – 50%, etc.
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
49
Other Estimation Factors
– Account for resource experience or skill
• Up to a point
• Often needed more on the “low” end, such as for a
new or junior person
– Allow for “non-project” time & common tasks
• Meetings, phone calls, web surfing, sick days
– There are commercial ‘estimation tools’ available
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
50
Other Estimation Factors
– Please download our tool TestPal
• Test Case Design assistance
• Test Result logging
• Test Effort Estimation
– STU
– Task Based
– Test Case Enumeration
– www.chemuturi.com/TestPal-Setup.exe
–Thank you Very much
6th Sep, 2011
Chemuturi Consultants,
www.chemuturi.com
51