Selecting a Tool
When to start evaluating:
•Current method is not effective
•Can solve an immediate problem/challenge
•Can increase productivity and deliverables
•Budget is available
1
Selecting a Tool
Management’s Priorities:
•Total Cost
•Return on Investment
•Does it help with the company’s deliverables
1
Selecting a Tool
Total Cost:
•Financial Costs
•Non Financial costs
1
Selecting a Tool
Financial Cost:
•Purchase Price
•Annual Maintenance / Support costs
•Training
•Cost for upgrades and changes
1
Selecting a Tool
Non Financial Costs:
•Time spent on evaluation
•How many people do we need to administer this tool
•How much work must be put into updating the work for
changes in our environment
•How much are we currently spending to accomplish these
tasks (i.e. maintaining internal tools etc.)
1
Selecting a Tool
Return on Investment:
•How much value are we going to get out of this tool
•How long will it take to realize a positive return
•How long will we be able to use this tool
•Do I feel confident that my group will use this tool and it
will not become shelf ware.
1
Selecting a Tool
Does this tool help with our deliverables
•Are we improving quality
•Are we reducing cost
•Are we speeding delivery of our product / project
1
Selecting a Tool
How to work with Vendors
(A view behind enemy lines)
•Minimize time spent on the evaluation
•What Information does a tool vendor need from our company
•What are vendors willing to provide during the sales cycle
•Helpful resources on tools selection
1
Selecting a Tool
Minimizing the time of the evaluation:
Planning!
Make sure you have defined and can articulate:
•Your current process
•Your Company’s goals
•How your application is developed and it’s architecture.
•What is a must have
•What is a nice to have
1
Questions to ask internally before you talk to a Vendor:
1. What are our primary objectives for a tool?
2. Describe the organizations development and testing process / methodology.
3. What type(s) of testing are we doing currently?
4. What tools do we currently own / maintain? (Purchased and Homegrown)
5. What language(s) is our application developed in?
6. Is the application web or browser based? If so, which browsers and versions do we test on.
7. What operating systems do we test on?
8. Are there any third party controls or grids?
9. Are there any Active X controls?
10. Do we need to test server side com objects or other parts of our application not accessed by a GUI?
11. Describe the application architecture?
12. What is the communication protocol between the different tiers of the application?
13. What databases does our application work with?
14. How do we create our test data?
15. Are we trying to validate data in the GUI or the back end database or both?
16. How often does our group test new builds of applications?
17. Do we have a dedicated Test Lab or would testers be using their own desktops?
18. What amount of money are we budgeting for this project / tool?
1
Selecting a Tool
Information a Tool Vendor needs from
a company:
•What are you hoping to accomplish with a tool.
•What is your decision process.
•What is your development and testing environment.
•What changes are you expecting in your
environment.
1
Selecting a Tool
Questions that few customers ask……….
But they would be much happier if the had!!!!!
•Will this tool work if our development environment changes?
•Are there any additional costs if we change our application?
•What if I have a problem that Technical support can’t answer?
•What are the ongoing costs?
•How stable is your Company?
1
Example Questions to ask a tool Vendor:
1. Given our environment, how will your tools help support it?
2. Will your tool work natively in our environment (i.e. with our own JVM) or is there an add on?
3. Will there be any additional modules if we change our environment?
4. Are those an additional cost?
5. What integration’s exist between you tools as well as the tools we already own?
6. How easy is it to maintain scripts?
7. What functions exist to help in error detection and error recovery.
8. What are the different components of your tool (i.e. Requirements Mgmt, Test Mgmt, Data Mgmt, Record and play back, Defect
tracking etc.)?
9. How is your tool licensed? (Floating or per seat)
10. What is the single user list price of your software?
11. What is the maintenance cost of your software?
12. What does maintenance include?
13. What are the hours of Technical support?
14. Is training available?
15. Is training and technical support provided by your company or a third party vendor?
16. Where is your development done?
17. Are they available for feature requests and more advanced technical support?
1
Selecting a Tool
What are Vendors willing to provide during an evaluation:
•Discovery meeting
•Demonstration
•Proof of Concept
•Trial / Evaluation
•Price
1
Selecting a Tool
Additional Resources:
www.stickyminds.com
•Ray Robinson “Automation Test Tools Comparison”
-A detailed functional evaluation of the 5 major functional
testing tools.
•Elizabeth Hendrickson “Evaluating Tools” -- Features
that you should consider
www.qaforums.com -- discussion groups and resources
www.qualitytree.com -- tips and techniques for testers
1
Selecting a Tool
Contact Information:
David Herness
Compuware -- QA Center
Phone: 425-201-2626
Email: David.Herness@Compuware.com
www.Compuware.com
1