Software Quality Assurance
Automated Testing Techniques
Michael Biwer
Apology
 I did not communicate to Dr.Yang my Abstract and
References by Monday 8:00AM.
 This is because I completely forgot and dropped the ball on
the simplest part of this course.
 This is why there had been a late communication saying that I
was to present today.
 Thankfully, Dr.Yang and me worked out a deal.
Agenda
 Software Quality Assurance: Automated Testing
 Overview
 Testing Methodology
 Overview of types of tests
 Automated Testing Software
 Versus Manual Testing, Pros, Cons, And Basic Functionality
 Types of Automated Testing Framework
 Modular, Data-Driven and Keyword-Driven Frameworks.
Definition
 “The management and performance of
test activities, to include the
development and execution of test
scripts so as to verify test requirements,
using an automated test tool”[2].
History
 Began to evolve with the rest of software development as pressure
was put on developers to:
 Reduce time, cut costs and increase quality.
 This began to create more incremental builds:
 Shown with the increase of Rapid Application Development methods
such as Agile.
 Posed the difficulties of faster changes and additions to project
requirements.
 Higher customer appreciation of quality control. (ISO 9000)
 Therefore, Software Quality Assurance: Automation Testing began
to evolve and third party companies began to create automation
suites.
Developer Testing
VS.
Quality Assurance Testing
 Developer Testing:
 Ensures that the all of the projects requirements have been met
through testing the processes.
 Usually, done through testing the code from an internal standpoint.
 Overall, looks to ensure that the process work correctly.
 Quality Assurance Testing:
 Ensures that all of the project requirements have been met through
testing the product.
 Usually, done through testing the product from an external
standpoint. Most instances mimic’s users view of the product.
 Overall, looks to improve the product as a whole.
Manual Testing Vs. Automated Testing
 Manual:
 Labor intensive, testing take many man-hours to complete.
 Does not require software knowledge or skills from the testers.
 Repetitive tests still require many hours to complete and may be
narrowed to fit a time schedule.
 Automated:
 Capital intensive.
 Large upfront cost for the software. Scalability might require hardware
upgrades.
 Will not receive any benefits until the 2 – 3 project, because the
creation of the test framework.
 Repetitive testing shows rates of returns around the 10th time it is run.
 Requires more specialist knowledge bases for the automation
engineer.
Automated Testing
Manual Testing
Test Case Creation
1
Need for test created
Need for test created
2
Business Analyst creates test requirements, test
plan and test cases which reflects the need in
automated testing tool software.
Business Analyst creates test requirements,
test plan and test cases which reflects the need
in a document.
3
Automation Engineer creates test script framework
which reflect the test plan and test cases
requirements.
Automation Testers create driver scripts to run the
test scripts generated by the automation Engineer.
Manual Tester carries out the test plan,
recording results of tests.
4
5
Tests are run automatically and results are
analyzed. Business Analyst logs bugs for next
build/run of test.
Results of tests and expected results are
compared and bugs are recorded for the next
build.
N/A
All Runs of the Test Case Afterwards
6
The Business Analyst reviews test, makes
adjustments if needed and re-run the test in the
automation software.
The Business Analyst reviews test, makes
adjustments.
7
N/A
Manual Tester carries out the test plan,
recording results of tests.
Why?
 Proper defect tracking.
 Aim to enhance the overall quality of the product, not to just
ensure it works correctly.
 Different eyes on the testing ensures that project
requirements were communicated efficiently between
customer and development team.
 Because not all developers are good testers, it is good to have
a specialist test the product to ensure consistent quality.
 Ensures that the customer has input, in order to “make sure
that they will be happy with the end result.”
Pros
 24/7 Testing.
 Repetitive testing is easily accomplished and cost effective in
long running projects or in maintenance.
 In depth reporting tools.
 Consistent testing.
 Fewer, more skilled workers.
Cons
 Cost
 Heavy upfront cost with the purchase of the software and the
time devoted to the creation of a automation framework.
 In addition, you will not see a return on investment right away.
 Must pay your employees more.
 You will require an automation engineer which has software
programming knowledge.
 You will have to train your automation testing team in the
automation software tool you choose.
Overview of Testing Types
 Two overall types of testing:
 White-Box Testing:
 Testing that is done from an internal viewpoint of the application.
Input
Process
Output
 Black-Box Testing:
 Testing that is done from an external viewpoint of the application with
only the working knowledge of how the application “should” work.
Input
Output
Overview of Testing Methods (cont..)
 Unit Testing
 Tests that target specific code to ensure that it can operate separately
from the system given the correct inputs.
 Integration Testing
 Testing that aims to test the applications code as a whole to ensure
that the components interact correctly.
 System Testing
 Testing that makes sure that the application meets the requirements
set down for the software and that it doesn’t interfere with any other
platforms/applications.
 Acceptance Testing
 Testing where either the user or developer/QA team is given a build
of the program in order to verify the requirements have been met.
Overview of Testing Methods(cont…)
 Smoke Testing:
 “Build Verification Testing”, Ensures that no serious problem has occurred
between build of the application.
 Regression Testing
 Testing normally performed after a major code overhaul has been made to make
sure that there are no persisting defects and that no functionality has been lost.
 Functional Testing
 Testing that only cover a particular flow or process within the program.
 Non-Functional Testing
 Testing that confirms the overall quality of the product in reference to the user.
 Performance Testing
 Testing that checks the scalability, reliability and resource usage of certain aspect
of the program.
 Usability Testing
 Testing of the Graphical User Interface (GUI) works correctly in the given
layout.
When to Automate a Test
 It makes the most sense to automate tests for products that
are going to have continuous development or maintenance.
 This ensures that the test cases can be reused, increasing the
return on investment of the automation software.
 Such as for the purposes of regression or smoke tests.
 Side Note:
 Developers often use automated testing methods for unit or integration
tests. (Junit Tests.)
Basic Automation Software Features
 Record/Playback
 Process:
 Click the play button in the automation suite.
 The Automation Software tracks your mouse movements, clicks and keyboard
actions.
 Click the stop button when the test completes.
 The Software then compiles the movements into a test cases for you to
insert into a test plan.
 You are then able to perform those exact actions multiple times.
 Sometimes features of the automation suite include auto-generation of
record/playback tests into test scripts.
Basic Automation Software Features
 Record/Playback (cont…)
 Benefits:
 Simple, does not require a software background to utilize.
 Quick, there is not much set up to create this type of test.
 Can be used for repetitive testing on a basic level.
 Drawbacks
 The types of tests that you can performs are limited to external options.
 If you have to change the test data, you must redo the entire test.
 Therefore, there can be a lot of maintenance.
Example of Record/Playback
 http://youtu.be/facNRr8SHpk?t=45s
Basic Automation Software Features
 Object Recognition
 Allows you to have an external look at the objects that are currently
being displayed in the application being tested.
 Aka. GUI.
 You are able to store these objects in an “Object Repository” in order to
easily reference them or you are able to “Call” them directly within
your test scripts.
 They are formed in a hierarchy of “container” objects.
 Automation software with this ability gives you libraries which allow
you to perform functions and retrieve/change properties of the
objects.
Basic Automation Software Features
 Object Recognition(cont…)
 Benefits:
 Allows you to set up more in-depth specific tests within test scripts.
 Drawbacks:
 Requires a test script framework to create test on these objects.
 Creates large initial overhead.
 Problems occur when custom objects/properties are created by the
developers which the automation suite cannot recognize.
 This would require you to create your own “Object Script Extender”
to recognize the object/property.
Example of HP QuickTest Pro
Object Recognition
Basic Automation Software Features
 Test Script Execution Framework
 Allows you to create libraries in order to test any portion of
your application.
 Code
 GUI
 API’s
 Only limited to the amount of coding you want to do.
 These scripts are accessed through drivers and can be used
across many different test cases.
 Commonly these test scripts are created using VBScript, Java,
Perl, or CGI.
 The automation tools have pre-loaded libraries to give you extra
capabilities with testing objects.
 Simple test script example.
 Uses a direct calling of objects to interact with them.
Basic Automation Software Features
 Test Cases Reports
 Able to view the passes and the fails of your automation tests in
a report format.
 Since this is based on your test scripts, you only get as much
detailed information as you allow the test case record to collect
from the tests.
HP QTP:
Example
Report
Components
 Test Plan
 A set of test cases which verify similar requirements of the
application.
 Test Case
 A test to verify requirements of the application.
 Driver Scripts:
 The overall scripts that call your test scripts that make up your test
cases.
 In essence, will reflect your test plan.
 Test Scripts:
 Code which allows for the tests actions to be carried out.
 Test Data:
 The data that you will be testing in your application.
Meta-Language Development
 Meta-Language is a abstract language used in running scripts. It





has been simplified for the average user with no knowledge of
programing to use.
This allows testers that only have the business knowledge of the
system to utilize testing tools.
Allows for manual testers to migrate onto the new system easily.
This is essential for the testers to take ownership of the test cases.
It is used at the highest visible level for the users to see, the Driver
Script.
In essence however, the Meta-Language, will be a list of
parameters that will be parsed/translated and used in the test
scripts.
 HP QTP Example:
Input parameters
that the
automation
engineer has set
for this Driver
Script
Where the automation tester
inputs the test parameters.
Types:
Modular Framework
 Creating specific test scripts in a library which are used
within the Driver Scripts to perform the tests.
 IE: In the windows calculator, having a script that would verify
that 5 + 5 = 10.
 Benefits:
 Quick and easy to put together.
 Drawbacks:





Difficult to maintain.
Less reusable code.
Data is within the test scripts.
Can easily have multiple copies of the same code.
Because the test scripts have all of the control, this means that
the ownership of the test cases lie in the automation engineer.
Types:Modular Framework (cou…)
Example of framework setup:
Driver Script
Test Library 1
Function_1
Function_2
Test Library 2
Function_3
Function_4
Example of Psuedocode for a Modular Framework Driver Script:
Main
Read Parameters
Window_1 Functions
Function_1 Call
Function_2 Call
Function_3 Call
Window_2 Functions
Function_4 Call
Function_5 Call
Function_6 Call
End Main
Function_5
Function_6
Example of Driver Script Parameters:
Action_Type
Name_Of_Script
Run_Script
Verify_Calc_Script
Types:
Data-Driven Framework
 This framework looks to fix the problem of data being within the
test script. The input is commonly moved to the driver script or to
an external database or spreadsheet which is fed into the test
scripts.
 IE. In the windows calculator, you have a spreadsheet with numbers
which are run through a script that adds them together.
 Benefits:
 Gives ownership of the test cases to the testers.
 Maintenance is more manageable.
 Drawbacks:
 Test case information can be split into different areas confusing large
operations.
 Scalability is a real problem.
Types: Data-Driven Framework (cou…)
Example of framework structure:
Driver File
Script_1
Script_2
Script_3
Data_File_1
Data_File_2
Data_File_3
Comparison of Expected and Actual Outputs
Pseudocode for Data-Driven Driver Script:
Main
Read in Parameters
Perform required Script with given parameters
Script_1 Call
Script_2 Call
Script_3 Call
End Main
Example of the Data-Driven Driver Script:
Script_Name
Data_File
Numbers
Expected_Result
Add_Calc_Script Value_Add_To_10.xml A1-A2
B1
Sub_Calc_Script Value_Sub_To_0.xml
A1-A2
B1
Mul_Calc_Script Value_Mul_To_49.xml A1-A2
B1
Types:
Keyword-Driven Frameworks
 This framework looks to provide maximum ownership to the
testers and maximum flexibility.
 Does this by providing a object by object look at the test cases and
providing actions that can be taken out on those objects.
 Benefits:
 Provides all ownership to the testers.
 Maximum flexibility and highly scalable.
 Maintenance is on the testers and is standardized.
 Drawbacks:
 Large overhead time for creating the framework.
 A driver file must be created for each “window” of the application.
 An iterative approach which can increase test case execution times.
Types: Keyword-Driven Framework (cou…)
Driver Script
Function_1 + Object_1 +
Data
Function_1 + Object_2 +
Data
Function_2 + Object_2 + Data
Test Libraries
Function_1
Function_2
Pseudocode for Keyword-Driven Driver Script:
Main
Read in Parameters
Parse parameters into correct Object+Function+Data groups
For Each Object in window
Selected Action
Object_1 Action Calls
Object_2 Action Calls
Object_3 Action Calls
End Main
Iteration
Action_Type
Button_1
1
Click
Click
2
Insert
“Hello”
3
Click
Highlight
4
Key_CNTRL_X
5
click
6
Key_CNTRL_V
7
Property
Edit_1
CheckBox_1
Click
Click
Text=Hello
Checked=True
Types: Hybrid Framework
 Combines the previous frameworks together in various levels
to provide maximum testing capabilities.
 Normally involves connecting to a test database for various
inputs.
Example of Hybrid Framework Psuedocode:
Main
Read Parameters
Group Parameters
Link to database
Select a specific window
For each object in a specific window
Perform the needed scripts
End Main
What we discussed
 Software Quality Assurance: Automated Testing
 Overview
 Testing Methodology
 Overview of types of tests
 Automated Testing Software
 Versus Manual Testing, Pros, Cons, And Functionality
 Types of Automated Testing Framework
References
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
Chaos manifesto 2012:The year of the executive sponsor. (2012). Retrieved from
http://versionone.com/assets/img/files/CHAOSManifesto2012.pdf
Elfriede, D., Rashka, J., & Paul, J. (2008). Automated software testing: Introduction, management and performance. (13 ed.). Castleton,
New York: Hamilton. Retrieved from
http://books.google.com/books?id=kl2H0G6EFf0C&lpg=PR5&ots=wN_PrWrBx1&dq=automation software test
qa&lr&pg=PA43
Mosley, D., & Posey, B. (2002). Just enough software test automation. Upper Saddle Hill, new Jersey: Prentice Hall PTR. Retrieved
from http://books.google.com/books?id=PEBvfWESIt4C&lpg=PR15&ots=xGxZPGvp2P&dq=software test automation
&lr&pg=PA3
Laukkanen, P. (2006). Data-driven and keyword-driven test automation frameworks. (Master's thesis)Retrieved from
http://eliga.fi/Thesis-Pekka-Laukkanen.pdf
Software testing. (n.d.). Retrieved from http://en.wikipedia.org/wiki/Software_testing
Allott, S. (1999, 10). Automate your tests - you won’t regress it!. Seventeenth annual pacific northwest software quality conference,
Oregon Convention Center Portland, Oregon. Retrieved from http://www.snapshots.pnsqc.org/2011-theme/pnsqc-04-062011/proceedings/pnsqc99.pdf
Kumar, S. (Designer). (2007, April 9). File:SampleCode.JPG [Print Photo]. Retrieved from
http://en.wikipedia.org/wiki/File:SampleCode.JPG
Kiranpaul , K. (n.d.). Quick test professional tutorials - qtp fundamentals, qtp features, vb scripting examples in qtp, qtp framework and vb
script.. Retrieved from http://qtp4free.blogspot.com/2010/05/object-spy-in-qtp-92.html
Jain, A. (Photographer). (2013, July 5). Good Looking and Enhanced Results Viewer [Web Graphic]. Retrieved from
http://2.bp.blogspot.com/_iTz5d0BSy6g/TPY5-7SOIGI/AAAAAAAAAPM/wxlEl9BHMqM/s1600/QTP_11_RV_SS.png
Walter , T. (Photographer). (2012, July 6). Add Parameters to specify the Ranorex Automation [Web Graphic]. Retrieved from
http://www.ranorex.com/blog/wp-content/uploads/2012/07/Add-Parameters-To-Test.png
Tutor. (2012, Aug 6). Software testing life cycle (stlc) – complete tutorial. Retrieved from http://www.sdlc.ws/software-testing-lifecycle-stlc-complete-tutorial/