Performance Optimization In QTP Execution
Over Video Automation Testing
Speaker : Krishnesh Sasiyuthaman Nair
Date
: 10/05/2012
1
Importance Of Performance Optimization
• Prevent unnecessary wastage of test resources.
• Get faster test results thus reducing cost in
automated testing.
2
Video Testing
• Involves different interlinked components like
streaming servers, settop box racks, IR blasters etc.
• Need support from multiple applications and
frameworks which provides OCRs, Image
comparisons, SNMPs, trace processing etc
3
HPQC-QTP Framework In Video Testing
• Requirements and test cases will be available in QC
Test Plan. QC provides a template in Test Plan for
storing the test scripts, allows them to be added to QC
Test Set, executes and saves result to QC test set.
• Provides feature to store defects in the defects area of
QC.
4
QTP Test Execution From QC
5
Problems : QC-QTP Automation
Development & Execution
• Interruptions during long execution due to QC timeouts
and network issues.
• In the case of multi-location, its not possible to sync
the local agile development version repository with
mapping and execution of test cases in QC test plan.
6
Problems : QC-QTP Automation
Development & Execution Continued
• Manual testers and other QC users will also be able to
view the scripts and edit configuration files. This is not
necessary.
• Network bandwidth utilization is too much during
execution due to frequent connecting local machine
QC client to QC server for transferring reports,
initiating new runs etc as execution needs multiple
active QC sessions.
7
Long Test Execution Cycle Time
In video testing each script will take around 10-15
minutes and a typical release testing will contain
around 200-400 scripts , will be around 20-25 box
types to validate. On a minimum there will be 4000 test
runs and will take around 27 days if executed in one
machine.
8
No Support For Agile Development &
Coupled Execution
• There is no continuous integration available in QCQTP default framework.
• In rigorous script development environment when
multiple people works on the same library files, there
are chances of bugs getting added to the impacted
areas of code change and need to executed and
validated frequently, with minimal effort, which is not
possible currently.
9
Need For Common Local Subversion For
Execution
• Since video automation framework using QTP is
huge(contains lot of images, xmls , data tables, vbs
file, csv files etc ), the execution need to be as local as
possible because of multiple reasons.
• Its always an overhead in maintaining script and
framework in two locations(subversion & QC) has
wastage of time and space.
10
Network Bandwidth Usage Due To Active
QC Sessions
As discussed in previous slide an execution can
extend up to 27 days, which need to be shortened
through parallel executions in multiple machines, say
10 machines it will come down to 2.7 days. But, this
will need 10 active QC sessions polling frequently to
client machines for execution triggering and result
upload. Since framework itself is big, the file transfer
between server to client will eat up lot of network
bandwidth.
11
Solution : Performance Optimization In
Framework
Root causes for the problems mentioned in the
previous slides were found out and the converged
causes are the following.
– Continuous integration support not available.
– Execution need to be triggered locally with minimum
interaction with the QC server.
A test driver created with QTP AOM and QC OTA will
be sufficient to solve the major bottle necks.
12
Continuous Integration In
Script Development
Continuous integration involves integrating code early
and often so as to avoid the pitfalls of ‘integration hell’.
Aim is to reduce rework and reduce cost and time.
There is a need to have version based execution and
this support is added with the help of test driver.
13
Minimal Interaction With QC Server
With local execution framework, the long download
times of script and framework files from QC is reduced
along with reducing the interaction of QC during
execution with local client executing machines. Also a
configurable tool reboot time is introduced which
removes the unexpected memory leaks associated
with long duration executions.
14
Video Test Automation With Test Driver
15
Video Test Automation With Test Driver
Continued
• Driver retrieves the details of test set created in QC.
• Test lab added test cases with supporting files are
downloaded to local QTP execution machine will be
the latest version or specific version in subversion.
• Execution trigger will be added to windows scheduler
for daily runs.
• Execution results and logs will be available in QC Test
Set on completion of execution.
16
Test Execution Duration Difference
Execution initiated Without Driver
Total Scripts
Duration Hrs
Execution Initiated With Driver
Total Scripts
Duration Hrs
200
34
200
28
400
67
400
54
There is more than 10 hrs execution duration saving during execution. This is
a definite advantage in video test automation space, since we can free up the
hardware and software tools and resources blocked for this execution cycle &
advantage is thus tremendous.
17
Script Level :
Performance Optimization
• Each individual scripts need to be optimized
separately for better results.
• There are a set of guidelines to be followed in the
scripts and tool settings for better performance .
18
Conclusion
Performance optimized framework provides excellent
foundation for automated testing in video test
automation space. As manual execution is really tiring,
the cost benefit a well optimized framework provides is
really immense.
19
Question
&
Answers
20
Reference
• QC, QTP Documentation Library & HP ALM Open Test
Architecture Reference.
• Tata Elxsi Automation Information Database.
21
Thank you
ITPB Road, Whitefield
Bangalore, India 560 048
Tel
+91 80 22979123
Fax
+91 80 28411474
e-mail info@tataelxsi.com
www.tataelxsi.com 22