Quality Assurance & Testing
Report On
Automated Software
Testing
By Mishna Haria
Mxh06u
1|Page
Contents
INTRODUCTION TO AUTOMATED SOFTWARE TESTING [3] ................................................................. 3
THE AUTOMATED TEST LIFECYCLE METHODOLOGY (ATLM) .............................................................. 4
REASONS TO AUTOMATE SOFTWARE TESTING [1] .............................................................................. 5
TYPES OF AUTOMATED TESTS............................................................................................................. 5
ADVANTAGES OF AUTOMATED SOFTWARE TESTING ........................................................................ 7
DISADVANTAGES OF AUTOMATED SOFTWARE TESTING [4] ............................................................... 7
CHOOSING AUTOMATION TOOLS ....................................................................................................... 7
SUMMARY ........................................................................................................................................... 7
REFERENCES ........................................................................................................................................ 8
2|Page
INTRODUCTION TO AUTOMATED SOFTWARE TESTING [3]
Due to the financial situations in the world today, to save money and make the most of every project,
software development organisations and software developers face a challenge of completing
software projects in lesser and lesser amount of time and using minimal resources.
Software testing “is usually the part of the project which makes grown men weep”.[1] It is a laborious,
tedious, and time consuming task; which means more resources needed and therefore more
expense. For this reason many organisations have turned to automated software testing.
Automated software testing is automating the manual process of testing software.
“Automation is the use of tools and strategies that reduce human
involvement or interaction in unskilled, repetitive or redundant tasks”.[2]
Things that can manually never be done, is accomplished by automation.
Example: simulating 1000 virtual users for volume testing. [1]
This done manually:
A thousand people asked to come in and carry out the allocated task would mean, a thousand
computers are needed and every user needs to be explained their task that needs to be carried out
simultaneously. Every user has a level of intelligence to grasp the allocated task and that needs to be
taken into account. This would take long and shoot up the costs. Repeating the test is not possible.
This automated:
One virtual user is created on a computer. This user is multiplied by a thousand on the same
computer. One person would program these users to carry out the assigned tasks simultaneously.
Finally, this would be executed. This test could be repeated over and over again.
3|Page
THE AUTOMATED TEST LIFECYCLE METHODOLOGY (ATLM)
This is a systematic approach to maximize test coverage. See Fig 1.
Figure 1 Automated Test Lifecycle Methodology (ATLM)
The first step, in the lifecycle, is making the decision to automate testing. In this stage the team
outlines the potential benefits of automation and also creates a test tool proposal.
The second step is the test tool acquisition. In this stage the entire testing tool is evaluated and
selected. The system’s engineering environment is reviewed whenever possible and a list of tool
evaluation criteria is also produced. The vendor is then contacted to bring in the selected tool that is
evaluated based on the provided criteria.
The third step is the automated testing introduction process. This stage outlines the procedure
needed to successfully introduce automated testing to a new project. Test process analysis is the
first part of the procedure. This ensures that the overall test strategy is in place and adapted to the
project. The second and final part is the test tool consideration. This looks into the test tools brought
in without any specific project and checks if it now is of any advantage to the project.
The fourth step is the test planning, design and development. This, I believe, is the most important
stage as it includes identifying the test procedures standards, defining the tests and the
development standard. The test plan contains the results of all other stages in the ATLM. The
procedure creation standards and guidelines are identified. Also the hardware, software and
network required to support the test environment is identified along with a preliminary test
4|Page
schedule. The test design step includes the number of tests to be performed, the ways the testing
will be approached and the test conditions that will be used. Finally the test development step is
carried out so that the automated tests can be reusable, repeatable & maintainable.
The fifth stage is execution and management of tests. All test plans are now executed according to
the standards discussed earlier in their plans.
The sixth stage is test program reviews and assessment. This stage is carried out throughout the
lifecycle so that there is continuous improvement and less faults at the end. This would reduce the
cost as faults found in earlier stages are cheaper to resolve than later stages.
So, in short ATLM “is structured methodology geared toward ensuring successful implementation of
automated testing”. [3]
REASONS TO AUTOMATE SOFTWARE TESTING [1]
There are quite a few reasons to automate. Manual testing is mundane and laborious task that can
lead to human errors. Small but important details of the software to the requirements can easily be
forgotten. Automated testing verifies that the developed software matches the requirements and
validates that that is what the organisation wants. Automation also helps avoid the defects and
failures. Many things can lead up to this, for instance, if the environments are setup incorrectly or
the wrong version of software is being deployed than that will cause a bug which will mean the
system will malfunction or fail.
Compatibility is also an issue when it comes to deploying a new version of software. It should be
compatible with the existing software for it to run smoothly. Automation ensures this.
The other reason to automate is that there are various other non-functional parameters that need to
be tested, for instance: performance, usability, security, scalability etc. These parameters are
skewed and vary from person to person for the level of tolerance. So Automation would be able to
provide the wide range needed.
TYPES OF AUTOMATED TESTS
The five types of automated tests are code auditing, coverage monitor, functional tests, load tests
and test management.
Code Auditing [4]
This test performs automated qualification testing. It checks the compliance of code to specified
standards and procedures of coding.
The code auditor verifies that the code is fulfilling code structure instructions and procedures. It
looks at the module size, levels of loop nesting and prohibited constructs such as GOTO.
5|Page
The code auditor checks that the coding style procedures are being followed by the coding style.
That is the naming conventions for variables and files and the unreachable code lines of the
program.
The auditor looks at the internal program documentation and help support sections to check if
they follow the coding style procedures. The code auditor also looks at the format and the size of
the comments.
Coverage Monitor [5]
This determines a quantitative measure of code coverage when the tests are run, which is an
indirect measure of quality. It finds areas of software that are not using a set of test cases. It also
adds more test cases to increase coverage. When implementing a given test file, it produces
reports about the line coverage achieved.
It is a vital tool for white-box testing. White box testing, that is also called glass box testing,
examines internal calculation paths in order to identify bugs.
Functional Tests
Functional tests are aimed at examining the overall functionality of the product. It replaces the
manual black box testing. Blok box testing is a type of testing that does not consider “inner
workings” of the software or its code. It only focuses on the externally visible behaviour.
The specification for this test is very detailed as every aspect of the system is being tested.
Functional testing tests interfaces thus it involves clients and uses alpha testing.
Alpha tests are tests of a new software package that are performed by customers at the
developer’s site.
Load Tests [6]
Load testing is when an application is tested under heavy loads and its response is measured.
There are some tests that are impossible to perform manually. For instance, if a website is being
tested manually, it is not feasible to get 100 people using different parts of the website at the
same time. So using load tests, virtual people can be created and allocated to use parts of the
website concurrently. Load testing simulates expected usage of a software program and records
on how the system reacts. Many things can be simulated. For example change of hardware,
change of scenario, change of hardware with different scenarios, virtual people, virtual events
and much more.
Test Management
This involves participants carrying out the tests and correcting the detected errors. The test
monitors performance of every item on the long lists of test case files. This workload makes
timetable follow up important to management.
6|Page
ADVANTAGES OF AUTOMATED SOFTWARE TESTING
It is important to choose the right tools for every aspect in an organisation. If this is done properly,
there are lots of advantages.
Automated test performs the same operation every time it is run and thereby eliminating human
error. Also each test can be repeated on difference versions of an application, thus saving money
and time. This results in the more tests run in less time using fewer resources, giving a good test
coverage.
This means that automation is faster and would overall work out to be cheaper than manual testing.
DISADVANTAGES OF AUTOMATED SOFTWARE TESTING [4]
Although automated testing has many advantages it also has its own disadvantages.
One big disadvantage of automated testing is that initially a high investment is needed to buy the
tools and train the staff to use the tools. Also in the test preparation stages a high man power is
needed. Another thing is that a lot of areas in testing are left uncovered. There are not enough tools
out there yet to cover all the tests so some testing has to be done manually.
Also a lot of skill is needed to write the automation test scripts, so it is expensive to develop the
tools to make them custom made.
CHOOSING AUTOMATION TOOLS
To choose the best tool for the task, there are a lot of things that need to be considered. First thing
is the ease of integration. The tool should easily integrate with the system and should be weighed
against the cost and performance. Also, the tool has to be compatible with the design and
implementation of the application. It should be able to identify all objects and different classes in the
application. The second thing that needs to be looked at is performance. How it performs in the
environment with the network traffic and the hardware is very important.
A type of tool is directly dependant on the type of tests. So the tests need to be taken into
consideration. Another important thing is maintainability. The support, online communities etc need
to be looked at. If there is enough support than problems will be resolved quicker.
Finally, the tools have to be affordable.
SUMMARY
To summarise this report, the pros outweigh the cons and even though manual testing can not be
entirely replaced, the features of automated testing surely helps to speed up the process of testing.
7|Page
REFERENCES
1. Reasons to automate: http://www.my-project-management-expert.com/softwareapplication-testing.html
2. http://www.exforsys.com/tutorials/testing/automated-testing-advantages-disadvantagesand-guidelines.html
3. Stages of lifecycle: http://www.informit.com/articles/article.aspx?p=21468&seqNum=5
4. Software quality assurance by Daniel Galin (Pages 235 – 245)
5. Coverage monitor: http://www.bullseye.com/coverage.html
6. Load testing: http://en.wikipedia.org/wiki/Load_testing
8|Page