SOFTWARE DEVELOPMENT FUNDAMENTALS: Software Testing and Maturity
A research report
Submitted to
University of South Africa
In fulfilment of the partial requirements for the
Baccalaureus Technologiae: Engineering: Electrical Computer Systems (BTELN - COS)
by
M.M Bosega
Student Number 47249897
47249897@mylife.unisa.ac.za
18 September 2015
Declaration of own work
I declare that this research is my own work. It is being submitted for the
BaccalaureusTechnologiae: Engineering: Electrical Computer Systems at the University
of South Africa. It has not been before for any degree, diploma, or other examination at
any tertiary educational institution.
Moses M Bosega
Signed:______________________by____________________________
Midrand
18
September
15
at_______________________on
this ___day
of_______________20___
2
Acknowledgements
The researcher would like to thank his mentor Albert Samathanda, and the research
participants for their support and participation in this study.
3
Abstract
Software solutions have become an integral part of our daily lives, given how much
technology has developed. Corporates and human beings put their trust in software
systems than in humans whether is for enormous financial transactions or life critical
systems or operations. In-depth studies have been conducted in the past, on improving
the quality of software and technology; however, there is always a possibility that
something will go wrong. Failure seems to be a part of every success, and this is the
approach that this study is taking to seek to investigate the software failures that are in
production of companies, successful or not. As technology is rapidly advancing, quality
measures and failure prevention methods are put in place. Software testing is a discipline
that has been in existence for over 30 years, and although the processes and
methodologies have been improved and upgraded, there seem to be always a room for
errors that are not detected and identified by the process. The inquiry is based on the
case study, where we will investigate the current software testing practice of company M,
and based on the results, recommendations will be made.
4
Table of Contents
Declaration of own work .................................................................................................... 2
Acknowledgements ........................................................................................................... 3
Abstract ............................................................................................................................. 4
Table of contents .............................................................................................................. 5
List of tables ....................................................................................................................... 8
List of figures ..................................................................................................................... 8
1.INTRODUCTION ........................................................................................................... 9
1.1 Background ............................................................................................................................................ 9
1.2 Problem Statement .............................................................................................................................. 13
1.3 Value of study....................................................................................................................................... 15
1.4 Research objectives.............................................................................................................................. 15
1.5 Research methodology ........................................................................................................................ 16
1.6 Summary .............................................................................................................................................. 18
2.RESEARCH IN CONTEXT .......................................................................................... 19
2.1 Overview .............................................................................................................................................. 19
2.2 Conceptual study.................................................................................................................................. 24
2.2.1 Conceptual Framework ............................................................................................................ 24
2.2.1 .1 SDLC Software Application testing ....................................................................................... 27
2.2.1 .2 Elements of Software tester ................................................................................................ 28
2.3 Summary .............................................................................................................................................. 29
3.RESEARCH DETAILED DESIGN ................................................................................. 30
3.1 Aims ...................................................................................................................................................... 30
3.2 Research methodology ........................................................................................................................ 31
3.3 Research type ....................................................................................................................................... 32
3.3.1 Qualitative research methodology .......................................................................................... 32
3.3.2 Quantitative research methodology ........................................................................................ 33
5
3.4 Sample description of case study ......................................................................................................... 34
3.4.1 Introduction ............................................................................................................................. 34
3.4.2 Departmental Structure ........................................................................................................... 35
3.4.3 Research design, case study..................................................................................................... 36
3.5 Conclusion ............................................................................................................................................ 37
4.RESEARCH FINDINGS – Case study .......................................................................... 38
4.1 Background in context of the case study .................................................................................... 38
4.2 Background in context of the company and the division ........................................................... 40
4.3 Research focus ............................................................................................................................ 47
4.4 Research questions ..................................................................................................................... 49
4.5 Research sub questions .............................................................................................................. 50
4.5.1 Development team ..................................................................................................... 50
4.5.2 System Integration Test team ..................................................................................... 55
4.5.3 User Acceptance Team ............................................................................................... 63
4.6. Case study data analysis ............................................................................................................ 68
4.6.1 Development team ..................................................................................................... 68
4.6.1.1 Testing skills, knowledge and competence................................................. 68
4.6.2 System Integration Test Team ....................................................................... 69
4.6.2.1 Testing skills, knowledge and competence................................................. 69
4.6.2.2 Does the test team know what to test? ..................................................... 70
4.6.2.3 Defect management ................................................................................... 70
4.6.2.4 Test planning and management ................................................................. 70
4.6.2.5 Testing techniques ...................................................................................... 71
4.6.2.6 Application and system level programming ............................................... 71
4.6.2.7 Matrix data.................................................................................................. 71
4.6.2.8 Metrics ........................................................................................................ 71
6
4.6.3 User Acceptance testing Team ................................................................................................ 72
4.6.3.1 Operations team performing UAT ........................................................................... 72
4.6.3.2 Business Analyst ....................................................................................................... 72
4.6.3.3 Project manager ....................................................................................................... 72
4.6.3.4 Test tools .................................................................................................................. 72
4.6.3.5 Testing knowledge, skills and competence.............................................................. 72
4.7 Literature review ......................................................................................................................... 73
4.7.1.1 Software Development Life Cycle ............................................................................ 73
4.7.1.2 Insufficient Software Test Training .......................................................................... 76
4.7.1.3 Test Automation ...................................................................................................... 77
4.7.1.4 Defect Prediction ..................................................................................................... 79
4.7.1.5 User Acceptance testing .......................................................................................... 79
4.7.1.6 Development Manager Role in Software testing ..................................................... 80
4.7.1.7 Risk based testing .................................................................................................... 80
4.7.1.8 Probability of failure ................................................................................................ 81
4.7.1.9 Testers not involved in JAD sessions........................................................................ 82
4.7.1.10 Test Maturity Model .............................................................................................. 82
4.8 Summary and conclusion ............................................................................................................ 84
7
List of Tables
Table 1: Development Team Responsibilities Matrix .................................................................................... 52
Table 2: SIT responsibilities Matrix ................................................................................................................ 58
List of Figures
Figure 1: High level conceptual Framework................................................................................................... 26
Figure 2: conceptual frame work (Hall,2011) ................................................................................................ 27
Figure 3: Skill set of Software Tester(Slavchev ,2015) ................................................................................... 28
Figure 4: Organizational structure of a production team .............................................................................. 35
Figure 5: Billing System Functions.................................................................................................................. 41
Figure 6: Levels of Roles and Responsibilities ................................................................................................ 42
Figure 7: Billing Process.................................................................................................................................. 46
Figure 8 : Traditional Software Testing Environment (TYSS,2014) ................................................................ 46
Figure 9: Development Team Structure ......................................................................................................... 50
Figure 10: Time Spent by Development team in testing ............................................................................... 54
Figure 11: SIT Team Structure ........................................................................................................................ 56
Figure 12 : Percentage Representation of SIT Team ...................................................................................... 60
Figure 13: Billing and Production Team Structure ......................................................................................... 63
Figure 14: Development and Unit Test process ............................................................................................. 69
Figure 15: Waterfall Model (Bassil,2011)....................................................................................................... 74
Figure 16: Generic Structure of Testing Process (Tian, 2005) ........................................................................ 75
Figure 17: Risk Definition and Structure(Schaefer, 2005) .............................................................................. 81
Figure 18: Failure Probability(Schaefer, 2005)............................................................................................... 82
Figure 19: TMM levels(Veenendaal,2006) ..................................................................................................... 83
8
CHAPTER 1
1. INTRODUCTION
The topic under discussion is Software Development Fundamentals: Software Testing
and Maturity, and the emphasis will be on Software testing process and maturity. The
primary purpose of this chapter is to outline the background of the study, problem
statement, the significance and the value of the study.
1.1 Background
The research is centred on the study field Software Engineering, which is defined as the
study of design, development, and maintenance of software (Beray & Mohommed, 2013).
Software testing is a phase in a conventional SDLC (Software Development Cycle Life)
and it is defined as a process, or a series of processes, designed to make sure computer
code does what it was designed to do and, conversely, that it does not do anything
unintended (Myers, 2012), furthermore, It is an activity which is aimed for evaluating
quality of a program and also for improving it, by identifying defects and problem (Quadri,
2010). In the RSA (Republic of South Africa) it is becoming popular as companies have
realized its importance as being the center of Information technology and software
development because uncovered defects in the production environment have greater
impact as they are affecting the customer(s) and operational process.
Good quality software and systems arguably inspires customer confidence, and improves
the company’s reputation, because with good reputation the market is sustainable and the
growth is always on the rise. Software quality is defined as (1) the degree to which a
9
system, component, or process meets specified requirements. (2) The degree to which a
system, component, or process meets customer or user needs or expectations (IEEE
610.12, IEEE Standard Glossary of Software Engineering Terminology).
Due to the rapid growth and development of technology, the increased demand of quality
software solutions, and the increasing complexity of the software, the testing team, and
personnel are faced with enormous testing challenges to deliver reliable, trust worthy and
quality solutions, to prevent operational failures. Software and technology have become
an integral part of our daily lives and we don’t realize the impact of it until something
wrong happens. Furthermore, Software is critical to life because its quality deficiency can
lead to permanent disability and/or even loss of life, and this makes this study even more
significant due to some of the events in the past that lead to loss of life
In February 1991 during the Gulf War, 28 American Soldiers were killed and 100 more
were injured and this was attributed to the Software problem that lead to system failure
which failed to track and intercept an incoming Iraqi Scud missile (Blair & Obenski &
Bridickas, 1992).Companies lose revenue and income due to software failures, even
though adequate testing has been performed using the proven software testing
methodologies. The national institute of standards and technology, did a study and
reported that software bugs and defects costs the United States economy $59,5 billion
annually, and the study estimated that a third of the amount could be eliminated by
improving testing(NIST, 2002).A national survey has been conducted in the Netherlands
that sheds light on the economic impact of software failure, and the results are alarming,
to say the least. According to the research, software errors lead to 1.6 billion euros in
productivity and revenue losses (Tahapary, 2014).
10
The end-user and customers are always on the receiving end of software failures as
reported in mybroadband that an Australian bank began giving out large sums of money
from 40 cash machines across one city. Officials at the company said they were operating
in stand-by mode, so could not identify the account balances of customers
(mybroadband, 2011).
The study seek to provide a comprehensive coverage of the ill-defined problems that
prohibits organizations, and testing teams from delivering the solution that functions with
99,99 % accuracy to prevent loss of life, loss of income, yields returns of investment and
increase the reputation of the organization. Although South African literature has very
few details about the software failure rate in South Africa, Ernst & Young reported that
Global Information Security Survey 2002, the top causes of business interruption failures
are hardware or software failure (56%) and telecommunications failure (49%),
(Danton,2002).The study will outline the maturity level of software testing given that
various researches and studies on different methodologies of testing have been
previously performed. Testing process has to verify and validate whether the software
fulfills conditions laid down for its release/use (Quadri, 2010), hence Verification and
Validation are the essential aspects of software testing. Verification means built it right
(ISO/IEC 12207, Software life cycle processes), and Validation means built the right thing
(ISO/IEC 12207, Software life cycle processes). Academic initiatives have been taken by
various institutions and organizations to improve the standard and quality of testing
worldwide and this include the introduction of various software testing certificates.
In 1989, ISEB (Information System Examination Board) which is an examination awarding
body and part of BCS (British Compute Society) introduced ISEB Certificate for Software
11
Testing from foundation level to higher level to educate the testing community and help
them understand the fundamentals and requirement of software testing and ultimately
release quality software, which conforms to the requirements(ISEB, 2007). In March 2010
the ISTQB® UK Testing Board (UKTB) and BCS, The Chartered Institute for IT agreed to
partner to provide software testing examinations at Foundation and Advanced levels. This
was a significant move by the Institute to update its software testing accreditation and
certification scheme at Foundation and Advanced level to the ISTQB® certification
program. The Institute recognised the importance of a globally recognised scheme to
support the career development of testers and with the UKTB agreement has committed
to promoting ISTQB® certification as the industry standard (ISTQB,2010). There are
about 4300 ISTQB foundation certified testers, 330 ISTQB Advanced certified, and 345
ISTQB Test Manager certified in RSA (SASTQB, 2015).Various Tertiary Education
Institutions are offering Information Technology, Computer Science, and Engineering
qualifications with the curriculum including Software Engineering, and software testing
process. Despite the tremendous effort spent on software quality improvement, more than
half of all faults (also referred to as defects) are still not found during testing, but after
shipment (Middleton & Sutton, 2005) and for decades, users of software solutions have
been suffering from poor solution quality (Whittaker & Voas,2002). This study will
fundamentally look and the current industry practice of software testing process in relation
with the existing literature, studies and curricula.
12
1.2 Problem Statement
Based on the background of software testing process, the problem statement will be
defined as the list of questions that this study will have to answer.
The main question however is,
Why do companies have unknown, and uncovered high severity defects in the
production environment, given all the studies that have been carried out about
software testing process?
The question is inspired by the operational experience of software testing in various
organizations. For the study to provide an adequate answer to the main question, the
following list of items will have to be discussed,

Competency encompasses the knowledge, skill, abilities, traits, and behaviors that
allow an individual to perform a task within specific job or function (Vathanophas,
2007). In RSA, Software testing has no official qualification, and yet it is a specialized
discipline. Should the testing team have the following skills and abilities?
o Application level programming
o System level programming
o System Analysis
o Business Analysis
o Data Modelling
o Time Management

Does the testing team know what to test?
13

Software Measurement. Halsted’s software science, A lexical analysis of source
code (Halsted,1977)

Complexity Metrics. The complexity of a program is defined in terms of its control
structure and is represented by the maximum number of “linearly independent”
path through the program (Lee &Chang,2013).

When should testing begin?

Software testing terminology

How does the testing team know that they have compiled the right requirements?

Reliability Metrics. The availability measure of software is the percentage that a
program is operating according to requirement at a given time and is given by the
formula, Availability = MTTP / (MTTF +MTTE)* 100% (Lee, Chang,2013).

Test cases

How much time should be allocated for testing?

Proven development and testing methodology

Testing Environments

Failure analysis

Implementation for the sake of meeting project time lines

Manual and test Automation

Test Tools

Outsourcing of software testing tasks

Impact Analysis

Risk Assessment

Software Release
14

Knowledge base

Project management

Defect Management

Test Reporting

Communication

Roles and responsibilities
1.3 Value of study
This is an empirical study and will be based primarily on the software testing experience,
operational experience and literature review, and may be used by the organization to
which I am employed to re-asses its testing structure and processes.
This study should provide an important feedback to management about the position of
software testing in the company, and will help in the future planning of software testing in
the organization, as restructuring is inevitable.
1.4 Research objectives
PRIMARY OBJECTIVE
The general objective of the research is to find the root cause of having high severity
defects in the production environment.
SECONDARY OBJECTIVE
The second objective is to identify alternatives of addressing the root cause.
15
1.5 Research methodology
 The first methodology for this research is to conduct a theoretical study of literature
reviews which (Hart, 1998) defined as the selection of available documents (both
published and unpublished) on the topic, which contain information, ideas, data and
evidence written from a particular standpoint to fulfil certain aims or express certain
views on the nature of the topic and how it is to be investigated, and the effective
evaluation of these documents in relation to the research being proposed.
Questions a literature review can answer (Hart, 1998)
o What are the key sources?
o What are the key concepts, theories and ideas?
o What are the epistemological and ontologica grounds for the discipline?
o What are the main questions and problems that have been addressed to
date?
o How is knowledge on the topic structured and organised?
o What are the origins and definitions of the topic?
o What are the political standpoints?
o What are the major issues and debates about the topic?
Data sources:
o Internet
o Textbooks
o Journals
o Articles
o Magazines
16
o Library Catalogues

The second methodology for this research is an empirical study.
To understand how software engineers construct and maintain complex, evolving
software systems, we need to investigate not just the tools and processes they use, but
also the social and cognitive processes surrounding them and this requires the study of
human activities. We need to understand how individual software engineers develop
software, as well as how teams and organizations coordinate their efforts (Easterbrook &
Singer & Storey & Damian, 2007).The case study methodology is well suited for many
kinds of software engineering research, as the objects of study are contemporary
phenomena in its natural context , which are hard to study in isolation (Höst,2008).
THE STRUCTURE OF AN EMPIRICAL STUDY

Research context

Hypotheses

Experimental design

Threats to validity

Data analysis and presentation

Results and conclusions
(Perry & Porter & Votta ,NO DATE)
17
1.6 Summary
The study focuses on determining the gaps in the software testing process that leads to
undiscovered critical defects after release, from both theoretical and empirical
perspective. A case study will be carried out pertaining to the organization that the
researcher is employed. Key questions were asked in the problem statement and the
background study was carried out to make this research significant and essential.
18
CHAPTER 2
2. RESEARCH CONTEXT
2.1 Overview
SOFTWARE IS EVERYWHERE, It's what lets us get cash from an ATM, make a phone
call, and drive our cars, and typical cell phone now contains 2million lines of software
code (Charette, Uknown), however, proving that a program is fault free is equivalent to
the famous halting problem of computer science, which is known to be impossible
(Jorgensen, 1995). (Norman; Omlin, Unknown) of the Department of Computer Science in
the University of Western cape stated that, Software is increasingly becoming an integral
part of our daily lives because of its presence in services and in consumer products and
South Africa is now fully integrated within the global economy and any products produced
here (e.g. software) must be of the highest quality in order to survive the competitive
offerings available to software users, and they further more explain that Software is also
present in safety-critical systems whose mal-function can cause loss of life and/or
destruction to the environment. To further outline the importance of Software in our
corporate lives, I have stabled through (Raj, Unknown) that an Enterprise Resource
Planning (ERP) is the latest high end software solution, Information Technology has lent
to the world of business application, and An ERP software solution seeks to streamline
and integrate operations, processes and information flows in an enterprise, to synergize
the resources of an organization namely men, material, money and machines. Given that
our lives are so dependent on Software, why is a fault free software not guaranteed?
James Whittaker, Chair of the software engineering program at the Florida Institute of
Technology, has noted that DESPITE THE COUNTLESS HOURS THAT GO IN TO CODE
19
DEVELOPMENT AND SEEMINGLY ENDLESS CODE REVIEWS, BUGS AND
DEFECTS STILL ARE FOUND IN THE PRODUCTION RELEASE, WHY? (Bentley,
Unknown).
The question asked by James Whittaker, as cited by Bentley, is the key question of this
research, and this chapter. As stated in the previous chapter of Problem Statement
Definition that, Why do companies have unknown, and uncovered high severity defects in
the production environment, given all the studies that have been carried out about
software testing process?. (Bentley, Unknown) continues to state that, A big part of his(i.e
James Whittaker) answer is a lack of understanding of software testing and,
consequently, inadequate software testing processes and procedures. The amount of
defects in the production systems seem to be across all the technological sectors, i.e
Automobile, Banking, Health, Telecommunication, e.t.c.
From South African perspective, (Stefano,Johan 2011) on their journal, stated that small
software companies make for the majority of software companies around the world, but
their software development processes are often not as clearly defined and structured as
in their larger counterparts, especially the test process is often the most neglected part of
the software process, and this contribution analyses the software testing process in a
small South African IT company, here called X, to determine the problems that currently
cause it to deliver software fraught with too many defects, and they furthermore explain
that solutions to those (or similar) problems often already exist, but a major part of the
problem addressed in this contribution is the unawareness, or unfamiliarity, of many
small-industrial software developers and IT managers as far as the scientific literature on
software science and engineering, and especially in our case: software testing, is
20
concerned. On a much more global perspective, a recent audit, the OIG identified 23,868
high-risk vulnerability instances in the JPSS ground system for the second quarter of the
fiscal year (FY) 2014, which is up from 14,486 high-risk vulnerabilities identified in the first
quarter of FY 2012 (Greenberg,2014). To further add on high-risk Vulnerabilities, and
production defects, the automobile industry seems to have a pattern of most common
production defects, of airbags. Several articles have been published of recalling cars with
airbags defect. An article titled : “Nissan recall: 990,000 vehicles for airbag software
defect” was published on csmonitor. The article further explained that, Nissan recall
includes Altima, Leaf, Pathfinder, Sentra, and Infiniti models with a software glitch that
could prevent certain airbags from deploying during collisions, and that Nearly 990,000
vehicles are affected by the Nissan recall (Read,2014).
Toyota recalled about 700,000 vehicles due to a software problem with the vehicles’
computers which caused them to simply stop running, and related to the same recall,
certain RAV4, Tacoma, and Lexus RX350’s, affecting another 300,000 vehicles, need a
reprogramming of the ECU software to alleviate a problem which shuts off the Vehicle
Stability Control system rendering it ineffective in a situation where it is needed. This also
affects anti-lock brakes and traction control (Tracy, 2014)
(Campbell,2015) published that Jaguar Land Rover North America will recall more than
61,000 vehicles after finding that some passenger-side air bags may not open on impact
because of faulty software, and to add to the never ending recall of cars due to airbag
software defects (Jensen,2014) published that Audi is recalling almost 102,000 A4 and
S4 sedans and All road station wagons from the 2013-15 model years because a
software problem could cause the front airbag to malfunction, according to a report the
21
automaker posted Wednesday on the National Highway Traffic Safety Administration
website. Techtimes in 2015 published that Fiat Chrysler Automobiles (FCA) is recalling
more than 228,000 Jeep Cherokees equipped with airbag software that can cause the
airbag to inflate even without a crash, the recall, which is the latest in a long series of
airbag troubles affecting the automotive industry, involves more than 168,000 Jeep
Cherokees from model years 2014 and 2015 in the United States and most of the
remaining vehicles are in Canada, while a small percentage is distributed across the
world(Arce,2015).
In a statement obtained by Bloomberg, Ford said it is recalling a total of 627,275 Escape
sports utility vehicles (SUVs) and 65,192 C-MAX gas-electric hybrids,the recall involves
2013 and 2014 models and concerns car owners in the U.S., Canada and Mexico and the
SUVs were built at the Louisville Assembly plant between October 2011 and February
2014, while the C-MAX hybrids were built from January 2012 to February 2014, and the
recall is due to a software defect that prevents the side curtain airbags from inflating in the
event of a rollover crash(Arce,2015). In what's now the largest vehicle recall ever, more
than 33.8 million vehicles made by 11 companies have been recalled for faulty air bags,
which can randomly explode and send shrapnel at passengers, the companies affected
by the recall include: BMW, Chrysler, Daimler Trucks, Ford, General Motors, Honda,
Mazda, Mitsubishi, Nissan, Subaru and Toyota(Bennett,2015). In the Banking Industry,
Exclusive The tech problems at the RBS banking group that left millions of people unable
to access money for four days last week were caused by a failure in a piece of batch
scheduling software, sources have told The Register, and at least some of the support
staff
for
that
software
have
been
outsourced
22
to
India
-
as
recently
as
February(Leach,2012). The software failures ranges from life threatening failures to
embarrassing failures. Earlier this year a man lost a $57 million jackpot when a casino
alleged a "software glitch" on the slot machine, Well, that's nothing compared to the
backlog of $9 billion in unprocessed payments that happened in Japan in
March(Diaz,2011). Here's a good one—for those who were able to enjoy the glitch. A
Commonwealth Bank ATM network bug caused the machines to dispense large amounts
of money to random people, Police actually arrested two people who took the mistakenly
spit-out money, saying that it was a crime. No word about the hundreds who took the
money and ran—and got away (Diaz,2011).
This study will present research finding, and more questions to the question asked above
by James Whitaker as stated earlier that despite the countless hours that go in to code
development and seemingly endless code reviews, bugs and defects still are found in the
production release, Furthermore, this chapter will address Software Testing in relation to
the following questions: WHAT, WHY, AND WHO.
(Bentley, Unknown) continues to define that Testers are the only IT people who will use
the system as heavily an expert user on the business side and that user testing
almost invariably recruits too many novice business users because they’re available and
the application must be usable by them, and he interestingly define that the problem is
that novices don’t have the business experience that the expert users have and might not
recognize that something is wrong, and that testers from IT must find the defects that only
the expert users will find because the experts may not report problems if they’ve learned
that it's not worth their time or trouble.
23
The interesting view of Bentley brings us to the next question of capacity, and the maturity
thereof. The Process Capability Maturity Model (CMM) was developed by the SEI at
Carnegie-Mellon University, and CMM is a conceptual framework that represents process
management of software development (Joseph Raynus, 1998)
2.2 Conceptual study
2.2.1 Conceptual Framework
A conceptual framework is described as a set of broad ideas and principles taken from
relevant fields of enquiry and used to structure subsequent presentation (Reichel &
Ramey, 1987), furthermore a conceptual framework is used in research
to outline
possible courses of action or to present a preferred approach to an idea or thought and It
is also a type of intermediate theory that attempt to connect to all aspects of inquiry e.g.,
problem definition, purpose, literature review, methodology, data collection and analysis,
(Pineda,2011). Software project failures have a lot in common with airplane crashes. Just
as pilots never intend to crash, software developers don't aim to fail. When a commercial
plane crashes, investigators look at many factors, such as the weather, maintenance
records, the pilot's disposition and training, and cultural factors within the airline. Similarly,
we need to look at the business environment, technical management, project
management, and organizational culture to get to the roots of software failures
(Charette,2005). This chapter introduces the theoretical study of Software Development
Fundamentals: Software Testing and Maturity. Theory is a belief or assumption of how
things relate to each other and it establishes a cause and effect relationship between
variables with a purpose of explaining and predicting phenomena based on inductive
reasoning (Refat, 2013).Studies have previously been performed about software failures,
24
although less emphasis has been done on the ability and capability of software testing
process to be able to uncover almost all the critical and high severity defects, and it is
critical to note that as according to (Dijkstra,1972) program testing can be used to show
the presence of bugs, but never to show their absence, however, is testing showing the
presence of critical defects that are life threatening ?
This study is fundamentally about the maturity of testing, and the process followed to
uncover defects.
The focus area will be on:
Resources
Environment
Tools
Application or System under test
Roles and Responsibilities
Test Data
Test Approach
Test Strategy
Capability Maturity Model (CMM)
User/Customer Experience
And this will be mapped to the list of ill-defined problems that the study will have to
answer as defined In the previous chapter of problem statement definition.
25
Business Analyst(s)
REQUIREMENTS
SPECIFICATIONS
User
Acceptance
testing
Business Case/
Project Scope
Manage
Implementat
ion
Project Co-ordination
Change Management Team
Project Manager
Business
Case
BUSINESS
Application
TEST
RESULTS/
TEST
REPORT
User Acceptance
Testing
BUSINESS USERS
DEVELOPMENT TEAM
Technical
Specification
TESTING TEAM
Unit Testing
SYSTEM TEST
Code/System
Development
Figure 1: High level conceptual Framework
The high-level conceptual framework depicted on Figure 1 defines: The bi-directional
relationship of different resources in facilitating, co-ordinating and implementing project
activities. The Project manager is at the centre of the study as the coordinator and
facilitator to bring unity of various elements that make up a project team. The key focus
will be on the Testing Team, the testing process and the testing maturity, and the inability
to uncover high severity and critical defects given the external and internal environments
effects.
26
2.2.1.1 SDLC and Software Application Testing
Figure 2: conceptual frame work (Hall,2011)
The conceptual frame work on Figure2 defines the high-level application software
development life cycle, with emphasis on the type of test, test levels and testing steps,
until roll out and implementation. Software testing services include
-
The full life cycle of testing service, i.e from requirement analysis, planning, design,
and development, to execution and reporting (Hall, 2011). In the software
development phase, testing begins at the component level, and different testing
techniques are appropriate at different points in time. Testing is conducted by the
developer of the software, as well as an independent test group
(McGraw-Hill,1999)
-
Development Test, system test, acceptance test and non-functional testing.
27
The framework depicts the close relationship software testing and development teams
have with each other. The literature review and research design will explain further the
testing processes and testing techniques of the application and system testing. The
framework furthermore evaluates Software development life Cycle as a phase within the
project life cycle, and within the project life cycle we will further define the software testing
process, and the ideal skills of the testing team.
2.2.1.2 Elements of Software Tester
Figure 3: Skill set of Software Tester(Slavchev ,2015)
Part of the problem statement definition was a question of the skill-set of the testing team.
This conceptual framework further reveals and defines the ideal skill-set of the software
test in an ideal world. The complexity of the system and rapid growth of technology
involve much more robust coding and technical development where the testing team is
more likely required to have the skill to can interpret the code, understand the
28
requirement and can speak the technical language with the development team. The
conceptual frame-work depicts the different elements of the testing team.
2.3 Summary
The chapter detailed the background and the context of the research to help the reader
understand why the research is carried out. The underpinnings of the research have been
explained, with the aid of theoretical review not limited to journals, articles and previous
studies of the same relevance.
Based on the foundation laid, the study will continue with the design and the detailed
description of the research in the next chapters to follow.
29
CHAPTER 3
3. RESEARCH DETAILED DESIGN
3.1 Aims
The aim of the research is to establish why software production environments in various
organizations are experiencing high severity defects even though the software has gone
through a fundamental test process to uncover defects. The number of high severity
defects in the production environment ranges across different business disciplines,
including but not limited to Motor-Vehicle , ICT, banking, insurance, Aviation, and health
Sectors. The study will unpack different elements that are involved in deploying software
solutions to production and this are all the elements that form part of the Fundamental
test Process of Software Development Life Cycle. This is a test process that is
documented in the standard BS7925-2 Software Component Testing, It therefore relates
most closely to component testing but is considered general enough to apply to all levels
of testing (i.e. component, integration in the small, system, integration in the large, and
acceptance testing) (Ghahrai,2008).
(Pal, Uknown) list the steps of Fundamental Test Process as:

Planning and control

Analysis and design

Implementation and execution

Evaluating exit criterion and coverage

Test closure activities
30
Specific research questions have been issued and detailed in the previous chapters and
they are essential towards the development of the research design. The summary of
questions is as below:
i.
Why do companies have unknown, and uncovered high severity defects in the
production environment, given all the studies that have been carried out about
software testing process?
ii.
Why testing cannot uncover all the defects ?
iii.
When Should testing begin and End?
iv.
Does the testing team know what to test?
v.
How does the testing team know that they have the right requirements
vi.
What are the most efficient and effective test tools
vii.
What skills set must the testing team posses
viii.
Minimum academic qualifications required for a testing personnel
For the above summarize questions to be fully addressed, a case study will need to be
formulated, as a base of qualitative and inductive research.
3.2 Research methodology
Research methodology is the way to systematically solve the research problem (C. R.
Kothari, 2004). It refers to the choices that the researcher makes about cases to study,
and methods to be used for data collection and data analysis planning as well as carrying
out the research study (Silverman,2004), and research methodology helps a researcher
in identifying the problems, formulating the problems and hypotheses, gathering
information, participating in the field work, using appropriate statistical tools, considering
31
evidences, drawing out inferences from the collected information or experiment (Sahu,
2013).
3.3 Research type
The design for a research project is literally the plan for how the study will be conducted
(Berg, 2001), furthermore the design stage of the research is concerned with a series of
important decisions having to do with the research idea or question(s) (Berg, 2001), and it
is defined in details as the glue that holds the research project together, and it is used to
structure the research, to show how all of the major parts of the research project -- the
samples or groups, measures, treatments or programs, and methods of assignment -work together to try to address the central research questions (Trochim,2006). The
primary focus will be addressing the why questions of the research. Answering the ‘why’
questions involves developing a causal explanation (De Vaus,2001).
3.3.1 Qualitative research methodology
Interviews are a primary source of data in qualitative research; so too are observations,
observations are common in many kinds of qualitative research, such as in case studies,
ethnographies, and qualitative action research studies (Merriam; Tisdell, 2015)
.Qualitative research yields detailed information report in the voices of participants and
contextualized in the settings in which they provide experiences and the meaning of the
experiences (Cresswell,2008). The goal of qualitative research is understanding issues or
particular situations by investigating the perspective and behaviour of the people in these
situations and the context within which they act, and to accomplish this, qualitative
research is conducted in natural settings and uses data in the form of words rather than
numbers (Kaplan and Maxwell, 2005) .Their primary interest in qualitative research is to
32
achieve understanding of a particular situation, or individuals, or groups of individual, or
(sub)cultures, etc., rather than to explain and predict future behaviors as in the so-called
hard sciences, with their arsenal of laws, theories, and hypotheses employed or rejected
on the basis of their predictive value. In summary, qualitative methods are primarily
inductive,
in
contrast
to
the
deductive
methods
of
experimental
science(Bendassolli,2013). Inductive reasoning is based on learning from experience.
Patterns, resemblances and regularities in experience (premises) are observed in order to
reach conclusions or to generate theory( Research-methodology.net, 2015). The article(
Research-methodology.net, 2015) further more list that the concepts associated with
qualitative methods are :
Type of reasoning
Type of question
Type of analysis
Induction
Subjectivity
Meaning
Open-ended
Process-oriented
Narrative description
Constant comparison
Qualitative research will gather information through interviews of directly involved
Software testing personnel and steak holders, statistics (skill Level), management style,
project management, and risk management disciplines.
3.3.2 Quantitative research methodology
Quantitative research has traditionally provided a measurement orientation in which data
can be gathered from many individuals, and trends assessed across large geographic
regions(Cresswell,2008) in addition, Quantitative research is all about measuring
relationships between variables and relies primarily on numbers as the main unit of
33
analysis, furthermore quantitative data provides a better understanding of the usability of
a product because it allows you to measure the effectiveness (Can users successfully
achieve their objectives?) and efficiency (How much effort and resource is expended in
achieving those objectives?) of your product (Breeze and Conomos, 2013). This is an
appropriate approach for software reliability, quality, and failure measures.
3.4
Sample description of case study
3.4.1 Introduction
The company which will be understudy will not be named because of the confidentiality
clause and agreement(s) in place. The company was given a licence to operate in 1993,
and its operations were officially launched in 1994, and it is South African based
company, with Africa entities in Lesotho, DRC, Tanzania, Mozambique, and Zambia. This
case study will be carried out in relation with the testing process and software maturity in
the IT : Billing, Production and Testing division of the company. The high-level functions
of the division is Business Solutions support, Operations System Support, System
Analysis and Development. The department serves as a central point and an essential
component in all the business proposed enhancements and change request. The
personnel are able to provide expert analysis, and the impact of the business
requirements, and how plausible or implausible the requirements are.
In context the division manages full delivery lifecycles from demand gathering, through
initial and on-going budget requests and re-balancing throughout the year; contract
management in consultation with legal and SCM. The division employs project managers
and business analysts that process Business As Usual(BAU) as well as strategic
transformation requests from the business from inception through assessment, delivery,
34
deployment to production and then 3rd level specialist support as needed. This includes
functional, non-functional, governance and reporting requirements.
3.4.2 Departmental structure
Figure 4: Organizational structure of a production team
The billing system Vendor has the consultant onsite for the maintenance of the
application and for any changes and enhancements in the system functionality as per
business needs and demands. Software release is continuous and the testing team is
constantly making decisions on what to test next. The testing function is outsourced to a
software testing company. The case study will depict only the team that is involved in the
delivery of the software, and no focus will be paid at other integrating divisions of the
company. The interaction and functional flow process will be explained in details under
the discussion section. From this we can define the research questions, data gathering
and collection as well as evaluation the data to prepare the report.
35
3.4.3 Research design, case study
The research design for this study is a descriptive and Interpretive case study. (Polit &
Hungler 1999) defines descriptive case study as the type of research that describes what
exists and may help to uncover new facts and meaning, and the purpose of descriptive
research is to observe, describe and document. The case study will be analysed through.
A Case study is not a methodological choice, but a choice of object to be studied, and the
name case study is emphasized because it draws attention to the question of what
specifically can be learnt from the single case (Stake, 1994). The basic idea is that one
case (or perhaps a small number of cases) will be studied in detail using whatever
methods and data seem appropriate, while there will be specific purposes and research
questions, the general objective is to develop as full an understanding of this case as
possible(Punch, 1998)
A case study analysis focuses on a small number of cases that are expected to provide
an insight into a causal relationship (Gerring, Unknown), and it will emphasize detailed
contextual
analysis
of
a
limited
number
of
events
or
conditions
and
their
relationships(Springs, 1997), and it proposed six steps that should be used:
This introduction to case study research draws upon their work and proposes six steps
that should be used(Springs, 1997),

Determine and define the research questions

Select the cases and determine data gathering and analysis techniques

Prepare to collect the data

Collect data in the field
36

Evaluate and analyze the data Prepare the report
Case studies do not claim to be representative but the emphasis is on what can be
learned from a single case (Tellis, 1997).
3.5
Conclusion
This chapter summarized research questions, and discussed qualitative, quantitative, and
inductive researches. The research methodology discussed will guide in collecting the
data as we attempt to identify variables, and to understand their relationship to build up
the theory.
37
CHAPTER 4
4. RESEARCH FINDINGS – Case Study
4.1 Background in context of the STUDY
The purpose of this research is to look at the reason(s) behind software critical defects in
production.
Software bugs are widespread, although, effective bug detection tools have been
proposed, many software bugs inevitably slip into production runs and they have led to
many severe production-run failures, causing huge financial loss and threatening people’s
lives (Arulraj & Jin & Lu, 2014). Software failures greatly reduce system availability and a
recent study showed that software defects account for up to 40% of system failures and
that Memory-related bugs and concurrency bugs are common software defects, causing
more than 60% of system vulnerabilities and for this reason, software companies invest
enormous effort and resources on software testing and bug detection prior to releasing
software, however, software failures still occur during production runs since some bugs
inevitably slip through even the strictest testing (Qin & Tucek & Sundaresan & Zhou,
2005).
To further put emphasis on software failures in production and operations, software glitch
forced the New York Stock Exchange to halt trading for nearly four hours Wednesday, an
outage that unnerved Wall Street and revived concerns about the fragility of the
technological systems that underpin financial markets, and to further accentuates the
gravity of the failure, President Barack Obama was briefed on the NYSE outage,
38
according to the White House press secretary, and the root cause was determined to be a
configuration issue (Hope & Vaishampayan,2015).
Different organizations are using different software testing processes and software testing
methodologies, and for that reason, a case study was chosen as a research approach to
narrow the scope of the research.
The study of an entire software project as it progresses overtime is however extremely
challenging and, as a result researchers tend to focus on some aspect aspects of a
software project for their case studies (Runeson & Host & Rainer & Regnel, 2012).
Basically, a case study is an in depth study of a particular situation rather than a
sweeping statistical survey, and It is a method used to narrow down a very broad field of
research into one easily researchable topic (Shuttleworth, 2008). Whilst it will not answer
a question completely, it will give some indications and allow further elaboration and
hypothesis creation on a subject (Shuttleworth, 2008). Case studies are very suitable for
industrial evaluation of software engineering methods and tools because they can avoid
scale-up problems, and the difference between case studies and experiments is that
experiments sample over the variables that are being manipulated, while case studies
sample from the variables representing the typical situation(Esernet,2003).
39
4.2 Background in context of the Company and the Division
The company division under study is the IT: Billing and Production division of a
telecommunications company. For adherence to privacy and confidentiality clauses and
policies, we will name the company, Company T. The division under study is a central
point of operations support, business solutions support, development, configuration and
software testing of business and operations for billing and invoicing solutions. Solutions
development and changes can either be core development where a code is generated to
build, change, or enhance the solution, or the development can be configuration based,
and all this is dependent on the functionality and changes requested by business. The
customers that the division service are:
a. Contact Centres
b. Business Users
c. Retail Shops
d. Network Users
e. Finance
f. And Reporting Team
The division manages and maintains 5 500 Products and 2 055 Services
The billing system comprises of the Customer Relationship Management, Product Catalogue, Subscriber Management, Billing Engine, and data repository.
As can be seen from Figure 5 the systems functions mapping which presents different
functions that can be executed within the billing system.
40
Figure 5: Billing System Functions
The division, and its staff personnel understand how each function relate to the other, and
the business process from one function to the other.
There are other various systems which integrates with the billing system using Web
services and Application Program Interfaces, and various system modules within the
billing systems which are not relevant for this study. The billing system Vendor has the
consultants’ onsite for the management and maintenance of the application and for any
changes and enhancements in the system functionality as per business requirements,
needs and demands. Software release is continuous and the testing team is constantly
41
making decisions on what to test next based on the urgency of the required functionality
to keep business operating as usual, as stated in the previous chapter.
Customer Relationship Manager and the Billing system Modules are managed and
maintain in various level. Figure 6, presents the different levels of roles and
responsibilities in relation to the application management.
Figure 6: Levels of Roles and Responsibilities
42
Application task team is the team of individual who perform different functions on the
Customer Relationship Manager which interfaces with the billing system. The different
roles and responsibilities are defined as below:
a) Task Team Coordination and measurement
The team is primarily responsible for First Line Support and management of user and
customer queries which are reported on incident reporting tool and then get assigned
to the relevant Operations and business support specialist.
b) Application management Monitor team
The team is responsible for Monitoring processes and jobs that are scheduled, and to
initiate unscheduled processes, and to process system backup.

Operations System Support and Business Solution Support team
is
responsible for the following tasks:

Handling of Users and Customer queries, fix and resolve incidents within the
agreement service level agreement(Incident Management)

Billing data integrity checks and maintenance

Processing of critical systems jobs

Processing Billing and generate Output

Pre and Post billing business and operations Support

Reconciliation of bill data

Reconciliation of data, packages and services between billing system and the
network

Revenue Assurance
43

Incident Management

System Analysis for Impact Assessment and advices on the feasibility of
proposed business requirement

Attend Joint Application Design Sessions

Attend Design Review Meetings

Review Functional Specifications

Review Business Requirements Specifications

Review Technical Product Development Document

Approve or Reject Proposed business and Functional changes

Perform Testing(User Acceptance)

Document Processes

Perform Configuration of Products and Services

Development

Identify Operating Systems Requirements

Identify Resources Requirements

Plan the project in terms of:
o Work breakdown structure
o Time lines
o Precedence analysis
o Resource assignment
o Costing
o Critical path and PERT chart
o Gantt charting
44
o Budgeting
o Resource alignment
o Generate Reports
c) Development Team is responsible for the following:

Manage Code Development and Delivery Expectations

Define Functional Specifications

Define Technical Design Specifications

Development of the system/Coding

Unit Testing

Environment Maintenance

System Configuration

System Analysis

Feasibility Study

Incident Management

Facilitate Changes to be implemented

Coordinate System Testing

Coordinate User Acceptance Testing
d) System testing Team is responsible for the following:

Review Specification Documents

Design Test Cases

Capture Test Cases on Testing Tools

Execute Test Cases

Sign Off test result and Implementation Sheet
45

Test Reporting
The core function of the division is retail billing, maintenance and management of
products and services. Figure 7 defines the functional process, and a high level billing
process until customer’s bill data records are distributed.
Figure 7: Billing Process
The division has 3(Three) environment on which development and software testing is
performed. Figure 8 presents software development and test environment layout:
Figure 8 : Traditional Software Testing Environment (TYSS,2014)
46
Development environment is a dedicated environment, and exclusive to analyst
programmers. As Figure 4 shows, unit testing is the testing type performed on the
development environment after the code has been developed. The code will be promoted
to QA environment, to allow the System Integration test team to perform system and
Integration tests depending on the scope of the requirements. The scope might include
integrating systems where interfaces will be tested on QA environment. Production Team
uses the same QA Environment to process User Acceptance Testing. Based on the signoff by SIT team, and UAT team, the solution will be deployed to production environment.
4.3 Research Focus
A case study is an empirical method and by this we mean a defined, scientific, method for
posing research questions, collecting data, analysing the data, presenting the results and
each of these steps is planned from the outset of the study and do not come about
through serendipity, furthermore case studies are well-suited to “how” and “why”
questions in settings where the researcher does not have control over variables and there
is a focus on contemporary events (Perry & Sim & Easterbrook, 2004).
The research focal point is driven by this question, Why test Software ?, and different
studies have provided different answers to the question, however, the basic answer to the
question would be, to find bugs or defects in the software. We stated in the previous
chapter that James Whitaker, has noted that despite the countless hours that go into code
development, and seemingly endless code reviews, bugs and defects still are found in the
production release, why ?(Bentley, Unkown). The case study will seek to address the
explanatory question and focal point above.
47
Explanatory case studies not only explore and describe phenomena but can also be used
to explain causal relationships and to develop theory (Harder,2010). Where explanatory
research is undertaken, a single case may provide the basis for developing explanations
of why a phenomenon occurs, and these may then be further investigated by applying
them to additional cases in other settings (Darke & Shanks & Broadbent, 1998). The unit
of analysis is a critical factor in the case study and it is typically a system of action rather
than an individual or group of individuals in addition case studies tend to be selective,
focusing on one or two issues that are fundamental to understanding the system being
examined (Tellis, 1997). The case study is very relevant to the research questions that
aim to explain the present circumstances of how and why production run and operational
software defects.
A common misinterpretation is that the various research methods should be arrayed
hierarchically, and many social scientist still believe that case studies are only appropriate
for the descriptive phase, that surveys and histories are appropriate for the descriptive
phase, and that experiments are the only way for doing explanatory or causal inquiries
(Herold, 2011).
‘How’ and ‘Why’ questions are more explanatory and likely to lead us to the use of case
studies, histories and experiments as the preferred research methods(Herold, 2011).
A source of evidence for this case study is from open-ended interviews, which are
regarded as non-structured. This approach of interviews was adopted to allow the
participants in this study to construct reality and provide answers to research questions
and think about situations specific to the study. This would also allow the participants to
make suggestions on how the current process can be improved.
48
The personnel involved in the interviews were 2 (two) Test Analyst who have been with
the company for more than 5 years. The Test Manager of the division has less than 6
months in the position. From the development team, 1 (One) Analyst Programmer formed
part of the interviews, and all the resource in the User Acceptance Team.
This is a Single case study where a single organization was studied with 3 different
divisions which are directly involved, and are at the centre of the study. One rationale for
selecting a single case rather than multiple case design is that a single case can
represent the critical test of a significant theory (Yin,2013).
4.4 Research Question
Research questions form the basis of the case study and seek to study the current
software testing process and practice of company M. Research questions designate what
researchers want to understand about the research problem that led to their study, and
they further specify the stated purpose of the study, which in turn addresses the stated
research problem and in addition qualitative research is the method of choice when the
research question requires an understanding of processes, events and relationships in
the context of the social and cultural situation. (Sandelowski, 2008).
The exploration of the central phenomenon is based on the main question:
I. Why critical software defects are found in production environment?
To satisfy this inquiry, we have tabled sub-questions which the study will give descriptive
answers to, and this will seek to explore the phenomenon in details as we seek to
discover, understand, and describe the stories.
49
4.5 Research Sub-Questions
To seek a further understanding to the study question, sub-question become very
important to outline the current testing practice within the organization, and with every
process, roles and responsibilities are defined, and based on the skill set level, allocation
of the responsibilities is then processed. The inquiry focussed on drawing out factual
information on the following Categories:
i.
Roles and Responsibilities.
ii.
Level of software testing skill, knowledge and competence.
iii.
Current software testing practice
4.5.1 Development Team
Figure 9: Development Team Structure
50
The development team is outsourced and they are with the client onsite. The team is
comprised of the resources that are permanent with the vendor and contract and
temporary basis to company M. Roles and responsibilities for the development team is
managed by the development manager as illustrated on Figure 9 of development team
structure, and is the single point of entry for requirements from business, and depending
on the magnitude of the change, the task will either be assigned to the Business Analyst,
or the developer, also referred to as Analyst programmers.
The technical team lead is managing special projects depending on the magnitude and
the urgency of the project. The development manager reports to the division portfolio
manager.
Project administrator facilitates and administers the deployment of code from
development environment to Quality Assurance (Test) environment by liaising with
change management, and different users who are required to sign-off the implementation
sheet
Analyst Programmer, are also known as developers. Their main function is to develop the
code, or/and make changes to the code. Perform unit testing and document unit testing
results which they send to the Production team for further analysis. The test results with
the respective screen-prints are handed over to the test team, to help them understand
the changes to the function, or to understand the new functionality. The team also defines
the technical design document which is sent to the System Integration Test team to test
against.
Business Analyst defines the functional design specifications , although with no detailed
use-cases. The functional specification is however not defined for cosmetic or less critical
51
changes. An email is good enough to serve as a requirement and the developer will be
assigned to the function based on that. Production Support Analyst is responsible for
incidents management and they are subject matter expert of the vendor’s system, and
they are sitting on the same environment with the production team. They drive the
discrepancies issues between the network system and the billing system, and fix them
accordingly because of the understanding they have of the system. The team also
process the scheduled job.
Portfolio
Development manager
Project Administrator
Technical Team Lead
Analyst Programmer
Business Analyst
Responsibilities
Manages Business Expectations
Estimates time and Effort with input from Analysts programmers
Assigns tasks
Business Liaison
Manage Projects, and Budget
Member of Change Advisory Board
Administers code deployment to QA
Facilitates Sign-Off process
Involved in projects administration tasks
Manages Escalations, and Projects
Subject Matter Expert
Define Solution Design Document
Code Development, and Unit Testing
Document Unit Test results
Impact Analysis
Member of CAB
Define Solution Design Document
Code Development
Unit Testing
Document Unit Test results, and handover the test results to Testingteam
Impact Analysis
Fix defects
Compile Functional Specification
Facilitates Sign-Off process
Manages code deployment from Development to QA
Production Support Analyst
Incident management
Discrepancies Management
Support Analyst
Incident management
Table 1: Development Team Responsibilities Matrix
52
Support Analyst are responsible for specific incident management, and they provide
further analysis of incidents causes and report to the development manager of required
actions. This are the incidents that the production team could not resolve, fix and close.
As can be seen of Table 1, the responsibility matrix of the development team is defined,
as this is to allow for management and coordination of tasks in the development team
Testing Skill Level of analyst programmers
This section seeks to explore the testing knowledge and skills of the development team.
The development team is essential performing unit testing. A unit test is code that
exercises a specific portion of your codebase in a particular context, typically, each unit
test sends a specific input to a method and verifies that the method returns the expected
value, or takes the expected action and unit tests prove that the code you are
testing does in fact do what you expect it to do(Lorenz, 2014).
Upon investigation, 0% of the analyst programmers have received a formal software
testing training and 100% received software testing training as it formed part of the
modules they were studying.
Modules include:
a. System Analysis
b. Logic design
c. Software Engineering
d. Database Principles
e. Software Development
f. Advanced System Development
53
The whole team have testing knowledge although it is not in a form of an external
certification. The development team does not use the testing tools, i.e Quality Center. The
test cases designed are in a form of screen prints with details underneath them on the
word document. The testing results are only sent to the production User Acceptance team
as a guideline on what functions have changed and what should be tested. The team is
applying waterfall development methodology which is comprised of the following phases:
a. Requirements gathering and Analysis
b. Design
c. Coding and Unit testing
d. System Testing
e. Implementation
f. Maintenance
Figure 10: Time Spent by Development team in testing
54
The emphasis is that the development team is only responsible for unit test, although the
unit test is based on positive scenarios. Defect found by the development team during
unit testing are not documented. The team simply fixes the defect and retest the code.
Figure 10 puts into context the time spent by the developers in testing.
50% of the team spend maximum of 2.5 hours of testing, and only 16% spends minimum
of 1 hour in doing unit testing of the code. After Unit testing has been performed, a
request to move the code to QA environment from Development is made. Change
management team is responsible for processing the move to QA. System test team is
informed through email communication that the code has been successfully moved to QA
so that they can resume with end to end system integration test. 100 % of the
development team make time to sit with the user Acceptance team in the production
environment to take them through the changes that they have implemented, and how
each should be tested. On some occasions the development team supply the UAT and
System Integration Team with the test data to test the various functional changes as
required by the business.
4.5.2 System Integration Testing team
Company M, has outsourced its software testing services to the company which will be
called ILQ Testing due to confidentiality clauses. Figure 11 is a representation of System
Integration Test team structure. ILQ Testing has been in operation for over a year. ILQ
testing is responsible for testing all the code enhancements on a small and big scale. The
testing coverage is performed across the CRM system and the billing system. The CRM
uses API (Application Programming Interface) which calls functions. of the billing system.
The team is managed by a Test Lead who is an employee of ILQ testing, and the test
55
lead reports to the portfolio manager of the department. The team is responsible for end
to end Systems and Integration Testing.
Figure 11: SIT Team Structure
The team is made out of Eight (8) technical resources including the test lead. The test
lead is responsible for allocation of tasks and overseeing the testing. The test-lead is the
interface link between the development manager, release manager and the testing team.
In addition the test lead defines the test plan only for projects that are regarded as big
projects due to functionality scope, complexity, resources and budget allocation.
Depending on the magnitude of the project, either the functional specification or technical
56
design document, or Implementation sheet with high level technical changes will be given
to the test team.
A functional change which might require maximum 5 days to develop and test will not
have a functional design specification or technical design specification. What will be
produced is a change request sheet and the implementation sheet required for sign-off,
and it details all the programs and functions to be tested. Software release and
implementation is on weekly basis, and the testing team is continuously testing and are
kept on their toes as they deal with production defects which prohibit a seamless
business process.
Quality Centre is a test tool used to capture business or user requirements, and to design
test cases which are mapped to the user requirements. After the tests cases have been
designed, and captured on quality centre, they are put on a ready status until they are
executed. A pass or fail status is assigned to the test cases based on the results. As
defined on Figure 11,that the different testing resources execute different functions, with
junior testers have less responsibilities on at their helm.
The following Test Process is undertaken by the Test Team:
a. Specification Documents review
b. Requirements Design
c. Test cases Design
d. Link test cases to requirements
e. Execute Test cases
f. Defect Logging
g. Sign-Off implementation Sheet
57
Because of the nature of the business and environment, the design of test cases
depending on the changes is in parallel with test cases execution due to the urgency of
the changes. This type of testing is referred to as exploratory testing.
The functional matrix of the test team is tabled as follows:
Test Lead
Test Analyst
Junior test
Organizes Kick-off meetings
Defines a Test Plan Document
Review Specifications
Reviews the test cases and
requirements designed by
Junior testers and advices
Design test cases
Arranges meeting with the
test analyst to review test
cases
Assigns tasks to Test Analysts
Extracts functional and user
requirements from design
specification
Captures test cases on
Quality Center
Arrange test tool access, and
system access
Arranges hardware for test
setup
Capture requirements on
quality center
Design the test cases on
Quality Center according to the
specification document(s)
Execute test cases
Organize status meetings
Map Requirements with Test
cases
Single point of contact for test
team
Keep track of new requirements
and changes
Test Reporting
Sign-off implementation
document
Estimation of test Effort
Evaluates the Entry criteria.
Monitors and control testing
according to calendars
Evaluates the Exit Criteria
Log defects, and retest.
Monitors the test coverage
Execute the test
Log Defects
Tracks, and re-test the fixed
defects
Generate a test Report
Table 2: SIT responsibilities Matrix
The plainest definition of exploratory testing is test design and test execution at the same
time and this is the opposite of scripted testing (predefined test procedures, whether
manual or automated) furthermore Exploratory tests, unlike scripted tests, are not defined
58
in advance and carried out precisely according to plan(Bach, Unknown).The team is
essentially addressing defects on the production environment which are discovered by
business users, and because they impact on the day to day business processes, they
become urgent as customers are affected.
The test team performs the following type of testing:
a. Black Box/Functional testing(Manual)
b. Regression testing (Manual)
Black box testing (also called functional testing) is testing that ignores the internal
mechanism of a system or component and focuses solely on the outputs generated in
response to selected inputs and execution conditions, and Regression testing is selective
retesting of a system or component to verify that modifications have not caused
unintended effects and that the system or component still complies with its specified
requirements (Williams, 2006). The testing performed by the test team is manual. The
system test team is responsible to source and manage the test data. When a bug is found
during testing, the testing resource logs the defect in the defect reporting tool, and jots
down the steps to reproduce the defect, and assigns the defect to the respective analyst
programmer, who will fix, and redeploy the fix to test for the tester to retest, although the
analyst programmers or developers do not make use of a defect reporting tool. The
application to log defects has the ability to generate an email and as soon as the defect is
logged, an email will be sent to the relevant analyst programmer to fix and redeploy the
fix.The test team is responsible the end to end functional testing which includes, the
CRM, API, and the back-end. SQL and unix skills are essential requirements for the team
to provide comprehensive coverage of the test artefacts, and all the reasonable aspects
59
of functionality. After the system test has been performed and signed-off, the solution is
then handed over to the User Acceptance Team which is within the Billing and
Production, Operations System and Business Solution Support team to execute user
acceptance testing so that the solution can be implemented in production.
Skill level of the System Test Team
All the team members of the test team have tertiary qualifications in Information
Technology, where a software testing is a subsection of either in System Analysis,
Business Analysis, or Software Engineering design. The level of Education is National
Diploma, and the experience in years of software testing is just below 3 years. The team
understand the different testing methodologies. Error guessing and negative testing are
synonymous amongst the test team. The test team only signs –off testing after a clean
cycle of testing. For every defect found in the testing cycle, the code is retested after the
development fix, and only upon a clean cycle of testing, or when the testing has met the
exit criteria as defined in the Test Plan.
25%
12.50%
Test Lead
37.50%
Senior TA
Junior TA
25%
Test Interns
Figure 12 : Percentage Representation of SIT Team
60
Figure 12, shows the percentage composition of the test team, which defines how much
testing skill and knowledge is within the team.
The following skills are possessed by the testing team:
I.
Test Analysts and Junior test analysis combined: Test analysts make up 37.5
percentage of the entire system test team, and junior test analysts 25 percentage
of the team
a. System Analysis skills where they are able to decompose the system into
individual components and understand the individual functions of different
components and the objective of all the components put together to function
as a system.
b. Business Analysis which requires the testing team to understand the user
requirements, business requirements and how they are translated into
functional requirements which should be tested to meet the and conform to the
specification documents, satisfy business and user needs and requirements.
c. Time Management skill because the Test analysts with the guidance of the test
lead are executing test according to the test plan and schedule. The team
however have indicated that overtime has to be put on to meet the time line as
outline in the test plan, or as communicated by the release manager.
d. Failure Analysis percentage employed by the test team is calculated as :
(Number of Test Cases Failed/ number of test cases executed ) * 100
This give the percentage failure percentage of the defect and this give an
indication of how error prone the solution or changes are, or how ready the
solution is to be implemented in production.
61
e. Requirements based testing : The testing team is processing the testing on
requirements based testing and not risk based testing. The requirements as
extracted by the testing team provides a comprehensive coverage of all
reasonable aspects of functionality.
f. Test Cases Design – The test cases are designed based on the use cases as
defined in the functional specification if provided. The test-case details the
functional process are detailed to enable any one to easily understand the
function being tested.
g. Requirements extraction – The test team extract the requirements based on the
design documents and functional or business requirements specifications
documents. On the cases of changes on a small scale, experience of the
system is used to gather requirements to be tested and error guessing is
applied as a technique.
h. Beginning of Testing - Testing begin as soon as the solution has been deployed
to QA environment and a communication has been sent out. On some cases,
software testing is in parallel with test cases design. Test cases a simply kept
and stored on the quality center testing tool for history and traceability
purposes.
i.
Test tools used by the test team is a standard test tool used called Quality
Center which can be used for structuring test process, Defining and adding test
sets, running tests, or executing tests, adding new defects, matching defects,
designing functional requirements, mapping functional requirements to test
cases, and generating test reports.
62
II.
Test Interns – test interns make up 25 percentage of the test team. The interns are
newly graduates and don’t possess much of the corporate experience and it was
outlined that they have never tested before and it was the first time that they
actually tested, although they have learnt about testing as part of their degree.
III.
Test Lead - Time management, project management, test management, test
analysis, system analysis, business analysis, complexity metrics, failure analysis,
test tools, impact analysis, risk management, test reporting and communication.
4.5.3 User Acceptance team
Figure 13: Billing and Production Team Structure
Billing and Production core functions are Operations Systems Support, Business Solution
Support, Configuration, Development and Billing Operations. The team as represented on
Figure 13, is composed of technically inclined resources who essentially are responsible
for processing billing functions as explained under Operations System Support and
Business Solution Support team section. In addition to their core functions, the team is
63
required to perform User Acceptance Testing for all the functional and business process
changes in the billing system. All the changes that directly impact on the core billing
processes and functions should be tested by IT:Billing and Production team. Functions
that affect billing are, but not limited to pricing, tariffs plans, contract duration, services,
move billing, invoices changes, pro-rata, discounts, packages, migrations, subscription
activations, subscription migration, and billing cycles. The team understands the technical
architecture, functional layout, functional process, and how the system integrate with
other applications because they support the system technically and operationally,
although the actual users of the solution are not sitting within the IT:Billing and production.
The users of the system are in the contact centers, retail shops, and various dealers. It
was a strategic executive decision to have the team directly involved and responsible for
UAT. The billing and invoicing system integrates directly with the network system which
gives subscribers access to the services that they are ultimately billed and invoiced for.
The product catalogue is on both the billing system and the network systems and at any
point in time the products, services and business rules in both systems should be in
synch to ascertain a seamless provisioning of subscriber services. Business Intelligence
forms part of the Billing and Production team, but they are not involved in the User
Acceptance Testing process.
The User Acceptance Software is carried out on the QA platform, which is the same
platform used for Systems Integration Testing. The team is responsible for sourcing their
own test data, and test scenarios. The following dynamics takes place within the UAT
phase of functional changes:
64
1. The resource that operationally manages the function under test is
responsible to test that particular function because of the knowledge,
operational experience and understanding they have of the function.
2. The development team hand over the test documents which consists of
the tests performed during Unit testing with screen shots as a guidance
on what changes have been effected, and where should the team focus
on testing the changes.
3. Based on the magnitude of the changes, Functional Design Specification
or Technical Design specification maybe provided together with
Implementation sheet, which consist of the actual program changes, and
the size of the program.
4. User acceptance testing is based on the limited time schedule, and is
often executed in parallel with System Integration Testing.
5. The test type and methodology executed is exploratory, and there are no
test cases designed. Error guessing and positive test are the most
applied test techniques.
6. Defects are reported over email facility and coordinated over email
facility.
7. Based on the magnitude of the project, some changes will be assigned
the Project Manager and the business analyst to liaise with the team to
facilitate and coordinate the User acceptance progress. This mostly
applies when there are multiple integrating systems that are impacted by
the changes and where different systems need to be tested.
65
Testing Skill Level of IT: Testing, Billing and Production team
1. 20% of the team possesses formal testing qualification and certificate, i.e
Practical Software Testing for Business Analyst and Testers, and ISTQB
Foundation.
2. The whole team have application development and configuration
development skills
3. 80% of the team has never processed software testing before out of the
company, and it is the 1st time they encountered testing at the
department. The team has been testing the functions that they are
responsible for operationally, for the past 8 years.
4. 100% of the team have system analysis skills and they understand the
system functionally and technically, and the relationship between the
billing system and the other integrating systems.
5. Because of the team having to juggle between operations support,
business support, configuration, development and operations, the team
have enormous time management skills and competency.
6. The test performed is manual testing, and only functional changes are
tested. The team understand and a familiar with only these two testing
techniques
7. Test cases and test report are in the form of screen prints of the actual
application under test with very few descriptive details. This is how test
reporting is managed.
66
8. User Acceptance Testing is facilitated and coordinated by the
development manager and respective analyst programmer who was
responsible for the changes, and the test management responsibilities
are with development team.
9. Test Process followed is Analysis of the functions and changes, Test
Execution, Test Reporting, Sign-off and Implementation.
4.6 Case Study Data Analysis
The study conducted focussed on the three teams that are executing software testing
responsibilities within Company M. Analyst programmers, also known as developers,
perform unit testing, System integration team performs systems’ functional, interface and
Integration testing, and billing and production team carry out user acceptance testing
responsibilities. After going through open-ended and questionnaire, and studying various
processes documentation as presented by the study participants, the following problems
were identified, in relation to current academic standards and Literature Review.
4.6.1 Development Team
4.6.1.1
Testing skills, knowledge and competence
1. The inquiry outlined that 0% of analyst programmers, also known as
developers have not undergone any formal software testing certification or
training. The knowledge received at tertiary education is theoretical as it
formed part of the degree modules, and software testing best practices, and
practical training was not explored.
67
2. Testing Tools are not used by the development team, and this suggest that
the functions that are being tested are not documented, recorder, and there
is no traceability in place.
3. From the interviews, it was found that 16% of the development team spends
maximum an hour on unit testing, 33% spends 1,5 hours, 50% spends 2.5
hours on unit testing, and this is because of the fact that the analyst
programmers are processing 1 or 2 positive tests to verifying that the code
is executable, and produces results. This practice is attributed to the lack of
software testing knowledge.
4. Analyst programmers understanding of testing is to execute the code and it
generates results.
5. Analyst programmers do not involve system integration test team in the
design review meetings, and technical review meetings to enable the test
team to have a better understanding of the solution under test.
6. Technical design documents produced are not detailed enough to enable
the test team to understand the technical architecture and layout of the
solution. Current test process as defined in figure 14, only depicts System
Integration Test team’s involvement at the end of the process when a test
document compiled by the development team is handed over to the team.
68
Figure 14: Development and Unit Test process
4.6.2 System Integration Test Team
4.6.2.1
Testing skills, knowledge and competence
1. Only 37.5 % of SIT team reported having software testing certification,
ISTQB in addition to I.T tertiary qualification.
2. The study outlined that 62.5% of the team has less than 2 years of
software testing experience.
3. Testing is Manual.
69
4.6.2.2
Does the test team know what to test ?
1. Testing model is not clearly defined, as the testing team have no idea
what type of testing methodology they have to use. There was a mention
of waterfall model by the team, although it is not fully adhered to.
2. Specification documents are not provided for all the existing changes
and new functions and if they are provided, they lack details including
use cases which form the base of what the test team should test against.
This leaves the test team with test cases that do not provide a
comprehensive coverage of all reasonable aspect of functionality.
3. The study revealed that the SIT team does not know when to start
testing. They are not involved in JAD (Joint Application Development)
sessions, review meetings (Design), and they are not involved in the
review of the functional design or business requirements specifications
as that forms part of static testing.
4.6.2.3
Defect Management
1. The defect meeting is not held on regular basis where testers,
developers, business analysts, and project manager meet to coordinate
and discuss defects. The study revealed that the, the coordination is
over the email facility.
2. The investigation has revealed that none of the test team members are
applying defect prediction
4.6.2.4
Test planning and Management
70
Test plan is only compiled by the test lead for the big project, which
involves complex changes that affect and impact on multiple systems.
Test Analysts, and Junior testers and test interns do not receive the copy
of the test plan and were unable to produce any at their disposal.
4.6.2.5
Testing techniques
The inquiry outlined that the most common used test technique is
positive testing, and negative testing, and this is attributed to the type of
legacy system under test where they cannot use other type of
techniques.
4.6.2.6
Application and system level programming
Code Review is not synonymous within the SIT team. The investigation
made a revelation that none of the SIT team member have any sort of
programming or development skills and knowledge, in addition, the team
is unable to read the unix scripts. The team can however, generate SLQ
queries to extract, manage and maintain the data in the database.
4.6.2.7
Matrix data
The study revealed the whole team does not have a knowledge of how to
define requirements traceability matrix
4.6.2.8
Metrics
The team does not employ any metrics for defects density, defect removal
efficiency, defects leakage, and the risk factor metrics.
71
4.6.3 User Acceptance testing Team
4.6.3.1
Operations Team performing UAT
User Acceptance Testing, or Billing, Production and testing team executes
UAT not as business or requirements owners, but as the functional owner.
The study reported that the production team is responsible for UAT on the
functions that each team is responsible for from operations perspective.
4.6.3.2
Business Analyst
The team functions without a Business Analyst.
4.6.3.3
Project Manager
The team is without project manager
4.6.3.4
Test Tools
The team uses Quality centre, but no investment has been made on
automated tools.The study reported that, none of the UAT team uses the
test tools for test execution, defect reporting and test reporting.
4.6.3.5
Testing knowledge, skills and competence
The inquiry revealed that 80% of the UAT team does not have software
Testing Training, and certificate. Testing is manual. The team only tests
positive scenarios, and all the other testing types and techniques are not
applied.There is no Software Development Life Cycle model that is being
followed to process testing.UAT is not independent, because development
team provides the test team with the results and screen prints of unit testing,
to give the team guidance of what to test, and the study revealed that the
team uses the same data that was used in unit testing.the study reported
72
that UAT is performed on the same test environment, where systems test
was performed, i.e QA. UAT environment in the organization as discovered
by the study is used for Pre-Billing checks and balances, and test run, but
not for testing of functional changes.Production team execution of UAT is
the same as the execution of System Integration Testing by SIT team,
because they are testing the actual functional requirements.
4.7 Literature Review
The inquiry has put into context the current testing practice in company M, and the
assessment of testing activities and testing process has been performed through
investigation and open-ended interviews. Critical questions were addressed with a view to
provide a guideline of the underlying problems behind software defects in production
environment. The literature review’s objective is to provide guidance on the current
recommended and best industry practice of software development and testing, with an
attempt to provide suggestion and to improve the testing process and practice within
company M.
4.7.1.1
Software Development Life Cycle Model
The development team, SIT team and the Production team responsible for software
testing were unsure about the development model that has been adopted. The model is
supposed to be waterfall model, however, the methodology is not fully adhered to as
some of the activities like UAT and SIT have been executed parallel. Figure 15 represents
a waterfall model of software development lifecycle.
73
Figure 15: Waterfall Model (Bassil,2011)
The Waterfall SDLC model is a sequential software development process in which
progress is regarded as flowing increasingly downwards (similar to a waterfall) through a
list of phases that must be executed in order to successfully build a computer software,
and originally, the Waterfall model was proposed by Winston W. Royce in 1970 to
describe a possible software engineering practice, furthermore the waterfall model
defines several consecutive phases that must be completed one after the other and
moving to the next phase only when its preceding phase is completely done(Bassil,2011).
Testing happens after the code has been developed and implemented in the test
environment. With the coding of the application complete, the testing of the written code
now comes into scene, and If there are any flaws, the software development process
must step back to the design phase. In the design phase, changes are implemented and
74
then the succeeding stages of coding and testing are again carried out (Akhilesh,2012). A
generic very high level structure of test process activities has been given by (Tian, 2005)
as represented on Figure 16.
Figure 16: Generic Structure of Testing Process (Tian, 2005)
The test process regardless of the SDLC model, has the same activities, and it is
recommended that the SIT team understands the testing activities, and the ultimate goal.
Non adherence to the test activities is futile to the course. Test process is divided into
three main groups of test activities which are, Test planning and preparation, to set the
goals for testing, select testing strategy, prepare specific test cases and the general test
procedures, Test execution and related activities which also include related observation
and measurement of product behavior, and final activity is Analysis and follow-up, which
include result checking and analysis to determine if a failure has been observed, and if
so, follow-up activities are initiated and monitored to ensure removal of the underlying
causes, or faults, that led to the observed failures in the first place (Tian,2005).
75
The study gives an emphasis of adherence and compliance to adopted software process
by Company M, and that non-compliance, to the process will open a room for defects.
4.7.1.2
Insufficient Software Testing Training
In order to produce quality products, companies require new-hire candidates that have
good problem solving, debugging, and analysis skills and in software engineering and
computer science many graduates enter the work force with exceptional development
skills, but lack proficiency in test, debugging, and analytical skill, and this is in part
because of the current academic curricula emphasis on software development at the
expense of teaching software testing as a formal engineering discipline(Astigarraga &
Dow & Lara & Prewitt & Ward, 2013). It is estimated that more than 60% of the
development cost is spent on testing for many software systems, however, a large part of
the problem is not so much the amount of testing that is performed on software, as it is
“who” the software is tested by, and “how” these testers go about doing it furthermore
many personnel responsible for software testing are software engineers with
a very basic background in testing, mostly restricted to the application of a small set of
testing tools and a simple knowledge of how to apply a few testing tools cannot hope to
substitute for a strong foundation in software testing principles and methodologies(Wong
,2011). There is one primary certifying body in software testing, the International Software
Testing Qualifications Board (ISTQB) which is Introduced about 10 years ago, the board
offers a series of certifications across three levels: Foundation CTFL, Advanced CTAL,
and Expert CTEL(Thongkham,2015). Organizations, researchers and HRD professionals
realized that if they want to improve the employees as well as organizational performance
they have to invest in training activities (Bhatti 7 Hoe, 2012). Although (Astigarraga &
76
Dow & Lara & Prewitt & Ward, 2013) brings on a different dimension that Programs such
as the “Certified Software Quality Engineer” from the American Society for Quality, the
Association for Computing Machinery “Certified Software Development Associate” and
“Certified Software
Development Professional” curriculum, as well as the International Software Testing
Qualification Board's certificate in software testing tend to focus on terms and
definitions rather than practice, and they further continue to outline that, In order to ensure
a supply of sufficiently skilled test-oriented engineers during the next five to ten years,
actions in higher education curricula must be taken, and It is our belief that
industry must partner with academia to elevate the status of testing as an equally crucial
component of software engineering as the development phase is held today. The study
(Astigarraga & Dow & Lara & Prewitt & Ward, 2013) continued to point out that as
complexity in the IT industry and customer solutions continues to grow, the need for
highly-skilled software test engineers will also continue to increase and in an industry
where quality is expected and defects can cause costly or even life-threatening errors it is
imperative that the shortage of software test skills among recent university graduates is
addressed, and If students are to be prepared for jobs in software test, curricula
guidelines and available course materials must be reformed.
4.7.1.3
Test Automation
The desire for repeatability and accuracy is a reason organizations are moving to
automate testing, both to uncover bugs and to ensure performance standards
(Hildreth,2004). As reported in the study that company M testing is manual, and the
77
regression test suite increases with every release. Regression testing intends to ensure
that software changes do not introduce new problems, or failures, in a software, also, it
ensures that you don't reintroduce previously fixed bugs and with automated incremental
tests run by a tool in a set of environments, or test bed, you save the time that is needed
to manually run a test on each environment, and allow better sharing of resources,
furthermore by setting regression tests to run once a day, for example, you do not
consume resources that you might otherwise use for development or incremental testing
(Negrello,2013). Software testing using an automatic test program will generally avoid the
errors that humans make when they get tired after multiple repetitions. The test program
won't skip any tests by mistake (Kelly,1997), and Some of the problems faced during
manual testing as discovered on the academic study by (Rishab & Kaluri, 2015) are :
1. Changes in the requirements.
2. Documenting of the bugs that were logged, bugs fixed and those that are still open
using an excel sheet.
3. Maintaining of Requirements Traceability Matrix (RTM) without using any tools
manually is very tedious job.
4. Carrying out regression testing on a daily basis or biweekly or as needed by
manual testers is not possible.
And (Rishab & Kaluri, 2015) further stipulated that to overcome the above problems,
testing team, prefers use of test management tools and specific type of testing
automation tools. We suggest that company consider automation of the regression test
suite.
78
4.7.1.4
Defect Prediction
Having defect prediction as part of the testing process allows testing team to strengthen
their test strategies by adding more exploratory testing and user experience testing to
ensure known defects are not escaped and re-introduced to end-users and test engineers
would be able to have better root cause analysis of the defects found where in the long
run, testing can achieve what is called as zero-known post release defects for that
particular software (Suffian & Ibrahim, 2012)
4.7.1.5
User Acceptance testing
As changes to the system are planned and scheduled, it is important that any changes to
the system are tested, and It is especially important to include User Acceptance Testing
(UAT) as it is one sure way to reduce or eliminate change requests, and drastically
reduce project costs, and the goal of UAT is to assess if the system can support day-today business and user scenarios and ensure the system is sufficient and correct for
business usage(Yin-Tantouri,2015). User Acceptance Testing (UAT) focuses primarily on
verifying that the functionality delivered, and proven in System and System Integration
Testing, meets the end users' business requirements and it is usually performed by the
business, while best practice would see these business users supported by professional
testers (Test-direct,2014). When performing user acceptance testing, the expectation is
that testing skills and application domain knowledge have a higher weight than the
specific technology skills for a project, and testing is typically performed from a user’s
perspective using the application as users normally may(René, 2015). In User
Acceptance Testing, manual testing is done by the user,generally, UAT is not automated
(Tahiliani & Pandit, 2015). The recommendation to company M is that business users and
79
business analysts should be integrated in executing User Acceptance Testing, as they
understand the business process, and usability thereof.
4.7.1.6
Development Manager Role in Software Testing
Literature review puts into context the roles and responsibilities of the development
manager in the SDLC. The secret to being successful as a Development Manager is
managing expectations and making sure everyone understands your role is the first
step.(McCabe, 2014). The role being to be responsible for managing multiple priorities of
conflicting projects. Also an escalation for issues from the team, which are unable to be
resolves internally (Bogue,2006). The development management role works closely with
the project management role to ensure that the projects are completed and they also
spend time working with the business owners on planning and preparing for new projects
that will soon be consuming the group, in addition they're constantly making small
changes in the current project to get a better product, develop a better skill set in the
group, or just generally preparing the development team for the next hurdle that they must
face(Bogue,2015). To develop and manage reliable software, it is essential for developers
and managers to acquire knowledge for testing procedures at all stages of software
development. Therefore, the knowledge of software testing is equally essential for
software developers and managers(Ahmad & Al-Abri & Shiginah, 2013).
4.7.1.7
Risk-Based Testing
No matter how much time we have for testing software, we can always use it up and wish
for more, and we never have enough time, therefore quick risk analysis will help decide
where to focus the testing efforts and where all the potential risks associated with the
code under test will be listed, including those that go beyond functionality, such as
80
security, performance and usability, and the likelihood of a failure and the impact of that
failure should be used to help decide what tests to do first, and leave testing of lower-risk
areas to last.(Crispin, 2009). Risk is the product of damage and probability for damage to
occur. The way to assess risk is outlined in figure 17 below. Risk analysis assesses
damage during use, usage frequency, and determines probability of failure by looking at
defect introduction (Schaefer, 2005)
Figure 17: Risk Definition and Structure(Schaefer, 2005)
4.7.1.8
Probability of failure
The worst areas are the ones having most defects, and the task is to predict where most
defects are located, which is done by analyzing probable defect generators, in addition,
some of the most important defect generators and symptoms for defect prone areas are
complex areas, changed areas, areas with many defects before, and to determine the
probability, functional size of the area is to be tested, i.e. the total weight of this area will
be "fault proneness / functional volume".(Schaefer, 2005)
81
Figure 18: Failure Probability(Schaefer, 2005)
4.7.1.9
Testers only not involved in JAD sessions
"Testing is not a phase!" We have to stop thinking of "testing" as some separate activity,
removed from the rest of software development. It takes a village to produce a highquality software product, and we can’t do it from isolated functional silos(Crispin,2009).
We suggest that the SIT team get involved from the kick-off meeting, and engage in
design meeting and requirements review meetings.
4.7.1.10 Test Maturity Model
Maturity models are widely used in process improvement and the users of maturity model
should be confident that the weak points of the assessed processes can be found and
that the most valuable changes are introduced(Helgessonn& Weyns, Unknown), and the
structure of the TMM is partly based on the CMM and the staged version of the Capability
Maturity Model-Integrated (CMMi), and The TMM consists of 5 maturity levels that reflect
a degree of test process maturity(Veenendaal,2006), as represented in Figure 15. Setting
sensible goals for process improvement requires an understanding of the difference
82
between immature and mature software organizations and In an immature software
organization, software processes are generally improvised by practitioners and their
management during the course of the project, even if a software process has been
specified, it is not rigorously followed or enforced furthermore, the immature software
organization is reactionary, and managers are usually focused on solving immediate
crises (better known as fire fighting)(Paulk & Curtis & Chrissis & Weber, Uknown), and
this is the finding that has been done by the study on Company M.
Figure 19: TMM levels(Veenendaal,2006)
83
It is recommended that Company M, work towards aligning to TMM for effective and
efficient test process and comprehensive test coverage for all aspects of functionality.
4.8 Summary and conclusion
The study "The Theory of Software Testing" conducted by (Lawanna, 2012), pointed out
Five problems found in software testing, which are as follows,the first problem is
concerned with the limitations of the testing team, where insufficiency may be the result of
limited resources, lack of training of the individual team members, or issues with the
leadership, and Also, suitable testing means may not be available to the team,
furthermore the second problem is failure of the test maintenance and this is because the
specification changes of the requirements result in abnormally long reverse times and the
third problem is with manual testing, where the software testing team is busy making
manual testing instead of building new test specifications, or modifying old ones to fit a
new or changed requirement, and the forth problem is about the uncertainty principle
where sometimes the uncertainty is in the exact testing conditions as well as how to view
the condition for replication, and finally the fifth problem is in selecting the right tests in
which software testing cannot address some of the essential aspects of the software
testing application or system when testing only some part of the required functions or
choosing only the expected interactions when executing a fault-tolerant application. This
problems are not unique to the study, and they are directly linked and associated with the
problems identified in the problem analysis of this study.
84
Bibliography
http://srmo.sagepub.com/view/sage-encyc-qualitative-research-methods/n164.xml
http://www.testingexcellence.com/fundamental-test-process-software-testing/
http://istqbexamcertification.com/what-is-fundamental-test-process-in-software-testing/
http://mrbool.com/software-testing-fundamental-test-process/30289
http://www.sagepub.com/sites/default/files/upm-binaries/41407_1.pdf
https://mthoyibi.files.wordpress.com/2011/05/qualitative-research-methods-for-the-social-sciences__brucel-berg-2001.pdf
https://student.unsw.edu.au/writing-case-study
http://study.com/academy/lesson/purposes-of-research-exploratory-descriptive-explanatory.html
http://www.nus.edu.sg/celc/research/books/cwtuc/chapter04.pdf
http://research.apc.org/images/5/5f/De_Vaus_chapters_1_and_2.pdf
http://uxmag.com/articles/quantitative-research-and-eye-tracking
http://isites.harvard.edu/icb/icb.do?keyword=qualitative&pageid=icb.page340344
http://srmo.sagepub.com/view/encyc-of-case-study-research/n138.xml
http://www.infotech.monash.edu.au/research/groups/rcrg/publications/framewrk.html
http://0-delivery.acm.org.oasis.unisa.ac.za/
https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf
http://dl.acm.org/citation.cfm?id=2591172
http://lup.lub.lu.se/luur/download?func=downloadFile&recordOId=1276781&fileOId=1276782
https://www.ischool.utexas.edu/~ssoy/usesusers/l391d1b.htm
http://dl.acm.org/citation.cfm?id=2729973
http://dl.acm.org/citation.cfm?id=1457528&dl=ACM&coll=DL&CFID=706807202&CFTOKEN=37033684
http://dl.acm.org/citation.cfm?id=1134292
http://www.wsj.com/articles/trading-halted-on-new-york-stock-exchange-1436372190
http://www.cs.toronto.edu/~sme/papers/2004/CaseStudiesTutorial2004.pdf
http://srmo.sagepub.com/view/encyc-of-case-study-research/n138.xml
https://www.uio.no/studier/emner/matnat/ifi/INF5500/h07/undervisningsmateriale/ISJ_case_study.pdf
85
http://www.managementissues.com/index.php/organisatietools/83-organisatietools/693-case-studyresearch-design-and-methods
http://www.testyantra.com/cloud-testing
http://www.satisfice.com/articles/what_is_et.shtml
https://researchrundowns.wordpress.com/intro/writing-research-questions/
http://dx.doi.org/10.1080/09518390902736512
https://uk.sagepub.com/sites/default/files/upm-binaries/47619_Sullivan.pdf
http://www.stibamalang.com/uploadbank/pustaka/RM/QUALITATIVE%20METHOD%20SAGE%20ENCY.pd
f
http://srmo.sagepub.com/view/sage-encyc-qualitative-research-methods/n397.xml
https://developer.salesforce.com/page/How_to_Write_Good_Unit_Tests
http://www.infoq.com/articles/development-manager-role
http://www.hskl.de/~amueller/vorlesungen/se/breaking%20down%20software%20development%20roles.pd
f
http://www.developer.com/java/ent/article.php/3529081/Anatomy-of-a-Software-Development-RoleDevelopment-Manager.htm
http://agile.csc.ncsu.edu/SEMaterials/BlackBox.pdf
http://www.researchgate.net/publication/256094322_Software_Testing_Education__A_Required_Knowledg
e_for_Software_Systems_Developers_and_Managers
http://people.clarkson.edu/~dowem/downloads/Emerging-Role-Test-PID1021060.pdf
http://www-ivs.cs.ovgu.de/sw-eng/agruppe/forschung/TR_Farooq.pdf
http://scienceandnature.org/IJEMS-Vol3(4)-Oct2012/IJEMS_V3(4)16.pdf
http://cprenet.com/uploads/archive/IJBBS_12-1155.pdf
http://paris.utdallas.edu/ccli/papers/CSEET-2011-testing-panel.pdf
http://people.clarkson.edu/~dowem/downloads/Emerging-Role-Test-PID1021060.pdf
http://www.journal.au.edu/au_techno/2012/jul2012/journal161_article05.pdf
http://www.sei.cmu.edu/productlines/frame_report/on_testing.htm
http://www.sei.cmu.edu/productlines/frame_report/on_testing.htm
http://www.computerworld.com/article/2566270/enterprise-applications/automating-the-40-monkeys.html
http://www.ibm.com/developerworks/library/os-test-stafstax/
86
http://www.mactech.com/articles/mactech/Vol.13/13.10/SoftwareTestAutomation/index.html
http://www.arpnjournals.com/jeas/research_papers/rp_2015/jeas_0415_1779.pdf
http://arxiv.org/ftp/arxiv/papers/1401/1401.5830.pdf
http://www.test-direct.com/blog/2014/03/the-challenges-of-user-acceptance-testing-uat
http://hilifehub.com/category/articles/
http://www.astqb.org/certified-tester-resources/professionalism-in-user-acceptance-testing/
http://research.ijcaonline.org/volume120/number10/pxc3903533.pdf
http://fileadmin.cs.lth.se/cs/Personal/Kim_Weyns/phd/sysrev.pdf
http://www.erikvanveenendaal.nl/NL/files/Guidelines%20for%20Testing%20Maturity.pdf
http://www.tmmi.org/pdf/TMMi.Framework.pdf
http://homepages.inf.ed.ac.uk/dts/pm/Papers/cmm.pdf
87