1
Software Development Process Audits - A General
Procedure1
Stewart G. Crawford & M. Hosein Fallah
AT&T Bell Laboratories, Holmdel, New Jersey 07733.
USA
ABSTRACT
To assist development organizations in improving
software quality and productivity, the AT&T Bell
Laboratories Quality Assurance Center created a
Design Quality Group to independently evaluate
the software development processes and associated
development products of our AT&T projects. These
software development process audits examine
software engineering techniques and tools in
practice, as they fit into the overall development
environment. Our strategy behind these audits is to
assemble a team which, with the involvement of the
developers and their managers, will: characterize
the existing development process; identify project
strengths and areas for improvements; and
recommend possible improvements. This paper
details the general approach behind this strategy.
1. BACKGROUND AND OVERVIEW
During the last few years, the AT&T-Bell
Laboratories Quality Assurance Center (QAC)
conducted several audits of software and hardware
development processes and of documented
requirements and craft interface specifications.
Our objectives centered around identifying areas
of
potential
quality
and
productivity
improvements. The QAC performed these audits
by invitation from the respective development
organizations, and their positive response led the
QAC to create a group dedicated to software
development process audits. This paper defines our
general approach; the specifics of any one
particular audit, however, will vary depending on
the breadth and depth of the audit scope negotiated
with the software development project.
The audits we describe here mark a departure from
the adversarial relationship generally implied by
the term "audit". Our primary objective is to assist
development organizations in improving software
quality and productivity through independent
Proceedings of the 8th international
conference on Software engineering
London, England, Pages: 137 – 141, 1985.
1
evaluations of their software development
process and associated development products;
additional benefits include the transfer of
proven ideas and technologies in software
development from other
projects, consolidated upward feedback to
project management, and pointers to further
opportunities for education and training.
Several projects within Bell Labs have
requested these development process audits to
date, and feedback from their management has
been.
positive.
QAC's
organizational
independence
from
the
development
organizations allows for a perspective free of
the local bias surrounding development issues;
our interviews and discussions with project
management provided time for them to stop
and take a long view of the global
development issues. We can audit a project at
anytime during the project lifecycle, but if
done early enough, the response to the audit
findings can improve the quality and
scheduling of the product currently under
development. In one particular project audited
late in the design phase, our findings and
recommendations led the project to enhance
the baselining of intermediate lifecycle
deliverables, comprehensively review the
remaining project schedule, and improve the
project tracking system; these activities
revealed several trouble areas looming in the
near future for that particular software release,
and the project then addressed the issues
before they became serious problems.
The distinction between a development process
audit and a project management audit is
primarily that of focus. The former focuses on
the techniques, tools, documents, and
procedures through which an idea becomes a
supported product, while the latter focuses on
the organization and management of that
whole effort. The scope of these two types of
audit will overlap somewhat since by their
2
natures, a process and its management are closely
coupled. Figure 1 illustrates these ideas: The upper
dashed circle shows the scope of concerns in a
process audit, while the lower circle depicts the
scope of a project management audit.
3
DEVELOPMENT PROCESS AUDIT
PROBLEM
SYSTEM
SOLUTION
SYSTEM
UNSATISFIED NEED
DEVELOPMENT
PROCESS
SATISFIED NEED
EXISTING
SYSTEM
PRODUCT
SUPPORT ENVIRONMENT
PLAN
&
ORGANIZE
MONITOR
&
CONTROL
MANAGEMENT
PROCESS
SUPPORT SYSTEM
PROJECT MANAGEMENT AUDITS
Figure 1. Overview of a Development Project
Our strategy behind these audits is to assemble a
team which, with the involvement of the
developers and their managers, will :
- Characterize the existing development
process by :
o Analyzing development
methodologies, software quality
assurance plans, and related practices
and procedures.
o Interviewing supervisory and nonsupervisory personnel
o Attending project meetings, reviews,
and inspections, and
o Reviewing
selected
development
products;
- Identify project strengths and areas for
further improvements; and
- Provide recommendations for improving the
software development process and the
quality of its software products.
Positive interactions and close coordination
between the projects management and the audit
team are important to the effectiveness of these
audits, and these contributed greatly to our
previous successes. The core audit is a time of
the team’s almost continuous presence within
the development process; this is the interval
when we interview the staff and sit in on many
project meetings. Our presence will be nonthreatening only when management support of
the audit is understood by the project team.
2.
A GENERAL AUDIT PROCEDURE
Prepare and Plan
Once a project requests an audit, some
groundwork is needed before the activities can
begin. The pre-audit activities include the
following :
2.1.1 Clarify Global Goals and Objectives: Due
to the flexibility built into our audit service, the
initial request may not be sufficient to define the
task adequately. Thus the first planning activity
4
is to clarify what the audit's customer, usually
some level of project management, is seeking
through the audit.
Discussion with he project of the managers'
needs and of our expertise are crucial to the
proper initial focus of our effort.
2.1.2 Assemble an Audit Team: Depending on
the size of the project, the team may consist of
three to six people: one person should be
designated as the team leader. Each team will
include members with experience and
expertise in process audits, software
development, and software quality assurance.
Additional subject matter experts may be
called in as necessary.
2.1.3 Understand the Project: Understanding
the project is a prerequisite. A tutorial or an
overview presentation to the team by the
project is a useful start. The team should also
learn about the project by reading project
descriptions and other related documents that
may be available, and by talking to the project
contacts.
2.1.4 Understand the Organization: In an
effective audit, it is crucial to know who is
doing what in the project. Organization charts
are often outdated and the group titles are
generally insufficient for identifying the
functions and activities of the project staff.
The organization chart should be reviewed
with the project contacts to identify the
functional responsibilities of the people.
2.1.5 Define the Audit Scope: Negotiation of
the audit's scope with the project's
management is the foundation for further
activities. Each audit will be customized to
the needs of the particular project, thus, the
breath and the depth have to be defined.
Depending on the scope, the audit may focus
on some particular phase(s) of the
development process, or its depth could vary
depending on the audit objectives, resource
constraints, and timing.
2.1.6 Develop an Audit Plan: The plan is the
"road map" for the audit. The plan should
reiterate and expand on the agreed scope (the
Why the audit), answering the following
questions: What will be audited? Who are the
auditors? When can the various audit
activities be expected to happen and finish?
Where and How will the audit activities take
place?. A preliminary draft of the plan should
be reviewed with the Project's management to
assure a mutual -consensus on the audit
activities and schedule.
2.2.
Characterize the Development Process
With the initial groundwork laid, the formal
audit activities begin in earnest. Our intent is
to uncover as much as possible within the
established scope and time constraints. The
team should analyze not only the methodology
as it is written down, but the "de facto"
methodology, i.e., the methodology as it exists
in practice. Our methods of collecting data
include analyzing the development process
documentation,
interviewing
project
personnel, observing project meetings, and
reviewing development process products.
2.2.1 Analyze methodologies: The first step in
characterizing the development process is to
study the documented methodology. Besides
providing a general process overview and
background for all the ensuing activities, we
can study completeness of lifecycle activity
documentation, evaluate adequacy of the
written methodology in providing clear and
consistent guidelines for the development
process, and formulate initial hypotheses to
follow up in interviews. Any documented
development methodology, Software Quality
Assurance Plan (SQAP), or other documented
practices and procedures should be included
in this review.
In analyzing development methodologies,
either in the form of a handbook or a series of
documents, we would look for: task
definitions, descriptions; and ownership;
solution procedures for tasks; review and
inspection procedures; a well defined
lifecycle, broken into activities; well-defined
handoffs between activities; and any standards
currently in use. We would want to look to
any related practices and procedures for
measurement and tracking procedures,
monitoring and control mechanisms, and
efficient organizational interfaces. The above
lists are not meant to be all-inclusive, but they
are a good starting point.
In general, we would get the necessary
documents through the project interface
contact or the project librarian. After reading
and annotating the documents, all the auditors,
in a group meeting, would discuss the
documents, evaluate their adequacy, and
5
formulate the first set of hypotheses regarding
the process as it is documented.
2.2.2- Interview Developers: The core audit is
the main time for interviewing developers on
site and attending any relevant project
meetings. The auditors should interview as a
group or in subgroups of at least two : multiple
interviewers provide multiple viewpoints
which helps balance any individual bias.
Interviewing people near the project site but
not in their offices, removes many distractions
and may help people open up more since
they're slightly removed from their usual
environment.
We would want to interview people in the major
functional areas of software development
included within the scope of the audit. A noninclusive list may include staff and their
managers responsible for:
system definition
system requirements
system architecture
software requirements
software architecture
software design
software code
software unit test
software integration test
system test
customer interface
end-user training installation
software maintenance
problem reporting & corrective action
project management & configuration
management
quality control
methodology documentation & updates
developer training
A primary reason for interviewing people is to
characterize the "de facto" methodology. Does it
follow the documented methodology? Is it
consistently applied across people? Is it
effective? Additionally, we would want to:
gather information to support or refute pending
hypotheses; formulate new hypotheses as new
information is gathered; characterize the
visibility of project tracking and the
effectiveness of project control; and gather
upward feedback for the project management.
Basic investigative and journalistic interview
techniques would be the auditors' tools for this
task; summary question checklists can help
guide initial interviews.
2.2.3 Attend Meetings: The auditors should
have a blanket invitation to any technical,
administrative, or informative meeting, either
formal or informal, the project holds during the
time-frame of the audit. The purpose behind
attending these gatherings is to: observe project
communication;
characterize
general
effectiveness of meetings, reviews, and
inspections; and characterize compliance of
reviews and inspections to any relevant
standards.
The entire team (for large meetings) or a
subgroup (for small meetings) would discretely
attend and observe a project meeting.
Discussing the meeting after it officially ends
with attendees who happen to linger on is often
of value in resolving any questions the auditors
may have. The meetings observed would
normally occur during the core audit, but they
may occur before or after that time also,
depending on when representative project
meetings are scheduled.
2.2.4
Review
Selected
Development
Products: The team may decide to review indepth a sample of products representative of the
various lifecycle activities within the scope of
the audit. Such reviews serve to: examine the
products' fitness-for-use in downstream
development activities; characterize compliance
of the products to existing standards (project or
industry); or explore project details for
necessary background.
Individual auditors examine selected products
within their field of expertise. These reviews
would be oriented toward substantive, not
editorial or typographical, comments. When
multiple auditors review the same set of
documents, they would need to discuss and
integrate their comments.
2.3
Evaluate and Report
The team should get together frequently during
the core audit period and review their
observations. These meetings help ensure that
the audit is progressing according to the plan,
provide opportunities to formulate and revise
hypotheses, and adjust the planned activities,
when necessary. The auditors may decide to
include other people in the list of interviewees,
plan to attend additional meetings, or review
other relevant documents.
6
Following the core audit, the teams findings
should be brought together, combined,
evaluated, and documented in an audit report.
As part of this process, any open items should
be followed-up, and the auditors should obtain
additional information and seek clarification on
issues that do not have the team's consensus.
In preparing the audit report the following
should be considered :
• Report strengths as well as areas for
improvement. Although the main objective
is to identify the areas with potential for
quality and productivity improvement, it is
equally important to identify those aspects
of the process that are done well. This
recognizes the successful efforts within the
project to produce quality software, aids
the project by itemizing things done well
which shouldn't be changed for the worse,
and helps us identify good methods and
practices that have potential for use on
other projects.
• Separate major from minor concerns. The
team may find many weak spots. The few
major problems should be separated from
the others and given special emphasis and
discussion; this separation assures they
don't get lost amidst everything else.
• State the problems succinctly and state
why they are problems. The problems
should be communicated to the project
very clearly. By stating why the team
thinks something is a problem, we provide
the project with the assumptions,
perceptions, and the reasoning that led to
the team's conclusions. This information
helps the project to assess the audit
findings in light of their more detailed
project knowledge.
• Present recommendations with due
caution. When possible, the team should
recommend viable solutions to problems
they find while examining the development
processes. But, the recommendations
should be expressed as suggestions that the
project may want to consider in developing
its own plan of action. The team should be
cognizant of the complexities of the
technological,
functional,
and
organizational aspects of a development
project as they relate to the recommended
solutions.
while
the
team's
recommendations may provide a good start
for addressing a problem area, the people
within the project are in a better position to
decide what solution is most likely to
work.
The audit report should be reviewed with the
project managers before being finalized. Such a
review helps to remove any misunderstanding
that the team may have. It also provides an
opportunity to find out if there are any major
disagreements between the team and the
project managers, and document the opposing
views if necessary in the report. With the
issuance of the final report, the team may also
make a formal presentation of the findings and
provide a forum for discussion of the
recommendations.
The report is meant only for the managers of
the project. It is their prerogative to distribute
the report either up or down managerial lines
or outside the project. The team members will
forward any request for an Audit Report to the
appropriate project manager. The team
members may report some generalities about
the audit to their management as needed to
keep them informed of the audit activities.
2.4 Follow-up
We will continuously try to improve the audit
process. Feedback from the projects provide
valuable information for achieving this
objective. A follow-up with the project right
after an audit may identify some of the changes
that could strengthen the audit process. The
effectiveness of the audit, however, can only be
assessed in due time. Hence, the team leader
should make another follow-up eight to twelve
months after the audit. In this follow-up, we
solicit an assessment of the benefits realized
(or expected to be realized) by the project so
that the cost-effectiveness of the audit can be
evaluated. The project may also have other
suggestions on ways to further improve the
audit process.