SOFTWARE DEVELOPMENT PROCESS
AUDIT
Description
A fixed price consultancy-led review of the software development lifecycle
(SDLC) or application lifecycle management (ALM) process.
Whether your software development process revolves around in-house,
offshore, “right-shore”, or out-sourced resources it needs to be effective and
efficient. Software is produced to directly serve the business that calls for it,
yet all too often it falls short of expectations.
This audit will reveal potential problem areas, risks, weaknesses as well as
direct opportunities for improvement in all aspects of the process.
The deliverables of the audit include:
● Executive summary
● Report that details strengths, weaknesses, and risks of the current
SDLC
● Recommendations for improvement
● Briefing session
Benefits
It’s all too easy to believe software that appears to work is actually doing what
it is supposed to. Metrics are often misplaced and regularly measure
superficial and irrelevant data.
Clients are vulnerable to questions of accountability when the problem really
relates to a line of carelessly-written code deep within a system.
Mismatches between requirements and delivered features are exactly as
common as are misunderstandings between business and technical people.
This is no coincidence.
Organisations that commission Storm’s software development audit gain the
benefits of:
●
●
●
●
Protection through independent validation of the process
Risk assessment and opportunity for mitigation
Due diligence
Advance notice of potential problem areas
Opportunities and recommendations for improvement
Storm’s software development process audit comprises 4 activities, which are
explained in more detail below. The activities are:
1.
2.
3.
4.
End to end process review
Requirements gathering audit
Solution provision audit
Report and presentation
The review is led by a Principal Consultant.
End to end process
review
Customer interviews are conducted to reveal their overall satisfaction with the
development process, and the ensuing results. The investigation seeks to
establish the defect rate and efficiency of dealing with defects.
Requirements gathering
Interviews with those people whose role is business analyst, although that
may not be their formal job title. This role is absolutely vital in defining the
needs that are to be addresses by software and the tests that define the
criteria for acceptance.
Solution provision
Interviewing the solution developers and technical architects is geared
towards revealing their diligence towards addressing actual requirements as
well as their technical capabilities.
Report
The report synthesises the research and findings into a single comprehensive
document which Storm customarily presents at executive level.
The report may be shared with all stakeholders in order to provide a startingpoint for overall process improvement, to the mutual advantage of all.
Gap Analysis and Recommendations
for VMware IT Value Model Solution
For
The audit of a software developer will include an
examination of:
●
●
●
●
●
●
●
●
●
Quality Systems & Procedures
Software Development Life Cycle (SDLC) Processes/Practices
Electronic Records/Electronic Signatures
Security, Access Controls & Audit Trail
Release Management
Change Management
Operational Controls
Validation Deliverables
Training Records
<Customer>
Prepared by
<Consultant>
VMware Professional Services
<consultant>@vmware.com
Version History
Date
Ver.
Author
Description
Reviewers
PSO Consultant: Amend all text highlighted in yellow as needed and address the consultant notes. After
completion, the comments and yellow highlighting can be removed. This text provides either sample
guidance information, configuration recommendations for specific services, or customer specific
configurations.
© 2020 VMware, Inc. All rights reserved. This product is protected by U.S. and international
copyright and intellectual property laws. This product is covered by one or more patents listed
at http://www.vmware.com/download/patents.html.
VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or
other jurisdictions. All other marks and names mentioned herein may be trademarks of their
respective companies.
VMware, Inc.
3401 Hillview Ave
Palo Alto, CA 94304
www.vmware.com
Contents
Contents
1.
Document Overview
2.
Executive Summary
2.1 Engagement Overview and Findings
3.
4.
Assessment Methodology
Findings and Recommendations
4.1 Prioritized IT Capabilities
4.2 Current State Summary
4.3 Desired State Summary
4.4 Recommendations
5.
Detailed Assessment
5.1 VMware Maturity Model Assessment
5.2 Current State Technology Assessment
5.3 Current State People and Process Assessment
5.4 Desired State Technology
5.5 Desired State People and Process
6.
Summary of Recommendations and Remediations
6.1 Organizational Strengths, Weaknesses, Opportunities and Threats
6.2 Recommended IT Outcomes, Capabilities and Priorities
6.3 Recommendations for Remediation
PSO Consultant: Some parts of this document have been auto-generated. Update Table of Contents to
see the latest section references
1.
Document Overview
This Gap Analysis and Recommendations document is created by VMware Professional Services to
determine the readiness and steps for successful implementation of the VMware IT Value Model Solution.
An assessment of readiness and complete consideration of next steps requires review of existing
technology, people and process related to this solution.
The readiness assessment conducted as part of this engagement may uncover additional challenges
faced by the organization while adopting new business processes enabled by the appropriate technology
and expertise.
2.
Executive Summary
PSO Consultant: Fill in the information below.
<Customer Name><Brief description of Customer organization:
· <Industry Vertical>
· <Number of employees>
· <Global or regional>
· <IT Organization centralized or distributed>
2.1
Engagement Overview and Findings
PSO Consultant: Choose whichever statement applies to the customers assessment outcome.
The assessment component of this engagement discovered that the current set of IT Capabilities under
development and the associated priorities are in alignment with the required maturity to achieve the
desired state.
or
The assessment component of this engagement discovered that the current set of IT Capabilities under
development and the associated priorities are not in alignment with the required maturity to achieve the
desired state.
For further detailed information, please refer to the Findings and Recommendations section of this
document.
3.
Assessment Methodology
VMware Professional Services leverages a model based upon the following principals:
·
Business models are enabled by IT systems.
·
IT systems fail due to IT Problems.
·
IT Problems are solved by becoming competent in an IT Capability.
·
Combinations of IT Capabilities deliver an IT Outcome.
·
Combinations of IT Outcomes provide business competitive differentiation.
VMware Professional Services has developed a number of customer maturity models that relates an IT
organizations IT capability maturity to the relationship the IT organization has with the broader business.
It is this relationship that defines whether the organization is merely effective in its market or a disrupter in
its market.
In this engagement, VMware has focused the assessment on the following IT Capabilities to solve the
following IT Problems.
IT Capabilities
Enable continuous integration and deployment
Modernize business-critical applications
IT Problems
Revenue/mission critical application technical debt
Lack of automation across the toolchain
Complex processes and multiple tools
4.
Findings and Recommendations
This section describes the high-level findings and recommendations for the gap analysis.
4.1
Prioritized IT Capabilities
To ensure the IT organization is investing in the right priorities based upon the higher-level IT Initiatives;
workshops were conducted to uncover the organizations defined top five IT Capabilities.
PS Consultant: Use the information gathered from the assessment workshop to complete the table
below.
Priority
1
2
3
IT Capabilities
4
5
4.1.1 Analysis
PS Consultant: Us the table above to develop commentary regarding the level of alignment between the
organization’s IT Capabilities and their IT Problems. Consider the capability and consider the alignment
from a priority perspective. Consider whether an IT Capability is a higher priority, but the related IT
Problem is a lower priority and also whether the IT priorities are reactive in nature or are they preemptive.
4.2
Current State Summary
PS Consultant: Briefly describe what you would consider are key identifying traits of the current state
considering people, process and technology. Please refer to the Solution Framework Guide for
information regarding any tooling that is relevant to products involved that can be used to complete this
section.
4.2.1 Current State Maturity Rating
VMware’s extensive global experience enabled the development of organization maturity models. The
data collected from VMware’s TAM customers globally, helped in bringing geographical and industrial
perspectives to the models. The ranking below is a ranking in the context of VMware TAM Customers
globally based upon the most recent TAM CLEAR report.
PS Consultant: Use the spreadsheet assessment tooling provided in the service content in conjunction
with the data you acquired from the assessment workshop to come up with the ranking. Please refer to
the Solution Framework Guide for information regarding any tooling relevant to products involved that can
be used to complete this section.
RANKING:
4.3
Desired State Summary
PS Consultant: Briefly describe what you would consider are key identifying traits of the desired state
considering people, process and technology. Describe the desired state in the context of the relevant
VMware value model (ITVM or DWJM). Describe the mandatory IT Capabilities that a customer needs to
achieve to be developed in order to have successfully achieved that Solution in the respective journey
model.
4.3.1 Desired State Maturity Rating
VMware’s extensive global experience enabled the development of organization maturity models. The
data collected from VMware’s TAM customers globally, helped in bringing geographical and industrial
perspectives to the models. The ranking below is a ranking in the context of VMware TAM Customers
globally based upon the most recent TAM CLEAR report.
PS Consultant: Use the spreadsheet assessment tooling provided in the service content in conjunction
with the data you acquired from the assessment workshop to come up with the ranking. Please refer to
the Solution Framework Guide for information regarding any tooling relevant to products involved that can
be used to complete this section.
RANKING:
4.4
Recommendations
This section describes a summary of the gap analysis and the proposed remediation plan.
4.4.1 Summary of Strengths, Weaknesses, Opportunities and Threats
SWOT Analysis is a useful technique for identifying <Customer> Strengths, Weaknesses, Opportunities
and Threats. The following table describes these items for this engagement.
PS Consultant: Use the table below to list out the SWOT analysis. You can achieve this by reviewing the
notes from the Assessment Workshop. A level of maturity was assigned to items such as Technology
Function and Operations Process during this workshop. Categorize these items accordingly.
Strengths
Weaknesses
·
·
Opportunities
Threats
·
·
4.4.2 Proposed Remediation
The following actions are recommended to realign <Customers> resources to solve the identified IT
problems through the development of IT capabilities.
Priority
Remediation Action
5.
Detailed Assessment
This section describes the detailed assessment to support the high-level findings and recommendations.
5.1
VMware Maturity Model Assessment
VMware IT Value Model Solution help customer to develop capabilities to address the following use
cases:
5.2
Current State Technology Assessment
The technology assessment has found that the current technology state in <Customer> environment.
PS Consultant: Please refer to the Solution Framework Guide for information regarding any tooling
provided relevant to products involved that can be used to complete this section.
5.2.1 Products, Versions and Support Status
The following table illustrates the current relevant products by function within the VMware Solution, the
current vendor, product, version and support status (if identifiable).
PS Consultant: Please fill out the Function and support status if known.
Technology Components
Version
Tanzu Kubernetes Grid
1.1.x
5.2.1.1. Analysis
PS Consultant: Brief paragraph that summarizes the technologies currently used and commentary on
fitness for purpose given IT Capabilities and priorities.
5.3
Current State People and Process Assessment
PS Consultant: Remove this section if there are no OTS related assessments done. Use information
acquired from the Assessment workshop relative to People and Process related micro-services to
complete the current state section. If no people and process related micro-services are in scope of this
engagement, delete all the relevant people and process related sections.
5.3.1 Organizational Structure
PS Consultant: Brief paragraph and simple diagram that illustrates the current IT organization – the
diagram should start at CIO and go down to team leaders at the most.
5.3.1.1. Analysis
PS Consultant: Brief paragraph that analyses the diagram – the aim is to draw out challenges identified
through assessment workshop.
5.3.2 Process Assessment
PS Consultant: Using the tooling provided in the Framework documents folder for this Solution, input the
data acquired from the Assess Workshop and enter the maturity rating into the table below. Where a row
is not appropriate please remove.
Process Type
Provisioning
Operating
Monitoring
Recovering
Removing
Process Maturity Rating
Marketing
Defining
Problem Management
Change Management
5.3.2.1. Analysis
PS Consultant: Add a brief paragraph that analyzes the table above – the aim is to draw out challenges
identified through assessment workshop.
5.3.3 Related Process Supporting Technology
IT business processes are supported by technology. During the assessment, the following products are
identified as supporting elements. The table below highlights where a maturity rating could be improved
through the change of the supporting technology.
Process Type
Provisioning
Operating
Monitoring
Recovering
Removing
Marketing
Supporting Technology
Potential Improvement
through tooling change
Defining
Problem Management
Change Management
5.4
Desired State Technology
The following section details the desired technology state for <Customer>.
5.4.1 Required Technology Functionality
5.4.1.1. Function - <Solution Element Technology Function>
Related Product
Minimum Required Version
5.4.2 IT Capability Alignment
PS Consultant: Use the information acquired in the Assess Workshop to assign a priority to technology
functionality, based upon the IT Capabilities and priorities.
The following table illustrates the alignment between the functionality outlined in the desired state and the
IT Capabilities and priorities.
Technology Functionality
Related IT Capability
Priority
5.5
Desired State People and Process
PS Consultant: Remove this section if there are no OTS related components. Use information acquired
from the Assessment workshop relative to People and Process related micro-services to complete the
current state section. If there are no people and process related micro-services that are in scope in this
engagement, delete all the relevant people and process related sections.
5.5.1 Organizational Structure
PS Consultant: Brief paragraph and simple diagram that illustrates your understanding of the desired
state IT organization – the diagram should start at CIO and go down to team leaders at the most.
5.5.2 Areas of Process and Desired State Description
The following table illustrates the desired state.
Process Type
Provisioning
Operating
Monitoring
Recovering
Process
Desired State Description
Removing
Marketing
Defining
Problem
Management
Change
Management
5.5.3 IT Capability Alignment
The following table illustrates the alignment between the functionality outlined in the desired state and the
IT Capabilities and priorities.
Process Type
Related IT Capability
Priority
6.
Summary of Recommendations and Remediations
This section describes recommended actions for Customer to remediate any gaps in the environment.
6.1
Organizational Strengths, Weaknesses, Opportunities and
Threats
PS Consultant: Using the tooling provided in the Solution Framework Guide for this Solution and
information gathered in the assessment workshop to develop the SWOT analysis for this section.
6.2
Recommended IT Outcomes, Capabilities and Priorities
PS Consultant: Using the information gathered from the assessment workshop along with the
information in Solution Builder regarding the IT Outcomes and Capabilities, assign a priority and complete
the table below.
IT Outcome
6.3
Related Capabilities
Recommended Priority
Recommendations for Remediation
PS Consultant: Using information gathered from the assessment workshop and the priorities assigned,
develop this section, outlining the work streams, the key outcomes of each work stream and the
recommended sequencing of the work streams.
Workstreams
Team structure:
Manager:
Developers:
Lead:
QA
UX
Admin
Support
BA
Other:
Process:
Tools:
Velocity:
Releases:
Sequence
Key Outcomes
People development:
Risk mitigation:
Quality assurance:
Security
Resiliency
Finding and recommendations:
System Development Life Cycle Audit
Program
AUDIT PROGRAM OVERVIEW
A system development life cycle (SDLC) is a methodology that can be used to develop or
modify application systems. Each organization should establish a SDLC methodology and
assign responsibility for each phase of the cycle so that system design, development, and
maintenance may progress smoothly and accurately. This cycle starts with a perceived need
and extends through feasibility study, design and development, testing, implementation, system
acceptance and approval, post-implementation review, and maintenance of the application and
systems software. Following each phase of this cycle ensures that the new or revised software
meets the organization's needs, that adequate internal controls are consistent with
management's objectives, and that the application is properly implemented.
This audit program assumes that an application system is developed by an in-house
programming staff. However, application systems in use by many state agencies were not
developed in-house but instead were purchased. In these instances, all the steps performed
during in-house development of an application are not applicable for purchased software.
Specifically, systems and programming standards, and file and programming specifications are
not needed. In these cases, document in the Summary Memo how the scope of this audit
program will be modified and answer Not Applicable (N/A) to any questions on the ICQ that do
not apply.
Suggested interviewees for ICQ:
A. System Programming Manager
B. Director of Data Processing
A. Control Objective #1 - SDLC Methodology
1. Determine the extent of the responsibilities of management, internal audit, users, quality
assurance, and data processing during the system design, development, and
maintenance.
2. Review SDLC workpapers to determine if the appropriate levels of authorization were
obtained for each phase.
3. Obtain and review requests for DP services. Determine if the University's procedures are
being followed.
B. Control Objective #2 - Needs Analysis
1. Review and evaluate the procedures for performing a needs analysis.
2. Review a needs analysis for a recent project and determine if it conforms to standards.
C. Control Objective #3 - Systems Design and Development
1. Review and evaluate the procedures for systems design and development.
2. Review design specifications schedules, look for written evidence of approval, and
determine if the design specifications comply with the standards.
3. Determine if an audit trail and programmed controls are incorporated in the design
specifications of a recent project.
4. Review samples of source documents used for data entry which are included in SDLC
workpapers of a recently developed application. Determine if they are designed to
facilitate accurate gathering and entry of information.
5. Obtain and review programs to determine if they comply with the University's
programming standards.
D. Control Objective #4 - Testing Procedures
1. Review and evaluate the procedures for system and program testing.
2. Review documented testing procedures, test data, and resulting output to determine if
they appear to be comprehensive and if they follow University standards.
3. Review the adequacy of testing performed on the manual phases of an application.
E. Control Objective #5 - Implementation Procedures
1. Review and evaluate procedures for program promotion and implementation.
2. Review documentation of the program promotion procedure. Determine if the standards
are followed and if documentation of compliance with the standards is available. Trace
selected program and system software changes to the appropriate supporting records to
determine if the changes have been properly approved.
3. Review documentation of the conversion/implementation of a newly developed
application. Determine if the University's implementation procedures were followed.
F. Control Objective #6 - Post-implementation Review
1. Review and evaluate the procedures for performing post-implementation reviews.
2. Review program modifications, testing procedures, and the preparation of supporting
documentation to determine if the University's standards are being followed.
G. Control Objective #7 - Maintenance of Applications
1. Review and evaluate the procedures for the maintenance of existing applications.
2. Review program modifications, testing procedures, and the preparation of supporting
documentation to determine if the University's standards are being followed.
H. Control Objective #8 - Control over Systems Software
1. Review and evaluate the procedures for modifying systems software.
2. Review systems software modifications, testing procedures, and the preparation of
supporting documentation to determine if the University's standards are being followed.
3. Review and evaluate documentation of in-house developed systems software and the
features/options of proprietary systems software in use.
I. Control Objective #9 - Documentation Standards
1. Obtain and review the documentation standards to determine if they are complete.
EFFECT OF WEAKNESSES
Because it has been estimated that a major portion of the cost of an application over its useful
life is incurred for maintenance after the application becomes operational, if little attention is
given to the SDLC in the creation of a system, excessive maintenance costs can be incurred,
especially if it is necessary to put controls in after the application is already in production.
Redesign is not only expensive, but difficult to accomplish.
If accurate and comprehensive documentation is not maintained, the auditor will have difficulty
assessing controls without expending substantial effort to obtain an accurate description of
significant applications and their relationships to one another.
If modifications to application and system software are not adequately controlled, the integrity of
the software may be compromised by unauthorized changes in programs, procedures, or data.
When an application is properly designed, systems development and documentation controls
can prevent or disclose the following types of errors:
1. implementation of applications that do not have adequate application controls;
2. development of applications that either do not meet management objectives or do not
operate in accordance with original specifications;
3. implementation of applications that have not been adequately tested, and;
4. implementation of applications that are susceptible to unauthorized modification.