SOFTWARE DEVELOPMENT PROCESS AUDIT Description A fixed price consultancy-led review of the software development lifecycle (SDLC) or application lifecycle management (ALM) process. Whether your software development process revolves around in-house, offshore, “right-shore”, or out-sourced resources it needs to be effective and efficient. Software is produced to directly serve the business that calls for it, yet all too often it falls short of expectations. This audit will reveal potential problem areas, risks, weaknesses as well as direct opportunities for improvement in all aspects of the process. The deliverables of the audit include: ● Executive summary ● Report that details strengths, weaknesses, and risks of the current SDLC ● Recommendations for improvement ● Briefing session Benefits It’s all too easy to believe software that appears to work is actually doing what it is supposed to. Metrics are often misplaced and regularly measure superficial and irrelevant data. Clients are vulnerable to questions of accountability when the problem really relates to a line of carelessly-written code deep within a system. Mismatches between requirements and delivered features are exactly as common as are misunderstandings between business and technical people. This is no coincidence. Organisations that commission Storm’s software development audit gain the benefits of: ● ● ● ● Protection through independent validation of the process Risk assessment and opportunity for mitigation Due diligence Advance notice of potential problem areas Opportunities and recommendations for improvement Storm’s software development process audit comprises 4 activities, which are explained in more detail below. The activities are: 1. 2. 3. 4. End to end process review Requirements gathering audit Solution provision audit Report and presentation The review is led by a Principal Consultant. End to end process review Customer interviews are conducted to reveal their overall satisfaction with the development process, and the ensuing results. The investigation seeks to establish the defect rate and efficiency of dealing with defects. Requirements gathering Interviews with those people whose role is business analyst, although that may not be their formal job title. This role is absolutely vital in defining the needs that are to be addresses by software and the tests that define the criteria for acceptance. Solution provision Interviewing the solution developers and technical architects is geared towards revealing their diligence towards addressing actual requirements as well as their technical capabilities. Report The report synthesises the research and findings into a single comprehensive document which Storm customarily presents at executive level. The report may be shared with all stakeholders in order to provide a startingpoint for overall process improvement, to the mutual advantage of all. Gap Analysis and Recommendations for VMware IT Value Model Solution For The audit of a software developer will include an examination of: ● ● ● ● ● ● ● ● ● Quality Systems & Procedures Software Development Life Cycle (SDLC) Processes/Practices Electronic Records/Electronic Signatures Security, Access Controls & Audit Trail Release Management Change Management Operational Controls Validation Deliverables Training Records <Customer> Prepared by <Consultant> VMware Professional Services <consultant>@vmware.com Version History Date Ver. Author Description Reviewers PSO Consultant: Amend all text highlighted in yellow as needed and address the consultant notes. After completion, the comments and yellow highlighting can be removed. This text provides either sample guidance information, configuration recommendations for specific services, or customer specific configurations. © 2020 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. This product is covered by one or more patents listed at http://www.vmware.com/download/patents.html. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies. VMware, Inc. 3401 Hillview Ave Palo Alto, CA 94304 www.vmware.com Contents Contents 1. Document Overview 2. Executive Summary 2.1 Engagement Overview and Findings 3. 4. Assessment Methodology Findings and Recommendations 4.1 Prioritized IT Capabilities 4.2 Current State Summary 4.3 Desired State Summary 4.4 Recommendations 5. Detailed Assessment 5.1 VMware Maturity Model Assessment 5.2 Current State Technology Assessment 5.3 Current State People and Process Assessment 5.4 Desired State Technology 5.5 Desired State People and Process 6. Summary of Recommendations and Remediations 6.1 Organizational Strengths, Weaknesses, Opportunities and Threats 6.2 Recommended IT Outcomes, Capabilities and Priorities 6.3 Recommendations for Remediation PSO Consultant: Some parts of this document have been auto-generated. Update Table of Contents to see the latest section references 1. Document Overview This Gap Analysis and Recommendations document is created by VMware Professional Services to determine the readiness and steps for successful implementation of the VMware IT Value Model Solution. An assessment of readiness and complete consideration of next steps requires review of existing technology, people and process related to this solution. The readiness assessment conducted as part of this engagement may uncover additional challenges faced by the organization while adopting new business processes enabled by the appropriate technology and expertise. 2. Executive Summary PSO Consultant: Fill in the information below. <Customer Name><Brief description of Customer organization: · <Industry Vertical> · <Number of employees> · <Global or regional> · <IT Organization centralized or distributed> 2.1 Engagement Overview and Findings PSO Consultant: Choose whichever statement applies to the customers assessment outcome. The assessment component of this engagement discovered that the current set of IT Capabilities under development and the associated priorities are in alignment with the required maturity to achieve the desired state. or The assessment component of this engagement discovered that the current set of IT Capabilities under development and the associated priorities are not in alignment with the required maturity to achieve the desired state. For further detailed information, please refer to the Findings and Recommendations section of this document. 3. Assessment Methodology VMware Professional Services leverages a model based upon the following principals: · Business models are enabled by IT systems. · IT systems fail due to IT Problems. · IT Problems are solved by becoming competent in an IT Capability. · Combinations of IT Capabilities deliver an IT Outcome. · Combinations of IT Outcomes provide business competitive differentiation. VMware Professional Services has developed a number of customer maturity models that relates an IT organizations IT capability maturity to the relationship the IT organization has with the broader business. It is this relationship that defines whether the organization is merely effective in its market or a disrupter in its market. In this engagement, VMware has focused the assessment on the following IT Capabilities to solve the following IT Problems. IT Capabilities Enable continuous integration and deployment Modernize business-critical applications IT Problems Revenue/mission critical application technical debt Lack of automation across the toolchain Complex processes and multiple tools 4. Findings and Recommendations This section describes the high-level findings and recommendations for the gap analysis. 4.1 Prioritized IT Capabilities To ensure the IT organization is investing in the right priorities based upon the higher-level IT Initiatives; workshops were conducted to uncover the organizations defined top five IT Capabilities. PS Consultant: Use the information gathered from the assessment workshop to complete the table below. Priority 1 2 3 IT Capabilities 4 5 4.1.1 Analysis PS Consultant: Us the table above to develop commentary regarding the level of alignment between the organization’s IT Capabilities and their IT Problems. Consider the capability and consider the alignment from a priority perspective. Consider whether an IT Capability is a higher priority, but the related IT Problem is a lower priority and also whether the IT priorities are reactive in nature or are they preemptive. 4.2 Current State Summary PS Consultant: Briefly describe what you would consider are key identifying traits of the current state considering people, process and technology. Please refer to the Solution Framework Guide for information regarding any tooling that is relevant to products involved that can be used to complete this section. 4.2.1 Current State Maturity Rating VMware’s extensive global experience enabled the development of organization maturity models. The data collected from VMware’s TAM customers globally, helped in bringing geographical and industrial perspectives to the models. The ranking below is a ranking in the context of VMware TAM Customers globally based upon the most recent TAM CLEAR report. PS Consultant: Use the spreadsheet assessment tooling provided in the service content in conjunction with the data you acquired from the assessment workshop to come up with the ranking. Please refer to the Solution Framework Guide for information regarding any tooling relevant to products involved that can be used to complete this section. RANKING: 4.3 Desired State Summary PS Consultant: Briefly describe what you would consider are key identifying traits of the desired state considering people, process and technology. Describe the desired state in the context of the relevant VMware value model (ITVM or DWJM). Describe the mandatory IT Capabilities that a customer needs to achieve to be developed in order to have successfully achieved that Solution in the respective journey model. 4.3.1 Desired State Maturity Rating VMware’s extensive global experience enabled the development of organization maturity models. The data collected from VMware’s TAM customers globally, helped in bringing geographical and industrial perspectives to the models. The ranking below is a ranking in the context of VMware TAM Customers globally based upon the most recent TAM CLEAR report. PS Consultant: Use the spreadsheet assessment tooling provided in the service content in conjunction with the data you acquired from the assessment workshop to come up with the ranking. Please refer to the Solution Framework Guide for information regarding any tooling relevant to products involved that can be used to complete this section. RANKING: 4.4 Recommendations This section describes a summary of the gap analysis and the proposed remediation plan. 4.4.1 Summary of Strengths, Weaknesses, Opportunities and Threats SWOT Analysis is a useful technique for identifying <Customer> Strengths, Weaknesses, Opportunities and Threats. The following table describes these items for this engagement. PS Consultant: Use the table below to list out the SWOT analysis. You can achieve this by reviewing the notes from the Assessment Workshop. A level of maturity was assigned to items such as Technology Function and Operations Process during this workshop. Categorize these items accordingly. Strengths Weaknesses · · Opportunities Threats · · 4.4.2 Proposed Remediation The following actions are recommended to realign <Customers> resources to solve the identified IT problems through the development of IT capabilities. Priority Remediation Action 5. Detailed Assessment This section describes the detailed assessment to support the high-level findings and recommendations. 5.1 VMware Maturity Model Assessment VMware IT Value Model Solution help customer to develop capabilities to address the following use cases: 5.2 Current State Technology Assessment The technology assessment has found that the current technology state in <Customer> environment. PS Consultant: Please refer to the Solution Framework Guide for information regarding any tooling provided relevant to products involved that can be used to complete this section. 5.2.1 Products, Versions and Support Status The following table illustrates the current relevant products by function within the VMware Solution, the current vendor, product, version and support status (if identifiable). PS Consultant: Please fill out the Function and support status if known. Technology Components Version Tanzu Kubernetes Grid 1.1.x 5.2.1.1. Analysis PS Consultant: Brief paragraph that summarizes the technologies currently used and commentary on fitness for purpose given IT Capabilities and priorities. 5.3 Current State People and Process Assessment PS Consultant: Remove this section if there are no OTS related assessments done. Use information acquired from the Assessment workshop relative to People and Process related micro-services to complete the current state section. If no people and process related micro-services are in scope of this engagement, delete all the relevant people and process related sections. 5.3.1 Organizational Structure PS Consultant: Brief paragraph and simple diagram that illustrates the current IT organization – the diagram should start at CIO and go down to team leaders at the most. 5.3.1.1. Analysis PS Consultant: Brief paragraph that analyses the diagram – the aim is to draw out challenges identified through assessment workshop. 5.3.2 Process Assessment PS Consultant: Using the tooling provided in the Framework documents folder for this Solution, input the data acquired from the Assess Workshop and enter the maturity rating into the table below. Where a row is not appropriate please remove. Process Type Provisioning Operating Monitoring Recovering Removing Process Maturity Rating Marketing Defining Problem Management Change Management 5.3.2.1. Analysis PS Consultant: Add a brief paragraph that analyzes the table above – the aim is to draw out challenges identified through assessment workshop. 5.3.3 Related Process Supporting Technology IT business processes are supported by technology. During the assessment, the following products are identified as supporting elements. The table below highlights where a maturity rating could be improved through the change of the supporting technology. Process Type Provisioning Operating Monitoring Recovering Removing Marketing Supporting Technology Potential Improvement through tooling change Defining Problem Management Change Management 5.4 Desired State Technology The following section details the desired technology state for <Customer>. 5.4.1 Required Technology Functionality 5.4.1.1. Function - <Solution Element Technology Function> Related Product Minimum Required Version 5.4.2 IT Capability Alignment PS Consultant: Use the information acquired in the Assess Workshop to assign a priority to technology functionality, based upon the IT Capabilities and priorities. The following table illustrates the alignment between the functionality outlined in the desired state and the IT Capabilities and priorities. Technology Functionality Related IT Capability Priority 5.5 Desired State People and Process PS Consultant: Remove this section if there are no OTS related components. Use information acquired from the Assessment workshop relative to People and Process related micro-services to complete the current state section. If there are no people and process related micro-services that are in scope in this engagement, delete all the relevant people and process related sections. 5.5.1 Organizational Structure PS Consultant: Brief paragraph and simple diagram that illustrates your understanding of the desired state IT organization – the diagram should start at CIO and go down to team leaders at the most. 5.5.2 Areas of Process and Desired State Description The following table illustrates the desired state. Process Type Provisioning Operating Monitoring Recovering Process Desired State Description Removing Marketing Defining Problem Management Change Management 5.5.3 IT Capability Alignment The following table illustrates the alignment between the functionality outlined in the desired state and the IT Capabilities and priorities. Process Type Related IT Capability Priority 6. Summary of Recommendations and Remediations This section describes recommended actions for Customer to remediate any gaps in the environment. 6.1 Organizational Strengths, Weaknesses, Opportunities and Threats PS Consultant: Using the tooling provided in the Solution Framework Guide for this Solution and information gathered in the assessment workshop to develop the SWOT analysis for this section. 6.2 Recommended IT Outcomes, Capabilities and Priorities PS Consultant: Using the information gathered from the assessment workshop along with the information in Solution Builder regarding the IT Outcomes and Capabilities, assign a priority and complete the table below. IT Outcome 6.3 Related Capabilities Recommended Priority Recommendations for Remediation PS Consultant: Using information gathered from the assessment workshop and the priorities assigned, develop this section, outlining the work streams, the key outcomes of each work stream and the recommended sequencing of the work streams. Workstreams Team structure: Manager: Developers: Lead: QA UX Admin Support BA Other: Process: Tools: Velocity: Releases: Sequence Key Outcomes People development: Risk mitigation: Quality assurance: Security Resiliency Finding and recommendations: System Development Life Cycle Audit Program AUDIT PROGRAM OVERVIEW A system development life cycle (SDLC) is a methodology that can be used to develop or modify application systems. Each organization should establish a SDLC methodology and assign responsibility for each phase of the cycle so that system design, development, and maintenance may progress smoothly and accurately. This cycle starts with a perceived need and extends through feasibility study, design and development, testing, implementation, system acceptance and approval, post-implementation review, and maintenance of the application and systems software. Following each phase of this cycle ensures that the new or revised software meets the organization's needs, that adequate internal controls are consistent with management's objectives, and that the application is properly implemented. This audit program assumes that an application system is developed by an in-house programming staff. However, application systems in use by many state agencies were not developed in-house but instead were purchased. In these instances, all the steps performed during in-house development of an application are not applicable for purchased software. Specifically, systems and programming standards, and file and programming specifications are not needed. In these cases, document in the Summary Memo how the scope of this audit program will be modified and answer Not Applicable (N/A) to any questions on the ICQ that do not apply. Suggested interviewees for ICQ: A. System Programming Manager B. Director of Data Processing A. Control Objective #1 - SDLC Methodology 1. Determine the extent of the responsibilities of management, internal audit, users, quality assurance, and data processing during the system design, development, and maintenance. 2. Review SDLC workpapers to determine if the appropriate levels of authorization were obtained for each phase. 3. Obtain and review requests for DP services. Determine if the University's procedures are being followed. B. Control Objective #2 - Needs Analysis 1. Review and evaluate the procedures for performing a needs analysis. 2. Review a needs analysis for a recent project and determine if it conforms to standards. C. Control Objective #3 - Systems Design and Development 1. Review and evaluate the procedures for systems design and development. 2. Review design specifications schedules, look for written evidence of approval, and determine if the design specifications comply with the standards. 3. Determine if an audit trail and programmed controls are incorporated in the design specifications of a recent project. 4. Review samples of source documents used for data entry which are included in SDLC workpapers of a recently developed application. Determine if they are designed to facilitate accurate gathering and entry of information. 5. Obtain and review programs to determine if they comply with the University's programming standards. D. Control Objective #4 - Testing Procedures 1. Review and evaluate the procedures for system and program testing. 2. Review documented testing procedures, test data, and resulting output to determine if they appear to be comprehensive and if they follow University standards. 3. Review the adequacy of testing performed on the manual phases of an application. E. Control Objective #5 - Implementation Procedures 1. Review and evaluate procedures for program promotion and implementation. 2. Review documentation of the program promotion procedure. Determine if the standards are followed and if documentation of compliance with the standards is available. Trace selected program and system software changes to the appropriate supporting records to determine if the changes have been properly approved. 3. Review documentation of the conversion/implementation of a newly developed application. Determine if the University's implementation procedures were followed. F. Control Objective #6 - Post-implementation Review 1. Review and evaluate the procedures for performing post-implementation reviews. 2. Review program modifications, testing procedures, and the preparation of supporting documentation to determine if the University's standards are being followed. G. Control Objective #7 - Maintenance of Applications 1. Review and evaluate the procedures for the maintenance of existing applications. 2. Review program modifications, testing procedures, and the preparation of supporting documentation to determine if the University's standards are being followed. H. Control Objective #8 - Control over Systems Software 1. Review and evaluate the procedures for modifying systems software. 2. Review systems software modifications, testing procedures, and the preparation of supporting documentation to determine if the University's standards are being followed. 3. Review and evaluate documentation of in-house developed systems software and the features/options of proprietary systems software in use. I. Control Objective #9 - Documentation Standards 1. Obtain and review the documentation standards to determine if they are complete. EFFECT OF WEAKNESSES Because it has been estimated that a major portion of the cost of an application over its useful life is incurred for maintenance after the application becomes operational, if little attention is given to the SDLC in the creation of a system, excessive maintenance costs can be incurred, especially if it is necessary to put controls in after the application is already in production. Redesign is not only expensive, but difficult to accomplish. If accurate and comprehensive documentation is not maintained, the auditor will have difficulty assessing controls without expending substantial effort to obtain an accurate description of significant applications and their relationships to one another. If modifications to application and system software are not adequately controlled, the integrity of the software may be compromised by unauthorized changes in programs, procedures, or data. When an application is properly designed, systems development and documentation controls can prevent or disclose the following types of errors: 1. implementation of applications that do not have adequate application controls; 2. development of applications that either do not meet management objectives or do not operate in accordance with original specifications; 3. implementation of applications that have not been adequately tested, and; 4. implementation of applications that are susceptible to unauthorized modification.