INTOSAI IT Audit
IT Methods Awareness
Outline
•
•
•
•
•
•
Scope
Overview
It Methods
Methods Description
Methods Usage
Audit Reporting
Scope
• It Methods Described For:
– Project Selection, Control, Evaluation
– Systems Development
– Systems Acquisition
– Enterprise Architecture Development
– Security Assessment
Overview
• Methods Listed Here Are Generally
Accepted in The Community
• Methods Assess or Prescribe “What”
Must Be Done Not “How” to Accomplish
Activity
• Methods Provide a Framework to Audit
It Activity
It Methods
http://www.gao.gov
/special.pubs/ai10
123.pdf
Project Selection, Control,
Evaluation
Information Technology
Investment Management: A
Framework for Assessing and
Improving Process Maturity
Systems Development
SEI Software Capability Maturity http://www.sei.cmu
.edu/cmm/
Model
Systems Acquisition
SEI Software Acquisition
Capability Maturity Model
http://www.sei.cmu
.edu/publications/d
ocuments/99.repor
ts/99tr002/99tr002
abstract.html
Enterprise Architecture
Development
A Practical Guide to Federal
Enterprise Architecture, Chief
Information Officer Council,
Version 1.0, February, 2001
http://www.itpolicy.
gsa.gov/mke/archp
lus/group.htm
Security Assessment
Information Technology Security
Assessment Framework,
November 28, 2000 (Security,
Privacy, and Critical
Infrastructure Committee)
http://www.cio.gov/
docs/federal_it_se
curity_assessment
_framework.htm
Methods Description
Module 1
Project Selection, Control, Evaluation
Project Selection, Control,
Evaluation
• Wisely Managed Investments in It Can
Improve Organizational Performance
• Internet and Local Area Networks Enable
Data Sharing and Research
• Data Warehouse Permits Organizations to
Discover Unknown Fiscal or Physical
Resources
Project Selection, Control,
Evaluation
• However, Along With the Potential to Improve
Organizations, It Projects Can Become Risky,
Costly, Unproductive Mistakes
• In Response, Gao Developed Guidance, That
Provides a Method for Evaluating and
Assessing How Well an Agency Is Selecting
and Managing Its It Resources
Project Selection, Control,
Evaluation
• The Select/control/evaluate Model Has Become a
Central Tenet of the It Investment Management
Approach
Project Selection, Control,
Evaluation
• During the Selection Phase the Organization
– Selects Those It Projects That Will Best Support
Its Mission Needs and
– Identifies and Analyzes Each Project’s Risks and
Returns Before Committing Significant Funds to a
Project.
Project Selection, Control,
Evaluation
• During the Control Phase the
– Organization Ensures That, As Projects Develop,
the Project Is Continuing to Meet Mission Needs
at Expected Levels of Cost and Risk
– If the Project Is Not Meeting Expectations Steps
Are Taken to Address the Deficiencies
Project Selection, Control,
Evaluation
• Lastly, During the Evaluation Phase,
– Actual Versus Expected Results Are Compared to
• Assess the Project’s Impact on Mission Performance,
• Identify Any Changes or Modifications to the Project That
May Be Needed, and
• Revise the Investment Management Process Based on
Lessons Learned
Project Selection, Control,
Evaluation
• Gao’s Information Technology Investment
Model (Itim) Model Is Comprised of Five
Stages of Maturity
• Each Stage Builds Upon the Lower Stages
and Enhances the Organization’s Ability to
Manage Its It Investment Stages
Project Selection, Control,
Evaluation
Five Stages Of Investment Maturity
Project Selection, Control,
Evaluation
Progressing Through the ITIM Stages of Maturity
Project Selection, Control,
Evaluation
• Itim Is a Tool for Assessing the Maturity of an
Organization
– An Itim Assessment Can Be Conducted for an
Entire Organization or For One of Its Lower
Divisions
– Itim Is Applicable to Organizations of Different
Sizes
Project Selection, Control,
Evaluation
• Itim Allows Auditors to Assesses the Maturity
of Organizations to Manage Investments
• Itim Provides a Maturity Stage or “Level” for
an Organization
• Each Maturity Stage or “Level” Has Required
Practices or Activities
Project Selection, Control,
Evaluation
ITIM Required Processes
Project Selection, Control,
Evaluation
• Applying the Model Requires Assessing
– Critical Processes, Such As the Processes Used to Create
an It Investment Portfolio
– Core Elements, (Purpose, Organizational Commitment,
Prerequisites, Activities, and Evidence of Performance)
Questions / Discussion
•
•
•
•
Questions
Comments
Discussion
Etc.
Methods Description
Module 2
Systems Development
Systems Development
• Systems Development Includes
Activities Such As
– Project Management,
– Requirements Management,
– Configuration Management
– Software Development, Testing, Etc.
Systems Development
• Many Organizations Rely on Software-intensive
Systems to Perform Their Missions
• Software Quality Is Governed by the Quality of the
Processes Used To Develop the Software
• (Provide Reference)
Systems Development
• The Software Engineering Institute Has
Developed a Number of Models That
Facilitate Assessing the Maturity of
Organizations Developing Software
• The Models Are Called Capability Maturity
Models (Cmm)
Systems Development
• What Is the Cmm?
– An Ordered Collection of Practices for the Acquisition,
Development or Maintenance of Systems
– Ordered by “Key Process Area”
– Practices Determined by the Community Through Broad
Peer Reviews
– Defines the Stages Through Which Organizations Evolve As
They Improve Their Acquisition Process
– Identifies Key Priorities, Goals and Activities on the Road to
Improving an Organization's Capability to Do Its Job
Systems Development
• The Cmm Provides a Framework for
– Identifying an Organization’s Process Strengths
and Weaknesses
– Assisting an Organization Develop a Structured
Plan for Process Improvement
Systems Development
• Who Uses the Sw-cmm?
– Organizations That Develop or Maintain Products That
Contain Software
– Organizations Who Want to Improve Their Software
Development Processes
– Audit Organizations Who Want to Assess the Maturity Of
Organizations Developing or Maintaining Software Products
Systems Development
• The Cmm Is Structured Into
– Five Maturity Levels
• Each Level Has Key Process Areas (Kpa)
– Each Kpa Has Goals
• Goals Require Certain Activities Be Performed
• Management Provides Support and Verifies
That Activities Are Being Performed
Systems Development
Systems Development
• The Five Levels Are
• 1. Initial: The Software Process Is Characterized As Ad Hoc
and Few Processes Are Defined
• 2. Repeatable: Basic Project Management Processes Are
Established; Improvement Activities Are Begun
• 3. Defined: Software Processes Are Documented and
Standardized; All Projects Use an Approved, Tailored Version of
the Organization’s Standard Software Processes
Systems Development
• The Five Levels Are (Contd.)
• 4. Managed/quantitative: Detailed Measures of the Software
Processes, Products, and Services Are Collected; the Software
Processes and Products Are Quantitatively Measured and
Controlled
• 5. Optimizing: Continuous Process Improvement Is Enabled by
Quantitative Feedback From the Process and From Piloting
Innovative Ideas and Technologies
Systems Development
Software CMM Levels and KPAs
Systems Development
• Cmm Common Features
– Commitment To Perform
– Ability To Perform
– Activities
– Measurement & Analysis
– Verification
Systems Development
• Commitment To Perform
– Describes What an Organization Must Do to ‘Set
the Stage’ for Process Improvement /
Implementation
• Involves Establishing Policy
• Assigning Responsibility
Systems Development
• Ability To Perform
– Describes the Preconditions That Must Be Present
to Facilitate Process Improvement /
Implementation
• Assignment of Duties to Groups
• Providing Trained or Experienced Personnel
• Ensuring Adequacy of Resources
Systems Development
• Activities
– Describe the Activities, Roles, and Procedures
That Are Necessary to Implement the Key Process
Area
• Requires Formal and Informal Planning Documents
• Requires Formally Documented Procedures
• Requires (Depending on Kpa) Coordination With Other
Affected Groups, Tracking Contractor Performance, Etc.
Systems Development
• Measurement & Analysis
– Describes the Practices That Must Be
Accomplished to Enable the Group to Track the
Status of the Kpa
• Effort & Funds Expended by the Project Team in
Conducting Its Activities
• Tracking Their Schedule and Progress (for Developing
Formal Plans, Requirements, Etc.)
Systems Development
• Verification
– Describes the Practices That Must Be Performed
to Ensure That Project and Senior Management
Oversee the Activities of the Group
• Includes Periodic or As Needed
– Project Level Reviews
– Senior Management Level Reviews
Systems Development
Example From Model
Questions / Discussion
•
•
•
•
Questions
Comments
Discussion
Etc.
Methods Description
Module 3
Systems Acquisition
Systems Acquisition
• Systems Acquisition Includes Activities
Such As
– Project Management,
– Requirements Management,
– Solicitation, Contractor Tracking
– Evaluation, Risk Management, Etc.
Systems Acquisition
• Many Organizations Rely on Software-intensive
Systems to Perform Their Missions
• Organizations Have Been Increasingly Contracting
Out for Software or Engineering Services
Systems Acquisition
• The Software Engineering Institute Has
Developed a Number of Models That
Facilitate Assessing the Maturity of
Organizations That Acquire Software or
Systems
• The Models Are Called Capability Maturity
Models (Cmm)
Systems Acquisition
• Just As For Software Development
There Is the Sw-cmm (or Just Cmm)
• For Assessing or Improving Acquisition
Related Activities, The Sei Has
Developed the Software Acquisition
Capability Maturity Model (Sa-cmm)
Systems Acquisition
• Who Uses The Sa-cmm?
– Organizations That Acquire or Support Acquisition of
Products That Contain Software, Including Software Support
and Maintenance
– Organizations That Are Responsible for Acquisition Life
Cycle From Requirements Development Through System
Delivery and Support
– Audit Institutions That Want To Assess How Effectively
Software or Services Are Being Acquired
Systems Acquisition
• The Sa-cmm Is Also Structured Into
– Five Maturity Levels
• Each Level Has Key Process Areas (Kpa)
– Each Kpa Has Goals
• Goals Require Certain Activities Be Performed
• Management Provides Support and Verifies
That Activities Are Being Performed
Systems Acquisition
Systems Acquisition
• The Five Levels Are
• 1. Initial: The Software Process Is Characterized As Ad Hoc
and Few Processes Are Defined
• 2. Repeatable: Basic Project Management Processes Are
Established; Improvement Activities Are Begun
• 3. Defined: Software Processes Are Documented and
Standardized; All Projects Use an Approved, Tailored Version of
the Organization’s Standard Software Processes
Systems Acquisition
• The Five Levels Are (Contd.)
• 4. Managed/quantitative: Detailed Measures of the Software
Processes, Products, and Services Are Collected; the Software
Processes and Products Are Quantitatively Measured and
Controlled
• 5. Optimizing: Continuous Process Improvement Is Enabled by
Quantitative Feedback From the Process and From Piloting
Innovative Ideas and Technologies
Systems Acquisition
Systems Acquisition
Example From Model
Questions / Discussion
•
•
•
•
Questions
Comments
Discussion
Etc.
Methods Description
Module 4
Enterprise Architecture Development
Enterprise Architecture
Development
• An Enterprise Architecture (Ea)
Establishes the Agency-wide Roadmap
to Achieve an Agency’s Mission
• Eas Are “Blueprints” for Systematically
and Completely Defining an
Organization’s Current (Baseline) or
Desired (Target) Environment
Enterprise Architecture
Development
• Eas Are Essential for Evolving
Information Systems and Developing
New Systems That Optimize Mission
Value
• For Eas to Be Useful and Provide
Business Value, Their Development,
Maintenance, and Implementation
Should Be Managed Effectively
Enterprise Architecture
Development
• An Ea Is A Strategic Information Asset,
Which Documents the Mission And,
– The Information , Technology, and the
Processes Required to Perform the
Mission
• An Ea Includes a Baseline Architecture,
Target Architecture, and a Sequencing
Plan
Enterprise Architecture
Development
• Eas Typically Include
– Business or Operational Architecture
– Work Processes and Locations
– Information or Data Architecture
– Data or Information Needed to Perform Business
– Technical or Systems Architecture
– Technology Standards, It Systems Description
Enterprise Architecture
Development
Section 3.1
EA Process
Section 7
Section 6
Obtain
Executive
Buy-In and
Support
Maintain the
Enterprise
Architecture
Use
the
Enterprise
Architecture
Control
Control
and
and
Oversight
Oversight
Develop the
Sequencing Plan
Section 5
Sections Refer to:
A Practical Guide To Federal
Enterprise Architecture
Section 3.2
Establish
Management
Structure
and Control
Develop
Target
Enterprise
Architecture
Section 5
Define an
Architecture
Process
and Approach
Develop
Baseline
Enterprise
Architecture
Section 5
Section 4
Enterprise Architecture
Development
• Obtain Executive Buy-in and Support
–
–
–
Ensure Agency Head Buy-in and Support
Issue an Executive Enterprise Architecture Policy
Obtain Support From Senior Executives and Business Units
• Establish Management Structure and Control
–
–
–
–
Establish a Technical Review Committee
Establish a Capital Investment Council
Establish an Ea Executive Steering Committee
Appoint Chief Architect
• Define an Architecture Process and Approach
–
–
–
Define the Intended Use of the Architecture
Define the Scope of the Architecture
Determine the Depth of the Architecture
Enterprise Architecture
Development
• Develop the Baseline Enterprise Architecture
–
–
Collect Information
Generate Products and Populate Ea Repository
• Develop the Target Enterprise Architecture
–
–
•
Collect Information
Generate Products and Populate Ea Repository
Develop the Sequencing Plan
–
–
–
Identify Gaps
Define and Differentiate Legacy, Migration, and New Systems
Planning the Migration
Enterprise Architecture
Development
• Use the Enterprise Architecture
–
–
–
Integrate the Ea With Cpic and Slc Processes
Train Personnel
Establish Enforcement Processes and Procedures
• Maintain the Enterprise Architecture As the Enterprise Evolves
–
–
Reassess the Enterprise Architecture Periodically
Manage Products to Reflect Reality
Questions / Discussion
•
•
•
•
Questions
Comments
Discussion
Etc.
Methods Description
Module 5
Security Assessment
Security Assessment
• Information and the Systems That
Process It Are Among the Most Valuable
Assets of Any Organization
• Adequate Security of These Assets Is a
Fundamental Management
Responsibility
Security Assessment
• Agency Must
– Ensure That Systems and Applications Provide
Appropriate Confidentiality, Integrity, and
Availability
– Protect Information Commensurate With the Level
of Risk and Magnitude of Harm Resulting From
Loss, Misuse, Unauthorized Access, or
Modification
Security Assessment
• Agencies Must
– Plan for Security
– Ensure Appropriate Officials Are Assigned Security
Responsibility
– Authorize (or ”Certify") System Processing Prior to
Operations and Periodically As Necessary
Security Assessment
• The Federal It Security Assessment
Framework Provides a Method for
– Determining the Current Status of Their Security Programs
– Establishing a Target for Improvements Where Necessary
• The Framework May Be Used to Assess
the Status of Security Controls
Security Assessment
• The Framework Comprises Five Levels
to Guide Assessments of Security
Programs and Prioritization of
Improvement Efforts
Security Assessment
• Level 1 Documented Policy
• Level 2 Documented Procedures
• Level 3 Implemented Procedures and
Controls
• Level 4 Tested and Reviewed Procedures
and Controls
• Level 5 Fully Integrated Procedures and
Controls
Security Assessment
• Level 1 of the Framework Includes
– Formally Documented and Disseminated Security Policy
• Level 2 of the Framework Includes
– Formal, Complete, Documented Procedures for
Implementing Policies Established at Level One
Security Assessment
• Level 3 of the Framework Includes
– Security Procedures and Controls That Are Implemented
• Level 4 of the Framework Includes
– Routinely Evaluating the Adequacy and Effectiveness of
Security Policies, Procedures, and Controls
• Level 5 of the Framework Includes
– A Comprehensive Security Program That Is an Integral Part
of an Agency’s Organizational Culture
Security Assessment
Security Assessment
Security Assessment
Security Assessment
Security Assessment
Questions / Discussion
•
•
•
•
Questions
Comments
Discussion
Etc.
Methods Usage
Methods Usage
• Most Methods
–
–
–
–
Have Specific Activities to Be Performed
Can Be Applied to Specific Projects
Need a Team of About 3 - 4 Auditors
Requires Training or Understanding of the Method
Methods Usage
• Since Methods Have Specific Activities
– Questionnaires Can Be Generated
– Results Can Be Tabulated
– Analysis Can Be Formed Quickly From the
Results
Methods Usage
• The Sw-cmm and Sa-cmm Methods
Require the Audit Lead Be Specifically
Trained
Methods Usage
Sample Data Collection Instrument
Audit Reporting
Audit Reporting
• Audit Report Can Be Briefing Slides or
Full Reports
• Briefing Slides Can Contain Both
Summary or Detailed Results
Audit Reporting
Sample
•
SA-CMM
Summary
Results
Audit Reporting
Sample SA-CMM Acquisition Risk Management Detailed Results
Questions / Discussion
•
•
•
•
Questions
Comments
Discussion
Etc.
Contacts
– Keith Rhodes
• Phone
• Email
1 202 512 6412
rhodesk@gao.gov
– Madhav Panwar
• Phone
• Email
1 202 512 6228
panwarm@gao.gov