United States Government Accountability Office
GAO
Report to Congressional Committees
April 2007
INFORMATION
TECHNOLOGY
Immigration and
Customs Enforcement
Needs to Fully
Address Significant
Infrastructure
Modernization
Program Management
Weaknesses
GAO-07-565
a
April 2007
INFORMATION TECHNOLOGY
Accountability Integrity Reliability
Highlights
Highlights of GAO-07-565, a report to
congressional committees
Immigration and Customs Enforcement
Needs to Fully Address Significant
Infrastructure Modernization Program
Management Weaknesses
Why GAO Did This Study
What GAO Found
The Department of Homeland
Security (DHS) fiscal year 2006
appropriations act provided $40.15
million for the Immigration and
Customs Enforcement’s (ICE)
program to modernize its
information technology (IT)
infrastructure. As mandated by the
appropriations act, the department
is to develop and submit for
approval an expenditure plan for
the program, referred to as “Atlas,”
that satisfies certain legislative
conditions, including a review by
GAO. In performing its review of
the Atlas plan, GAO (1) determined
whether the plan satisfies certain
legislative conditions and
(2) provided other observations
about the plan and management of
the program. To do this, GAO
analyzed the fiscal year 2006 Atlas
expenditure plan and supporting
documents against the legislative
conditions, federal requirements,
and related best practices. GAO
also interviewed relevant DHS
officials.
The fiscal year 2006 Atlas expenditure plan, in combination with related
program documentation and program officials’ statements, satisfies or
partially satisfies the legislative conditions set forth by Congress (see table).
DHS’s Satisfaction of Legislative Conditions
Legislative conditions
1. Expenditure plan must meet the capital planning and investment control
review requirements established by the Office of Management and Budget
(OMB).
2. DHS must ensure Atlas is compliant with DHS’s enterprise
architecture.
3. The plan must comply with the acquisition rules, requirements,
guidelines, and systems acquisition management practices of the federal
government.
4. The plan must include a certification by DHS’s chief information officer
that an independent verification and validation agent is currently under
contract for the project.
5. The plan must be reviewed and approved by the DHS Investment
Review Board, the Secretary of Homeland Security, and OMB.
Status
partially satisfies
partially satisfies
partially satisfies
partially satisfies
satisfies
Source: GAO.
This satisfaction, however, is based on plans and commitments that provide
for meeting these conditions rather than on completed actions to satisfy
them. For example, to address the legislative condition related to capital
planning and investment control review requirements, the program plans to,
among other things, update its cost-benefit analysis in September 2007 to
reflect emerging requirements and other program changes and to complete a
privacy impact assessment by April 2007. In addition, the program is in the
process of defining how it plans to use its independent verification and
validation agent.
What GAO Recommends
GAO is recommending that DHS
minimize Atlas program risks by
fully adhering to requirements
development and management
practices, implementing key
contract management and
oversight practices, completing
planned risk management
activities, and implementing critical
performance management
practices. DHS concurred with
GAO’s recommendations and
described actions planned and
under way to implement them.
www.gao.gov/cgi-bin/getrpt?GAO-07-565.
To view the full product, including the scope
and methodology, click on the link above.
For more information, contact Randolph C.
Hite at (202) 512-3439 or hiter@gao.gov.
GAO also observed that DHS has not implemented key system management
practices. Specifically,
•
rigorous practices are not being fully adhered to in developing and
managing system requirements,
•
key contract management and oversight controls have not been fully
implemented,
•
planned risk management practices have yet to be implemented, and
•
performance management practices that are critical to measuring
progress against program goals are still being implemented.
Thus, much still needs to be accomplished to minimize the risks associated
with the program’s capacity to deliver promised IT infrastructure capabilities
and benefits on time and within budget. Given that hundreds of millions of
dollars are to be invested and the program is critical to supporting the ICE
mission, it is essential that DHS follow through on its commitments to build
the capability to effectively manage the program. Proceeding without it
introduces unnecessary risks to the program and potentially jeopardizes its
viability for future investment.
United States Government Accountability Office
Contents
Letter
Compliance with Legislative Conditions
Observations on the Expenditure Plan and Management of Atlas
Conclusions
Recommendations for Executive Action
Agency Comments and Our Evaluation
1
2
3
5
6
7
Briefing to the Staffs of the Subcommittees on Homeland
Security, Senate and House Committees on Appropriations
8
Appendixes
Appendix I:
Appendix II:
Appendix III:
Comments from the Department of Homeland Security
64
GAO Contact and Staff Acknowledgments
67
Abbreviations
CCE
CIO
CMMI
COTR
DHS
EA
ICE
IT
IV&V
OMB
SACMM
SEI
Common Computing Environment
chief information officer
Capability Maturity Model Integrated
contracting officer’s technical representative
Department of Homeland Security
enterprise architecture
Immigration and Customs Enforcement
information technology
independent verification and validation
Office of Management and Budget
Software Acquisition Capability Maturity Model
Software Engineering Institute
This is a work of the U.S. government and is not subject to copyright protection in the
United States. It may be reproduced and distributed in its entirety without further
permission from GAO. However, because this work may contain copyrighted images or
other material, permission from the copyright holder may be necessary if you wish to
reproduce this material separately.
Page i
GAO-07-565 Information Technology
A
United States Government Accountability Office
Washington, D.C. 20548
April 27, 2007
Leter
The Honorable Robert C. Byrd
Chairman
The Honorable Thad Cochran
Ranking Member
Subcommittee on Homeland Security
Committee on Appropriations
United States Senate
The Honorable David E. Price
Chairman
The Honorable Harold Rogers
Ranking Member
Subcommittee on Homeland Security
Committee on Appropriations
House of Representatives
The 2006 Department of Homeland Security (DHS) appropriations act1
provided $40.15 million for Immigration and Customs Enforcement’s (ICE)
program to modernize its information technology (IT) infrastructure. The
goals of the program—which consists of seven related IT projects and is
referred to by ICE as the “Atlas” program—include improving information
sharing and collaboration, strengthening information security, and
improving workforce productivity. The act provides that none of the funds
may be obligated until the department develops a plan for how the funds
are to be spent that satisfies certain legislative conditions. The conditions
included, among other things, having GAO review the plan. On November 9,
2006, DHS submitted its fiscal year 2006 expenditure plan to the Senate and
House Appropriations Subcommittees on Homeland Security. Pursuant to
the act, we reviewed the plan. Our objectives were to (1) determine
whether the plan satisfies legislative conditions specified in the act and
(2) provide other observations about the expenditure plan and the
department’s management of the Atlas program.
On February 2, 2007, we provided our review results via a briefing to the
staffs of the Senate and House Appropriations Subcommittees on
Homeland Security. This report summarizes our findings, conclusions, and
1
Pub. L. No. 109-90, Oct. 18, 2005.
Page 1
GAO-07-565 Information Technology
recommendations from that briefing. The full briefing, including our scope
and methodology, is reprinted in appendix I.
Compliance with
Legislative Conditions
DHS’s fiscal year 2006 Atlas expenditure plan, including related program
documentation and program officials’ stated commitments, satisfies or
partially satisfies key aspects of (1) meeting the capital planning and
investment control review requirements of the Office of Management and
Budget (OMB);2 (2) complying with the DHS enterprise architecture;3
(3) complying with acquisition rules, requirements, guidelines, and systems
acquisition management practices of the federal government; (4) including
certification by the department’s chief information officer (CIO) that an
independent verification and validation agent is currently under contract
for the project; and (5) having the plan reviewed and approved by the
department’s Investment Review Board, the Secretary of Homeland
Security, and OMB.
Regarding the fourth legislative condition, the department certified that an
independent verification and validation agent is under contract for Atlas,
and the department is in the process of defining how it plans to use its
agent, but unresolved issues remain. For example, we have previously
reported4 that the independence of a verification and validation agent is
critical to providing management with objective insight into program
processes and associated work products. We also reported5 that to be
effective, the verification and validation function should be performed by
an entity that is independent of the processes and products that are being
reviewed. When the agent engaged to provide the service is a contractor, a
practice to help ensure independence is to include in the agent’s contract a
provision that prohibits the agent from soliciting, proposing, or being
2
OMB Circular A-11, part 7, establishes policy for planning, budgeting, acquiring, and
managing federal capital assets.
3
An enterprise architecture provides a clear and comprehensive picture of an organization’s
operations and its supporting systems and infrastructure. It is an essential tool for
effectively engineering business processes and for implementing and evolving supporting
systems in a manner that maximizes interoperability, minimizes overlap and duplication,
and optimizes performance.
4
GAO, Homeland Security: Recommendations to Improve Management of Key Border
Security Program Need to Be Implemented, GAO-06-296 (Washington, D.C.: Feb. 14, 2006).
5
GAO-06-296.
Page 2
GAO-07-565 Information Technology
awarded program work other than the independent verification and
validation services and products. However, the contract that Atlas awarded
to its agent did not include such a provision.
Observations on the
Expenditure Plan and
Management of Atlas
Our observations address program efforts regarding (1) requirements
development and management, (2) contract management and oversight,
(3) risk management, and (4) performance management. An overview of
these observations follows:
• Rigorous practices are not being fully adhered to in developing and
managing system requirements. On its three key projects,6 the
program has not fully implemented federal IT guidance, relevant best
practices, and agency policies and procedures (i.e., ICE System
Lifecycle Management Handbook) that provide for effective
requirements development and management activities. For example, on
its Common Computing Environment project, while the project
established a policy to guide project efforts in developing and managing
requirements, it did not among other things:
• document stakeholder comments as part of the requirements
elicitation process and review and obtain stakeholder approval on
requirements until approximately 1 year after deploying system
capabilities;
• follow a disciplined change management process for several years
(between 2003 and 2006) after the project was initiated in March 2003
(during this time, requirements were introduced and implemented in
an ad hoc fashion); and
• develop supporting analysis showing traceability among
requirements, systems design, and the contract.
• Key contract management and oversight practices are not fully
implemented. The program has not fully implemented practices that
(1) the Software Engineering Institute’s CMMI®7 has identified as
6
These three key projects are Common Computing Environment, Integration, and ICE
Mission Information.
7
CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
Page 3
GAO-07-565 Information Technology
essential to effectively managing and overseeing contracts that support
IT projects and (2) the program has largely established in its existing
policies and procedures. Specifically, the program has implemented two
key practices, but has not fully implemented four others. For example,
the program has assigned responsibility and authority for performing
contract management and oversight, and has trained staff performing
contract management and oversight. However, it has not, among other
things, fully documented each contract—including clearly identifying
the work to be performed and acceptance criteria—and verified and
accepted deliverables.
• Key risk management practices have yet to be implemented. We
previously reported8 that the program had developed a risk management
plan and process to guide program office staff in managing risks
throughout each project’s life cycle and had recently begun to
implement the risk management process. Since then, the program has
taken additional steps to implement risk management, such as hiring a
risk management coordinator and conducting quarterly program
management meetings to review the program’s progress in managing
risks. However, the program has not fully implemented key practices
essential to effectively managing project risks. For example, it does not
have a transparent, documented, and traceable process for inventorying
and resolving risks. Further, while the risk management plan defines a
process and associated practices for developing risk mitigation
strategies, the program has not fully implemented them. In addition,
although the risk management plan calls for developing a process for
elevating risks to management’s attention, the program has not
developed and implemented such a process.
• Performance management practices are still being implemented and
available data show mixed results and may not be reliable.
We reported9 in July 2006 that Atlas had begun taking steps to
implement performance goals and measures. Since then, the program
has taken additional steps to define and implement them. For example,
the program identified 13 goals—for 5 of its 7 projects—that it has been
using to measure progress during fiscal year 2006. However, it has yet to
8
GAO, Information Technology: Immigration and Customs Enforcement Is Beginning to
Address Infrastructure Modernization Program Weaknesses but Key Improvements Still
Needed, GAO-06-823 (Washington, D.C.: July 27, 2006).
9
GAO-06-823.
Page 4
GAO-07-565 Information Technology
fully implement and institutionalize its performance goals and
measures. For example, the program has not yet developed
performance goals for 2 projects.
In addition, progress against performance goals, as reported by the
program, show mixed results. For example, of the 12 goals, the program
has met 3 of them. Further, underlying data used by the program to
measure performance may not be reliable in all cases. For example, on 1
goal, the program uses project cost and schedule data from the 2005 Atlas
business case in measuring progress against the goal, even though these
data are out of date (e.g., they do not reflect project cost and schedule
slippages that have occurred since 2005).
Conclusions
The fiscal year 2006 Atlas expenditure plan, in combination with related
program documentation and program officials’ statements, satisfies or
partially satisfies the legislative conditions set forth by Congress. However,
this satisfaction is based on plans and commitments that provide for
meeting these conditions, rather than on completed actions to satisfy the
conditions. In addition, while steps are being initiated that are intended to
address significant program management weaknesses, a number of
improvements, including those recommended in our past reports, have yet
to be implemented.
Further, the program has not fully achieved many performance goals that it
set out to accomplish over the past year. The above factors continue to put
the program at risk and call for heightened and sustained management
attention to expeditiously address and resolve the issues. Thus, there is
much that still needs to be accomplished to minimize the risks associated
with the program’s capacity to deliver promised IT infrastructure
capabilities and benefits on time and within budget. This includes
demonstrating better progress against established performance goals in the
coming year.
Given that hundreds of millions of dollars are to be invested and the
program is critical to supporting the ICE mission, it is essential that DHS
follow through on its commitments to build the capacity to effectively
manage the program. Proceeding without this capacity introduces
unnecessary risks to the program and potentially jeopardizes its viability
for future investment.
Page 5
GAO-07-565 Information Technology
Recommendations for
Executive Action
To minimize risks to the Atlas program, we recommend that the Secretary
of Homeland Security direct the Assistant Secretary for Immigration and
Customs Enforcement to ensure that ICE follows through on its
commitments to implement effective management controls and capabilities
by implementing the following five recommendations:
• Employ practices essential to ensuring that the Atlas program’s
independent verification and validation agent is and remains
independent, including incorporating requirements in future contracting
actions, such as the renegotiation or recompetition of the current
independent verification and validation contract, which will prohibit the
agent from soliciting, proposing, or being awarded program work other
than providing independent verification and validation services and
products.
• Fully adhere to requirements development and management practices,
including those specified in ICE’s policies and procedures. This should
also include having the Atlas program manager develop a process
improvement plan for all of the projects that is consistent with the ICE
System Lifecycle Management Handbook and provide for making the
program manager and project managers responsible and accountable
for rigorously adhering to the requirements in the handbook.
• Fully implement key contract management and oversight practices,
including those specified in ICE’s policies and procedures. This should
also include ensuring that the Atlas program manager, working with the
program’s contracting officer’s technical representative, follow through
on planned efforts to strengthen the program’s compliance with these
practices by (1) revising the acquisition plan by May 2007; (2) revising
the ICE System Lifecycle Management Handbook by June 2007 to
incorporate key contract management activities such as contract
tracking and oversight; and (3) preparing statements of work (beginning
in February 2007) for future contracts to clearly define Atlas work
statements and acceptance criteria.
• Complete implementation of planned risk management activities. This
should include (1) updating the risk inventory to include risks for all
projects and risk areas and (2) fully implementing and institutionalizing
procedures for reporting to management on the existence and status of
risks and progress in implementing mitigation strategies.
Page 6
GAO-07-565 Information Technology
• Improve program performance management, including developing
performance goals for projects that do not have goals and reporting on
their progress in the fiscal year 2007 expenditure plan. Further, the
program should assess that the data being used to measure progress are
reliable, complete, and accurate.
Agency Comments and
Our Evaluation
In DHS’s written comments on a draft of this report, signed by the Director,
Departmental GAO/Office of Inspector General Liaison Office, the
department concurred with all of our recommendations and described
actions planned and under way to implement them. The department also
orally provided technical comments updating Atlas obligation amounts and
the milestone for revising the acquisition plan. We incorporated these
comments in the report and briefing as appropriate. DHS’s written
comments are reprinted in appendix II.
We are sending copies of this report to the Chairman and Ranking Members
of other Senate and House committees and subcommittees that have
authorization and oversight responsibilities for homeland security. We are
also sending copies to the Secretary of Homeland Security, the Assistant
Secretary for Immigration and Customs Enforcement, and the Director of
the Office of Management and Budget. Copies of this report will also be
available at no charge on our Web site at www.gao.gov.
Should you or your offices have any questions on matters discussed in this
report, please contact me at (202) 512-3439 or at hiter@gao.gov. Contact
points for our Offices of Congressional Relations and Public Affairs may be
found on the last page of this report. Key contributors to this report are
listed in appendix III.
Randolph C. Hite
Director, Information Technology Architecture
and Systems Issues
Page 7
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees on
Homeland Security, Senate and House
Committees on Appropriations
Appendx
ies
Append
x
Ii
Information Technology: Immigration and Customs
Enforcement Needs to Fully Address Significant Infrastructure
Modernization Program Management Weaknesses
Briefing to the staffs of the
Subcommittees on Homeland Security
Senate and House Committees on Appropriations
February 2, 2007
1
Page 8
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Outline of Briefing
Introduction
Objectives, Scope, and Methodology
Results in Brief
Background
Results
ƒ Legislative Conditions
ƒ Observations
Conclusions
Recommendations for Executive Action
Agency Comments
Attachment I: Detailed Scope and Methodology
Attachment II: GAO Analysis of Requirements Management
2
Page 9
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Introduction
The Department of Homeland Security’s (DHS) Bureau of Immigration and Customs Enforcement
(ICE)1 is responsible for enforcing immigration, border security, trade, and other laws by, among other
means, investigating and collecting intelligence on individuals and groups that act to violate these
laws. ICE is also responsible for protecting federal facilities.
Atlas is the ICE program developed to modernize the bureau’s information technology (IT)
infrastructure, which includes the hardware (e.g., servers, routers, storage devices, communication
lines) and system software (e.g., database management and operating systems and network
management) that provide an environment for operating and maintaining software applications.
According to ICE, the goals of the Atlas program include improving information sharing, strengthening
information security, and improving workforce productivity.
The fiscal year 2006 Department of Homeland Security Appropriations Act2 appropriated $40.15
million for Atlas3 and stated that none of the funds may be obligated until the department submits for
approval to the Senate and House Committees on Appropriations a plan for how the funds are to be
spent that satisfies the following legislative conditions:
1
ICE was formed from the former Immigration and Naturalization Service, U.S. Customs Service, and other entities. Atlas began in 2002
under the former Immigration and Naturalization Service.
2
Pub. L. No. 109-90, Oct. 18, 2005.
3
In the act, appropriations for Atlas are under the heading Automation Modernization for Immigration and Customs Enforcement.
3
Page 10
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Introduction
ƒ meets the capital planning and investment control review requirements established by the Office of
Management and Budget (OMB), including OMB Circular A-11, part 7;4
ƒ complies with DHS’s information systems enterprise architecture;
ƒ complies with the acquisition rules, requirements, guidelines, and systems acquisition management
practices of the federal government;
ƒ includes certification by the DHS chief information officer (CIO) that an independent verification and
validation agent is currently under contract for the project;
ƒ is reviewed and approved by DHS’s Investment Review Board,5 the Secretary of Homeland
Security, and OMB; and
ƒ is reviewed by GAO.
DHS submitted its fiscal year 2006 expenditure plan for $39.756 million on November 9, 2006, to the
Senate and House Appropriations Subcommittees on Homeland Security.
4
OMB Circular A-11 establishes policy for planning, budgeting, acquisition, and management of federal capital assets.
The purpose of the Investment Review Board is to integrate capital planning and investment control, budgeting, and acquisition. It is
also to ensure that spending on investments directly supports and furthers the mission and that this spending provides optimal benefits
and capabilities to stakeholders and customers.
6
DHS’s fiscal year 2006 appropriations (Pub. L. No. 109-90, Oct. 18, 2005) provided $40.15 million for Atlas. In December 2005, this
amount was reduced to $39.75 million via a 1 percent governmentwide rescission in Pub. L. No. 109-148, Division B, Dec. 30, 2005.
5
4
Page 11
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objectives, Scope, and Methodology
As agreed, our objectives were to
ƒ determine whether the Atlas fiscal year 2006 expenditure plan satisfies the legislative conditions
and
ƒ provide other observations about the expenditure plan and DHS’s management of the Atlas
program.
To address our first objective, we analyzed the fiscal year 2006 Atlas expenditure plan and supporting
documents against legislative conditions and relevant federal requirements and related best practices
to determine whether the conditions were being met.
To address our second objective, we evaluated supporting documentation and interviewed relevant
officials to determine progress in establishing program management capabilities in areas such as
requirements development and management, contract management and oversight, risk management,
and performance management. In assessing requirements management and contract management
and oversight, we focused on three projects—Common Computing Environment, Integration, and ICE
Mission Information—that collectively account for approximately 75 percent of the funds provided to
the program and that the Atlas program manager identified as being the most critical to the program’s
success and ICE’s mission. We conducted our work at DHS and ICE headquarters in Washington,
D.C., from July 2006 through January 2007 in accordance with generally accepted government
auditing standards. Details of our scope and methodology are provided in attachment 1.
5
Page 12
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Results in Brief
Objective 1: Satisfaction of Legislative Conditions
Legislative conditions
Status
1. Meets the capital planning and investment control review requirements established by OMB,
including OMB Circular A-11, part 7.
partially satisfies7
2. Complies with the DHS information systems enterprise architecture.
partially satisfies
3. Complies with the acquisition rules, requirements, guidelines, and systems acquisition
management practices of the federal government.
partially satisfies
4. Includes a certification by the chief information officer of the Department of Homeland Security
that an independent verification and validation agent is currently under contract for the project.
partially satisfies
5. Is reviewed and approved by the DHS Investment Review Board, Secretary of Homeland
Security, and OMB.
satisfies8
6. Is reviewed by GAO.
satisfies
Source: GAO.
7
Partially satisfies=the plan, in combination with supporting documentation and stated commitments by program officials, either satisfies
or provides for satisfying many, but not all, key aspects of the condition that we reviewed.
8
Satisfies=the plan, in combination with supporting documentation and stated commitments by program officials, either satisfies or
provides for satisfying every aspect of the condition that we reviewed.
6
Page 13
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Results in Brief
Objective 2: Other Observations
DHS has not implemented key system management practices essential to establishing an effective program
management control environment. Specifically, in managing the Atlas program, the department has not
ƒ
ƒ
ƒ
ƒ
fully adhered to rigorous practices called for in developing and managing system requirements,
implemented key contract management and oversight controls,
completed implementation of planned risk management practices, or
implemented performance management practices critical to measuring progress against program
goals.
These shortfalls are attributable in part to a number of factors, including the Atlas program’s expedited
schedule to deploy modernized IT infrastructure capabilities to users in the wake of the events of September
11, 2001, and in the haste of doing so, not following these practices or related ICE policies and procedures.
Until these practices are fully implemented, the program is at an increased risk of not being able to deliver
modern IT infrastructure capabilities on time and within budget. Program officials acknowledge the shortfalls
and have efforts planned or under way to address some, but not all, of the missing practices.
Accordingly, we are making recommendations to the Secretary of Homeland Security to strengthen the Atlas
program by implementing and institutionalizing these key system management practices, thereby helping to
establish an effective program management control environment. In commenting on a draft of this briefing,
ICE officials, including the deputy chief information officer who is also the Atlas program manager, agreed
with our results.
7
Page 14
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Background
ICE
ICE was formed as a component agency of DHS in 2003 when the law enforcement functions of the
Department of Justice’s former Immigration and Naturalization Service and Department of the
Treasury’s former Customs Service and other agencies were merged into DHS.
The ICE mission is to ensure the security of the American people and homeland by, among other
means, investigating violators of and enforcing the nation's immigration and customs laws; policing
and securing federal buildings and other facilities; and collecting, analyzing, and disseminating
intelligence to assist in these endeavors. Headed by the Assistant Secretary of Immigration and
Customs Enforcement, ICE has approximately 15,000 employees in more than 400 offices in the
United States and in other countries.
The Atlas program was started by the Immigration and Naturalization Service in 2002. Responsibility
for the program was transferred to ICE in 2003. The figure on the following slide is a simplified version
of the current ICE and Atlas organizational placements within DHS.
8
Page 15
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Background
ICE and Atlas Organizational Placement
Department
Department of
of
Homeland
Homeland Security
Security
Secretary
Secretary
Deputy Secretary
Under
Under Secretary
Secretary for
for
Management
Management
Chief
Chief Information
Information Officer
Officer
Transportation
Transportation
Security
Security
Administration
Administration
Under
Under Secretary
Secretary for
for
Preparedness
Preparedness
Chief
Chief Financial
Financial Officer
Officer
Customs
Customs and
and
Border
Border
Protection
Protection
Office
Office of
of
Federal
Federal
Protective
Protective
Service
Service
Under
Under Secretary
Secretary for
for
Science
Science and
and Technology
Technology
Secret
Secret
Service
Service
Office
Office of
of
Detention
Detention and
and
Removal
Removal
Operations
Operations
Immigration
Immigration
and
and Customs
Customs
Enforcement
Enforcement
Chief
Chief
Information
Information
Officer
Officer
Citizenship
Citizenship
and
and
Immigration
Immigration
Services
Services
Office
Office of
of
Investigations
Investigations
Federal
Federal
Emergency
Emergency
Management
Management
Agency
Agency
Coast
Coast Guard
Guard
US-VISIT
US-VISIT
Office
Office of
of
Intelligence
Intelligence
Atlas Program
Source: GAO analysis of DHS data.
9
Page 16
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Background
Program Impetus and Goals
Impetus for Program
According to ICE officials, Atlas was initiated to address information sharing and security limitations
within the former Immigration and Naturalization Service caused by, for example,
ƒ obsolete hardware/software;
ƒ incompatible, noninteroperable information systems; and
ƒ system security weaknesses.
These officials stated that these challenges were exacerbated by the formation of ICE because the
organizations merged into ICE had different hardware/software environments (e.g., multiple e-mail
systems) and different missions and customer needs.
Program Goals
The stated goals of Atlas are, among other things, to
ƒ promote information sharing and collaboration,
ƒ strengthen information security, and
ƒ enhance workforce productivity.
10
Page 17
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Background
Atlas Projects
Atlas Consists of Seven Projects
Project
Description
Common Computing
Environment
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
ƒ
Integration
ƒ
ICE Mission Information
ƒ
ƒ
ƒ
Deploy compatible desktops, servers, and standard office automation applications.
Deploy a common e-mail platform.
Integrate ICE with a shared departmental active directory structure.
Migrate from the ICE network to the DHS OneNetwork infrastructure.
Provide streaming video.
Refresh local area network cabling and electronics and replace legacy firewalls.
Refresh the wide area network and local area network communications hardware at the overseas ICE
attaché offices.
Provide network support for all applications and provide converged communications among all system
users.
Provide enterprise data warehousing capabilities and integrate related ICE business processes.
Organize information and support user access so ICE users can find relevant, timely, and reliable
information.
Improve information searching and indexing capabilities and integrate legacy applications with serviceoriented techniques.
Institute strong information, system/application platform, network, and computer security measures.
Information Assurance
ƒ
Architecture Engineering
ƒ Provide state-of-the-art engineering facilities and tools to ensure new technologies align to the DHS
enterprise architecture (EA).
ƒ Migrate ICE hardware and applications from the Department of Justice data centers to the common
DHS data center.
ƒ Ensure compliance with the DHS EA.
ƒ Oversee and coordinate the various Atlas projects.
ƒ Ensure compliance with the DHS EA.
ƒ Ensure adherence to project cost, schedule, and performance goals.
Data Center Migration
Transformation Planning
Source: GAO analysis of DHS data.
11
Page 18
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Background
Atlas Appropriations
Appropriated Funds
a
Counterterrorism funding came from Pub. L. No. 107-117, Jan. 10, 2002; H.R. Rep. 107-350, Dec. 19, 2001.
Pub. L. No. 108-90, Oct. 1, 2003.
c
Pub. L. No. 108-334, Oct. 18, 2004.
d
DHS’s fiscal year 2006 appropriations (Pub. L. No. 109-90, Oct. 18, 2005) provided $40.15 million for Atlas. In December 2005, this
amount was reduced to $39.7 million via a 1 percent governmentwide rescission in Pub. L. No. 109-148, Division B, Dec. 30, 2005.
e
Pub. L. No. 109-295, Oct. 4, 2006. The law mandates that DHS also submit an expenditure plan prior to obligation of these funds.
b
12
Page 19
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Background
Atlas Obligations
ICE reports that it has obligated $84.7 million of the $113.0 million available for Atlas in fiscal years
2002, 2003, 2004, and 2005.
Atlas funds obligated (by project) for fiscal years 2002-2005
(as of December 31, 2006)
Project
(in order presented in the plan)
Obligations
(in millions)
Total
FY 2002 FY 2003 FY 2004 FY 2005
Common Computing Environment
Integration
ICE Mission Information
Information Assurance
Architecture Engineering
Data Center Migration
Transformation Planning
TOTAL
$0
$10.5
$3.2
$4.3
$18.0
6.1
20.4
2.5
0.5
29.5
1.3
4.6
0.8
4.0
10.7
10.2
9.6
0
0.6
20.4
0
0
0
0.7
0.7
0
0
0
0
0
0
0.9
3.3
1.2
5.4
$17.6
$46.0
$9.8
$11.3
$84.7
Source: GAO analysis of DHS data.
13
Page 20
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Background
Planned Use of Fiscal Year 2006 Funding
According to the fiscal year 2006 expenditure plan, the $39.75 million is to be spent on the projects as
follows.
Distribution of funding for Atlas projects (in order presented in the plan)
Project
Funding
(in millions)
Common Computing
Environment
Integration
ICE Mission Information
Information Assurance
Architecture
Engineering
Data Center Migration
Transformation
Planning
TOTAL
Planned use
$23.88 ƒ Complete acquisition of hardware refresh
ƒ Deploy active directory and e-mail exchange system
2.30 ƒ Replace outdated communications electronics at foreign attachés to ensure
compatibility with current ICE Net WAN and LAN infrastructure standards
ƒ Complete acquisition of streaming video for field sites
5.84 ƒ Complete acquisition of technology to efficiently integrate all related ICE business
processes
ƒ Complete 60 percent of acquisition of technology to support data warehousing
capabilities
0.92 ƒ Acquire IT security specialists to design and develop single sign-on and audit log
capabilities
0 ƒ No use of fiscal year 2006 funds identified
0.89 ƒ Plan for the migration of ICE hardware and applications from two Department of
Justice data centers to a common DHS data center
5.92 ƒ Provide $2.3 million to continue performing and planning program management
activities, including maintaining the Atlas business case
ƒ Provide $3.6 million in management reserve for unanticipated program costs or to
fund priority emerging IT modernization requirements
$39.75
Source: GAO analysis of DHS data.
14
Page 21
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
The Atlas expenditure plan satisfies or partially satisfies each of the legislative conditions.
Condition 1 partially satisfied. The expenditure plan, including related program documentation and
statements from the program manager, partially satisfies the capital planning and investment control
review requirements established by OMB, including Circular A-11, part 7, which establishes policy for
planning, budgeting, acquisition, and management of federal capital assets.
Examples of our analysis are included in the following table. As the table shows, not all OMB
requirements have been satisfied, but oral commitments have been made for doing so. Given that
ICE has reportedly already invested $75.4 million on Atlas projects and plans to invest another $39.7
million this year, it is important for ICE to follow through on these commitments.
15
Page 22
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
GAO Analysis of Atlas Compliance with Legislative Conditions
Examples of OMB Circular
A-11 requirements
Results of our analysis
Indicate whether the investment
was approved by an investment
review committee
The plan was approved on May 11, 2006, by DHS’s Deputy Secretary, who chairs the DHS
Investment Review Board.
Provide justification and describe
acquisition strategy
A business case, including a cost-benefit analysis, was issued in December 2005 to provide
economic justification for the program. It estimated the program’s life cycle costs to be
approximately $1 billion through the year 2024. The program plans to issue an updated costbenefit analysis in September 2007 to reflect emerging requirements and other program
changes (e.g., cost and schedule slippages). In addition, the expenditure plan and supporting
documents (e.g., fiscal year 2006 Atlas budget submission to OMB known as an exhibit 300)
provide aspects of a high-level acquisition strategy, such as identifying the ICE contracts that
are being used and are to be used to acquire hardware and software products and program
support services. An acquisition plan was approved in February 2006 that includes a statement
of need and the capabilities to be delivered through these contracts. To help ensure compliance
with contract criteria and Atlas program objectives, a contracting officer’s technical
representative (COTR) was hired in November 2006. The program’s efforts to ensure
compliance with contract management and oversight practices are more fully discussed in the
Observations section of this briefing.
The December 2005 cost-benefit analysis provides costs and benefits for the life cycle of Atlas,
which is through the year 2024. The analysis also included an estimated return on investment
9
for three alternative approaches. As we previously reported, the analysis was in large part
consistent with OMB and best practices guidance, but it did omit key practices such as key
mission requirements. According to the program manager, the analysis is considered to be a
“living document” and the program plans to issue an update in September 2007 that, among
other things, addresses these requirements.
Summarize life cycle costs and
cost-benefit analysis, including the
return on investment
9
GAO, Information Technology: Immigration and Customs Enforcement Is Beginning to Address Infrastructure Modernization Program
Weaknesses but Key Improvements Still Needed, GAO-06-823 (Washington, D.C.: July 27, 2006).
16
Page 23
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
GAO Analysis of Atlas Compliance with Legislative Conditions (cont.)
Examples of OMB Circular
A-11 requirements
Results of our analysis
Provide performance goals and
measures
The expenditure plan and supporting documentation (e.g., prior expenditure plans) identify 13
proposed goals and measures, but two projects do not have measures. Proposed measures have
been drafted for these projects, but target dates for their completion have not been identified by
the program. In addition, as part of the December 2005 Atlas business case, the program
mapped Atlas’s goals to ICE’s mission and goals. Our analysis of Atlas’s performance
management is discussed in more detail in the Observations section of this briefing.
The plan and supporting documentation state the importance of security and privacy and provide
high-level information on intended security measures, including one proposed project—
information assurance—that is intended to implement an ICE security program. The plan
allocates $0.9 million to this project in addition to the $1.9 million provided in the fiscal year 2005
plan. In addition, the program issued an Atlas system security plan, as well as a security test and
evaluation plan, in April 2006. The program also recently issued an updated system security plan
and security test and evaluation plan in December 2006. ICE also developed a draft Atlas privacy
impact assessment in August 2005, submitted a revised assessment in August 2006 to the DHS
privacy office, and plans to complete and submit the final Atlas privacy impact assessment by
April 2007.
The program issued a risk management plan in January 2006. This plan provides guidance for
identifying, analyzing, and resolving program risks before they occur. In addition, a risk
management coordinator was hired in August 2006 to help identify and monitor risks. According
to the program manager, this official recently vacated the position, and the program is currently in
the process of hiring a new coordinator. (At our Feb. 1, 2007, exit meeting with DHS and ICE
officials, the program manager told us a coordinator had been hired on Jan. 22, 2007.) Further,
while the program procured an automated tool in May 2006 to help complete the inventory and
manage risks, it does not have, among other things, a complete inventory of all program risks.
Our complete findings related to risk management are addressed in the Observations section of
this briefing.
Address security and privacy
Provide for managing risk
Source: GAO analysis of DHS data.
17
Page 24
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
Condition 2 partially satisfied. The plan, including related program documentation and DHS
officials’ statements, partially satisfies the condition that the department ensures Atlas is compliant
with DHS’s enterprise architecture (EA).
An EA provides a clear and comprehensive picture of an organization’s operations and its supporting
systems and infrastructure. It is an essential tool for effectively and efficiently engineering business
processes and for implementing and evolving supporting systems in a manner that maximizes
interoperability, minimizes overlap and duplication, and optimizes performance. We have worked with
Congress, OMB, and the federal Chief Information Officers Council to highlight the importance of
architectures for both organizational transformation and IT management. An important element of EA
management is ensuring that IT investments are compliant with the EA, including basing such
assessments on documented analysis.
The DHS Enterprise Architecture Board, supported by the Enterprise Architecture Center of
Excellence,10 is responsible for ensuring that projects demonstrate adequate technical and strategic
compliance with the department’s EA. To this end, the board conducts compliance reviews at key
decision points in an investment’s life cycle. Specifically, DHS guidance11 directs the board prior to an
investment’s acquisition milestone (referred to by DHS as key decision point 2) to assess the
investment against the transition strategy, data architecture, application component, and technology
architecture.
10
A review group made up of subject matter experts that recommends EA compliance to the DHS Enterprise Architecture Board and
ultimately to the DHS Investment Review Board.
Department of Homeland Security Enterprise Architecture Board: EAB Governance Process Guide, August 9, 2004, draft version 2.0.
11
18
Page 25
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
We previously reported12 in May 2005 that the Atlas program manager requested that the center
assess the program’s compliance with the EA. The center staff compared Atlas to version 2.0 of the
DHS EA and reported the results of its assessment to the board, stating that Atlas was in compliance
with the DHS EA. In July 2005, the board approved this compliance determination.
Subsequently, we reported13 that ICE’s compliance determination was not based on a documented
analysis mapping Atlas’s infrastructure to the DHS EA. Specifically, the department did not have a
documented methodology for evaluating programs for compliance with the DHS EA, other than
relying on the expertise of the Center of Excellence members.
Since the July 2005 compliance determination, DHS issued version 2006 of its EA.14 In addition, Atlas
has undertaken an effort to define an architecture framework and technical environment for a data
warehouse aimed at providing reporting and analytical capabilities to investigative and enforcement
personnel. Despite these changes, no new compliance reviews have been conducted. According to
the Atlas program manager, a compliance review was not required during this time as the program
had not yet reached a phase requiring it. The program manager stated the department has scheduled
an EA compliance review in March 2007.
12
GAO-06-823.
GAO-06-823.
On August 14, 2006, we reported that version 2006 of DHS’s EA was at stage 2 of the Enterprise Architecture Management Maturity
Framework, demonstrating that DHS has established the foundational commitments and capabilities needed to manage the
development of the architecture. See GAO, Enterprise Architecture: Leadership Remains Key to Establishing and Leveraging
Architectures for Organizational Transformation, GAO-06-831 (Washington, D.C.: Aug. 14, 2006).
13
14
19
Page 26
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
In addition, program officials told us that they are taking other steps aimed at ensuring compliance.
These steps include
ƒ participating in the DHS Enterprise Architecture Center of Excellence and the Data Management
Working Group; these groups are currently tasked with developing the DHS EA and the DHS Data
Reference Model,
ƒ participating in DHS EA initiatives, such as serving on the Service Oriented Architecture Tactical
Working Group,15 which is to provide the framework for departmentwide service alignment and
interoperability efforts, and
ƒ establishing an Atlas Interoperability Lab—under the Architecture Engineering project—to among
other things, help ensure the program’s projects are compliant with the EA.
These steps notwithstanding, until DHS demonstrates, through verifiable documentation and a
methodologically based analysis, that ICE is aligned with the DHS enterprise architecture, the
program will remain at risk of being defined and implemented in a way that does not support optimal
departmentwide operations, performance, and achievement of strategic goals and outcomes.
15
The Tactical Working Group is responsible for developing strategies to address service oriented architecture design and operational
issues.
20
Page 27
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
Condition 3 partially satisfied. The plan, including related program documentation and statements
from the Atlas program manager, partially satisfies the condition to comply with the acquisition rules,
requirements, guidelines, and systems acquisition management practices of the federal government.
These practices provide a management framework based on the use of rigorous and disciplined
processes for planning, managing, and controlling the acquisition of IT resources, including:
ƒ acquisition planning, which ensures, among other things, that reasonable plans, milestones, and
schedules are developed and that all aspects of the acquisition effort are included in these plans;
ƒ procurement, which involves making sure that (1) a request for proposals delineating a project’s
requirements is prepared and (2) consistent with relevant acquisition laws and regulations, a
contractor is selected that can cost-effectively satisfy these requirements;
ƒ requirements development and management, which includes establishing and maintaining a
common and unambiguous definition of requirements among the acquisition team, the system
users, and the development contractor;
ƒ project management, which provides for management of the activities within the project office and
supporting contractors to ensure a timely, efficient, and cost-effective acquisition;
ƒ contract tracking and oversight, which ensures that the development contractor performs according
to the terms of the contract; needed contract changes are identified, negotiated, and incorporated
into the contract; and contractor performance issues are identified early, when they are easier to
address; and
21
Page 28
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
ƒ evaluation, which determines whether the acquired products and services satisfy contract
requirements before acceptance.
These acquisition management processes are also embodied in published best practices models,
such as the Software Engineering Institute’s Software Acquisition Capability Maturity Model®.16
Examples of our analysis of ICE performance of these processes and practices are shown in the
following table. They show that not all aspects of the processes and practices have been
implemented, but that oral commitments have been made for doing so.
Developed by Carnegie Mellon Software Engineering Institute (SEI), Software Acquisition Capability Maturity Model (SA-CMM®)
version 1.03 (March 2002).
16
22
Page 29
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
GAO Analysis of Atlas Compliance with Legislative Conditions
Example of practices
Acquisition Planning
Ensures that reasonable plans,
milestones, and schedules are
developed and that all aspects of
the acquisition effort are included
in these plans.
Results of our analysis
The expenditure plan and supporting documents (e.g., fiscal year 2006 Atlas budget submission
to OMB known as an exhibit 300) provide aspects of a high-level acquisition strategy, such as
identifying the ICE contracts that are being used and are to be used to acquire products and
program support services. An acquisition plan was issued in February 2006, and it includes a
statement of the capabilities to be delivered through these contracts.
To manage and oversee the contracts, including helping ensure compliance with contract
performance criteria and Atlas program objectives, the program hired a COTR in November
2006. In addition, the business case issued in December 2005 provides details on alternatives,
cost, and schedule. The program plans to issue an updated business case and cost-benefit
analysis by September 2007 to reflect emerging requirements and other program changes (e.g.,
project cost and schedule slippages). In addition, Atlas is also updating and revising its January
2006 acquisition program baseline, which is to include cost and schedule baselines for its
projects, with the goal of finalizing it by March 2007.
According to the program manager, Atlas is currently following the February 2005 ICE System
Lifecycle Management Handbook.17 The handbook addresses a number of key process areas
such as project management and requirements development and management; however, it does
not address certain key acquisition management activities such as solicitation and contract
tracking and oversight. According to the Atlas program manager, he acknowledges this problem
and plans to update the handbook by June 2007 to include the missing acquisition management
activities. This area is more fully discussed in the Observations section of this briefing.
17
U.S. Immigration and Customs Enforcement: System Lifecycle Management Handbook, The Systems Development Lifecycle of
Immigration and Customs Enforcement, February 2005, version 1.0.
23
Page 30
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
GAO Analysis of Atlas Compliance with Legislative Conditions (cont.)
Example of practices
Project Management
Provides for the management of
activities within the project office
and supporting contractors to
ensure a timely, efficient, and costeffective acquisition.
Results of our analysis
ICE has established a program office with responsibility for managing the acquisition,
deployment, operation, and sustainment of Atlas. The Atlas program management office is to be
allocated funding of $2.3 million via the Transformation Planning project in the expenditure plan.
The current staffing of the program office consists of a program manager, who is also the deputy
chief information officer; a deputy program manager; a manager for each of the seven Atlas
projects; and other personnel (e.g., a business analyst or a communications specialist).
According to the program manager, the program office is now fully operational. In addition, a
project plan for each of the projects was approved by the program manager in the latter part of
2006.
Source: GAO analysis of DHS data.
24
Page 31
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
Condition 4 partially satisfied. DHS partially satisfies the legislative condition requiring that the plan
include a certification by the department’s CIO that an IV&V agent is currently under contract for the
project.
In an October 25, 2006, letter to ICE’s CIO, DHS’s Deputy CIO certified that an IV&V agent is under
contract for Atlas. However, there are unresolved issues with the Atlas program’s efforts in this area.
First, in the Deputy CIO’s letter, he requested additional information and made recommendations
regarding the IV&V agent, asking for a response by January 31, 2007. Specifically, this official
ƒ requested a copy of the IV&V contract and the agent’s work plan in order to validate, among other
things, whether planned IV&V activities are in accordance with industry quality standards and
whether the agent has sufficient staff and
ƒ recommended that the Atlas program (1) review whether the information assurance and security
experience of key IV&V personnel was sufficient and (2) appoint a full-time COTR before the end of
the first quarter of 2007.
The program manager stated that he is currently drafting the response on behalf of ICE’s CIO with the
goal of transmitting it to DHS by the first or second week of February 2007.
25
Page 32
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
Second, as we have previously reported,18 the independence of an IV&V agent is critical to providing
management with objective insight into program processes and associated work products. We also
reported that to be effective, the verification and validation function is to be performed by an entity
that is independent of the processes and products that are being reviewed. When the agent engaged
to provide the service is a contractor, a practice to help ensure independence is to include in the
agent’s contract a provision that prohibits the agent from soliciting, proposing, or being awarded
program work other than the IV&V services and products.
The contract that Atlas awarded to its IV&V agent (dated Sept. 13, 2006) does not include such a
provision. Instead, the contract states that the contractor will notify the agency of any potential
conflicts of interest in performing work under the contract. The Atlas program manager told us he did
not believe incorporating the contractual provision called for by the key practice was necessary
because the agent’s only line of work is performing IV&V activities, and he agreed to provide
documentation to support this. He also said that in the event the agent solicited, proposed, or is
awarded other work on the Atlas program, he would replace the agent with one that was independent.
However, the program manager did not provide documentation to IV&V being the agent’s sole line of
business. In addition, having to replace an agent could delay program activities and is not consistent
with the key practice to avoid such problems. Consequently, until the practice to ensure
independence is included in the agent’s contract, there is an increased risk that the agent’s
independence could become impaired, resulting in the program being unable to rely on the agent’s
work.
18
GAO, Homeland Security: Recommendations to Improve Management of Key Border Security Program Need to Be Implemented,
GAO-06-296 (Washington, D.C.: Feb. 14, 2006).
26
Page 33
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 1 Results
Legislative Conditions
Condition 5 satisfied. DHS and OMB satisfied the legislative condition requiring that the plan be
reviewed and approved by the DHS Investment Review Board, Secretary of Homeland Security, and
OMB.
x The DHS Deputy Secretary, who chairs the DHS Investment Review Board, approved the plan
on May 11, 2006.
x OMB approved the plan on September 14, 2006.
Condition 6 satisfied. GAO satisfied the condition that it review the plan.
Our review was completed in January 2007.
27
Page 34
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 1: Requirements Management
Observation 1: Atlas did not fully adhere to rigorous requirements development and
management practices
Federal IT management guidance and relevant best practices19 recognize the importance of
effectively developing and managing system requirements. According to this criteria, well-defined
requirements are important because they establish agreement among the various stakeholders on
what the system is to do, how well it is to do it, and how it is to interact with other systems. Effective
requirements development and management involves, among other things:
ƒ establishing a requirements management policy;
ƒ eliciting desired or required system capabilities from users and translating them into system
requirements and obtaining approval from all stakeholders before deploying system capabilities;
ƒ establishing a system concept of operations, which describes the business process to be supported
and how users will interact with the proposed system, and defines system requirements;
ƒ ensuring that changes to requirements are controlled and approved via a disciplined change control
process as the system is developed and implemented; and
ƒ maintaining bidirectional traceability, meaning that a given requirement can be traced backward to
systems documentation and forward to the appropriate contract vehicle and system components
that will satisfy the requirement and the test that will verify that it is satisfied.
19
See, for example, Software Engineering Institute, Capability Maturity Model Integrated (CMMI), version 1.1 (March 2002).
28
Page 35
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 1: Requirements Management
As we reported in July 2006,20 ICE incorporated these practices in its ICE System Lifecycle
Management Handbook,21 which was issued in February 2005. However, the program did not fully
follow these practices on its three key projects: Common Computing Environment (CCE), Integration,
and ICE Mission Information. For example, on CCE, while the project established the ICE System
Lifecycle Management Handbook as the policy to guide project efforts in developing and managing
requirements, it did not
ƒ document stakeholder comments as part of the requirements elicitation process and review and
obtain stakeholder approval on requirements until approximately 1 year after deploying system
capabilities;
ƒ develop a concept of operations until several years after the project had been initiated, specifically,
the project was initiated in March 2003 and began deploying system capabilities in December
2005, but the concept of operations was not developed until June 2006;
ƒ follow a disciplined change management process for several years (between 2003 and 2006) after
the project was initiated in March 2003 (during this time, requirements were introduced and
implemented in an ad hoc fashion); and
ƒ develop supporting analysis showing traceability between requirements, system design, and the
contract.
20
GAO-06-823.
U.S. Immigration and Customs Enforcement: System Lifecycle Management, The Systems Development Lifecycle of Immigration and
Customs Enforcement, version 1.0 (Washington, D.C.: February 2005).
21
29
Page 36
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 1: Requirements Management
Our assessment of the extent to which the program complied with these practices on the three
projects is shown in attachment II.
According to ICE officials, including the program and project managers, the reason that the Atlas
projects did not fully adhere to these practices is due to the agency’s expedited schedule to deploy
modernized IT infrastructure capabilities to users in the wake of the events of September 11, 2001.
Specifically, the officials stated that in their haste to deliver capabilities, the project manager skipped
certain system management life cycle practices such as these.
The program manager acknowledged that requirements management needs to be improved and
stated that key steps are being taken to ensure the projects closely follow the practices specified in
the ICE System Lifecycle Management Handbook. He cited recent efforts by the CCE project
manager to follow such practices on work recently initiated to refresh hardware, such as desktops
and laptop computers across the agency. For example, in January 2007, the project conducted a
workstation requirements analysis survey that was sent out to ICE users to identify requirements
regarding the current workstation being used and issues that needed to be addressed in order to
perform functions associated with their jobs. In accordance with the handbook, project staff drafted
requirements, which they plan to finalize and approve by February 2007. While these are steps in the
right direction on this particular project, the program manager was not able to provide us with a
process improvement plan on how the program going forward would ensure compliance with
requirements development and management practices across all of the projects.
30
Page 37
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 1: Requirements Management
Consequently, until Atlas fully implements effective requirements development and management
practices programwide, it faces the increased likelihood that its projects will not meet user needs,
operate as intended, or be delivered on time and within budget. In addition, there is evidence that this
has already occurred. For example, on CCE, the delivery of its initial operating capability (the date
when it was to begin providing common e-mail capabilities to ICE users) was delayed from
September 2005 to December 2005. According to the project manager, this delay is attributable to,
among other things, the project not following these practices. Specifically, the project manager told us
that the project experienced numerous changes to requirements when it was transferred from the
Department of Justice to DHS and that these changes were not documented well. As a result, there
was extensive system rework to address missing requirements, which caused schedule delays.
31
Page 38
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 2: Contract Management and Oversight
Observation 2: Key contract management and oversight practices are not fully implemented
The Software Engineering Institute’s CMMI®22 identifies best practices that are essential to effectively
managing and overseeing contracts that support IT projects. These best practices include:
ƒ establishing and maintaining a plan for managing and overseeing contracts,
ƒ assigning responsibility and authority for performing contract management and oversight,
ƒ training staff performing contract management and oversight activities,
ƒ documenting each contract, including the statement of work and acceptance criteria,
ƒ verifying and accepting contractor-provided products and services (i.e., deliverables), and
ƒ conducting reviews with contactors to ensure that cost and schedule commitments are being met
and risks are being managed.
As we reported23 in June 2006, the Atlas program has largely established policies and procedures for
these practices. While the program has implemented two of the practices, it has not fully implemented
the other four. Specifically, our analysis of the program’s performance of these practices, including
efforts planned and underway, are shown in the following table.
22
CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
GAO, Homeland Security: Contract Management and Oversight for Visitor and Immigrant Status Program Need to Be Strengthened,
GAO-06-404 (Washington, D.C.: June 9, 2006).
23
32
Page 39
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 2: Contract Management and Oversight
Implementation of Contract Management and Oversight Practices for Atlas Program
1
2
24
CMMI practice
Establish and
maintain a plan for
managing and
overseeing
contracts.
Assign
responsibility and
authority for
performing
contract
management and
oversight.
Implemented GAO’s assessment of compliance
no
The program office issued a plan in February 2006 (referred to as an acquisition plan by
program officials), but it does not describe contract management and oversight processes.
Instead, the plan simply states that Atlas is to have a COTR work with program and
contracting officials to administer the contracts. In November 2006, the DHS Acquisition
Policy and Oversight Office, which is responsible for reviewing and approving the
acquisition plan, stated that the plan lacked details on contract management and oversight
and recommended that Atlas develop, among other things, a contract management and
oversight plan and incorporate it into the acquisition plan. In addition, we reported in July
2006 that the ICE System Lifecycle Management Handbook, which is used to help manage
the Atlas program, does not address certain key acquisition management activities, such
as contract tracking and oversight.24 Atlas’s program manager stated that the acquisition
plan and the handbook are being revised to include contract management and oversight
activities. The acquisition plan is to be revised by March 2007 and the handbook by June
2007.
yes
The program office hired a COTR in November 2006, and ICE assigned the individual the
responsibility and authority for Atlas contract management and oversight. The program
office also provided one contractor staff member to assist the COTR. The contractor’s
responsibilities include helping develop statements of work, delivery schedules, and
inspection and acceptance procedures. Further, each of the project managers has been
tasked by the program manager to assist the COTR in identifying requirements and
providing assistance in determining whether contractor deliverables meet requirements.
GAO-06-823.
33
Page 40
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 2: Contract Management and Oversight
Implementation of Contract Management and Oversight Practices for Atlas Program (cont.)
3
4
CMMI practice
Train staff
performing
contract
management and
oversight
activities.
Implemented
yes
Document each
contract, including
clearly identifying
the work to be
performed and
acceptance
criteria.
no
GAO’s assessment of compliance
The program’s COTR has extensive contract oversight training and experience. For
example, during 2005 and 2006, the official took 120 hours of training on topics such as
systems acquisition management, COTR responsibilities, the use of government purchase
cards, and contracting officer’s representative mentoring. Further, this official’s prior
experience includes 5 years of managing and overseeing contracts at the Department of
Defense, at a civilian agency, and in private industry. Also, all but one of the Atlas project
managers received contract management and oversight training in October 2006. The
remaining project manager is to receive the training by June 2007.
On its three key projects—namely CCE, Integration, and the ICE Mission Information
projects—the program did not satisfy this practice. Specifically, the contracts for these
three projects did not clearly identify the work to be performed and the criteria for accepting
contractor deliverables. Instead, the contracts identified the work to be performed and the
acceptance criteria in general terms. For example, the contract for deploying CCE (a
standard e-mail system agencywide) states that the contractor is responsible for providing
deployment services, but does not identify that deploying a standard e-mail system is one
of those services. Similarly, regarding acceptance criteria, the CCE contract states that the
contractor will provide monthly status reports, but does not identify how the program will
determine whether the deliverable is acceptable, such as describing tests to be performed.
Program officials, including the program manager and the COTR, acknowledged that the
statement of work and the acceptance criteria are not clearly defined, and that this makes
contract administration difficult. They added that this difficulty extended to the other Atlas
contracts because the program managed these contracts in the same manner as the other
three. To address the problem, the program manager stated that he has tasked the COTR
to work with project managers in developing clear statements of work and acceptance
criteria for new contracts that are to be issued by late February 2007 to replace existing
Atlas contracts that will begin to expire at that time.
34
Page 41
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 2: Contract Management and Oversight
Implementation of Contract Management and Oversight Practices for Atlas Program (cont.)
CMMI practice
Implemented
GAO’s assessment of compliance
5
Verify and accept
the deliverables.
partially
The COTR’s responsibilities include verifying and accepting contract deliverables. On the
contracts of the three projects reviewed, the COTR, in assessing whether to accept
deliverables, did not compare them with specific criteria in the contract because none
existed (see the previous slide). Instead, the COTR, in consultation with the staff from the
involved projects and ICE offices of budget and acquisition management, reviewed each
contractor’s invoices to assess whether it appeared what the contractor was charging was
reasonable for the deliverables provided. If consensus was reached among these parties,
then the COTR would approve the invoices for payment.
6
Conduct reviews
with contractors to
ensure cost and
schedule
commitments are
being met and
risks are
managed.
partially
The program manager stated that he and the ICE CIO conduct reviews with program
contractors on a quarterly basis to determine whether cost and schedule commitments are
being met and to review project risks. However, he could not provide documentation that
the meetings occurred due to the program not documenting the meetings. In addition, the
contractors provide reports to program officials on their progress in meeting commitments
and to identify risks related to each project. However, according to the program manager,
assessing meaningful contractor progress from these reports is difficult without clearly
defined work-to-be-performed statements and acceptance criteria.
Source: GAO analysis of DHS data.
35
Page 42
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 2: Contract Management and Oversight
The Atlas program manager stated that the program did not fully adhere to these practices due to it
not having until recently a COTR to manage and ensure compliance with contract management and
oversight requirements. He also stated that to strengthen the program’s compliance with such
practices, he has tasked the COTR with (1) revising the acquisition plan (by March 2007) to provide
detailed guidance on how Atlas will manage and oversee contracts, (2) revising the ICE System
Lifecycle Management Handbook by June 2007 to incorporate key contract management activities
such as contract tracking and oversight, and (3) preparing statements of work (beginning in February
2007) for future contracts to clearly define Atlas work statements and acceptance criteria. While these
are steps in the right direction, they are works in progress and, as such, have not been completed.
Until these steps and the other practices are fully implemented and institutionalized, Atlas faces the
risk that it will not deliver required capabilities and promised benefits on time and within budget. It also
makes the program vulnerable to contract mismanagement.
36
Page 43
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 3: Risk Management
Observation 3: Key risk management practices have yet to be implemented
Federal guidance and related best practices25 identify key practices essential for effectively managing
risks that may impact an IT project. They include, among other things:
ƒ identifying and prioritizing risks as to their probability of occurrence and impact, as well as
documenting them in an inventory;
ƒ developing and implementing the appropriate risk mitigation strategies; and
ƒ reporting to management on the existence and status of risks and progress in implementing
mitigation strategies.
In July 2006, we reported26 that the program had developed a risk management plan and process to
guide program office staff in managing risks throughout each project’s life cycle. We also reported
that although the program had recently begun to implement the risk management process, a
complete inventory of risks had not been developed. Since then, the program has taken additional
steps to implement risk management. For example, the program hired a risk management coordinator
to maintain and update the risk management plan, facilitate risk assessments, track efforts to reduce
risks to acceptable levels, and develop and conduct risk management training. In addition, the
program has conducted periodic risk management meetings and quarterly program management
meetings to review the program’s progress in managing risks.
25
26
See, for example, Software Engineering Institute, Capability Maturity Model Integrated (CMMI), version 1.1 (March 2002).
GAO-06-823.
37
Page 44
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 3: Risk Management
Further, the program conducted risk management training with project managers on October 24,
2006. The training included guidance on, among other things, adhering to key risk management
practices such as preparing risk assessments and developing risk mitigation strategies.
However, the program has yet to implement other key practices. First, it does not have a transparent,
documented, and traceable process for inventorying and resolving risks. For example, the January
23, 2006, inventory identified three risks related to data/information, security, and privacy. These risks
were previously reported as having a medium probability of occurrence and a low impact. However,
the January 23, 2007, inventory did not include these risks. Program officials explained that the
discrepancy with the number of risks in the inventory is that the program periodically closes risks and
removes them from the inventory. However, these officials provided documentation on only one risk
that had been removed from the inventory, and it did not include a clear rationale for why the risk had
been removed. In addition, the officials were unable to provide documentation showing that the other
two risks had been closed and removed and the justification for doing so.
Second, while the risk management plan defines a process and associated practices for developing
risk mitigation strategies, the program has not fully implemented them. Specifically, the plan specifies
that mitigation strategies are to include:
ƒ a rationale for mitigation strategy chosen;
ƒ an explanation of the impact on program performance;
38
Page 45
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 3: Risk Management
ƒ a proposed schedule showing milestones for initiation, significant risk mitigation activities,
and completion; and
ƒ the official responsible for implementing and tracking mitigation activities.
Out of the 61 risk mitigation strategies documented in the January 23, 2007, inventory:
ƒ all included a rationale for mitigation strategy chosen;
ƒ 13 did not include an explanation of the impact on program performance;
ƒ none included a proposed schedule showing milestones for initiation, significant risk mitigation
activities, and completion; and
ƒ all identified officials responsible for implementing and tracking mitigation activities.
Third, although the risk management plan calls for developing a process for elevating risks to
management’s attention, the program has not developed and implemented such a process. According
to the program manager, in lieu of a defined escalation process, program officials do make ICE’s
deputy CIO (and sometimes the CIO) aware of program risks as part of quarterly program
management meetings, but acknowledged that, for the most part, such reviews—and thus the
escalation of risks—occur in an ad hoc fashion and are not documented.
39
Page 46
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 3: Risk Management
The program manager stated that the reason for the current state of the program’s risk management
was that the process is new and that it will take time for the risk program to fully mature. He added
that the program is in the process of addressing these weaknesses by, for example, revising the risk
management plan to include procedures for (1) reporting to management on the existence and status
of risks and (2) closing risks. More recently, on January 25, 2007, the program manager provided us
with the revised plan; it calls for the risk management coordinator to, among other things, establish
processes for elevating and reporting to management risks that are likely to occur and have a high
impact. Further, at our February 1, 2007, exit meeting with program officials, they provided a
document, approved on January 25, 2007, defining their processes for elevating risk. While these are
steps in the right direction, the program manager acknowledged that program staff were just
beginning to implement the processes.
Until the Atlas program fully implements and institutionalizes risk management, there is increased
probability that program risks will not be proactively mitigated and, thus, will become actual program
cost, schedule, and performance shortfalls.
40
Page 47
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 4: Performance Management
Observation 4: Atlas program still implementing performance management practices essential
to measuring progress against commitments
As we have previously reported, 27 to enable Atlas program management to measure progress and
make well-informed investment decisions, it is important for the program to develop and implement
rigorous performance management practices that include, among other things, properly aligned goals
and anticipated achievements that are defined in measurable terms. In our September 2005 report,28
we recommended that the Atlas program develop and implement an effective performance
management system that includes such goals and measures. More recently, we reported29 in July
2006 that Atlas had begun taking steps to implement performance goals and measures, but had not
yet completed defining and implementing them. For example, the program had developed goals for
four of its seven projects but had not done so for its three other projects, which were
ƒ Transformation Planning,
ƒ Architecture Engineering, and
ƒ Data Center Migration.
We also reported that the program had developed measures to gauge progress on the goals, but had
not yet begun to track their progress.
27
GAO-06-823.
GAO, Information Technology: Management Improvements Needed on Immigration and Customs Enforcement’s Infrastructure
Modernization Program, GAO-05-805 (Washington, D.C.: Sept. 7, 2005).
29
GAO-06-823.
28
41
Page 48
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 4: Performance Management
Since then, the program has taken additional steps to define and implement performance goals and
measures. For example, the program identified 13 goals for five of its seven projects that it has been
using to measure progress during fiscal year 2006. Twelve of these goals either were expected to be
finished by fiscal year 2006 or were expected to have some interim measure completed during fiscal
year 2006. The expenditure plan identified an additional goal (goal 12) that did not have an interim
measure, but rather was expected to be finished by December 2006.
In addition, the program now reports it is measuring progress against these performance goals by, for
example, using them to discuss each project’s status and progress at quarterly program management
reviews.
The following table provides our analysis of the program’s performance goals against its reported
progress (as of Sept. 30, 2006).
42
Page 49
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 4: Performance Management
Atlas Project Progress in Meeting Fiscal Year 2006 Goals
Project
Goal
Overall
Program
Management/
Transformation
Planning
Goal 1: 100 percent of Atlas project activities that are managed with
less than a 10 percent variance in cost and schedule per fiscal year
Goal 2: 30 percent reduction in ICE's operations and maintenance
contribution for network services
Goal 3: The enterprise project management tool, Primavera, was
scheduled for installation, configuration, and deployment in third
quarter of fiscal year 2006
Goal 4: Performance management processes and balanced scorecard
to be developed and implemented in fourth quarter of fiscal year 2006
Common
Computing
Environment
Did Atlas
meet goal?
Goal 5: 100 percent of ICE personnel using the DHS standard
Microsoft Outlook E-mail platform
Goal 6: 33 percent progress in implementing a refresh of ICE
personnel desktop equipment
Goal 7: 100 percent of ICE office sites achieving workforce productivity
upgrades through implementation of ICE e-mail standard, backend
server upgrades, and ICENet connections
no
If no, what is
progress against
goal?
93 percent
yes
not applicable
no
deployed in 4th quarter
of FY 2006
no
*
no
95 percent
no
20 percent
no
95 percent
*While the program did not meet this goal in the fourth quarter of FY 2006 as planned, program officials told us that the program has efforts underway to reach this
goal, but did not provide a date for when it would be accomplished.
43
Page 50
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 4: Performance Management
Atlas Project Progress in Meeting Fiscal Year 2006 Goals (cont.)
Did Atlas
meet goal?
Project
Integration
ICE Mission
Information
Information
Assurance
Goal
Goal 8: 100 percent of ICE sites using DHS standard high-speed
network circuits
Goal 9: 100 percent of ICE office sites achieving workforce productivity
upgrades through implementation of ICE e-mail standard, backend
server upgrades, and ICENet connections
Goal 10: 14 percent of investigative and enforcement personnel with
access to decision support system data marts
Goal 11: 8 percent increase in ICE investigative and enforcement
systems incorporated into ICE decision support system consolidated
data marts
Goal 12: The ICE Mission Information project will implement a
technology model in the first-quarter of FY 2007 for the integration of
systems and information throughout ICE within the framework of the
DHS EA.
Goal 13: 30 percent of ICE users employing single sign-on capability
for systems access.
no
If no, what is
progress against
goal?
73 percent
no
73 percent
yes
not applicable
yes
not applicable
Not
applicable
(target goal
set for FY
2007, but
not for FY
2006)
No
**
0 percent
Source: GAO analysis of DHS data.
**The program reported that it met this goal in fiscal year 2007, as planned.
44
Page 51
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 4: Performance Management
However, while the program has taken additional steps, it has yet to fully implement and
institutionalize its performance goals and measures. Specifically, the program has not yet developed
goals for these two projects—Architecture Engineering and Data Center Migration. According to
program officials, they have developed draft goals for these projects and are in the process of having
them reviewed by management, but officials did not provide a date for when the goals are to be
approved and implemented.
In addition, the progress on the performance goals, as reported by the program, shows mixed results.
As shown in the chart, of the 12 goals, the program has met only 3 of them. Of the remaining 9, Atlas
was close to meeting its fiscal year 2006 targets. For example, on goal 5, the fiscal year 2006 target
was 100 percent of ICE personnel using the DHS standard Microsoft Outlook e-mail platform, and the
program reported 95 percent of users have been migrated. Also, on goal 8, the fiscal year 2006 target
was 100 percent of ICE sites using DHS standard high speed network circuits, and the program
reported 73 percent of ICE sites are using the circuits.
Further, underlying data used by the program to measure performance may not be reliable in all
cases. For example, on goal 1, the program uses project cost and schedule data from the 2005 Atlas
business case in measuring progress against the goal, including determining whether variances have
occurred. However, these data are out of date (i.e., they do not reflect project cost and schedule
slippages that have occurred since 2005) and are currently being revised as part of a larger effort to
update the entire business case to reflect overall program changes.
45
Page 52
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Objective 2 Results
Observation 4: Performance Management
In addition, the program relies on its project managers to develop estimates of where their projects
should be (with respect to schedule) given funds used to date. The program official responsible for
determining progress against goal 1 told us that this is a key weakness of the performance
management process because it allows project managers to report on their progress without the
program having a means to verify and validate their assessments. The program manager
acknowledged that having and using reliable information to support measurement against goals is a
challenge across all of the program measures, and the program plans to address the problem by,
among other things, updating the business case and supporting cost and schedule analyses (which
are to be issued in September 2007) and by implementing and using an enterprise project
management tool (planned to start in October 2007) that calculates progress against funds used for
each project. While these are steps in the right direction, it also means it will be months before the
program has data it can rely on to measure performance.
According to the program manager, the program’s limited progress in implementing performance
management practices is attributable in large part to (1) the program’s evolving of its performance
goals as the program evolves; (2) performance management being new to the program, and it will
take time for the program’s processes to fully mature; and (3) the enterprise project management
tool’s taking longer to implement than originally planned. According to the program manager, the
program recently purchased the tool and plans to begin using it for all projects beginning in October
2007. Nonetheless, until the program completes implementation and institutionalization of its
performance management capabilities, including well-defined goals and measures, its managers will
not have the necessary information for measuring progress and making well-informed investment
decisions.
46
Page 53
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Conclusions
The fiscal year 2006 Atlas expenditure plan, in combination with related program documentation and
program officials’ statements, satisfies or partially satisfies the legislative conditions set forth by
Congress. However, this satisfaction is based on plans and commitments that provide for meeting
these conditions rather than on completed actions to satisfy the conditions. In addition, while steps
are being initiated that are intended to address significant program management weaknesses, a
number of improvements, including those recommended in our past reports, have yet to be
implemented. Further, the program has not fully achieved many performance goals that it set out to
accomplish over the past year. These factors continue to put the program at risk and call for
heightened and sustained management attention to expeditiously address and resolve the issues.
Thus, there is much that still needs to be accomplished to minimize the risks associated with the
program’s capacity to deliver promised IT infrastructure capabilities and benefits on time and within
budget. This includes demonstrating better progress against established performance goals in the
coming year. Given that hundreds of millions of dollars are to be invested and the program is critical
to supporting the ICE mission, it is essential that DHS follow through on its commitments to build the
capacity to effectively manage the program. Proceeding without this capacity introduces unnecessary
risks to the program and potentially jeopardizes its viability for future investment.
47
Page 54
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Recommendations for Executive Action
To minimize risks to the Atlas program, we recommend that the Secretary of Homeland Security
direct the Assistant Secretary for Immigration and Customs Enforcement to ensure that ICE follows
through on its commitments to implement effective management controls and capabilities by
implementing the following five recommendations:
ƒ Employing practices essential to ensuring that the Atlas program’s IV&V agent is and remains
independent, including incorporating requirements in future contracting actions such as the
renegotiation or recompetition of the current independent verification and validation contract, which
will prohibit the agent from soliciting, proposing, or being awarded program work other than
providing independent verification and validation services and products.
ƒ Fully adhering to requirements development and management practices, including those specified
in ICE’s policies and procedures. This should also include having the Atlas program manager
develop a process improvement plan for all of the projects that is consistent with the ICE System
Lifecycle Management Handbook and provide for making the program manager and project
managers responsible and accountable for rigorously adhering to the requirements in the
handbook.
ƒ Fully implementing key contract management and oversight practices, including those specified in
ICE’s policies and procedures. This should also include ensuring that the Atlas program manager,
working with the program’s COTR, follow through on planned efforts to strengthen the program’s
compliance with these practices by (1) revising the acquisition plan by March 2007; (2) revising the
ICE System Lifecycle Management Handbook by June 2007 to incorporate key contract
management activities such as contract tracking and oversight; and (3) preparing statements of
48
Page 55
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Recommendations for Executive Action
work (beginning in February 2007) for future contracts to clearly define Atlas work statements and
acceptance criteria.
ƒ Completing implementation of planned risk management activities. This should include (1) fully
implementing and institutionalizing procedures for reporting to management on the existence and
status of risks and progress in implementing mitigation strategies and (2) updating the risk
inventory to include risks for all projects and risk areas.
ƒ Improving program performance management, including developing performance goals for projects
that do not have goals and reporting on their progress in the fiscal year 2007 expenditure plan.
Further, the program should assess whether the data being used to measure progress are reliable,
complete, and accurate.
49
Page 56
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Agency Comments
In their February 1, 2007, oral comments on a draft of this briefing, ICE officials, including the Deputy
CIO and Atlas program manager, agreed with our findings, conclusions, and recommendations.
These officials also provided technical comments, which we incorporated in the briefing as
appropriate.
50
Page 57
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Attachment I
Detailed Scope and Methodology
To accomplish our objectives, we
ƒ analyzed the fiscal year 2006 Atlas expenditure plan and supporting documents against
legislative conditions and other relevant federal requirements, guidance, and best practices
to determine whether the conditions were met (in doing so, we considered the conditions
met when the expenditure plan, including supporting program documentation and program
officials’ representations, either satisfied or provided for satisfying the conditions) and
ƒ evaluated supporting documentation and interviewed program and other involved ICE and
DHS officials to determine progress in establishing capabilities in program management
areas, such as acquisition planning, enterprise architecture, project management,
requirements development and management, contract management and oversight, and risk
management and performance management.
In assessing the management of requirements development and contract management and oversight,
we focused on three projects—Common Computing Environment, Integration, and ICE Mission
Information—that collectively accounted for approximately 75 percent of the funds provided to the
program and the Atlas program manager identified as being the most critical to the program’s success
and ICE’s mission. For DHS and ICE data that we did not substantiate, we made appropriate
attribution indicating the data source. We conducted our work at ICE and DHS headquarters in
Washington, D.C., from November 2006 through January 2007 in accordance with generally accepted
government auditing standards.
51
Page 58
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Attachment II
GAO Analysis of Requirements Management
ICE Requirements Management Efforts Compared with Federal IT Management Guidance and
Best Practices
Establish a requirements management policy
Project
Implemented
CCE
yes
Integration
yes
ICE Mission
Information
yes
GAO’s assessment of compliance
In February 2005, the CCE project established the ICE System Lifecycle Management
30
Handbook as the policy to guide project efforts in developing and managing requirements. The
handbook requires requirements development and management activities, such as eliciting
desired system capabilities from users and translating them into requirements and obtaining
approval by all stakeholders.
In February 2005, the Integration project established the ICE System Lifecycle Management
Handbook as the policy to guide project efforts in developing and managing requirements.
In February 2005, the ICE Mission Information project established the ICE System Lifecycle
Management Handbook as the policy to guide project efforts in developing and managing
requirements.
Source: GAO analysis of DHS data.
Legend: Yes=established/implemented; No=not established/implemented; Partially=some, but not all, actions are implemented or actions are in progress of being
implemented
30
U.S. Immigration and Customs Enforcement: System Lifecycle Management, The Systems Development Lifecycle of Immigration and
Customs Enforcement, version 1.0. (Washington, D.C.: February 2005).
52
Page 59
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Attachment II
GAO Analysis of Requirements Management
ICE Requirements Management Efforts Compared with Federal IT Management Guidance and Best
Practices (cont.)
Elicit desired or required system capabilities from users and translate them into system requirements and
obtain approval by all stakeholders
Project
Implemented
Common
Computing
Environment
partially
Integration
partially
ICE Mission
Information
partially
GAO’s assessment of compliance
In 2003, the CCE project elicited system capabilities from component and departmental users.
Although the capabilities elicited from the users were not documented, project officials told us
they translated the users’ input into system requirements, issuing a draft requirements
document in June 2004. Although the draft requirements had not yet been reviewed and
approved by system stakeholders, the project began deploying system capabilities in December
2005. Project officials told us that they planned to finalize the requirements document, including
obtaining stakeholder approval, and in January 2007, they provided us with a finalized and
approved version of the requirements document.
In 2003, Integration elicited system capabilities based on a directive from the DHS Deputy
Secretary and departmental users. The capabilities were elicited from users at requirements
workshops and documented. Project officials told us that they translated the users’ input into
system requirements, issuing a draft requirements document on June 15, 2005. Although the
draft requirements have not yet been reviewed and approved by system stakeholders, the
project began deploying system capabilities in March 2006. Project officials told us that they
have obtained stakeholder approval on the requirements document; however, they could not
provide documentation showing this had occurred.
Divided into four phases, the ICE Mission Information project is currently in the process of
eliciting user system capabilities for each of the respective phases. For the first phase, the
project has elicited system capabilities from its users, but has not documented them. Project
officials told us that they translated the user capabilities into system requirements, issuing a
draft requirements document dated August 7, 2006. Project officials stated that they planned to
finalize the requirements document, including obtaining system stakeholder approval, but could
not provide a date by when this was to occur. For the other three phases, the project has not
completed eliciting requirements and documenting capabilities, but project officials told us they
planned to do so in accordance with the practices. However, they were not able to provide a
plan and associated timetable for when they are to complete the practices.
53
Page 60
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Attachment II
GAO Analysis of Requirements Management
ICE Requirements Management Efforts Compared with Federal IT Management Guidance and Best
Practices (cont.)
Develop a system concept of operations
Project
CCE
Integration
ICE Mission
Information
Implemented
partially
no
partially
GAO’s assessment of compliance
CCE did not develop a concept of operations, which is integral to defining system requirements,
until 3 years after the project had been initiated. Specifically, the project was initiated in March
2003 and began deploying system capabilities in December 2005, and the concept of
operations was developed in June 2006.
According to the Integration project manager, a concept of operations was developed. However,
Integration was not able to provide documentation showing it had been developed.
The project issued a draft concept of operations on October 12, 2006. The ICE Mission
Information project manager told us that the project is in the process of finalizing the document
but did not provide a date for when the document would be completed.
Source: GAO analysis of DHS data.
54
Page 61
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Attachment II
GAO Analysis of Requirements Management
ICE Requirements Management Efforts Compared with Federal IT Management Guidance and Best
Practices (cont.)
Ensure that changes to requirements are controlled and approved via a disciplined change control process
as the system is developed and implemented
Project
CCE
Implemented
partially
Integration
partially
ICE Mission
Information
partially
GAO’s assessment of compliance
CCE did not follow a disciplined change management process for 3 years after the project was
initiated in March 2003; during this time, requirements were introduced and implemented in an
ad hoc fashion. The program has taken steps to improve change management efforts by (1)
developing an Atlas Configuration Management Plan on February 15, 2006; (2) developing a
CCE change control board charter on November 1, 2006; and (3) developing a change process
that provides the procedures for reviewing and approving CCE change requests. The project is
currently in the process of implementing these changes. Project officials told us that the project
recently began conducting CCE change control board meetings in September 2006. According
to the change process, CCE is required to ensure that technical, cost, and schedule impacts are
considered before approval is granted. Although program documentation showed the technical
impact of the change, the cost and schedule impacts were not documented.
The configuration control board and change control process that Integration follows was
developed by DHS’s Infrastructure Transformation Program, which is the departmental
organization (commonly referred to as the “network steward” by DHS and ICE officials)
responsible for ensuring that all DHS components migrate from their existing networks to the
DHS network. However, Integration officials could not provide evidence of compliance with the
network steward’s change control process.
The project plans to control and approve changes via a change control process when the
requirements are developed and finalized. The project plans to do so consistent with the
guidance specified in the ICE System Lifecycle Management Handbook. As previously noted,
project officials were not able to identify when the requirements would be finalized and
approved.
55
Page 62
GAO-07-565 Information Technology
Appendix I
Briefing to the Staffs of the Subcommittees
on Homeland Security, Senate and House
Committees on Appropriations
Attachment II
GAO Analysis of Requirements Management
ICE Requirements Management Efforts Compared with Federal IT Management Guidance and Best
Practices (cont.)
Maintain bidirectional traceability, meaning that a given requirement can be traced backward to systems
documentation and forward to the appropriate contract vehicle and system components
Project
CCE
Implemented
partially
Integration
partially
ICE Mission
Information
partially
GAO’s assessment of compliance
In December 2006, CCE developed a requirements traceability analysis document—commonly
referred to as a matrix—to show bidirectional traceability between the requirements and system
documentation. While the project identified requirements, it did not trace requirements to the
system documentation, such as the system design and test plans. Additionally, the project did
not trace requirements to the existing contract task order.
In June 2005, Integration developed a requirements traceability analysis document—commonly
referred to as a matrix—to show bidirectional traceability between the requirements and system
documentation. While the project identified requirements, it did not trace requirements to the
system documentation, such as the system design and test plans. Additionally, the project did
not trace requirements to the existing contract task order.
The project plans to ensure traceability among requirements, system documentation, and the
existing contract task order when the requirements are developed and finalized. The project
intends to do so in accordance with the guidance specified in the ICE System Lifecycle
Management Handbook. As previously noted, project officials were not able to identify when the
requirements would be finalized and approved.
Source: GAO analysis of DHS data.
56
Page 63
GAO-07-565 Information Technology
Appendix II
Comments from the Department of Homeland
Security
Page 64
Appendx
Ii
GAO-07-565 Information Technology
Appendix II
Comments from the Department of Homeland
Security
Page 65
GAO-07-565 Information Technology
Appendix II
Comments from the Department of Homeland
Security
Page 66
GAO-07-565 Information Technology
Appendix III
GAO Contact and Staff Acknowledgments
GAO Contact
Randolph C. Hite (202) 512-3439
Staff
Acknowledgments
In addition to the individual named above, Gary Mountjoy, Assistant
Director; Nancy Glover; James Houtz; Jennifer Mills; Freda Paintsil;
Madhav Panwar; Karl Seifert; and Kelly Shaw made key contributions to
this report.
(310640)
Page 67
Appendx
iI
GAO-07-565 Information Technology
GAO’s Mission
The Government Accountability Office, the audit, evaluation and
investigative arm of Congress, exists to support Congress in meeting its
constitutional responsibilities and to help improve the performance and
accountability of the federal government for the American people. GAO
examines the use of public funds; evaluates federal programs and policies;
and provides analyses, recommendations, and other assistance to help
Congress make informed oversight, policy, and funding decisions. GAO’s
commitment to good government is reflected in its core values of
accountability, integrity, and reliability.
Obtaining Copies of
GAO Reports and
Testimony
The fastest and easiest way to obtain copies of GAO documents at no cost
is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts
newly released reports, testimony, and correspondence on its Web site. To
have GAO e-mail you a list of newly posted products every afternoon, go to
www.gao.gov and select “Subscribe to Updates.”
Order by Mail or Phone
The first copy of each printed report is free. Additional copies are $2 each.
A check or money order should be made out to the Superintendent of
Documents. GAO also accepts VISA and Mastercard. Orders for 100 or
more copies mailed to a single address are discounted 25 percent. Orders
should be sent to:
U.S. Government Accountability Office
441 G Street NW, Room LM
Washington, D.C. 20548
To order by Phone: Voice: (202) 512-6000
TDD: (202) 512-2537
Fax: (202) 512-6061
To Report Fraud,
Waste, and Abuse in
Federal Programs
Contact:
Congressional
Relations
Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400
U.S. Government Accountability Office, 441 G Street NW, Room 7125
Washington, D.C. 20548
Public Affairs
Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800
U.S. Government Accountability Office, 441 G Street NW, Room 7149
Washington, D.C. 20548
Web site: www.gao.gov/fraudnet/fraudnet.htm
E-mail: fraudnet@gao.gov
Automated answering system: (800) 424-5454 or (202) 512-7470