Budgeting for Performance in the U.S.
Using the
Program Assessment Rating Tool
J. Kevin Carroll
U.S. Office of Management and Budget
July 2008
1
Agenda
•
•
•
•
•
•
Overview of U.S. R&D spending
Budget Players and Process in the U.S.
Why PART Was Created
PART Basics
How OMB Uses PART Results
Supplemental Information
2
R&D as a Share of Discretionary Spending
It’s approximately constant over the last 30 years!
30%
25%
20%
15%
10%
5%
0%
1962
1967
1972
1977
R&D/ Discretionary, Civilian
R&D/ Discretionary, Total
1982
1987
1992
1997
2002
Civilian R&D share, excluding Apollo
Total R&D share, excluding Apollo
3
4
90
80
70
60
50
East
West
North
40
30
20
10
0
1st Qtr
2nd Qtr
3rd Qtr
4th Qtr
5
Agenda
•
•
•
•
•
•
Overview of U.S. R&D spending
Budget Players and Process in the U.S.
Why PART Was Created
PART Basics
How OMB Uses PART Results
Supplemental Information
6
The Administration
• 15 Cabinet-level Departments
• > 100 agencies, boards, & commissions
• Executive Office of the President
7
The Executive Office of the President
•
•
•
•
•
•
•
•
•
•
•
•
•
Office of the Vice President*
Chief of Staff*
Council of Economic Advisers
Council on Environmental Quality
Domestic Policy Council
National Economic Council
National Security Council
Office of Homeland Security
Office of Management and Budget*
Office of National Drug Control Policy*
Office of Science & Technology Policy
Office of the United States Trade Representative*
Etc…
* Cabinet rank members
8
What does OMB do?
• Leads or participates in the development
and resolution of all budget, policy,
legislative, regulatory, procurement, and
management issues on behalf of the
President.
• Oversees the implementation,
coordination, and management of agency
programs.
9
Congress
~250 Committees and
Subcommittees
About a dozen appropriations
subcommittees in both the
House and the Senate
"No money shall be drawn from the treasury, but in consequence
of appropriations made by law; and a regular statement and
account of receipts and expenditures of all public money shall
be published from time to time."
– U.S. Constitution, Article 1, Section 9
10
Overlapping Budget Processes
Oct 1, 2007
Oct 1, 2006
FY 2007
FY 2008
OMB oversees agency budget execution and evaluates performance
OMB
prepares
FY 2008
Budget
Congressional
Action
On FY 2008 Budget
Agencies
prepare
FY 2009
budget
requests
We are here
OMB
assesses
FY 2009
requests
OMB
Congressional
prepares
Action
FY 2009
On FY 2009 Budget
Budget
11
Agenda
•
•
•
•
•
•
Overview of U.S. R&D spending
Budget Players and Process in the U.S.
Why PART Was Created
PART Basics
How OMB Uses PART Results
Supplemental Information
12
The Challenge in 2001
– Agency performance information inadequate,
inconsistent or not available
– Performance measures not outcomeoriented and not tied to agency mission
– Performance results not integrated into
agency decisions and budget requests.
• How do we incorporate program
performance into funding and
management decisions?
13
Budget – Performance Integration
• One of five government-wide initiatives of
the President’s Management Agenda
• PART is a tool for assessing agency
performance in this initiative
– The “stoplight” scoring system used to assess
agency progress in the BPI initiative includes
criteria related to PART
14
Agenda
•
•
•
•
•
•
Overview of U.S. R&D spending
Budget Players and Process in the U.S.
Why PART Was Created
PART Basics
How OMB Uses PART Results
Supplemental Information
15
PART Basics (1)
The PART has four sections:
I. Purpose and Design
II. Strategic Planning
III.Management
IV.Results
(20 percent)
(10 percent)
(20 percent)
(50 percent)
• Each section has 5 to 10 questions.
• Sections I, II, and III allow only Yes or No
responses.
• Section IV allows partial credit.
16
PART Basics (2)
• PART Guidance document
– elements of a “Yes” response
– acceptable documentation of evidence
• Products posted to ExpectMore.gov:
•
•
•
•
program score (0 to 100)
effectiveness rating
summary of findings
Improvement plan / follow-up actions
17
PARTs Tailored to Program Type
• The PART has seven versions, one for each
program “type”:
–
–
–
–
credit
- block/formula grant
competitive grant
- direct federal
regulatory
- research and development
capital assets & service acquisitions
• Questions in the “Direct Federal” version are
common to all PARTs
• Other versions include extra questions tailored
for program type
• Question weighting can be adjusted
Issue: Definition of a Program
18
Section I: Purpose & Design
Highlights
• Clear purpose?
• Address a specific and existing problem?
• Not redundant or duplicative?
• Free of major design flaws that limit the
program’s effectiveness or efficiency?
• Resources effectively targeted?
19
Section II: Strategic Planning
Highlights:
• Long-term and annual performance measures?
• Baselines and ambitious targets?
• Plan for independent evaluations of sufficient
scope, quality, and frequency?
• Budget requests tied to accomplishment of
goals?
• Address strategic planning deficiencies?
20
Section III: Program Management
Highlights:
• Collect and use performance info?
• Federal managers and contractors held
accountable for cost, schedule, performance?
• Funds spent for intended purpose?
• Effective collaboration and coordination?
• Strong financial management practices?
• Address management deficiencies?
21
Section IV: Program
Results/Accountability
Highlights:
• Meeting long-term and annual goals?
• Demonstrate improved efficiency?
• Results of independent evaluations?
• Scoring: partial credit allowed
– Yes
– Large Extent
– Small Extent
– No
22
Agenda
•
•
•
•
•
•
Overview of U.S. R&D spending
Budget Players and Process in the U.S.
Why PART Was Created
PART Basics
How OMB Uses PART Results
Supplemental Information
23
What Do We Get Out of PART?
1. Overall program “effectiveness” ratings, based
on banding of scores that can range from 0 to
100:
–
–
–
Effective
- Ineffective
Moderately Effective - Results Not Demonstrated
Adequate
2. Specific follow-up actions aimed at improving
performance
3. Data to inform decision making processes,
including both budget and management
decisions
24
“Effectiveness Ratings”
(cumulative number of programs assessed)
100%
80%
60%
6%
11%
15%
15%
26%
29%
17%
18%
30%
31%
24%
26%
15%
5%
20%
5%
40%
26%
28%
28%
29%
4%
4%
50%
20%
38%
29%
3%
3%
24%
22%
19%
2005 (793)
2006 (977)
2007 (1,004)
0%
2002 (234)
2003 (407)
Resu lts No t Demo nstra ted
2004 (607)
In effe cti ve
Ad equa te
Mode ratel y Effective
Effective
25
(a) Data for Budget Decision Making
Funding
Guidance
Administration
Priorities
Programmatic
trends/external
factors
PART
Agency
Requests
Budget Process
Director’s Review
-funding decisions
-management recommendations
-legislative proposals
Departments/agencies
-budget proposals
-actions to improve performance
26
(b) Data for Budget Decision Making
• PART informs budget decisions; not sole
basis for decisions.
– A good PART score does not necessarily
mean more funding.
– A bad PART score does not necessarily mean
less funding.
• PART helps identify needed planning and
management improvements.
27
Agenda
•
•
•
•
•
•
Overview of U.S. R&D spending
Budget Players and Process in the U.S.
Why PART Was Created
PART Basics
How OMB Uses PART Results
Supplemental Information
28
Portion of Discretionary funds
spent on R&D by OMB division
100%
% BA, FY 2004
80%
60%
40%
20%
0%
Nat. Res.
Hum. Res.
F S&T
"D"
Gen. Govt.
Non-R&D
Non-R&D
Natl. Sec.
29
Total funds spent on R&D by OMB Division
100%
% $, FY 2004
80%
60%
40%
20%
0%
Nat. Res.
Hum. Res.
F S&T
"D"
Gen. Govt.
Natl. Sec.
Non-R&D
30
Implementation Issues
•
•
•
•
•
•
•
Definition of a “program”
Reassessment frequency
Tracking PART follow-up actions
Quality control
Data collection, analysis, and presentation
Congressional and public interest
Increasing transparency: ExpectMore.gov
31
R&D Specific Questions
• 2RD2) If applicable, does the program assess and
compare the potential benefits of efforts within the
program and (if relevant) to other efforts in other
programs that have similar goals?
• 2RD2) Does the program use a prioritization process
to guide budget requests and funding decisions?
• 3RD1) For R&D programs other than competitive
grants programs, does the program allocate funds and
use management processes that maintain program
quality?
32
Performance Measures
The PART makes the assessment approach
consistent across programs; performance
measures make assessment appropriate for
each program.
•Performance measures are key to:
– assessing program effectiveness
– incentivizing the right behavior
•Performance measures should be:
– Salient and meaningful
– Capture the most important aspects of a program’s
purpose and priorities
33
Goals, Measures, and Targets
Goal = performance measure + target
• Targets should be quantitative and
trendable over time
• Quantitative baseline required.
34
Outcomes and Outputs
Performance goals may focus on outcomes or outputs.
• Outcomes: Intended result, effect, or consequence of
program. Public benefit should be clear. External
factors influence.
• Outputs: What the program produced or provided.
• The PART strives for measures of outcomes.
– Output or process measures should clearly tie to outcomes.
35
Consistency Check Purpose
• Ensure OMB examiners are applying
PART guidance consistently
– PART response vs. PART guidance
– NOT PART response vs. other PART
response
• Assessing quality and content of
Explanation and Evidence
• Generally no review of evidentiary
documents
36
Available at www.omb.gov/part
• PART guidance
• Link to ExpectMore.gov
• Supporting materials on performance
measurement and evaluation
• Frequently asked questions
37