Agency Capabilities and
Performance in Applying BCA
Another Modest Proposal for Reform
Richard B. Belzer
Regulatory Checkbook
Mt. Vernon, VA
rbbelzer@post.harvard.edu
History of Federal Practice
 Formalized since 1981
 Agency responsibility to perform BCA
 OMB responsibility to perform peer review
 Controversies and conflicts are hard-wired
 With Congress and among agencies
 Staffing, organization, and reprogramming
 Methodological issues and research priorities
Society for
Benefit Cost Analysis
‘CSI: Executive Branch’
 Bureaucratic biology
 BCA: A bacterial infection
 Agencies have developed antibodies


Program control over / co-option of agency economists
Methodological corruption
 Benefits can be found if you look hard enough
 Costs can be ignored if you are determined enough
 OMB: Dissociative Identity Disorder (ICD-9-CM Dx 300.14)
 Performs external, independent peer review of RIAs…

…but advance the President’s agenda…
 …without being held publicly accountable for the difference
where these functions conflict
 Administrative Procedure Act: Placebo therapy
Society for
Benefit Cost Analysis
Evidence of ICD-9-CM Dx 300.14
Caveats
 Agency (not OMB)
estimates
 From a draft report
Invalid Assumptions
 All ‘major rules’ are
included
 Agencies adhered to
OMB Circular A-4
 Estimates of costs, risks,
and benefits are unbiased
 Uncertainty is unimportant
Society for
Benefit Cost Analysis
Agency Use of Science in BCA:
April 1998 -- March 2000
 73 rules, 36 about risk
 Embedded bias




7 (‘judgment’)
2 (assumptions)
6 (upper bounds)
1 (sensitivity analysis)
 Uncertainty ignored



6 (point estimates)
2 (ranges)
1 (distribution)
 Motive for performing BCA


Statutory reqmt: 100%
EO 12866: 14%
Society for
Benefit Cost Analysis
Examples:
Major Rules from 1988-1998
 ‘Hazardous Waste Corrective Action’
 C: ~$1 trillion; B: not estimated
 ‘Partially-Cooked Beef Patties’
 C: < $10 million; B: not estimated
 ‘Seafood HACCP’: impossible benefits
 Reduced risks from seafood
(‘consumers underestimate true risk’)
 Increased consumer confidence in seafood safety
(‘consumers overestimate true risk’)
 ‘Meat & Poultry HACCP’: imaginary benefits
 Assume regulation reduces risk 90%
Society for
Benefit Cost Analysis
Examples:
Major Rules from 1988-1998
 ‘Hazardous Waste Corrective Action’
 C: ~$1 trillion; B: not estimated; R: ~0
 ‘Partially-Cooked Beef Patties’
 C: < $10 million; B: not estimated; R: ~ 30 deaths/yr
 ‘Seafood HACCP’: impossible benefits
 Reduced risks from seafood
(‘consumers underestimate true risk’)
 Increased consumer confidence in seafood safety
(‘consumers overestimate true risk’)
 ‘Meat & Poultry HACCP’: imaginary benefits
 Assume regulation reduces risk 90%
Society for
Benefit Cost Analysis
Examples:
Major Rules from 2007-08
 5 Major Rules
 PTO ‘Limits on Claims’
 PTO ‘Limits on
Continuations’
 PTO ‘IDS’
 PTO ‘Appeals’
 DHS ‘No Match’
 OMB made aware of
costs but deemed
none of them ‘major’
Society for
Benefit Cost Analysis
Examples:
Major Rules from 2007-08
 5 Major Rules
 PTO ‘Limits on Claims’
 PTO ‘Limits on
Continuations’
 PTO ‘IDS’
 PTO ‘Appeals’
 DHS ‘No Match’
 OMB made aware of
costs but deemed
none of them ‘major’
Society for
Benefit Cost Analysis
$1 billion!
BCAs Can Be Validated,
But They Aren’t
 Often easier than ex
ante analysis
 Very limited federal
agency interest
 Limited external WTP
for ‘academic’ work
HACCP Implementation Period
Society for
Benefit Cost Analysis
Dx 2001:
Centralized Review Doesn’t Work










Limited and progressively smaller review staff
Transaction-driven, and hence reactive
Review occurs after decisions have been made
Asymmetrical rules of engagement
Limited enforcement tools
Enforcement has limited utility
Presidential initiatives are exempt
Congressional sensitivity is unbounded
Statutory/judicial deadlines trump OMB review
OMB has limited political and public support
Society for
Benefit Cost Analysis
Rx 2001:
Failed Remedies
 Persuasion
 Co-option
 RIA Guidance
 Return to sender
 Suspension of review
Society for
Benefit Cost Analysis
Rx 2001:
First Modest Proposal
Failed Remedies
 Persuasion
 Co-option
 RIA Guidance
 Return to sender
 Suspension of review
Society for
Benefit Cost Analysis
Possible Ideas
 Regulatory budgeting
 ‘Prompt’ letters
 Publish peer reviews
 BCA blueprints
 Integrate BCA & PRA
Rx 2001:
Implemented 2001-09
Failed Remedies
 Persuasion
 Co-option
 RIA Guidance/Cir. A-4
 Return to sender
 Suspension of review
Society for
Benefit Cost Analysis
Possible Ideas
 Regulatory budgeting
 ‘Prompt’ letters
 Publish peer reviews
 BCA blueprints
 Integrate BCA & PRA
 Information quality
 Peer review policy
How Have These Remedies Fared?
 Guidance, ‘prompts’, and returns
Society for
Benefit Cost Analysis
OMB Review:
Lion or Pussycat?
Total OMB Reviews and Rules
Returned, 1981-2007
 Lion
Returned
 Reagan I
 Bush 41
 Bush 43 (2001 only)
3,000
60
2,500
50
2,000
40
1,500
30
1,000
20
500
10
0
0
1981
Society for
Benefit Cost Analysis
1985
1989
1993
1997
2001
2005
Rules Reviewed
 Reagan II
 Clinton I & II
 Bush 43 (2002-07)
Total
70
Rules Returned
 Pussycat
Prompts
How Have These Remedies Fared?
 Guidance, ‘prompts’, and returns
 Peer review policy (2005)
Society for
Benefit Cost Analysis
How Have These Remedies Fared?
 Guidance, ‘prompts’, and returns
 Peer review policy (2005)
 Covers ‘influential’ scientific, technical, & statistical
information used in BCA’
 Good idea, badly implemented
 Too much agency discretion
 Agencies control charge and reviewers
 Reviewers cannot veto
 Agency BCAs reviewed by OMB are exempt!
Society for
Benefit Cost Analysis
How Have These Remedies Fared?
 Guidance, ‘prompts’, and returns
 Peer review policy (2005)
 Information quality
Society for
Benefit Cost Analysis
Information Quality
Law and Guidelines
 ‘Information Quality Act’ (2001)
44 U.S.C. 3516 note
 OMB government-wide guidance (2002)
 Reduce error: Pre-dissemination review
 Correct error: Administrative procedures
 Agency responses
 All complied except DHS
 Almost all were not serious
Society for
Benefit Cost Analysis
Information Quality
Terms and Definitions
 Utility
 Integrity
 Objectivity
 Substance
 “accurate, reliable, and unbiased”
 Presentation
 “accurate, clear, complete, and unbiased manner”
 “within a proper context”
 Procedure (transparency, reproducibility)
Society for
Benefit Cost Analysis
Types of Information Quality Error






Substance
Transcription
Units
Excess precision
Misapplication
Misuse
Bias (not policy-neutral)






Presentation
Caveats omitted
Sources omitted
Context omitted
Data/results censored
‘Best’ models only
Bias (not policy-neutral)
Procedural
 Non-transparency
 Irreproducibility
Society for
Benefit Cost Analysis
Types of Bias
 Incidental

Not material to decision-making
 Unavoidable

No model is perfect, data are limited and expensive
 Purposeful






‘2 + 2 = 5 for large values of 2’
Best justification given: ‘avoids false negatives’
Consequence: many false positives
Conflates analytic and decision-making functions
Lets analysts make policy decisions
Fundamentally incompatible with information quality and BCA
Society for
Benefit Cost Analysis
An Example of
Freely Advertised Purposeful Bias
“EPA’s policy is that risk
assessments should not knowingly
underestimate or grossly
overestimate risks. This policy
position prompts risk assessments
to take a more ‘protective’ stance
given the underlying uncertainty
with the risk estimates generated.”
U.S. Environmental Protection Agency Office of the Science Advisor. 2004. An Examination of EPA
Risk Assessment Principles and Practices. Available: http://www.epa.gov/osainter/pdfs/ratf-final.pdf
Society for
Benefit Cost Analysis
Why Biased Risk Assessment
Is So Toxic to BCA
 Even if every other methodological problem in
BCA is solved…
 BCA results are invalid and highly misleading
 Baseline risks are overstated
 Benefits of regulation are overstated
 CE-analysis yields unreliable rankings
 Rank order would be correct if bias was constant
 Bias is not constant
 Tolerating bias incentivizes bias
 Some interests are made better off if risk
assessment and BCA are discredited
Society for
Benefit Cost Analysis
Flexible Principles:
Activists & BCA
Complaint
Regulation
Sophisticated Sabotage
(2004)
Undercounted
Benefits
Overcounted
Costs
Too hard/unethical to quantify
Benefits
Double-counting
Costs
Discounting
Rate too high
Intergenerational benefits
Overwhelming
Existence value
Massive
Biases in data & analysis
Must be ‘protective’
Society for
Benefit Cost Analysis
Flexible Principles:
Activists & BCA
Complaint
Regulation
Sophisticated Sabotage
(2004)
Roads, Bridges, Dams, etc.
Damming the West
(1973)
Undercounted
Benefits
Costs
Overcounted
Costs
Benefits
Too hard/unethical to quantify
Benefits
Costs
Double-counting
Costs
Benefits
Discounting
Rate too high
Rate too low
Intergenerational benefits
Overwhelming
Overwrought
Existence value
Massive
Negligible
Biases in data & analysis
Must be ‘protective’
Inexcusable always
Society for
Benefit Cost Analysis
Regulatory Checkbook Study of
5-Year Agency Performance
 Procedural compliance




Timely?
Informative?
Interpretively correct?
Forgiving of limited sophistication?
 Substantive compliance
 Accurate: Is response true?
 Honest: Are legitimate complaints admitted?
 Responsive: Is the remedy reasonable?
Society for
Benefit Cost Analysis
Agency Performance on the
Least Demanding Procedural Index
Requests for Correction of Information Quality Errors, 2002-2007:
Selected Agency Response Times (Days)
Agency
N
Cases
Avg
(days)
SD
(days)
Min
(days)
Max
(days)
‘Timely’
%
(days) ‘Timely’
HHS
33
171
128
35
605
60
12%
EPA
41
175
128
17
689
90
15%
Society for
Benefit Cost Analysis
Why Is
Agency Performance So Bad?
 Institutional
 Externally imposed
 Some errors are useful
 Acknowledging error increases litigation and
compromises enforcement
 Legal




WH decision to issue guidance, not promulgate a rule
Limited enforcement of guidance by OMB
No enforcement of guidance by courts
No congressional oversight
Society for
Benefit Cost Analysis
What Can the New
Society for Benefit Cost Analysis Do?
 Publish a journal
 Some theory
 Lots of applications
 Publish an external, authoritative manual
 OMB guidance documents (1990, 2000, 2003) need
improvements that OMB staff cannot make
 An external authority is better
 Peer review federal agency BCAs
 Use information quality levers (insist on objectivity)
 Convene genuinely independent panels
 M*A*S*H approach (‘meatball economics’)
Society for
Benefit Cost Analysis