Evaluating OSHA inspections for
intended and unintended outcomes
Mike Toffel
Associate Professor
Harvard Business School
Liberty Mutual Research Institute for Safety
August 2012
1
May 18, 2012
2
OSHA, a much-criticized agency
Too lenient! (?)
• Inspections too seldom
• Penalties too small
• Lengthy process to adopt new regulations
compromises worker safety
Too costly! (?)
• Stifling job creation / job killer
• Increasing labor costs
• Eroding America's competitiveness
3
4
Challenges evaluating impact of OSHA inspections:
Causality
Most OSHA inspections are not random:
 After accidents and deaths
 When employees complain
If accidents/deaths are rare events, outcomes will
feature mean reversion:
 Problems likely decline after inspections…
but even without inspections
Our approach
 Examine random inspections and compare to a
control group.
5
Challenges evaluating impact of OSHA inspections:
Measuring outcomes
Several studies have relied on company logs
 But inspections can lead companies to improve
logs’ comprehensiveness, increasing reported
injuries
 This cloaks changes in actual injury rates
Our approach
 Rely on workers’ compensation claims
 Annual number of claims
 Annual cost of all claims
6
3 data sources
OSHA IMIS
Inspections
Dun & Bradstreet
Industry
Single-plant
Employment
Sales
WCIRB
Number of claims
Cost of claims
Occupation classes
Payroll
7
Data restrictions
 California
 Cal/OSHA
 Some high hazard industries
 Single-plant firms
(randomization targets)
Data issues
WCIRB data
• MOU between California Department of Insurance
(CDI) and Commission on Health and Safety and
Worker’s Compensation listing us as
subcontractors
• Required to keep data anonymous
OSHA data
• Complex data structure
• Limited documentation
• Many conversations with Cal/OSHA to understand
implementation of randomization process
8
Developing a matched sample
Treatments
 Single-plant establishments randomly selected for a
programmed inspection
 High-hazard industries Cal/OSHA targeted for random
inspection each year
Matched controls
 Find population of single-plant establishments at risk of
random inspection, but not selected
 Exclude if < 10 employees or recently inspected
 For each treatment, select one control:
 Same industry and region
 Closest size
Result: 409 matched pairs
9
Matching led to a balanced sample of
very similar treatments and controls
In the two years before the match year,
the 409 matched pairs had very similar:
 Sales
 Employment
 Payroll
 Credit scores (PAYDEX, Comprehensive Credit
Appraisal)
 Annual number of WC claim
 Annual total cost of WC claims
10
Figure S1: Indistinguishable levels
11
Figure S2: Indistinguishable trends
12
Industry distribution of matched sample (Table S2)
13
Evaluation model
Did treatments experience a greater decline in
annual injuries (or injury-related costs) after
inspections than the controls, examined over the
same time period?
 Fixed effects regression
 Control for establishment characteristics
 Difference-in-differences approach
 Compares two groups over time
14
Randomized inspections reduce annual
injuries by 9.4%
Persistent
15
Randomized inspections reduce annual
injuries by 9.4%
Persistent
16
Randomized inspections reduce annual
injuries by 9.4%
Persistent
effect
17
Randomized inspections reduce annual
injury costs by 26%
Persistent
18
Randomized inspections reduce annual
injury costs by 26%
Persistent
19
Randomized inspections reduce annual
injury costs by 26%
Persistent
effect
20
Unanticipated consequences of inspections?
Inspections (and consequences) cause
interruptions, but are they substantial?
 Sales impact?
 Credit worthiness?
 Employment, payroll?
 Firm survival?
21
Unanticipated consequences of inspections?
22
Unanticipated consequences of inspections?
No difference in employment, payroll, sales (Table 2)
 Tight confidence intervals enable us to rule out that
inspections caused big declines of employment or payroll
No difference in credit ratings (Table S8)
 Late bills, etc. more sensitive to financial burden than firm
death
 Two D&B metrics of financial distress: PAYDEX & CCA
No difference in firm survival (Table S7)
 Approx. 5% of treatments and of controls died
Difference not statistically significant
Result robust to survival regressions
23
Summary of results
Evidence of intended results
 Annual injuries reduced by 9.4%
 Annual injury costs reduced by 26%
No evidence of unintended consequences
 Sales impact
 Credit worthiness
 Employment, payroll
 Firm survival
24
Conclusions
Cal/OSHA is not entirely toothless
• Injuries and injury costs decline a lot
• Social value of very approximately $350,000 per
inspection
• Includes medical costs and lost wages, but not productivity loss,
pain and suffering, etc.
• With many assumptions (similar effects across all states and
inspection type, etc.) translates to $22 B per year nationwide
OSHA is not detectably a job killer
• But we cannot rule out some costs
Levine, David I., Michael W. Toffel, and Matthew S. Johnson. "Randomized
Government Safety Inspections Reduce Worker Injuries with No Detectable
Job Loss." Science 336, no. 6083 (May 18, 2012)
25
Future research in this domain
1. When do random inspections bolster regulatory compliance?
 High vs. low hazardous industries
 Strong vs. weak compliance history
 VPP participants versus others
 Prior OSHA consultation versus not
2. Do random inspections have spillover effects?
 Organizational: bolster compliance at corporate siblings?
 Geographic: bolster compliance at neighboring facilities?
 Regulated domains: bolster compliance with EPA regulations?
3. Your ideas?
 Loss prevention vs. OSHA inspections?
26
Other research…
Environmental reporting & carbon disclosure
• Responding to public and private politics: Corporate disclosure of
climate change strategies, Strategic Management Journal (2009)
• Engaging supply chains in climate change. Working Paper (2012)
• When do firms greenwash? Corporate visibility, civil society scrutiny,
and environmental disclosure, Working Paper (2012)
Environmental ratings
• How well do social ratings actually measure corporate social
responsibility? Journal of Economics & Management Strategy (2009)
• How firms respond to being rated, Strategic Management Journal
(2010)
27
Environmental policy evaluation
Voluntary violations disclosure (EPA Audit Policy)
• Coerced confessions: Self-policing in the shadow of the regulator,
Journal of Law, Economics and Organization (2008)
• Coming clean and cleaning up: Does voluntary self-reporting indicate
effective self-policing, Journal of Law and Economics (2011)
• Making self-regulation more than merely symbolic: The critical role of
the legal environment, Administrative Science Quarterly (2010)
Mandatory performance disclosure
• How firms respond to mandatory information disclosure, Strategic
Management Journal (forthcoming)
Tailpipe emissions fraud
• Competition and illicit quality, Working Paper (2012)
• The role of organizational scope and governance in strengthening
private monitoring, Working Paper (2012)
28
Quality management: ISO 9001 evaluation
• Quality management and job quality: How the ISO 9001 standard for
quality management systems affects employees and employers,
Management Science (2010).
Occupational health and safety
• Randomized government safety inspections reduce worker injuries
with no detectable job loss, Science (2012).
29
Thank you!
Prof. Michael Toffel
Harvard Business School
Morgan Hall 497
Boston, MA 02163
mtoffel@hbs.edu
www.people.hbs.edu/mtoffel/
30