C I V I T A S T R A I N I N G
7 M A Y 2 0 1 4
F L O R E N C E , I T A L Y
The Long and Short of It
Eric N. Schreffler
Transport Consultancy
ESTC San Diego, CA USA

6.
7.
8.
9.
10.
1.
2.
3.
4.
5.
My experience
Why evaluate? and for whom?
Forecasting vs. measurement
Principles of good evaluation
Overall framework for MM evaluation
U.S. examples – California and Washington D.C.
Transitioning from short-term to long-term evaluation
The politics of MM evaluation
Common mistakes
Top ten tips for effective evaluation

 MM/TDM evaluation for over 30 years
 Written/contributed to several seminar guidebooks



Involved in several EC-funded projects
MOST – MOST MET
MAX – MaxEVA


CIVITAS GUARD
CIVITAS II –final brochure
 CIVITAS PLUS training
 Currently leading evaluation of TDM component of 6
U.S. Urban Partnership Agreement projects

 Good management practice!
 Measure progress against objectives
 Output or outcome objectives
 Program or public policy objectives
 Integral to performance-based planning

 Benchmark program against peers
 Compare cost effectiveness to other solutions
 Marginal cost to accommodate a car vs. cost to reduce a car trip
 e.g., LA Metro evaluation performance-based planning
 Satisfy funding entities
 Satisfy policy or oversight boards

Know your evaluation,
Love your evaluation,
One day your evaluation may save your (life) program.
Don’t be afraid of evaluation!!!




Evaluation can be defined as:
A priori estimation of expected impacts (forecasting)
Ex post measurement of outcomes (results)
 Forecasting
 Predicting what could/should occur
 Using comparative case studies, models, sketch planning, SWAG
 Measurement
 Quantifying what has actually occurred
 Using counts, surveys, etc.

P
R
O
S
T



Important to plan for evaluation
Have plan to frame approach and tasks
 Should include:
 Purpose and objectives
 Data collection methods
 Analysis methods
 Budget
 Schedule
 Reporting
 Use conceptual framework - MaxEva

Assessment of Services
Provided
Assessment of Mobility
Options
Offered
Overall Effects
C
D
A
B
E
F
G
H
I
Project activities and outputs
Awareness of mobility services provided
Usage of mobility services provided
Satisfaction with mobility services provided
Acceptance
Take up of mobility option offered
Satisfaction with the mobility option offered
Long-term
System of mobility option offered attitudes and behavior impacts
MaxEva – The EPOMM Evaluation Tool – www.epomm.eu/maxeva

European Example – Sweden
Employee Public Transit “Test Rider” Pass Pilot
Assessment of Services
Provided
Assessment of Mobility
Options
Offered
Overall
Effects
A Outputs: posters, intranet info, meetings, 1,000 employees
B Awareness : 48% of public transit ticket offer (480 emp)
C Usage : 28% participated in meeting, sought info
D Satisfaction : 80% satisfied with info and pilot concept
E Acceptance : 6% signed a contract to participate (54 emp)
F Take up: 85% of these used PT; 5% used before
G Satisfaction: 90% of participants satisfied with PT
H Long-term : after one year, 40% still using PT
I System impacts: car use  110K km; CO2  20 tons/year
MaxEva – The EPOMM Evaluation Tool – www.epomm.eu/maxeva

TDM and Highway
Reconstruction


Reconstruction mitigation
$730,000/year for TDM
 Three elements:



More commuter express buses
Vanpool promotion/subsidies
Carpool fuel incentive
 Surveyed all participants
 Removed 310 cars daily
 Carpool incentive most cost effective ($3.36/VTR/day)
US101 Cuesta Grade CA

Regional TDM Program




Triennial evaluation since 1997
Consistent approach (MaxEva)
Evaluate total impacts = travel, emissions, energy (e.g.,
118K fewer car trips per day)

Evaluate separate impacts
Ridematching







Employer outreach
Mass marketing
Guaranteed ride home
Bike to work
Carshare
Telework
Incentives
Commute Connections
Washington D.C.



Short-run generally covers one year or duration of project funding
Uses before and after data year or duration.



Long-term can include projecting impacts into future (BCA, lifecycle)
Or can involve time series data for program over years
Key is planning and consistency

RESEARCH
There may be pressure to:
- subvert findings
- document desired outcomes
- spin the results

1.
2.
3.
4.
5.
6.
Putting off evaluation
Underfunding evaluation
Assuming “projected” = “actual” results
Incorrectly comparing projected to actual
Projecting target group findings to entire population
Ignoring causality and externalities

7.
8.
Confusing outputs with outcomes
Changing methods/assumption mid-evaluation
9.
Assuming all mode shift from drive alone
10.
Ignoring access mode to new travel options
11.
Ignoring non-response in surveys
12.
Giving into pressure to change findings

1.
Get help, at least the first time
2.
Plan, plan, plan
 Plan for evaluation
 Performance-based plan
 Stick to the plan
3.
Budget for evaluation

4.
Keep findings simple!
5.
Don’t be afraid of what you may find!
6.
Be confident!
No whining!

7.
Seek local default factors (e.g. trip length)
8.
Use standardized methods/tools/guidance
9.
Learn to talk like an engineer

10.

MaxEva – The EPOMM Evaluation Tool www.epomm.eu/maxeva
Evaluation Matters: A Practitioners’ Guide to Sound Evaluation for
Urban Mobility Measures www.civitas.eu/content/evaluation-matters
Integrating Demand Management into the Transportation Planning
Process: A Desk Reference (Chapter 9) www.ops.fhwa.dot.gov/publications/fhwahop12035/index.htm
Canadian TDM Measurement Toolbox www.tc.gc.ca/media/documents/programs/tdm-toolbox.pdf

Eric N. Schreffler
Transportation
Consultant
ESTC
San Diego, CA,
USA estc@san.rr.com
Grazie per la vostra attenzione!