University of Florida Survey
Research Center
Bureau of Economic and Business Research
(BEBR)
• Established in 1929
• Population Program (State estimates and
Projections)
• Publications (Florida Statistical Abstract)
• Economic Analysis (Ad hoc studies, such as Florida
Price of Living Index)
• University of Florida Survey Research Center
(Established 1979)
Some of Our Clients
Selected Health Related Surveys
Survey
Population
BRFSS Cell Phone Survey
FL Cell Phone users
College of PHHP--CAHPS:
Eval. Medicaid for Reform
Implementation Plan
Children enrolled in
Medicaid from Broward,
Duval, Baker, Clay and
Nassau counties
Sample
Type
Completed
Interviews
Number of
Attempts
Total Sample
Released
Total
Number of
Calls
Response Rate
Refusal
Rate
% of Refusals
Converted
RDD Cell
Phone
300
12
7,200
24,569
0.32
0.22
3.00%
Listed
3091
21
14,695
68,006
0.38
0.07
14.43%
Listed
422
13
3,357
13,188
0.62
0.08
6.45%
Listed
3714
20
18,397
72,553
0.72
0.06
16.85%
Listed
2205
15
20,000
94,521
0.37
0.07
2.52%
FL ICHP--FL KidCare eval.
Adults who received
substance abuse
treatment from state
Children enrolled in a
Medicaid HMO
FL children (11-18 yrs
old)
Children enrolled in FL
KidCare
Listed
1714
31
7,859
70,600
0.57
0.15
9.64%
FL ICHP--Eval. health care
in Partners in Care
Children enrolled in
PIC(terminally ill
children)
Listed
140
33
352
2,904
0.76
0.12
23.75%
TX STAR+PLUS--Eval.
elderly Medicaid enrollees
with disabilities
Adults enrolled in
STAR+PLUS
Listed
955
34
9,128
47,618
0.26
0.14
13.80%
Listed
422
11
3,953
14,507
0.66
0.07
7.69%
Listed
4004
20
22,209
90,889
0.66
0.12
15.08%
Listed
2255
50
20,133
173,000
0.77
0.05
11.65%
FL DCF --Eval. substance
abuse programs
FL AHCA--Report Card
FL DOH--Eval. antitobacco program
FL AHCA--Report Card
Adults who received
substance abuse
treatment from state
Children enrolled in a
Medicaid HMO
TX STAR--Medicaid HMO
health care eval.
Adults in a Medicaid
HMO under TX STAR
FL DCF --Eval. substance
abuse programs
Organizational Chart
Director :
Chris McCarty
Application
Developer:
Eric Lavigne
Project Manager:
Scott Richards
Field Director:
Gene Averkiou
Project Manager:
Mark Girson
Payroll &
Personnel:
Stefanie Jones
Department Supervisors and Graduate Assistants
Hiring – Scheduling – Training – Mentoring – Attendance
Translation – Programming –Evaluations
Interviewers
Refusal converters – Spanish interviewers – General interviewers – Mentors – Supervisor Assistants
Breakdown of OPS Staff
•
•
•
•
•
11 Supervisors ($13.00/hr)
10 Assistant Supervisors/Interviewers ($9.00)
15 Refusal Convertors ($9.50/hr)
25 Spanish Speaking Interviewers ($9.00/hr)
100-150 General Interviewers ($8.50/hr)
– Bonus based on productivity for interviewers and
listening to recordings for supervisors
Refusal convertors
• Our most skilled interviewers
– Higher pay, higher productivity standard
– Allowed to call our most difficult cases
• Promotion process
– Minimum of 208 hours of interviewing
– Maintained higher productivity standards
consistently for last 40 hours
– No penalty boxes in last two months
Shift Staffing
• 1 Shift Supervisor
• 0-4 Assistant Supervisors
• 0-5 Mentors
Hours of Operation
• Mon-Friday 9am-12am
• Saturday 12pm-9pm
• Sunday 3pm-12am
FACILITIES
Recently expanded location
Dedicated network backbone
Security
• Building security guards and alarm system
• Dedicated on-site servers kept in a locked room
• Off-site backups to prevent data loss
• Virus and malware detection
• All network infrastructure protected with filtered power
and battery backup
• Unauthorized access detected 24 hours a day, seven days a
week by IT staff
Telephone system
•
•
•
•
•
93 telephone interviewing stations
69 outgoing lines with VOIP overflow option
Extra server in case first server fails
Asterisk VOIP software
Reducing background noise
– Headsets
– Noise absorbent dividers and ceiling
OTHER SERVICES
• Focus Groups
• Data entry
• Transcription
Software
• Wincati – Sample management software
(Sawtooth)
• Ci3 – Questionnaire authoring software
(Sawtooth)
• Scheduler – Lab management software
WINCATI
Sample Management Software
Survey Specifications
Example Record
Disposition Report
Response Rate
• Proportion of completed interviews out of all
eligible respondents
• What is eligible?
– Listed sample -- All cases may be eligible
– RDD sample – There are rules to determine which
cases to count
AAPOR RR1
I
RR1 
( I  P)  ( R  NC  O)  (UH  UO)
•
•
•
•
•
•
•
I = Complete interview
P = Partial interview
R = Refusal and break-off
NC = Non-contact
O = Other
UH = Unknown if household/occupied housing unit
UO = Unknown, other
Monthly survey response rates by
county
Ci3
Questionnaire Authoring Software
Lab Management and Quality
Control
Scheduler
Interviewers receive feedback
Reviewing recorded calls
Reviewing call recordings
• Automatically select random calls – targeted at
mistakes and policy violations
• Supervisors listen to and evaluate calls
• Determine whether interviewer correctly
described the outcome of the call
• Check for policy violations
Real-time call monitoring
• Assistant supervisors listen to calls during a
shift
• Evaluate six skills: verbatim, probing,
clarifying, feedback, pace, and voice
• Meet with interviewer during same shift to
discuss their performance
Real-time call monitoring
Miscoding Penalty Box
• Receive penalty box for
one call with low rating
on call skills
• Can also receive penalty
boxes for frequently
choosing the wrong
description of a call’s
outcome
• May be terminated for
third penalty box
Productivity
Top performer
This interviewer needs help
Measuring employee productivity
• Compare completed interviews per hour to average
for survey
• Average across time spent on each survey
• Interviewer score calculated over large 45-day
window to smooth out bad days
• Survey average calculated over smaller 15-day
window for higher accuracy
Rewarding top performers
• Productivity incentive for keeping score above
100%
• Incentive proportional to hours worked and
how far above 100%
• Some top performers earn over $600 in
productivity incentives for a 2-month
incentive period
Penalty box
• Required to keep productivity score above
– 84% for general interviewers
– 91% for spanish-speaking interviewers
– 107% for refusal convertors
– 117% for spanish-speaking refusal convertors
• Hold most skilled to higher standard
• Meet with supervisor after second box
• Terminated or demoted after third box
How Much Does A Survey Cost?
Negotiating a survey
• Estimate productivity per interviewer hour
• We try to get $26 per interviewer hour
• A survey where we complete one interview per hour
costs $26/complete
• A survey where we complete one interview every
two hours costs $52/complete
Relation between hours worked and cost
per hour
Price per interviewer hour we must charge to break even
600
500
400
300
200
100
Total interviewer hours worked
96000
91000
86000
81000
76000
71000
66000
61000
56000
51000
46000
41000
36000
31000
26000
21000
16000
11000
6000
0
1000
Price per interviewer hour
700
Survey Cost Structure
• Major factors that influence Survey Costs include:
–
–
–
–
–
Population
Mode
Length
Sample
Subject/Salience
Survey Mode Characteristics
2010 Surveys by Mode
Results from AASRO survey
Telephone survey projects
Mail survey projects
Web survey projects
In-person survey projects
Mail and web projects
Telephone and mail projects
In-person and telephone projects
In-person and mail projects
Telephone and web projects
Mean number of
projects
NonMembers
members
11.8
9.1
4.0
1.3
7.7
3.0
1.5
1.0
2.2
1.1
1.6
0.4
1.0
0.3
0.7
0.1
1.2
1.3
Proportion (%) of all
projects
NonMembers
members
35
51
12
7
23
17
4
6
6
6
5
2
3
2
2
1
4
7
Mean number of
projects
Small
10.4
2.4
4.8
0.7
1.6
1.0
0.4
0.1
1.3
Big
16.8
6.2
11.4
2.4
3.1
2.2
2.3
1.9
1.6
Proportion (%) of all
projects
Small
43
10
20
3
7
4
2
0
5
Big
33
12
22
5
6
4
4
4
3
Effect of Population on Cost
• How easy is it to identify, locate
and/or select the group being
studied?
• Survey of the Elderly (Elder)
Statewide study of Florida Seniors
Selection & screening: simple
$38/complete
Cost per Com plete
$60
$55
$50
$40
$38
$30
Oral Cancer Awareness (Port9)
Oversamples African Americans, age
25+ in selected counties
Selection & screening: challenging
$55/complete
$20
$10
$0
Port 9
Elder
Effect of Length on Cost
• Generally, a shorter survey =
lower costs
• Survey of Star Plus program
members (Star)
36 min
$53/complete
Cost per Com plete
$60
$53
$50
$40
$30
$26
• Consumer Confidence Index
(CCI)
11 min
$26/complete
$20
$10
$0
St ar
CCI
Effect of Mode on cost
• Type of survey– telephone,
mailout, etc.-- affects survey
cost.
Effect of Mode on Cost
$30
$28
$25
• Dept. of Fine Arts Evaluation
Web survey $10/complete
$20
$15
• FL Dept of Transportation Survey
of Drivers
Telephone
$28/complete
$10
$10
$5
$0
Art s
FDOT
Effect of Subject/Salience on Cost
What is the survey about? How important is
it to the respondent? Surveys with
threatening questions (disability, alcohol
abuse, etc.) may increase costs.
Effect of Subject on Cost
$80
$75
$70
Florida BRFSS
Subject: Health practices & care
Inc. threatening questions
$75/complete
$60
$50
$40
$30
$25
CAHPS
Subject: Health care
No threatening questions
$23.50/complete
$20
$10
$0
BRFSS
CAHPS
Effect of Sample on Cost
Sample type (Random Digit Dial vs.
Listed sample) and quality can
substantially affect costs.
Records per com plete
12
10
10
8
HMO Report Card Survey (HMO)
Listed sample
3.2 recs/comp.
6
4
Consumer Confidence Index (CCI)
Random Digit Dial 10 recs/comp.
3.2
2
0
HMO
CCI
Sampling
Telephone survey
sampling schemes
RDD – random digit dialing
• Numbers are created using information
about existing area codes (GENESYS)
• Best for obtaining a representative sample
• Less productive than listed sample
SAMPLING SCHEMES
Targeted RDD sample
• Overlay U.S. census tracts with working
telephone banks
• Targets specific subgroups
• Regions
• Ethnic groups
• Levels of income
SAMPLING SCHEMES
• Predetermined lists
• Efficient
• Cost effective
• Quality depends on the frequency of
updates
Sample size estimate
H
z p(1  p)
n

1.96 .5(1  .5)
n

Margin of error
Margin of Error by Sample Size
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1
101
201
301
401
501
601
Sample size
701
801
901
1
n
Do Not Call Lists
• National Do Not Call Registry (www.donotcall.gov)
• Only calls that solicit the sales of goods or services
are covered
• Surveys and charities are exempt
• Respondents usually do not know that
Common sources of lists
• Telephone numbers and households listed in telephone
directory
– Can pull national sample
– Unlisted numbers vary a lot by geographic area and respondent
characteristics
• Drivers licenses from state Department of Motor Vehicles
– Must select samples by state, and states vary in laws regarding drivers
licenses
– Data may be old as people move without informing Department of
Motor Vehicles
– Nor every one drives and there are biases (old and young, people in
urban settings with public transportation and high insurance costs)
Common sources of lists (continued)
• Voter Registration
– Potentially more updated than driver’s license database
– Not everyone votes – potentially very biased unless survey concerns
potential voters
• Lists from behavioral surveys and credit card evaluation
– Usually expensive
– Can often select people with particular characteristics (e.g. smokers)
– Potentially biased based on source
• Member and User Lists such as patient records, HMO
membership, recipients of Temporary Assistance for Needy
Families (TANF)
– Source is often variable in maintaining records (e.g. HMOs do not have
common database practices for recording membership data)
Telephone sample options and costs
•
$300 minimum on all orders
•
Random Digit Dialing (RDD)
uncleaned – $.04/record
•
RDD with business purging from
yellow pages – $.05/record
•
RDD with business purge and
attended dialing using automated
detection – $.09/record
•
Claritas Modeled data
•
Experian Behavior Bank – $.35/record
Example:
Consumer Confidence Survey
What is Consumer Confidence?
• Telephone survey based measure calculated each month from
the responses of approximately 500 Florida households
• Designed to predict consumer spending which accounts for
70% of Gross Domestic Product (GDP)
• Barometer of public opinion about economy
How is consumer confidence measured?
Consumer Sentiment from the
University of Michigan
Consumer Confidence from the
Conference Board
• 500 Random Digit Dial telephone
interviews a month across the
U.S.
• Approximately 3,500 mail out
surveys each month across the
U.S.
• Preliminary release second Friday,
final release last Friday
• Released last Tuesday
The University of Florida Survey Research Center uses
the University of Michigan method for Florida, but
releases the results the last Tuesday of each month
Comparison of Index Questions
University of Michigan Index of
Consumer Sentiment
Conference Board Consumer
Confidence Index
Current Conditions
Overall Buying Conditions
Business Conditions in Area
Current Personal Financial Condition
Current Job Availability in Area
Future Expectations
Business Conditions Over Next 12 Months Business Conditions in 6 Months
Business Conditions Over Next 5 Years
Job Availability in Area in 6 Months
Personal Financial Condition in 12 Months Total Family Income in 6 Months
Comparison of U.S. Consumer Sentiment and Consumer Confidence
120.0
160.00
140.00
100.0
120.00
80.0
100.00
60.0
80.00
60.00
40.0
40.00
20.0
20.00
Consumer Sentiment
Consumer Confidence
Jan-10
Jan-08
Jan-06
Jan-04
Jan-02
Jan-00
Jan-98
Jan-96
Jan-94
Jan-92
Jan-90
Jan-88
Jan-86
Jan-84
Jan-82
Jan-80
0.00
Jan-78
0.0
Comparison of Florida and U.S. Consumer Sentiment
What are some drivers of Consumer
Confidence?
• Economic
Shocks
• Prices
• Stocks
• Employment
Our Survey Design
• Approximately 5,000 Random Digit Dial phone numbers released
each month
• Phone numbers are released proportionate to households by
county
• A preliminary consumer confidence release is made the last
Tuesday of each month with at least 400 completed interviews
• By the end of the month we collect at least 500 interviews
• Faculty can add questions to our monthly survey
News sources that have used our data