"Developing Surveys to Measure Student
Satisfaction and Learning Outcomes“
Nathan Lindsay & Larry Bunce
February 19, 2014
www.campuslabs.com/blog
@CampusLabsCo #labgab
Like us on Facebook!
There are many types of surveys
to consider…
 Satisfaction Surveys
 Learning Outcome Surveys
 Needs Assessments
 Exit Surveys
 Alumni Surveys
 User Surveys
 Non-User Surveys
 Student/Faculty/Staff/General Public Surveys
 Other?
Steps in survey design
Outline topic(s) and draft items
Choose response formats
Write and edit items
Determine item sequence
Determine physical characteristics of survey
Review and revise survey
Pilot test survey and revise
D
E
S
I
G
N
• Determine your purpose
Begin with the end in mind…
What do you want/need to show?
Why do you need to show it?
Who is the source of your data?
How will you use the data?
Who will need to see results?
The purpose of this assessment is…
• To better understand what the needs of our
veteran students are, and how the new Veteran’s
Center can meet them
• To evaluate if students achieved the stated
learning outcomes of our workshop, and what
additional training needs they have
• To demonstrate to stakeholders the impact that
living in the residence halls has on student
development
• To assess student awareness of services in order
to develop our marketing and communications
plan
D
• Determine your purpose
E
• Examine past assessments
S
I
G
N
Examine past assessments
Those who cannot remember the past are condemned to repeat it.
Philosopher, George Santayana 1905
• Did you use the data?
o If not, what kept you from examining it?
• If you used the data…
o
o
o
o
What was useful?
Was any of the data difficult to analyze?
Were there questions you wished you had asked?
Did any question wording make you unsure of what the data
meant?
• What feedback did you receive from those who
participated?
D
• Determine your purpose
E
• Examine past assessments
S
• Select the appropriate method
I
G
N
Select an appropriate method
Indirect vs.
direct
Quantitative
vs.
qualitative
Formative
vs.
summative
Population
vs. sample
Quantitative
• Focus on numbers/numeric values
• Who, what, where, when
• Match with outcomes about
knowledge and comprehension
(define, classify, recall, recognize)
• Allows for measurement of variables
• Uses statistical data analysis
• May be generalize to greater
population with larger samples
• Easily replicated
Qualitative
• Focus on text/narrative from
respondents
• Why, how
• Match with outcomes about
application, analysis, synthesis,
evaluate
• Seeks to explain and understand
• Ability to capture “elusive” evidence
of student learning and
development
Sampling
Population
Sample
The whole group
A subsection of that group
Example: survey goes to entire
campus
If entire campus: use sparingly and
coordinate with Institutional Research
Example: survey goes to 30% of
campus
Sampling strategies
Simple Random Sample
gives everyone in
sampling population an
equal chance of
selection; a probability
sample
Stratified Random
Sample
breaks total sample into
subpopulations and then
selects randomly from
each stratum
Sample suggestions
Number of Students in Population
Random Sample Size (suggestion)
1,000
278
500
217
350
184
200
132
100
80
50
44
Based on 5% margin of error
Suggestion in: Assessing Student Learning by Linda Suskie
Sample size is the desired number of respondents NOT the
number of individuals invited to participate.
Direct Methods
Indirect Methods
Any process employed to
gather data which
requires subjects to
display their knowledge,
behavior, or thought
processes.
Any process employed to
gather data which asks
subjects to reflect upon
their knowledge,
behaviors, or thought
processes.
Where on campus would you go or
who would you consult with if you
had questions about which courses
to register for the fall?
I know where to go on campus if I have questions
about which courses to register for in the fall.
Strongly agree
Moderately agree
Neither agree nor disagree
Moderately disagree
Strongly disagree
Formative
Summative
• Conducted during the program
• Purpose is to provide feedback
• Use to shape, modify or improve
program
• Conducted after the program
• Makes judgment on quality, worth,
or compares to standard
• Can be incorporated into future
plans
Is a survey right for you?
Pros:
• Include large numbers
• Relatively fast and
easy to collect data
• Lots of resources
available
• Requires minimal
resources
• Fast to analyze
• Good for surface level
or basic data
Cons:
• Survey fatigue and
response rates
• Non-responsive
• Limited in type of
questions asked
• Lacks depth in data
• Skills set in both
designing questions
and analyzing data
properly
Focus Groups
• Group discussions where the facilitator supplies
the topics and monitors the discussion.
• The purpose is to gather information about a
specific (or focused) topic in a group
environment, allowing for discussion and
interaction by participants.
• Similar to interviews, but use when the group
interaction will give contribute to a richer
conversation
Is a focus group right for you?
Pros:
• Helps to understand
perceptions, beliefs,
thought processes
• Small number of
participants
• Focus groups encourage
group interaction and
building upon ideas
• Responsive in nature
• Relatively low costs
involved
Cons:
• Getting participants (think of
time/places)
• Data collection and analysis
takes time
• Data is as good as the
facilitator
• Beware of bias in analysis
reporting
• Meant to tell story, may not
help if numbers are needed
• Data is not meant to be
generalizable
Quick, 1-Minute Assessments
On a notecard, write a real-world example of how
you can apply what you learned.
Pass an envelope containing notecards with quiz
questions. Students pick one and have 60
seconds to answer and pass along.
At the end of a workshop, ask students to write
down 1 thing they learned, and 1 lingering
question.
Is a quick assessment right for you?
Pros:
• Provides a quick summary of
take away from student
perspective
• Quickly identifies areas of
weakness and strengths for
formative assessment
• Can track changes over time
(short-term)
• Non-verbal (provides
classroom feedback from all
students)
• Captures student voice
• Short time commitment
• Provides immediate feedback
Cons:
• Non-responsive
• Short (so you may lose
specifics)
• Sometimes hard to interpret
• Need very specific prompts
in order to get “good” data
• Plan logistics ahead of time
and leave time during
program/course
• May need to be collected
over time
Mixed Methods
Look for same results across multiple data
collections
Build upon or relate results from one
assessment to another
Use data from one method (e.g., a survey) to
inform another method (e.g., a focus group)
Able to increase scope, number, and types of
questions
D
• Determine your purpose
E
• Examine past assessments
S
• Select the appropriate method
I
G
N
• Identify ethical/logistical considerations
Identify ethical/logistical considerations
 Do you have the necessary resources and brain
power?
 Do you need to go through IRB?
 Do you need to identify respondents for follow up,
merging of data, tracking of cohorts, or pre/post
analysis?
 Do you need to include demographic questions to
drill down or separate data?
 Who needs to be involved at planning stage to
avoid problems when results are in? Does anyone
need to approve the project?
 Are there any political issues to be aware of?
D
• Determine your purpose
E
• Examine past assessments
S
• Select the appropriate method
I
G
N
• Identify ethical/logistical considerations
• Generate the best question and answer format
What to consider
• Scales that match
• Mutually exclusive
• Exhaustive
• Neutral/Not applicable/Non-response options
•
•
•
•
Choose not to respond
Not applicable
No opinion
Neither ___ nor ___
Don’t know
Unable to judge
Neutral
Pairing Question Text with Answer Choices
Question text should be compatible with the answer
choices
e.g., “How satisfied were you with the following?”
Excellent
Good
Fair
Meals at the
conference
Location of the
conference
Date of the
conference
e.g., “Did you enjoy the Black History Month speaker?”
Strongly agree
Somewhat agree
Somewhat disagree
Strongly disagree
Poor
Mutually Exclusive Answer Choices
Response options should never overlap
e.g., How many hours per week do you work?
0-10
10-20
20-30
30-40
Response options should exist independently of
one another
e.g., Which of the following statements describes your peer mentor?
He/she is helpful and supportive
He/she is difficult to get a hold of
Exhaustive Answer Choices
Respondents should always be able to choose
an answer
e.g., How often do you use the University website?
Daily
2-3 times a week
Weekly
Monthly
Non-response options
Always consider a non-response option
Choose not to respond
Not applicable
No opinion
Neither ___ nor ___
Don’t know
Unable to judge
Neutral
Customize the non-response option when possible
e.g., How would you rate the leadership session?
Excellent
Good
Fair
Poor
Did not attend
Pitfalls to avoid
Socially desirable responding – based on social norms
– Can never be eliminated
– Consider sensitive topics like race, drug and alcohol use, sexual activity,
and other areas with clear social expectations
Leading questions – suggesting there is a correct answer
e.g., “Why would it be good to eliminate smoking on campus?”
Double-barreled questions – asking more than one question
e.g., “What were the strengths and weaknesses of orientation?”
Double negatives – including negative phrasing which makes
responding difficult
e.g., “I do not feel welcome in my residence hall.”
Response Formats
• Open ended responses
– Free response - text
– Numeric
– Yes/No with please explain
• Types of multiple choice responses
–
–
–
–
–
Yes/No
Single response
Multiple response (e.g., Check all that apply, Select 3)
Ranking
Scales
Yes/No
When to Use:
– There is no response between “Yes” and “No”
e.g., “Have you ever lived on campus?”
– You consciously want to force a choice even if other
options might exist
e.g., “Would you visit the Health Center again?”
When Not to Use:
– There could be a range of responses
e.g., “Was the staff meeting helpful?”
Single response
When to Use:
– All respondents would only have one response
e.g., “What is your class year?”
– You consciously want to force only one response
e.g., “What is the most important factor for improving the Rec
Center?”
When Not to Use:
– More than one response could apply to respondents
e.g., “Why didn’t you talk to your RA about your concern?”
Multiple response
Options: “Check all that apply” or “Select (N)”
When to Use:
– More than one answer choice might be applicable
e.g., “How did you hear about the Cultural Dinner?” (Check all that apply)
– You want to limit/force a certain number of responses
e.g., “What were your primary reasons for attending?” (Select up to 3)
When Not to Use:
– It’s important for respondents to only be associated with one
response
e.g., “What is your race/ethnicity?”
Ranking
When to Use:
– You want to see the importance of items relative to one
another
e.g., “Please rank how important the following amenities are to you in
your residence: (1=most important)”
– You are prepared to do the analysis and interpretation!
When Not to Use:
– You want to see the individual importance of each item
e.g., “How important are the following amenities to you?”
Scales
When to Use:
– You want to capture a range of responses
e.g., “How satisfied were you with your meeting?”
– When you would like statistics
e.g., 4 = strongly agree
3 = agree
2 = disagree
1 = strongly disagree
When Not to Use:
– The question is truly a Yes/No question
e.g., “My mother has a college degree.”
Scales
Unipolar – no negative
A great deal
Considerably
Moderately
Slightly
Not at all
Bipolar – positive or negative (with
or without a midpoint)
Very safe
Somewhat safe
Somewhat unsafe
Very unsafe
Consider…
• Number of points
• Inclusion of neutral
• Whether labels are needed
• Order (e.g., 1, 2, 3, 4, 5 or 5, 4, 3, 2, 1)
Recommended Scales
D
• Determine your purpose
E
• Examine past assessments
S
• Select the appropriate method
I
• Identify ethical/logistical considerations
G
• Generate the best question and answer format
N
• Note the purpose for each data point
Note the reason for each data point
• Bubble next to question
• Compare against
purpose to identify gaps
• Look for overlap
• Eliminate “nice to know”
• Help with ordering
• Retain for analysis step
DATA COLLECTION
METHODS
Paper surveys
Things to consider
• Captive audience
• Administrator available for
questions
• No technology issues or
benefits
• Data entry necessary
Web surveys
Things to Consider:
• No data entry
• Technology issues and
benefits
• Immediate results
• Can be anonymous or
identified
• Not a captive audience
Data Collection Methods
Pros
Web
Mobile
Paper
 No data entry
 Accuracy is excellent
 Technology benefits (e.g., display
rules, required questions)
 Immediate results
 Anonymous
 No data entry
 Accuracy is good
 Technology benefits (e.g., display
rules, required questions)
 Captive audience
 Administrator is available for ?s
 Captive audience
 Administrator is available for ?s
 No technology issues
Cons




Audience is not usually captive
Possible misinterpretation (can’t ask ?s)
Technology issues
Response sample unrepresentative




Technology issues
Response sample unrepresentative
Limited formatting
Anonymity is questionable




No benefits of technology
Accuracy can be compromised
Data entry necessary
Anonymity is questionable
SURVEY FATIGUE
General information
• Survey response rates have been falling
o Difficult to contact people
o Refusals to participate increasing
• Strategies for correcting low response rates:
o Weight the data for non-response
o Implement strategies to increase response rates
Non-response may not be
random
• Correlation exists between demographic characteristics
and survey response
• Higher response has been found among certain subpopulations:
o Women
o Caucasians
o High academic ability
o Living on campus
o Math or science majors
• Research is inconsistent
IMPROVING
RESPONSE RATES
Specific techniques
•
•
•
•
•
•
•
•
•
•
•
Survey length
Preannouncement
Invitation text
Reminders
Timing of administration
Incentives
Confidentiality statements
Salience
Request for help
Sponsorship
Deadlines
Survey length
• Greater attrition at 22 questions or 13 minutes
• What to consider:
–
–
–
–
–
Excluding “nice to know”
Eliminate what you already know
Outlining how results will be used
Number of open-ended questions
Number of required questions
Invitations
•
•
•
•
•
•
•
•
Importance/Purpose
Relevancy to respondent
Request for help
How and by whom results are
used
How long it will take to respond
Deadline
Incentives/Compensation
Contact information
Timing of contact/administration
• Avoid busy times or
holidays
• Send email/
preannouncement 2-3
days prior to survey
mailing
• First half of
semester/term may be
better if you are
surveying in an
academic environment
Piloting
• 1. Take it as if you were respondent
• 2. Seek reviews from colleagues with no prior
knowledge
• 3. Administer to sample of actual population
being studied
– Focus group
– Questions at end of survey
– Observing
Reliability & Validity
• Reliability – yielding the same results
repeatedly
– Test/Re-test – consistency over time
– Inter-rater – consistency between people
• Validity – accurately measuring a concept
– Internal – confidence results due to independent variable
– External – results can be generalized
– Face validity – does this seem like a good measure?
• If a survey is valid, it is almost always reliable!
USING THE
SURVEY RESULTS
PR or advertising campaign
Results in survey invitations
Some Final Advice
• Google your area to see what other
surveys have been conducted
• Contact Larry Bunce (Director of
Institutional Research) or Nathan Lindsay
for help in designing your survey
• Online surveys should be coordinated
through the Office of Institutional Research
QUESTIONS?
Nathan Lindsay
Assistant Vice Provost for Assessment
University of Missouri-Kansas City
lindsayn@umkc.edu
816-235-6084