the PowerPoint file presented in this video

advertisement
Survey 101: Survey
Questions and Responses
Paula Harmer, Director of Institutional
Research, Office of Academic Affairs
November, 2012
Outline of Workshop
•
•
•
•
Review “Why use surveys”?
Outline of survey preparation steps
Overview of question and response types
Guidelines for writing survey questions and
selecting response sets
• Practicum: identify survey issues and revise
questions & responses
Fink, Arlene. How to Ask Survey Questions volume 2, Sage
Publications, 2003.
Why use surveys?
• Obtain feedback with a fast turnaround
• Self-administered ones are resource efficient
compared to phone/in person interviews
• Anonymity/confidentiality allows respondents
various levels of protection for candid responses
• Accepted research methodology for applied and
academic research
• Can gather holistic feedback of total population
versus a select few
Survey – Getting Ready for Launch
• Step 1: Develop a purpose and specific objectives for your survey. Examples:
How satisfied are recent alumni with their educational experiences and
learning support at Regis University? Does student engagement impact
course learning outcomes?
• Step 2: Define the human population(s) being surveyed. Sample/whole
population?
• Step 3: Select survey type. Self administered? Paper? Online? Phone? In
person interviews or focus group?
• Step 4: Plan number of questions that measure objectives, and how much
time respondent will have to complete survey.
• Step 5: Timing of the survey – Once? Twice? At a time point in student’s life
cycle? 6 months after graduation? Pre-test/Post-test?
• Step 6: Determine security level: if survey is identifiable, anonymous,
confidential
Getting Ready + Administration
• Step 7: Permissions – Does the respondent population require
authorization for recruitment? Eg. Regis IRB? What level of risk do
respondents have for participation, and how can you minimize that
risk?
• Step 8: Survey data collection – online or paper-based surveys?
Automated or manual data entry? Paper scanning?
• Step 9: Write your survey questions geared to answer the research
question purpose and objectives, following writing guidelines. You
MUST formulate questions so respondents may answer easily and
accurately.
• Step 10: Review and test survey questions with potential
respondents and content experts.
Why Questions are Critical
• For self-administered surveys, you must be sure
that the questions can be understood by
respondents without assistance from survey
team
• Using pretested & proven questions is best
• Enhances validity (are you measuring what you
think you are measuring) and reliability (if the
respondent took the same survey again, would
they respond the same way?)
Guidelines for Writing Survey
Questions
One concept per question
If you include more than one concept,
respondents will be torn between responding to
differing concepts, giving poor quality data.
Poor: How satisfied were you with the availability of
faculty outside of class and opportunities to engage in
research?
Better: How satisfied were you with availability of faculty
outside of class?
Ask Precise Questions
• Logically related to survey objectives
• Asks precise and unambiguous questions
Eg. Use specific time periods rather than ‘typical’
Vague: How would you describe your emotional well being?
Better: How would you describe your emotional well being in
the past three months?
• More detail will increase reliability of answers
• Respondent recall guideline: Major life events (more than 1 year is
OK) versus unimportant events (less than 1 month is best)
Use simple & conventional
language
• Use complete sentences
• Do not confuse respondents by using disciplinary jargon or
acronyms/abbreviations. This measures comprehension, not perceptions or
self-reported behavior. This is not a test!
Q: The faculty member utilized a variety of pedagogical approaches in
this course.
Q: The faculty member utilized a variety of teaching and learning
approaches in this course.
Q: Indicate the types of learning approaches utilized in the course:
Lecture
Group Discussion
Games
Lab
Seminar
Case studies
Use simple & conventional
language
• Avoid negative questions if possible
• Avoid loaded questions if possible
• Avoid biased words or phrases that trigger
emotional response
Select Open or Closed Questions
• Closed questions have pre-specified responses, are more
reliable and efficient, good for ranking and rating, allows
statistical analysis
• Open questions allow responses in own words, require
qualitative analysis. Good for exploration or quotes
Note: requires more effort to code and analyze, and
fewer respondents will make the effort to respond, so
don’t make the questions required to complete the
survey
• Good practice to have majority closed Qs with a few open
Qs
Closed Question Response Types
• Categorical: responses are categories, no numerical or
preferential values, they are true or false.
Eg. Male & Female; Alumni/Student/Faculty/Staff;
• Categories should be mutually exclusive; should include
all possible answers (inclusive); should exhaust possible
categories (exhaustive)
• Ordinal: scale highest to lowest, rate or order items Eg
age categories, high-low scales, etc
• Numeric: asks for number Eg age in years
Select Best Response
Sets
• Categorical response sets should be inclusive of all possible answers
Eg. Select the reasons you chose to enroll in your program:
academic reputation/geographical location/faculty/ tuition /
scholarship opportunities / other
• Categorical: It is good practice to use a ‘I don’t know/not applicable’
or ‘other’ when it makes sense to, in case a question has little or no
relevance to a respondent – helps prevent survey abandonment by
respondents
• On the other hand, it can encourage ‘lazy’ responses
• Ordinal: Good for analysis, balances between negative and positive
responses
Ordinal Response Scales
5-point scales
URL (as of Nov 28, 2012):
http://www.utexas.edu/academic/ctl/assessment/iar/teaching/plan/method/survey/writing.php
Ordinal Response Scales
• Use a meaningful scale
• Use a balanced scale – 4 or 5 point is the ‘norm’
• Provide a neutral category such a ‘Don’t Know’ only if it makes sense
1
2
3
4
5
Endorsement
Definitely
true
True
Don’t Know
False
Definitely
False
Frequency
Always
Very Often
Sometimes
Almost
Never
Never
Intensity
None
Very Mild
Mild
Moderate
Severe
Influence
Big
problem
Moderate
problem
Small
problem
Very small
problem
No problem
Comparison
Much more
than others
Somewhat
more than
others
About the
same as
others
Somewhat
less than
others
Much less
than others
Ordinal Response Scales
• Put the socially undesirable/negative response
first
• Tell respondent how and where to mark the
responses
• When possible, group similar questions &
response scales together
• Rank order scales – best not to exceed three
choices in surveys
Numeric Responses
• Responses are a number Eg. height, weight, age in years,
number of books, etc.
• Numeric scales/measures are when respondents select a
number along a scale, such as a pain scale from 0 (No Pain)
to 10 (Worst Possible Pain).
Questions Measuring Knowledge,
Behaviors and Attitudes
• Attitudes are complex and challenging to
define and measure
• Psychometricians use methods to examine
statistical properties of attitudes
• Knowledge and behavior is different
Eg What a person feels about gun control, knows
about gun control, and does about can be very
different
• Finding and developing attitudinal questions
and scales is hard work!
Behavioral Questions
• Measures what respondents actually do
• Involve time, duration and frequency
• Guideline: Major life events can be recalled
longer than one year
Poor: In the past year, which of the following
items did you purchase?
Better: In the past 3 months, which of the
following items did you purchase?
Knowledge Question
• A correct answer is available
• Used to assess respondents’ knowledge of a
topic
• For example, if knowledge of a group is slim, an
educational campaign may be warranted
Demographic information
• Asking a respondent facts about age, income,
race/ethnicity, education, gender, etc.
• If known, compare respondent sample with total
population to gauge ‘representativeness’
• Allows comparisons Eg adult versus
traditionally-aged students
• Sensitive information (Eg income) should be
asked at end of survey and should be optional
Identify Issues and Fix! Exercise
Each of the following survey questions & responses have a problem(s).
Identify the problem and fix!
Q1. In the past year, how many times did you buy groceries?
0-10
11-20
21-30
31-40
41 or more
Q2. How frequently do you attend church services and meet with your pastor?
Never
Almost Never
Sometimes
Very Often
Always
More exercises
Q3. What are the reasons you selected Regis University?
_____________________________________________________
_____________________________________________________
____________________________________________________
Q4. Please select the top 5 reasons why you withdrew from Regis?
- Cost of tuition
- Location of main campus
- Better offer at another institution
- Personal reasons
- Financial constraints
- Availability of online courses or programs
- Academic challenge
- Curriculum
- Etc
Exercises
Q5. Rate your agreement with the following statement(s).
The professor was readily available outside of class.
Strongly Agree
Partly Agree
Agree
Disagree
Q6. Based upon your positive experiences at Regis, how likely are you to
recommend Regis to other prospective students?
Very likely
Somewhat likely
Somewhat unlikely
Very unlikely
Exercises
Q7. Do you think the program should enhance faculty qualifications and revise
the curriculum?
Yes
No
Maybe
I don’t know
Q8. What is your current annual household income?
0-25,000
25,000-50,000
50,000-75,000
75,000-100,000
100,000 or over
Exercises
Q9: As a recent alumni, do you think the university should seek NCATE
accreditation?
• Yes
• No
Q10. Select the factors you used when choosing your insurance plan.
• Cost
• Deductibles
• Insurers’ reputation
• Availability of providers in my area
• Coverage
• Others: _____________________
Review Offer
To assist your survey development, I will offer to
review your survey questions and responses.
Paula Harmer
[email protected]
303-964-5199
Exercise #2
• Develop a survey research question
• Develop 4 research objectives for your survey
• Review the 10 steps and write your steps
Download
Random flashcards
State Flags

50 Cards Education

Countries of Europe

44 Cards Education

Art History

20 Cards StudyJedi

Sign language alphabet

26 Cards StudyJedi

Create flashcards