the PowerPoint file presented in this video

advertisement

Survey 101: Survey

Questions and Responses

Paula Harmer, Director of Institutional

Research, Office of Academic Affairs

November, 2012

Outline of Workshop

• Review “Why use surveys”?

• Outline of survey preparation steps

• Overview of question and response types

• Guidelines for writing survey questions and selecting response sets

• Practicum: identify survey issues and revise questions & responses

Fink, Arlene. How to Ask Survey Questions volume 2, Sage

Publications, 2003.

Why use surveys?

• Obtain feedback with a fast turnaround

• Self-administered ones are resource efficient compared to phone/in person interviews

• Anonymity/confidentiality allows respondents various levels of protection for candid responses

• Accepted research methodology for applied and academic research

• Can gather holistic feedback of total population versus a select few

Survey – Getting Ready for Launch

• Step 1: Develop a purpose and specific objectives for your survey. Examples:

How satisfied are recent alumni with their educational experiences and learning support at Regis University? Does student engagement impact course learning outcomes?

• Step 2: Define the human population(s) being surveyed. Sample/whole population?

• Step 3: Select survey type. Self administered? Paper? Online? Phone? In person interviews or focus group?

• Step 4: Plan number of questions that measure objectives, and how much time respondent will have to complete survey.

• Step 5: Timing of the survey – Once? Twice? At a time point in student’s life cycle? 6 months after graduation? Pre-test/Post-test?

• Step 6: Determine security level: if survey is identifiable, anonymous, confidential

Getting Ready + Administration

• Step 7: Permissions – Does the respondent population require authorization for recruitment? Eg. Regis IRB? What level of risk do respondents have for participation, and how can you minimize that risk?

• Step 8: Survey data collection – online or paper-based surveys?

Automated or manual data entry? Paper scanning?

• Step 9: Write your survey questions geared to answer the research question purpose and objectives, following writing guidelines. You

MUST formulate questions so respondents may answer easily and accurately.

• Step 10: Review and test survey questions with potential respondents and content experts.

Why Questions are Critical

• For self-administered surveys, you must be sure that the questions can be understood by respondents without assistance from survey team

• Using pretested & proven questions is best

• Enhances validity (are you measuring what you think you are measuring) and reliability (if the respondent took the same survey again, would they respond the same way?)

Guidelines for Writing Survey

Questions

One concept per question

If you include more than one concept, respondents will be torn between responding to differing concepts, giving poor quality data.

Poor: How satisfied were you with the availability of faculty outside of class and opportunities to engage in research?

Better: How satisfied were you with availability of faculty outside of class?

Ask Precise Questions

• Logically related to survey objectives

• Asks precise and unambiguous questions

Eg. Use specific time periods rather than ‘typical’

Vague: How would you describe your emotional well being?

Better: How would you describe your emotional well being in the past three months?

• More detail will increase reliability of answers

• Respondent recall guideline: Major life events (more than 1 year is

OK) versus unimportant events (less than 1 month is best)

Use simple & conventional language

• Use complete sentences

• Do not confuse respondents by using disciplinary jargon or acronyms/abbreviations. This measures comprehension, not perceptions or self-reported behavior. This is not a test!

Q: The faculty member utilized a variety of pedagogical approaches in this course.

Q: The faculty member utilized a variety of teaching and learning approaches in this course.

Q: Indicate the types of learning approaches utilized in the course:

Lecture

Group Discussion

Games

Lab

Seminar

Case studies

Other

Use simple & conventional language

• Avoid negative questions if possible

• Avoid loaded questions if possible

• Avoid biased words or phrases that trigger emotional response

Select Open or Closed Questions

• Closed questions have pre-specified responses, are more reliable and efficient, good for ranking and rating, allows statistical analysis

• Open questions allow responses in own words, require qualitative analysis. Good for exploration or quotes

Note: requires more effort to code and analyze, and fewer respondents will make the effort to respond, so don’t make the questions required to complete the survey

• Good practice to have majority closed Qs with a few open

Qs

Closed Question Response Types

• Categorical: responses are categories, no numerical or preferential values, they are true or false.

Eg. Male & Female; Alumni/Student/Faculty/Staff;

• Categories should be mutually exclusive; should include all possible answers (inclusive); should exhaust possible categories (exhaustive)

• Ordinal: scale highest to lowest, rate or order items Eg age categories, high-low scales, etc

• Numeric: asks for number Eg age in years

Select Best Response

Sets

• Categorical response sets should be inclusive of all possible answers

Eg. Select the reasons you chose to enroll in your program: academic reputation/geographical location/faculty/ tuition / scholarship opportunities / other

• Categorical: It is good practice to use a ‘I don’t know/not applicable’ or ‘other’ when it makes sense to, in case a question has little or no relevance to a respondent – helps prevent survey abandonment by respondents

• On the other hand, it can encourage ‘lazy’ responses

• Ordinal: Good for analysis, balances between negative and positive responses

Ordinal Response Scales

5-point scales

URL (as of Nov 28, 2012): http://www.utexas.edu/academic/ctl/assessment/iar/teaching/plan/method/survey/writing.php

Ordinal Response Scales

• Use a meaningful scale

• Use a balanced scale – 4 or 5 point is the ‘norm’

• Provide a neutral category such a ‘Don’t Know’ only if it makes sense

1

Endorsement Definitely true

Frequency Always

2

True

3 4

Don’t Know False

Intensity None

Influence Big problem

Comparison Much more than others

Very Often Sometimes Almost

Never

Very Mild Mild Moderate

Moderate problem

Somewhat more than others

Small problem

About the same as others

Very small problem

Somewhat less than others

5

Definitely

False

Never

Severe

No problem

Much less than others

Ordinal Response Scales

• Put the socially undesirable/negative response first

• Tell respondent how and where to mark the responses

• When possible, group similar questions & response scales together

• Rank order scales – best not to exceed three choices in surveys

Numeric Responses

• Responses are a number Eg. height, weight, age in years, number of books, etc.

• Numeric scales/measures are when respondents select a number along a scale, such as a pain scale from 0 (No Pain) to 10 (Worst Possible Pain).

Questions Measuring Knowledge,

Behaviors and Attitudes

Attitudes are complex and challenging to define and measure

• Psychometricians use methods to examine statistical properties of attitudes

• Knowledge and behavior is different

Eg What a person feels about gun control, knows about gun control, and does about can be very different

• Finding and developing attitudinal questions and scales is hard work!

Behavioral Questions

• Measures what respondents actually do

• Involve time, duration and frequency

• Guideline: Major life events can be recalled longer than one year

Poor: In the past year, which of the following items did you purchase?

Better: In the past 3 months, which of the following items did you purchase?

Knowledge Question

• A correct answer is available

• Used to assess respondents’ knowledge of a topic

• For example, if knowledge of a group is slim, an educational campaign may be warranted

Demographic information

• Asking a respondent facts about age, income, race/ethnicity, education, gender, etc.

• If known, compare respondent sample with total population to gauge ‘representativeness’

• Allows comparisons Eg adult versus traditionally-aged students

• Sensitive information (Eg income) should be asked at end of survey and should be optional

Identify Issues and Fix! Exercise

Each of the following survey questions & responses have a problem(s).

Identify the problem and fix!

Q1. In the past year, how many times did you buy groceries?

0-10

11-20

21-30

31-40

41 or more

Q2. How frequently do you attend church services and meet with your pastor?

Never

Almost Never

Sometimes

Very Often

Always

More exercises

Q3. What are the reasons you selected Regis University?

_____________________________________________________

_____________________________________________________

____________________________________________________

Q4. Please select the top 5 reasons why you withdrew from Regis?

Cost of tuition

Location of main campus

Better offer at another institution

Personal reasons

Financial constraints

Availability of online courses or programs

Academic challenge

Curriculum

Etc

Exercises

Q5. Rate your agreement with the following statement(s).

The professor was readily available outside of class.

Strongly Agree

Partly Agree

Agree

Disagree

Q6. Based upon your positive experiences at Regis, how likely are you to recommend Regis to other prospective students?

Very likely

Somewhat likely

Somewhat unlikely

Very unlikely

Exercises

Q7. Do you think the program should enhance faculty qualifications and revise the curriculum?

Yes

No

Maybe

I don’t know

Q8. What is your current annual household income?

0-25,000

25,000-50,000

50,000-75,000

75,000-100,000

100,000 or over

Exercises

Q9: As a recent alumni, do you think the university should seek NCATE accreditation?

• Yes

• No

Q10. Select the factors you used when choosing your insurance plan.

• Cost

• Deductibles

• Insurers’ reputation

• Availability of providers in my area

• Coverage

• Others: _____________________

Review Offer

To assist your survey development, I will offer to review your survey questions and responses.

Paula Harmer pharmer@regis.edu

303-964-5199

Exercise #2

• Develop a survey research question

• Develop 4 research objectives for your survey

• Review the 10 steps and write your steps

Download