STUDENT AFFAIRS RESEARCH AND ASSESSMENT
TO SURVEY OR NOT TO
SURVEY? THAT’S A REALLY
GOOD QUESTION!
It is tempting, if the only tool you have is a
hammer, to treat everything as if it were a nail.
~ Abraham Maslow, 1966
SARA Assessment Brown Bag
February 10, 2011
Why is assessment important?

Helps determine if you are meeting
your educational objectives

Helps ensure that you have the
resources you need

Helps prioritize efforts

Can contribute to our understanding
of student learning and development
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Key steps - PDCA

Identify outcomes

Identify appropriate measures

Choose an appropriate assessment method

Choose and appropriate research design

Collect the data

Analyze the data

Disseminate the findings

Take action

Wash, Rinse, Repeat!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Presentation outline

Why do you ask?

Review types of outcomes and measures

Benefits and drawbacks to surveys

Other types of assessments

It’s your turn - practice makes perfect!
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Why not use a survey?

Ease of web-based surveys has lead to
their proliferation

Result is that surveys are the “go-to”
research tool, but…
◦ Not every question is a nail
◦ Survey fatigue is a significant threat
◦ Multiple types of evidence (assessment
triangulation) build a stronger case
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Starting questions
1.
2.
3.
What type of outcome do you want to measure?
◦
Cognitive (knowledge)
◦
Affective (attitudes)
What type of data do you want to collect?
◦
Psychological (personal traits)
◦
Behavioral (observable activities)
Time frame?
◦
Short-term
◦
Long-term
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Taxonomy of Student Outcomes
(Astin, 1993)
TYPE OF OUTCOME
TYPE OF DATA
Cognitive
Affective
Psychological
Subject-matter knowledge
Academic ability
Critical thinking ability
Basic learning skills
Special aptitudes
Academic achievement
Values
Interests
Self-concept
Attitudes
Beliefs
Satisfaction with college
Behavioral
Degree attainment
Vocational achievement
Awards or special
recognition
Leadership
Citizenship
Interpersonal relations
Hobbies and avocations
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Time: Examples of Short- and Longterm Outcomes (Astin, 1993)
TYPE OF
OUTCOME
TYPE OF DATA
SHORT-TERM (DURING
COLLEGE)
LONG-TERM
(AFTER COLLEGE)
Cognitive
Behavioral
Completion of
AlcoholEdu (vs.
noncompletion)
Cognitive
Psychological
MCAT score
Score on medical
licensing exam
Affective
Behavioral
Participation in student
government
Involvement in local
or national politics
Affective
Psychological
Satisfaction with college
Job satisfaction
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Exhibits responsible
drinking behaviors
Types of measures

Direct measures

Indirect measures

Norm-referenced

Criterion-referenced

Self-referenced
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Although direct measures are
typically preferred, practically
speaking, your overall assessment
plan should contain a mix of these
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Your question should guide
your choice of assessment
tool, and not the other way
around!
Surveys
“Surveys represent one of the most
common types of quantitative, social
science research. In survey research, the
researcher selects a sample of respondents
from a population and administers a
standardized questionnaire to them.”
http://writing.colostate.edu/guides/research/survey/index.cfm
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Surveys can be a good tool if you
are interested in:

Perceptions

Beliefs

Motivations

Future plans

Past behavior

Private behavior
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
What about student learning?

Research indicates that aggregate self-reports of learning can
provide a reasonable estimation of actual learning. *

**Self-reported data is most valid when:
◦ the information is known to the respondents,
◦ the questions are unambiguous and refer to recent activities,
◦ the respondents take the questions seriously, and
◦ responding has no adverse consequences nor does it encourage
socially desirable, rather than truthful, answers.
* (Anaya, 1999; Kuh, Kinzie, Schuh, Whitt, & Associates, 2005; Laing, Swayer, & Noble, 1989; Pace, 1985; Pike, 1995).
**(Kuh et al., 2005 & Pike, 1995)
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Validity and reliability
Surveys tend to be weak on validity and strong on
reliability. The artificiality of the survey format puts a
strain on validity. Since people's real feelings are hard to
grasp in terms of such dichotomies as "agree/disagree,"
"support/oppose," "like/dislike," etc., these are only
approximate indicators of what we have in mind when
we create the questions. Reliability, on the other hand, is
a clearer matter. Survey research presents all subjects
with a standardized stimulus, and so goes a long way
toward eliminating unreliability in the researcher's
observations. Careful wording, format, content, etc. can
reduce significantly the subject's own unreliability.
http://writing.colostate.edu/guides/research/survey/index.cfm
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Representativeness

If your respondents are not
representative of your population, then
your results may be misleading
◦ Example:You want to survey all
undergraduate students about their attitudes
towards a student honor code.Your friend is
in charge of the FYS program and offers to
have your survey passed out in all first-year
seminars.You’re excited to get this direct push
for your survey, but how might this effect
your results?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Strengths of surveys

Relatively inexpensive

Can reach large numbers of people

Large numbers allow for multivariate analyses

Can ask many questions relatively quickly

Standardized instruments (like NSSE) allow for
comparisons between groups

Can be confidential or anonymous

Can have high reliability
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Strengths continued

Can provide student/alumni/employers
perspective of the institution/program.

Can make respondents feel that their
opinions matter.

Ease of response can provide information
from hard to reach individuals

Results easily understood
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Weaknesses of surveys

Questions have to be general enough to
apply to all or most respondents

Inflexible - Forced-response choice may not
allow respondents to express their true
opinions

Require good response rates to achieve
representative results

May be hard for respondents to recall
information or answer truthfully

Can seldom deal with “context”
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Weaknesses continued

Validity can be questionable – results tend to
be highly dependent on wording of items,
salience of survey, and organization of the
instrument

Socially desirable responses

Indirect evidence which may have less
legitimacy with stakeholders

Better for measuring and comparing the
responses of groups rather than individuals
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Finally, it might not be a survey if….

The questions you want to ask don’t have
a limited number of known, well-defined
possible answers

You want to be able to ask about
relationships rather than inferring them

Your population of interest differs in
culture or language from the majority
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Some other types of assessments

Standardized exams

Test of abilities or knowledge

Simulation or performance appraisals

Interviews and focus groups

External examiners

Archival records and transcript analysis

Portfolios

Behavior observations

Student self-evaluations

Reflective writing

Minute papers/muddiest point
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Final thoughts on choosing an
assessment tool

Go back to your assessment question(s)
◦ What do you want to know?
◦ What are the resource limitations? (e.g., time,
money, staff)
◦ One shot or longitudinal?
◦ Experimental design?
◦ What type of analysis is appropriate?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
NOW IT’S YOUR TURN
Think about a program you want to
assess…

What do you want to know?

What type of measure?

How many students are involved?

What type of evidence do you already
have (if any)?

What type of evidence is most effective
with your intended audience?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Sample assessment questions

Do participants in an alternative spring break
(n=20) develop an increased awareness of
social injustice and subsequently a greater
commitment to working for social justice?

Do students (n=5,000) who go through an
alcohol intervention drink less as a result?

Do participants in the PRCC’s Learning
Circle (n=15) exhibit improved ability to
engage in positive cross-racial interaction
with other participants?
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
Resources

Astin, A. W. (1993). Assessment for excellence: The philosophy and practice
of assessment and evaluation in higher education. American Council on
Education Series on Higher Education. Phoenix, AZ: Oryx Press.

Improving Educational Programming online training module:
https://www.sa.psu.edu/workshops/edprogram/index.htm

Pope, R. L., Reynolds, A. L., Mueller, J. A., & Cheatham, H.E. (2004).
Multicultural competence in student affairs. San Francisco: Jossey-Bass.

SARA Assessment Resources website:
http://studentaffairs.psu.edu/assessment/resources.shtml

Writing Guide: Survey Research, Colorado State University:
http://writing.colostate.edu/guides/research/survey/index.cfm

Yin, A. C., & Volkwein, J. F. (2009). Assessing General Education
Outcomes. In J. F.Volkwein (ed.), Assessing Student Outcomes: Why, who,
what, how? New Directions for Institutional Research Assessment
Supplement (pp. 79-100). San Francisco: Jossey-Bass.
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
STUDENT AFFAIRS RESEARCH AND ASSESSMENT
QUESTIONS?