WHERE WE NEED TO GO:
MEASURING SKILLS PSYCHOLOGY
BACCALAUREATES POSSESS
R. Eric Landrum
Department of Psychology
Boise State University
elandru@boisestate.edu
APA Symposium: Essential Skills for Psychology Majors: Do Students Actually Acquire Them?
Toronto, Ontario, Canada
August 8, 2009
Thinking about this talk…
 Statistics:
 18 days, 5000 miles, 12 states, 2 teenagers (2009)
Thinking about this talk…
Belief #1
 What we deem as important to
our students’ education and to
our discipline is worthy of and
necessitates assessment.
Belief #2
 Not everything that is
important in higher education
can be adequately assessed
with a multiple choice test.
APA Guidelines for the
Undergraduate Psychology Major
APA Undergraduate Guideline
Goal 1: Knowledge Base of Psychology
Students will demonstrate familiarity with the major concepts, theoretical perspectives, empirical findings, and historical
trends in psychology.
Goal 2: Research Methods in Psychology
Students will understand and apply basic research methods in psychology, including research design, data analysis, and
interpretation.
Goal 3: Critical Thinking Skills in Psychology
Students will respect and use critical and creative thinking, skeptical inquiry, and, when possible, the scientific approach to
solve problems related to behavior and mental processes.
Goal 4: Application of Psychology
Students will understand and apply psychological principles to personal, social, and organizational issues.
Goal 5: Values in Psychology
Students will be able to weigh evidence, tolerate ambiguity, act ethically, and reflect other values that are the underpinnings
of psychology as a discipline.
APA Guidelines for the
Undergraduate Psychology Major
APA Undergraduate Guideline
Goal 6: Information and Technological Literacy
Students will demonstrate information competence and the ability to use computers and other technology for many
purposes.
Goal 7: Communication Skills
Students will be able to communicate effectively in a variety of formats.
Goal 8: Sociocultural and International Awareness
Students will recognize, understand, and respect the complexity of sociocultural and international diversity.
Goal 9: Personal Development
Students will develop insight into their own and others’ behavior and mental processes and apply effective strategies for
self-management and self-improvement.
Goal 10: Career Planning and Development
Students will emerge from the major with realistic ideas about how to implement their psychological knowledge, skills, and
values in occupational pursuits in a variety of settings.
APA Guidelines for the
Undergraduate Psychology Major
APA Undergraduate Guideline
Goal 1: Knowledge Base of Psychology
Students will demonstrate familiarity with the major
concepts, theoretical perspectives, empirical findings, and
historical trends in psychology.
Skill-Based Assessments (Mostly)



GRE Subject Test in Psychology
Major Field Test for Psychology
Area Concentration Achievement Test (ACAT) in
Psychology


California Critical Thinking Skills Tests
Collegiate Assessment of Academic Proficiency
(CAAP) Critical Thinking Test
Collegiate Learning Assessment (CLA) Critical
Thinking, Analytic Reasoning, and Problem Solving
Cornell Critical Thinking Test
Watson Glaser Critical Thinking Appraisal
Measure of Academic Proficiency and Progress
(MAPP)
Goal 2: Research Methods in Psychology
Students will understand and apply basic research methods
in psychology, including research design, data analysis, and
interpretation.
Goal 3: Critical Thinking Skills in Psychology
Students will respect and use critical and creative thinking,
skeptical inquiry, and, when possible, the scientific approach
to solve problems related to behavior and mental processes.




APA Guidelines for the
Undergraduate Psychology Major
APA Undergraduate Guideline
Goal 4: Application of Psychology
Students will understand and apply psychological principles
to personal, social, and organizational issues.
Goal 5: Values in Psychology
Students will be able to weigh evidence, tolerate ambiguity,
act ethically, and reflect other values that are the
underpinnings of psychology as a discipline.
Skill-Based Assessments
APA Guidelines for the
Undergraduate Psychology Major
APA Undergraduate Guideline
Skill-Based Assessments
Goal 6: Information and Technological Literacy
Students will demonstrate information competence and the
ability to use computers and other technology for many
purposes.





iSkills
Internet and Computing Core Certification (IC3)
Computer Skills Placement (CSP)
North Carolina Computer Skills Test
Assessment of Basic Computer Proficiency
Goal 7: Communication Skills
Students will be able to communicate effectively in a variety
of formats.

Collegiate Assessment of Academic Proficiency
(CAAP) Writing Skills Test
Collegiate Assessment of Academic Proficiency
(CAAP) Writing Essay Test
Collegiate Level Assessment (CLA) Written
Communication
College-Level Academic Skills Test (CLAST) English
Language Skills and Reading Skills
WorkKeys Foundational Skills Assessment: Writing




APA Guidelines for the
Undergraduate Psychology Major
APA Undergraduate Guideline
Skill-Based Assessments
Goal 8: Sociocultural and International Awareness
Students will recognize, understand, and respect the
complexity of sociocultural and international diversity.

Goal 9: Personal Development
Students will develop insight into their own and others’
behavior and mental processes and apply effective strategies
for self-management and self-improvement.








Goal 10: Career Planning and Development
Students will emerge from the major with realistic ideas
about how to implement their psychological knowledge,
skills, and values in occupational pursuits in a variety of
settings.
Student Portfolio and Information Form (SPIF) /
ePortfolio
Beliefs, Events, and Values Inventory
Global Awareness Profile
Global Perspective Inventory
Intercultural Development Inventory
WorkKeys Personal Skills Assessment (Performance,
Talent, Fit)
WorkKeys Foundational Skills Assessment: Teamwork
WorkKeys Foundational Skills Assessment:
Observation
WorkKeys Foundational Skills Assessment: Listening
Employer Expectations
 Employer survey: What graduates lack
(AAC&U, 2008)—percent of employers
reporting the item lacking in parentheses
 Global knowledge (46%)
 Self-direction (42%)
 Writing skills (37%)
 Critical thinking skills (31%)
 Adaptability (30%)
 Self-knowledge (26%)
Assessment Centers
 First industrial use in 1956 at AT&T
 Tasks:
 Leaderless group discussion
 In-basket task
 Business game
 Projective tests
 In-depth interview
 Personal history form
Assessment Center Guidelines
 Multiple assessment techniques are used, at




least one of which must be a simulation.
Multiple trained assessors are used.
The overall judgment regarding the applicant
must be based on a combination of multiple
assessors and multiple assessments.
Simulation exercises must be reliable,
objective, and job-related.
All behaviors measured must be job-related.
AP Psychology Exam
 Over 150,000 high school students took the
AP Psychology Exam in 2009
 Two-thirds of the AP score is based on 100
multiple choice items
 One-third of the AP score is based on two
free-response essays
 Students typically must go beyond definition
of terms and make real-world connections to
terminology via application of information
DMV Requirements for a
Driver’s License (Idaho)
 Vision screening
 Written knowledge test (may miss 6 out of
40)
 Skills test (evaluates your ability to drive a
vehicle safely, demonstrate good driving
habits, and obey traffic laws in a variety of
driving situations)
 Pre-drive check
 Driving test
Useful Rubrics Available
From: Halonen, Bosack, Clay, McCarthy, Dunn, Hill, McEntarffer, Mehrotra,
Nesmith, Weaver, and Whitlock (2003)
Assessment Resource Examples
 Assessment Cyberguide
http://www.apa.org/ed/critique_study.html
 Assessing student learning: A collection of
evaluation tools (Johnson & Vosmik, 2009)
More Resources

American Psychological Association. (2007). APA guidelines for the undergraduate psychology major. Washington, DC: Author. Retrieved
from www.apa.org/ed/resources.html

Appleby, D. (2000, Spring). Job skills valued by employers who interview psychology majors. Eye on Psi Chi, 4(3), 17.

Association of American Colleges and Universities. (2008). How should college assess and improve student learning? Employers’ views on
the accountability challenge: A survey of employers conducted on behalf of the Association of American Colleges and Universities.
Washington, DC: Author. (ERIC Document Reproduction Service No. ED 499718)

Beins, B. (2003). What should psychology majors know and what should they be able to do? Eye on Psi Chi, 7(3), 38-39.

Fried, C. B., & Johanson, J. C. (2003). Curriculum review using a knowledge, skills, and abilities-based assessment of alumni. Teaching of
Psychology, 30, 316-318.

Gaither, G., & Butler, D. (2005). Skill development in the psychology major: What do undergraduate students expect? College Student
Journal, 39, 540-552.

Graham, S. E. (1998). Developing student outcomes for the psychology major: An assessment-as-learning framework. Current Directions
in Psychological Science, 7, 165-170.

Halonen, J. S., Bosack, T., Clay, S., McCarthy, M., Dunn, D. S., Hill, G. W., IV, McEntarffer, R., Mehrotra, C., Nesmith, R., Weaver, K. A., &
Whitlock, K. (2003). A rubric for learning, teaching, and assessing scientific inquiry in psychology. Teaching of Psychology, 30, 196-208.

Halpern, D. F. (1988). Assessing student outcomes for psychology majors. Teaching of Psychology, 15, 181-186.

Hettich, P. I., & Helkowski, C. (2005). Connect college to career: A student’s guide to work and life transitions. Belmont, CA:
Thomson/Wadsworth.

Kruger, D. J., & Zechmeister, E. B. (2001). A skills-experience inventory for the undergraduate psychology major. Teaching of
Psychology, 28, 249-253.

Lawson, T. J. (1999). Assessing psychological critical thinking as a learning outcome for psychology majors. Teaching of Psychology, 26,
207-209.

Milton, O., Pollio, H. R., & Eison, J. A. (1986). Making sense of college grades. San Francisco: Jossey-Bass.

McGovern, T. V., Furumoto, L., Halpern, D. F., Kimble, G. A., & McKeachie, W. J. (1991). Liberal education, study in depth, and the arts
and sciences major—Psychology. American Psychologist, 46, 598-605.

Robbins, A., & Wilner, A. (2001). Quarterlife crisis. New York: Penguin Press.
Belief #1
 What we deem as important to
our students’ education and to
our discipline is worthy of and
necessitates assessment.
Belief #2
 Not everything that is
important in higher education
can be adequately assessed
with a multiple choice test.
You only get two…
Bloom’s Taxonomy (Revised)
Creating
• Generating, hypothesizing, planning,
designing, producing, constructing
Evaluating
• Checking, coordinating, monitoring,
testing, critiquing, judging
Analyzing
Applying
Understanding
Remembering
• Differentiating, discriminating, distinguishing, focusing,
selecting, organizing, integrating, outlining, parsing, structuring,
attributing, deconstructing
• Executing, carrying out, implementing,
using
•Interpreting, clarifying, paraphrasing, representing, translating, exemplifying,
illustrating, classifying, categorizing, subsuming, summarizing, abstracting,
generalizing, inferring, concluding, extrapolating, predicting, comparing,
contrasting, mapping, matching, constructing models
• Recognizing, identifying, recalling,
retrieving
(Anderson & Krathwohl, 2001)
In-Basket Task (Pelfrey, 1986)
“The assessees were told that they had just been appointed to
the position they were seeking. They were to arrive in their
new job on Sunday and leave in exactly 3 hours to catch a plane
for a meeting. They would be out of town for several days and
must attend to all of the things left by their predecessor…. The
assessees were then given a packet of materials including 25
distinct but interrelated problems. These problems ranged
from minor but thorny problems such as a secretary going on
vacation to a complicated report requiring an immediate
response. The assessee worked individually and had to deal
with each of these problems, write notes, report on actions
taken, delegate tasks where appropriate, and recognize the
interrelationship of some of the problems. Following the inbasket exercise, the AT&T team interviewed each assessee to
see how the problems were addressed, the thought processes
used, and the logic of the decisions. This interview became
part of the assessment of the candidate” (p. 66).