Measuring Psychological Literacy:
Accuracy of undergraduates’ selfrated critical thinking abilities
Dr Suzanne Czech
Suzanne.Czech@usq.edu.au
Scientific Literacy
Psychologically
literate graduates
engage in critical
thinking about how to
apply and use scientific
explanations regarding
knowledge about the
real world.
Krause, M & Corts, D (2012)
Psychological Science: Modeling Scientific Literacy
ISBN-13: 9780131739857
Psychological Literacy
•
•
•
•
•
•
•
•
•
basic knowledge of psychology content
use scientific thinking; information analysis; evaluate
creative approach to problem solving
applying psychological principles in all contexts
acting ethically
information and technology competencies
effective communication
respect for diversity
inter / intra personal insight into mental processes
McGovern et al. (2010; p 11)
Promoting psychological
literacy
In addition to higher economic and political
profile for psyc’s, we required a coherent,
relevant & science-based undergraduate (UG)
curriculum
(Cranney et al. 2012)
which requires students to have “…the capacity to
intentionally use psychological knowledge and
skills to achieve personal, professional and
societal goals”
(Cranney & Dunn, 2011)
Graduate Attributes (GAs)
Four-Year Australian UG Psychology Program*
1. Knowledge and Understanding of Psychology
2. Research Methods in Psychology
3. Critical Thinking Skills in Psychology
4. Values in Psychology
5. Communication Skills in Psychology
6. Learning and the Application of Psychology
*produced by Cranney et al. (2009) as part of the
Carrick Associate Fellowship project and adopted by the
Australian Psychology Accreditation Council (APAC; 2008)
Student Self-Reports
GA & PL ratings were moderately high among 213
UNSW UG’s, which reflects:
“…substantial awareness, perceived development,
and perceived importance of these concepts”
(Morris, et al. 2013)
CYO Adventure tutorial programme delivered to
273 3rd years increased self-reported critical
analysis and problem-solving skills
(Karantzas et al, 2013)
Research Question
Are students’ CT self-appraisals accurate?
Research Aim:
Examine accuracy of psyc UG students’
self-appraised critical thinking skills by
comparing SR ratings to an objective
measure.
Defining CT
American Philosophical Association’s
“Delphi Report”
(Facione, 1990)
 qualitative research methodology
 interactive panel of experts
 work toward a consensus resolution of matters of
opinion
 46 persons, widely recognized by their
professional colleagues to have special experience
and expertise in CT instruction, assessment or
theory, made the commitment to participate in
this Delphi project
CT Variable(s)
Critical Thinking
Operationalised
as 6 CT skills
identified in the
Delphi report
(and definitions
sourced for these)
1. Self-appraised
2. Objectively
measured
CT Skills
Definition
EVALUATE
judging the value of
something
SELFREGULATION
self-awareness, particularly of
biases and self-interests
INFERENCE
conclusion based on evidence
and reasoning
INTERPRET
understanding of underlying
logic
ANALYSIS
EXPLAIN
examine elements of structure
restating information with
added clarity and perspective
CT Self-Appraisal Measure:
For each…(GAs) please indicate how much you have developed the attribute so far in your …
Definition
courses: 0 = Not at all to 10 = an extremely high … (i.e., Graduate level)
APAC Graduate Attribute 3: Critical Thinking
• Apply knowledge of the scientific method in thinking about problems related to
behaviour and mental processes.
• Question claims that arise from myth, stereotype, pseudo-science or untested
assumptions.
• Demonstrate an attitude of critical thinking that includes persistence, openmindedness, and intellectual engagement.
• Demonstrate a capacity for higher-order analysis, including the capacity to
identify recurrent patterns in human behaviour.
• Evaluate the quality of information, including differentiating empirical evidence
from speculation.
• Identify and evaluate the source and context of behaviour.
• Recognise and defend against the major fallacies of human thinking.
• Evaluate issues and behaviour using different theoretical and methodological
approaches.
• Use reasoning and evidence to recognise, develop, defend, and criticise
arguments and persuasive appeals.
• Demonstrate creative and pragmatic problem solving.
EVALUATE, SELF-REGULATION, INFERENCE, INTERPRET, ANALYSIS, EXPLAIN
CT Objective Measure
(Possibilities Considered)
• Halpern Critical Thinking Assessment
(HCTA)
• Watson-Glaser Critical Thinking Appraisal
• California Critical Thinking Test
• Lawson’s Psychological Critical Thinking
Exam (PCTE)
Australian Psychological Critical
Thinking Test
Developed specifically to measure Delphi
Report 6 CT skills mapped on to APAC GA 3
CT Skills
Definition
EVALUATE
Apply theory to case scenarios; identify source, context,
recurrent patterns in human behaviour; evaluate strength
and type of theoretical and empirical evidence
SELFREGULATION
Identify psychological misconceptions (pseudo-science)
forced-choice; sentence stem completion to reflect
intellectual engagement and appreciation for learning
INFERENCE
Identify experimental design errors (Lawson PCTE); Apply
scientific method; creative problem solving
INTERPRET
Recognise logical fallacies
ANALYSIS
Sentence stem completion reflecting stages of knowledge
(capacity for higher order analysis); recognise argument
structures in paragraphs;
EXPLAIN
defend arguments in paragraphs (explaining why or why not
text is an argument); develop logical argument from
statements
Primary Aim
Hypothesis:
UG students SR ratings on APAC GA CT
skills will positively correlate with scores
on the Australian Psychological Critical
Thinking Test (APCTT)
Secondary Aim
Establish evidence for the validity of
the APCTT.
Hyp that APCTT scores:
 reflect theoretical expected differences
between 1st year and 3rd year scores
and T1 and T2 scores
 concurrently predict SR grade results
and knowledge of psychology content
 converge with endorsing psychology as
a science
Method
Procedure & Measures
Each of T1 & T2 conditions required participants to
complete a 1-hour survey and the APCTT (identical for
both samples)
Survey:
-Both T1 and T2 survey included Psychology as a Science
(Friedrich, 1996)* and participant self-appraisals on
APAC GA3: CT skills (0 – 10).
-T1 survey comprised questions related to career path
and level of knowledge acquired or needed to achieve.
-T2 survey comprised questions related to individual
course offerings (e.g., preference, knowledge, skills),
and views on the science-practitioner model.
*permission to use from Taylor & Francis
Sample
Participants included in data analysis:
Age range 17 to 67 (M = 35; SD = 11.9)
89% Non-indigenous Australian
2% Indigenous Australian
Cross-sectional Design:
Time 1 (T1): 1st years N=38; 3rd years N=103*
62% enrolled in Bachelor of Science (Psyc)
32% enrolled in Graduate Diploma Studies (Psyc)
6% enrolled in Bachelor of Science – Other
Median number of psyc courses taken: 8 (Max 28)
Time 2 (T2): 3rd years N = 76*
*represents ~80% agreed to research participation
Feedback to Students
Results
Findings for APCTT
Validity Evidence
Construct Validity
Internal consistency
All subscale factors sig corr with total APCTT scores
r = .28 - .52, 95% CI, p < .0001 (N = 208-217)
Theoretical expected diffs
On avg, at T1, 3rd Years scored sig higher total APCTT
scores (M = 65.8; SD 10.5) than 1st years (M = 60.4; SD
11.4). This diff (5.42, 95% CI [-9.442, -1.393]), was sig
(t139 = 2.662, p = .009, d = 0.47)
However, 3rd Years did not improve sig from T1 to T2
APCTT Scores Group Diffs
EVALUATE, SELF-REGULATION, INFERENCE, INTERPRET, ANALYSIS, EXPLAIN
APCTT Scores Group Diffs
3rd Years did not improve sig from T1 to T2
Except evaluate
theory, problem
solve, higher order
analysis
EVALUATE, SELF-REGULATION, INFERENCE, INTERPRET, ANALYSIS, EXPLAIN
Findings for APCTT
Validity Evidence
Convergent Validity
 Total APCTT scores (N = 217; M = 64.5; SD 12.4) and
PAS total score (N =210; M =65.1; SD 10.3) were sig
corr (r = .27, 95%, p < .001)
Criterion Validity
 Total APCTT scores and number completed units and selfrated knowledge of psych courses were sig corr (N = 76,
r = .27, 95%, p = .019), but not with actual SR grade
results in these courses (No sig corr between these var’s and
the PAS )
Research Question
Findings of Evidence
• 3rd YEARs Total
APCTT scores
sig lower than
self-appraisal
critical thinking
skills at both
T1 and T2
• No sig diff for
1st YEAR scores
Research Question
Findings of Evidence
Total APCTT scores
 sig lower for 3rd Years at T1 (M = 5.5; SD .87)
and T2 (M = 5.0, SD = 1.15) than SR GA3 CT
ratings (M = 7.1, SD = 1.37; M =7.2, SD =
1.19).
In both instances this diff (-1.60, 95% CI [1.904, -1.296]; -2.21, 95%, CI [-2.544, -1.885])
was sig (t102 = -10.440, p < .001; t73 = -13.391,
p < . 001).
 sig positive correlation with SR development
on GA3 CT for 3rd Years at T2 (N = 74; r = .26,
95%, p < .024), but not T1 for 1st or 3rd Years
Conclusions
APCTT Measurement Validity
 Evidence for construct and criterion
validity from the 3rd year data.
However, lack of improvement over the semester
was surprising despite no direct CT instruction.
• Some instruction was given in developing a logical
argument, which may explain ns gains in this area.
• Sig gains in problem solving may be due to
assignment on systematic lit review
• Sig gains in capacity for higher order analysis may
be due to rigour of course content
• Sig gains in theory evaluation may be due to other
course content instruction (e.g., Abnormal)
Accuracy of self-appraisals
on critical thinking skills
As a group, first year students were
fairly accurate
However, data was not sig correlated between
discrete subscales and the graduate attribute,
with the exception of evaluating theory (r = .45,
p < .01).
Third year students performed sig
relatively poorly on the APCTT
compared to self appraisals.
Limitations
Methodological limitations:
• Selection bias. 1st Years self-selected;
3rd years partially self-selected
• 3rd Year students were not motivated to
exhibit best performance
• Students were not given specific
instructions on how marks would be
allocated to responses
• APCTT has not been validated against
other existing measures
Other studies which
objectively measured CT
• Tutorial exercises which required
students to develop hyp’s, analyse
data, assess internal validity of
conclusions.
• Two hours each week to discuss to
promote higher order reasoning
• Measured improvement over time
with PCTE
(Haw, 2011)
Future studies with
APCTT
 First years in current study provided
student number for follow up in third
year of studies at USQ
 Currently collecting data from
randomly selected Australian
community members on psychological
misconceptions
 Measurement refinement of the APCTT;
validity study by concurrent
administration with e.g., HCTA
Thank you
Volunteer Research Assistants:
Ms Bronwen Noller
Ms Victoria Drummond