SLCI Scores- ten brave Utah
Valley Conference Participants
Metacognitive Tools For Expert Learners
Tools
Knowledge
surveys
Learning
modules
Reflective
learning
journals*
Reading
reflections*
Knowledge
Knowledge)
Skill
Reasoning
Metacognition
Skill%
Reasoning)
Metacogni*on+
Skill%
Reasoning)
Metacogni*on+
Knowledge)
Skill%
Reasoning)
Metacogni*on+
Knowledge)
Skill%
Reasoning)
Metacogni*on+
Knowledge)
Skill%
Reasoning)
Metacogni*on+
Skill%
Reasoning)
Metacogni*on+
Exam wrappers
Learning
documents
* Evaluated through rubrics
Unifying Concepts - Metadiscipline of Science
1.
Science explains physical phenomena based upon testable information about
the physical world.
2. In modern life, science literacy is important to both personal and collective
decisions that involve science content and reasoning.
3. Doubt plays necessary roles in advancing science.
4. Scientists use evidence-based reasoning to select which among several
competing working hypotheses best explains a physical phenomenon.
5. A theory in science is a unifying explanation for observations that result from
testing several hypotheses.
6. Peer review generally leads to better understanding of physical phenomena
than can the unquestioned conclusions of involved investigators.
7. Science can test certain kinds of hypotheses through controlled experiments.
8. All science rests on fundamental assumptions about the physical world.
9. Science differs from technology.
10. Scientific knowledge is discovered, and some discoveries require an important
history.
11. Science employs modeling as a method for understanding the physical world.
12. Scientific knowledge imparts power that must be used ethically.
Metadisciplinary Concepts Converted to Outcomes
STUDENTS WILL BE ABLE TO…
1. Define the domain of science and determine whether a statement constitutes a
hypothesis that can be resolved within that domain.
2. Describe through example how science literacy is important in everyday life to an
educated person.
3. Explain why the attribute of doubt has value in science.
4. Explain how scientists select which among several competing working hypotheses
best explains a physical phenomenon.
5. Explain how "theory" as used and understood in science differs from "theory "as
commonly used and understood by the general public.
6. Explain why peer review generally improves our quality of knowing within science.
7. Explain how science employs the method of reproducible experiments to
understand and explain the physical world.
8. Articulate how science’s way of knowing rests on some assumptions.
9. Distinguish between science and technology by examples of how these are
different frameworks of reasoning.
10. Cite a single major theory from one of the science disciplines and explain its
historical development.
11. Explain and provide an example of how modeling is used in science.
12. Explain why ethical decision-making becomes increasingly important to a society
as it becomes increasingly advanced in science.
DRAFT: Metadisciplinary Outcomes for the Arts
Students should be able to…
1. Explain the significance of creative expression and art to the
human experience.
2. Discern objective vs. subjective scholarship, criticism and analysis
of the arts.
3. Articulate in his/her own words a definition for what constitutes
the arts.
4. Communicate ideas and emotions through the practice and study
of the arts.
5. Recognize and value creative expression from various cultural
and historical perspectives.
6. Explain in his/her own words reasons why critical thinking and
problem solving have value in the arts.
7. Describe, using at least two specific examples, how art literacy is
important in everyday life.
A realization that should arise from
becoming educated: every metadiscipline
offers a valuable way of knowing
•
•
•
•
•
Articulate concepts
Rewrite concepts as outcomes
Develop assessment instruments
Teach the outcomes
Use assessment results to try improve learning
OK…Can we assess this stuff?
The Instrument
• For each outcome
– Construct several concept inventory items.
– Use established methods for drafting items that have been
developed in other concept inventories.
• In addition…
– Test reasoning, not factual knowledge.
– Administer the inventory under the conditions in which a
citizen will use common information.
…initial instrument constructed 2008-2010 through
the collegial efforts of
•
Edward Nuhfer, Faculty Development & Geology, Channel Islands
•
Jerry Clifford, Physics, Channel Islands
•
Christopher Cogan, Environmental Sciences & Resource Management,
Channel Islands
•
Anya Goodman, Biochemistry, San Luis Obispo
•
Carl Kloock, Biology, Bakersfield
•
Beth Stoeckly, Physics, Channel Islands
•
Christopher Wheeler, Geology, Channel Islands
•
Gregory Wood, Physics, Channel Islands
•
Natalie Zayas, Science Education & Environmental Sciences, Monterey Bay
Science Literacy Concept Inventory
• Incorporates 25 validated items that map to
the twelve concepts
• Reliability of .85
• Tested on over 8000 students in about 30
institutions
Outcome. Student can Define the domain of science and
determine whether a statement constitutes a hypothesis that
can be resolved within that domain.
Concept
• Science explains
physical phenomena
based upon testable
information about
the physical world.
Some Misconceptions
• Science is on a mission to
refute religion; scientists
study the paranormal;
untestable statements are
like scientific hypotheses.
Which of the following statements presents a hypothesis that science
can now easily resolve?
A. Warts can be cured by holding quartz crystals on them daily for a week.
B. A classmate sitting in the room can see the auras of other students.
C. Radio City Music Hall in New York is haunted by several spirits.
D. People with chronic illnesses have them as punishment for past misdeeds.
0, 1, or 2?
Which of the following statements presents a hypothesis that science
can now easily resolve?
A. Warts can be cured by holding quartz crystals on them daily for a week.
B. A classmate sitting in the room can see the auras of other students.
C. Radio City Music Hall in New York is haunted by several spirits.
D. People with chronic illnesses have them as punishment for past misdeeds.
What did we learn that made this so
interesting?
• Starting with…Do experts (professors)
outscore novices (students)?
YES! They do! Also, students on average do come to us with some
science literacy: zero literacy = about 25% (random guessing).
Does knowing “stuff” advantage a person?
Professors in Metadisciplines by
Mean Score %
We can learn about our students’
science literacy
Our current GE science courses don’t produce
science literacy…
Development seems more apparent long
term through ranks…
We can learn much about our
students
Gender-equality (n = 12,120)
Science Major or Not: 73% vs 67%
Science Major or Not: difference generally
increases through undergraduate ranks
Freshmen
Juniors
Sophomores
Seniors
First Generation Students: 67% vs 72%
English as a First Language: 63% vs 71%
Cumulative first-generation and ESL:
9 point disadvantage
• 64% versus 73% for those who have English as a
first language AND are not first-generation
students.
• That may be another way of showing that
economic status and struggles of students and
their families infuse differences.
SLCI by Institutions with Other
Measures
SAT
Verbal
1.00
SAT
Math
0.94
ACT
Composite
0.97
SLCIFrsh
0.87
SLCISoph
0.86
SLCIJr
0.85
SLCISr
0.87
SAT Math
ACT
Composite
SLCI-Frsh
0.94
1.00
0.98
0.86
0.82
0.77
0.78
0.97
0.87
0.98
0.86
1.00
0.88
0.88
1.00
0.84
0.94
0.81
0.95
0.84
0.87
SLCI-Soph
SLCI-Jr
0.86
0.85
0.82
0.77
0.84
0.81
0.94
0.95
1.00
0.96
0.96
1.00
0.95
0.93
SLCI-Sr
0.87
0.78
0.84
0.87
0.95
0.93
1.00
SAT Verbal
As Eric Gaze and the Quantitative Literacy test researchers have noted, these little
literacy tests may be powerful predictors of student success.
Institutions
Given that our GE science courses don’t produce
much increase in reasoning of science literacy…
What are some ways that we can
convey citizen competency in
science literacy to our students?
Metacurriculum for Metacognition
Activity
Knowledge or Skills
Knowledge Surveys
Goal-setting, Monitor. & Eval.
Reading Reflections
Reflection & Monitoring
How I Earned an “A” Goal-setting & Monitoring
Learning Reflections Refl., Monitoring & Evaluation
Exam Wrappers
Evaluation & Goal-setting
Critical Thinking
Strategies for Thinking
Metacognitive Tools For Expert Learners
Tools
Knowledge
surveys
Learning
modules
Reflective
learning
journals*
Reading
reflections*
Knowledge
Knowledge)
Skill
Reasoning
Metacognition
Skill%
Reasoning)
Metacogni*on+
Skill%
Reasoning)
Metacogni*on+
Knowledge)
Skill%
Reasoning)
Metacogni*on+
Knowledge)
Skill%
Reasoning)
Metacogni*on+
Knowledge)
Skill%
Reasoning)
Metacogni*on+
Skill%
Reasoning)
Metacogni*on+
Exam wrappers
Learning
documents
* Evaluated through rubrics
Knowledge Surveys
Example Survey Item
• I can outline Piaget’s four main stages of cognitive development, and
comment on how children’s thinking changes during these four stages.
Response Options
2 = I have current ability to address this challenge very well.
1 = I can now only partially address the challenge.
0 = I currently have insufficient skill/knowledge to address the challenge.
Knowledge Surveys
• Metacognition
– If I think I can do this, how well can I do this?
– Can I look at the items and start to distinguish
which emphasize knowledge, skills or reasoning?
– Am I able to do this at the targeted level of
understanding?
• For construction, consult the tutorials at
http://elixr.merlot.org/assessment-evaluation/knowledge-surveys
Correlations between performance
and several metacognitive selfassessments
(n = 1011)
First
Impression
Global Selfassessment
Knowledge
Survey itemby-item SelfAssessment
PostKnowledge
Survey
Overall SelfAssessment
Post-SLCI
Overall Selfassessment
(*n=590)
SLCI
Actual
Score
r = 0.29
r = 0.58
r = 0.45
r = 0.62*
Average
72 %
Average
75%
Average
79%
Average
76%
Average*
76%
Stages of Intellectual Development
• Level 1 & 2 thinkers believe that all problems have right and wrong answers,
that all answers can be furnished by authority (usually the teacher), and that
ambiguity is a needless nuisance that obstructs getting at right answers.
• Level 3 thinkers realize that authority is fallible and doesn't have all answers.
They respond by concluding that all opinions are equally valid, that arguments
are just about proponents thinking differently. Evidence doesn't change this.
• Level 4 thinkers recognize that not all challenges have right or wrong answers,
but they do not yet recognize frameworks through which to resolve how
evidence best supports one among several competing arguments.
• Level 5 thinkers can use evidence and begin to accept that evaluations that lead
to best solutions can be relative to the context in which a problem occurs.
• Level 6 thinkers appreciate ambiguity as a legitimate quality of many issues,
can use evidence well to explore alternative viewpoints. They recognize that
the most reasonable answers often depend upon context and value systems.
• Levels 7, 8 and 9 thinkers incorporate metacognitive reflection in their
reasoning, and they increasingly perceive how their personal values act
alongside context and evidence to influence chosen decisions and actions.
William J. Perry Jr. (1968) Forms of Intellectual and Ethical Development in the College Years
“Critical Thinking” and Metadisciplinary Reasoning
Traditions of
Critical Thinking*
Metadisciplines
Where Concentrated
Some realms of
Inquiry**
Logic and
philosophy
Humanities
What is
truthful/ethical?
Science
Science and
quantitative
reasoning
What is
testable/probable?
Pragmatism
All metadisciplines
including technology
What is
consequential?
Psychoanalysis
Social science, arts
What is
authentic/valued?
Critical theory
Humanities
What is privileged?
* From Brookfield, 2012
**Modified from Carole Huston, San Diego University
What might a rubric look like…
• To guide development of a sophisticated
written teaching philosophy for a faculty
member
• To guide development of a sophisticated
written learning philosophy…
• For anyone?
Reflective Exercises & Learning Journals….
Strategic Reading
Expert Readers:
•read with a purpose and in “extensive mode”
•accomplished in use of prior knowledge
•utilize a wide variety of strategies for monitoring and
comprehension (e.g., prediction, integration, selfquestioning, reflecting)
Novice Readers:
•focus on decoding single words/phrases
•fail to adjust for different texts/purposes
•seldom use text-processing strategies
Paris et al. (1996)
Reading and Reflecting
Reading Reflections:
• Completed after each reading assignment
• Short responses to a few questions
• What is the main point of this reading?
• What did you find surprising? Why?
• What did you find confusing? Why?
• Submitted online before class
• Credit awarded for “reflective” submissions
Metacognitive Goals of Reflections
What is the main point of this reading?
• Summarizing (Anderson & Thiede, 2008)
• Keywords (Thiede et al., 2005)
What did you find surprising? Why?
• Misconceptions (Bransford et al., 2000)
• Affect (Winne & Hadwin, 1998; Pintrich and Zusho, 2007)
What did you find confusing? Why?
• Monitoring, Evaluation, and Reflection
(Ertmer and Newby, 1996; Zimmerman, 2002)
Early Results
Reading Reflections vs. Course Grades
MACALESTER GEOLOGY
Pearson = 0.842
p-value = <0.001
HAMLINE ECONOMICS
Pearson = 0.779
p-value = <0.001
Classroom Activities
Effect Size = 1.35 (Large)
Clicker Questions
Effect Size = 1.08 (Large)
Exam
“Wrapper”
• Self Evaluation
• Preparation Strategies
• Performance Analysis
• Planning
Achacoso (2004)
Lovett (2008)
Backwards Design
Worth being
familiar with
Important to
know and do
Highest Priority
Wiggins and McTighe (1998)
We Must Act Beyond the Scale of Courses
Worth being
familiar with
Important to
know and to do
HIGHEST
PRIORITY
1
Start with articulating
what we most want as
the final result.
3
Complex Iterative Form
Coordinate our courses in
programs and degrees
Developing Form
Align the lessons
in our courses
On a day to day basis, try to find
small ways in which we can act to
support what we most want to do.
2
Become VERY attentive to the kind
of generator that can grow
to support our aspiration.
Initial form
Employ the generator
in our lessons
Generator
Choose the attributes
that we want for our graduates.
Initiator
(Our aspirations)
A Guide for Learning to Learn…
•
•
•
•
•
•
•
•
•
•
•
Purpose of Education
Levels of Thinking
Affective Domain
Significant Learning
Meanings of Learning
Research on The Brain
Intellectual Development
Critical Thinking & Reasoning
Metacognition
Learning Styles
Behavioral Dimensions of Grades