Eastern Michigan University
Peggy Liggit – Director of Academic Assessment
Steven Pernecky – Professor of Chemistry
Micheal Tew-Professor of Communications, Media, Theatre Arts
Peggy Liggit
Shared Assumption:
Conscious Competent Learning Model
Unconscious
Conscious
Incompetent
Incompetent
Conscious Competent
Unconscious Competent
Most faculty who achieve tenure are
operating at this level in their discipline
y “The connections made by good teachers are held not in their methods but yg
in their hearts – meaning heart in its ancient sense, as a place where intellect and emotion and spirit and will converge in the human self.” ‐‐Parker Palmer, Courage to Teach.
y “You should never worry about your good ideas being stolen in educational reform, because even when people are sincerely motivated to y ,
y
g
learn from you, they have a devil of a time doing so.” ‐‐Michael Fullan, Change Forces: The Sequel.
y “One important feature of embedded assessment is that it “blurs the lines” between teaching and assessment.” ‐‐James J. Gallagher, Improving Science Teaching and Student Achievement through Embedded Assessment.
Why There are Few ‘Good’ Assessment models
Michael Fullan (1999, 2006)
y Innovative ideas are “difficult to disseminate and replicate.”
y Not easily transferable ‐ difficult to capture “subtleties of the reform practice.” y The inability to replicate another’s model, “replicating the Th i bili
li
h ’
d l “ li i
h
wrong thing” y
“The
The reform itself, instead of the conditions which spawned reform itself instead of the conditions which spawned
the success.” y
Problems with scale: small ‐scale may not work well on a y
wider‐scale. Reflect about a misconception/problem in g
y
g
student learning that you did something about.
y What/how were you teaching at the time?
y What activity were you doing to determine student’s y
y
g
thinking/reasoning?
y What did students understand/not understand about what you were teaching?
y What did you do to help students better understand?
Care to share?
Care to share?
You have just documented a, presumably,
d
t d
bl
unconscious competent activity. Teach, Assess, Analyze, and Adj t
Adjust = Embedded Assessment
E b dd d A
t
Include an activity that intentionally
Gather information about learning through
focuses on a learning outcome and
work samples that expose
reveals what students’ are
students’ ideas/skills students
“thinking” – not just
e.g. constructing
Assess
focusing on a
representations
final or
Assess
of what they
correct
know.
Assess
answer.
Teach
Teach
Teach
Start
Analyze
Analyze
Analyze
y
Adjust
Adjust
Decide and
i l
implement
t nextt
instructional moves.
Adjust
Analyze what
students struggle
with as learning
g
happens in
real time?
Embedded Assessment Cycle
Embedded Assessment Cycle “Blurring the lines” between instruction and assessment. • Read how other’s have found misconceptions
R d h th ’ h f
d i
ti
• Try it yourself!
E b dd d A
t
Embedded Assessment:
Identifying Misconceptions Exercise
Identifying Misconceptions Exercise
Located ~10 diverse articles about misconceptions in varied disciplines. Looked for these trends:
1) Wh t
1) What was the misconception?
th
i
ti ?
2) How was the misconception discovered? 3) What was the suggestion on how to improve student understanding about the misconception. E
l A i l
Example Articles:
Mapping the structures of resonance
(M ld 2009.
(Malde.
2009 N
National
ti
lA
Association
i ti off T
Teachers
h
off Si
Singing)
i )
“Body Mapping helps people discover and correct
misconceptions about the way their bodies are built…”
Misconceptions Articles: Results
y Misconceptions found through various activities:
y performance or application activities,
y disciplinary writing,
y problem solving “show your work” exercises
y Most remedies included instruction with an activity
(hands-on or some kind of problem set.)
Embedded assessment is a useful model for Embedded assessment is a useful model for
finding and fixing misconceptions and supports the conscious competent learning model
conscious competent
learning model
AND
serves as documentation for reporting on p
g
accountability, …thus humanizing the process.
Writing Better Exam Essays in Human Physiology
With permission from D H
Dr. Howard Booth (EMU‐
d B th (EMU Biology)
Bi l
)
ASSESS
HUMAN PHYSIOLOGY
TEACHING IMPROVEMENT PROJECT
ANALYZE
TEACH
ADJUST
y and evolution into a SOTL p
j
History
project
My conviction: science students need to communicate in their discipline
Revealing misconceptions: Student writing is a prime place to asses their
understanding of content but poor writing skills distorts this
assessment and lowers exam scores disproportionately
disproportionately.
(Good students were losing many essay points due to poor writing)
The challenge: good writing is hard work for teachers and students
Traditional resolutions: ignore it, require it, or teach it
Hypothesis: If students understood and applied good organizational and
proofreading techniques to their writing, essay scores would move up to be
comparable to the rest of the exam and provide a truer assessment of their learning
learning.
Goal of the project: to improve the Big Essay scores (worth 25% of the exam; 4 exams per semester)
-Through a focused effort to teach students better essay writing skills
-Through
g a secondary
y focus on improving
p
g organization
g
and coherence
Students Learning Outcome 2: Students will communicate scientific knowledge and concepts in written form.
Students Learning Outcome 3: Students will synthesize scientific knowledge in higher level thinking problems.
17
ASSESS
TEACH Phase 1: Getting Organized
The SOTL Experiment
ANALYZE
TEACH
ADJUST
Control group: Student exam and essay scores from 2008 (before essay focus)
Experimental Group: Fall 2009 and Winter 2010 classes
-All students: essay score and exam scores
Subgroup: student scores from Rewrite Workshop
-Subgroup:
Time Line --- Methods and materials
Pre exam 1: All lecture instructions “how to take the exam” (5 min)
P
Post
exam1:
1
-All lecture illustration of an excellent essay answer (10 min)
-Essay rewrite workshop: For those “performing poorly” or “want to do better”
Instruction-discussion
Instruction
discussion on writing & organization skills (40 min)
Re-write exam1 essay: take home, self timed (30 min)
Re-Grade: scores, written comments & personal discussion
Postt exam 2:
P
2 lecture
l t
di
discussion/illustration
i /ill t ti off a conceptt map off essay (15 min)
i )
Pre exam 4: Opinion survey on rubric & essay scoring
Inclusion of an additional rubric statement.
18
ASSESS:
Essay 1 Example
“Bill”,
A good
Student
89%=“A-” on
Non-essay
Part of exam
Earning
A
Poor
essay
Score
16 of 25
is
64%=“D”
19
ASSESS
TEACH Phase 2: The Workshop
ANALYZE
TEACH
Doing the “Experiment”
Workshop Highlights
•
•
•
•
•
•
ADJUST
Blue print - how to write a good paragraph: Theme introduced in
topic sentence, supporting sentences in logical order, a conclusion.
Science
Sc
e ce writing
g iss concise,
co c se, de
detailed,
a ed, accu
accurate,
a e, b
breadth
ead o
of the
e theme
e e
Building blocks- analyze the question (see printout)- finding out what to include
Main theme, mentioned areas of interest, required 9 terms/phrases
Organizing the construction site- organization of logical sequence
Outline areas of interest and terms-terms flow diagram & concept map
Building it-- apply above -- also, some hints on good construction
Set aside enough time, avoid “wordy” superficial statements,
use the space
p
p
provided- not more , not less. Write neatly,
y 14-18point
p
too small hard to read, too large wastes space. Spell correctly.
Building inspection-- proof read, all “areas of interest” included?’
& terms used correctly, explained, circled and numbered?
match your essay with rubric - avoid losing unnecessary points
Avoid building disasters-- study ! If your blocks crumble----if you don’t know the content, good writing won’t help much
20
ASSESS:
“Bill’ ”
“Bill’s”
Rewrite
Scored
92%
On
The
Rewrite
28%
Better
From a
“D”
p to
Up
an “A-”
21
ASSESS
ANALYZE: What we were looking for
and what we found
The Results
ANALYZE
TEACH
• The quality of the re-writes improved:
F’09, 13/28 (46%) to 23/28 (82%) a gain of 36 %
ADJUST
p
y
• Improvement
carried into exam 2 and beyond
13/28 to 19.8/28 (up 70.1%) on exam 2.
• Overall exam scores moved up over 5% from 74.4% to 79.7% .
• Numerous positive student comments about the experience and their increased
comfort with essay writing.
writing “ I loved the essay format
format- very helpful”
helpful F
F’09
09.
•Participation increased from 15/60(25%) F’09 to 38/61 W’10(62%) (up 37%)
•Rubric Opinion Survey to assess student misconceptions or concern on scoring
•Change was greatest in the poorest writers, about 65% improved dramatically
Data Chart of Essay Score Averages before (W’08) and now (W’10)
Exam 1
Exam 2
Exam 3
Exam 4
17.8
18.3
18.5
19.8
63.5%
65.2%
66.1%
70.8%
16.8
21.0
22.0
22.0
% chang e numbe r/28 possibl e
59.9%
75.5%
78.6%
77.8%
W'08 C ontrol all stude nts' essays
61.7%
63.3%
63.0%
66.1%
All students Averag e
% chang e numbe r/28 possible
Workshop
p stude nt averages
g
And see
the Graph
next slide
lid
22
ASSESS
ANALYZE
TEACH
ANALYZE : Results Graph and Summary
Graph of Essay Score Averages
Before and Now
ADJUST
90.0%
W'08 Control
80.0%
W'10 All
Students
W'10 Workshop
students
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
Series1
control
RESULTS SUMMARY
1. W'10 Workshop participants impoved most from a D+ to a B
2. W'10 all students improved modestly from a C- to a C+
3. W'08 control students improved the least from C- to a C
4. Workshop participants improved most from exam I to exam 2
and continued to improve in exams 3&4
5. All W'10 students benefited compared to W'08 control
6. Essay scores no longer "Disproportionately low"
Workshop 5% above, AllW'10 at all exam averages.
All
W'10
Series2
Workshop
Series3
10.0%
0.0%
1
2
Exams
3
1, 2, 3
&
4
4
I c o mpleted th e s u rvey f o r a B onu s pt. Name: _ ______ _ _____ _ _____ _ _
-- ----- ----- ------- ------- ------- ----- ---tear o ff o r c u t h ere -- --- ------- ----- ------- ---
ANALYZE:
Rubric
Rubric Survey Results
Opi n ion Su rvey on Big Essay P o in ts Dis trib ut ion (sc o ri n g r u bric)
P lease circ le th e lett e r b e fo re
a= s tro n g ly agre e
b = agree
c= n eith er agree n or d isagree
d= d isagree
e= s tro n g ly d isagree
th e statem e n t to in dica te y o ur le v e l of agree m en t w ith it.
5p t
4p t
3p t
2p t
1p t
(An y co mm en ts e x p lain in g y ou r th o ug h ts w o u ld a lso b e v er y h e lp fu l,. T h an ks)
40 s tude n ts resp
p o n ded ( o u t o f 53 still atte n d in g n ear th e en d of t h e se m es ter )
Conclusions:
1 U
1.
Understanding
d
t di
the essay
scoring is not
much of a
problem
2. It’s easy to fix it
for the few who
were concerned
((add a statement to
the exam)
1. ( a b c d e ) A s I read th e essa y ques tio n itse lf , I thin k I h a v e a c lear idea o f h ow I ca n ear n fu ll p o in ts.
Resu lts: 4 @ 2p t, 3@ 3p t, 24 @ 4p t 9 @ 5p t = a v erage 4 .0 o u t o f 5.0
An aly sis: M o st d id n' t see ru b ric sc o rin g as a p ro bl em , b u t 7 (2 0% ) d id h a v e co n cer n s h er e.
C o mm en ts: " v er y c lear"
" th e essa y ques tio n s are usua lly s tra ig h t fo rwar d an d y o u k n o w y ow th ey ar e supp o sed to b e an swer in g." ,
" o n ce I we n t to th e w orks h o p I was ab le to b ett e r u n ders ta n d h o w th e essa y was to b e w ritt en "
2. ( a b c d e ) A separa te ch eck li st a n d e x p lana tio n , n o t par t of th e ques tio n itse lf e x p lain in g p o in t
d istrib u tio n (sc o rin g ru b ric), w o u ld m ake th is p roc ess m o re u n ders ta n da b le.
Resu lts: 1 @ 1, 5 @ 2p t, 9 @ 3 pt ,18 @ 4, 7 @ 5p ts = average o f 3. 6 o u t of 5.0
An aly sis: Qu ite a stro n g fee li n g t h is w o u ld h elp. Bu t read in g t h e co mm en ts left m e th in k in g th is was n' t
v er y im p o rtan t to th em .
("ca n' t h ur t")
C o mm en ts:
"g
g oo d idea, T h is w ill a ll ow stude n ts to h av e a n e x ac t acc o u n t for a ll p o in ts
ts"
"y o ur sc o rin g m eth o d is a lread y c lear"
"h elp fu l for fi rs t e x a m , I f e lt v er y u n co mf o rtab le w ritin g it b ecause I h ad n o c lue w h a t to e x pec t. A g rad in g
ru b ric w o u ld let m e k n o w w h at y ou were e x pec ting"
3. ( a b c d e ) C o n fus io n a b o u t h ow I ear n th e essa y p o in ts c o n cer n s m e as I w rite th e essa y (" n ot
k n o w in g" w h ere y o u mi g h t ga in o r l o se p o in ts in ter feres w ith y ou r th o ug h t pr og ress io n as y o u are p lann in g
a n d wr itin g th e essa y ).
Resu lts: 2 @ 1p t, 13 @ 2 pt, 9 @ 3p t, 8 @ 4p t, 5@5p t, = av erag e o f 2.9 out o f 5
An aly sis: Near n eu tra l n ot stro n g fee li n g o r c o m m e n ts. N o t m uc h o f a n issue for m o st
C o mm en ts:
"Wh il e I'm wr itin g t h e essa y I 'm jus t tryi n g t o d o th e b es t I ca n . I'm n ot w orr ied ab o u t th e p o in ts ." " N o ,
dep th o f a n swer an d lack o f tim e ar e my c o n cer n s", "T h is ca n b e t ro u b li n g b ecause t h e p o in t sy stem isn' t
cu t a n d dr y "
4. (a b c d e ) I th in k m o re in fo rm atio n o n th e sc o rin g ru b ric w o u ld im pro v e m y essa y sc o res.
4
res
Resu lts: 3 @ 1p t, 7 @ 2 pt, 1 4 @ 3p t, 14 @ 4, 1@ 5p t = a v erage o f 3.1 o u t o f 5
An aly sis: A ga in alm o st n eu tra l o v er all b u t a b o u t 30% at leas t fa v o red th e idea. C o mm e n ts:
"m o re in fo rm atio n n e v er h urts", "a sc o rin g ru b ric d o es n' t e nh a n ce o r inh ib it my w ritin g."
5. An y ot h e r sugges tio n s?
24 y ,
"A grad in g ru b ric a t th e b eg inn in g of th e se m es ter w o u ld b e h e lp fu l, I f e lt v er y n er v o us ab o u t th e fi rs t essa
a fter th at it ge ts eas ier"
ASSESS
ADJUST: Continuous fine tuning
ANALYZE
TEACH
ADJUST
1. Already have incorporated a series of changes
•Three “editions” of workshop notes
half page outline of my talking points
full page on writing
two page handout on writing,
organizing, proofreading and rubric
•Adding concept map to lecture presentation W’10
•Expanded “ workshop
workshop” to a take
take-home
home option W’10
W 10
•Last week upgraded the handout and sent e-copies to all
of my spring students as a Pre Exam 1 study sheet.
•Plan to subdivide workshop participants into working
groups this spring.
25
ASSESS
Conclusions
Yes, teach
it again
!
g
TEACH
ANALYZE
Experiment was a success:
ADJUST
-hypothesis supported
-embedded
data
b dd d d
t collected
ll t d
-students engaged & learning outcome goals were met
-essay scores improved and now more accurately
reflect learning (no longer disproportionately low)
-improved essays: better assessment of misconceptions
Cost/ Benefit Analysis
-costs: clocked about 10 hours per semester
-benefits:
students
t d t write
it better--realize&
b tt
li & appreciate
i t itit-- “you
“
care””
course & the department get assessment quantified
I followed my convictions --teaching “good” writing in science
improvement keeps teaching interesting and enjoyable
Bottom line: It was, and is a success at many levels
26
y
Steve Pernecky EMU Biochemistry Program Assessment
EMU Biochemistry Program Assessment
What did we need to get started?
y
• Passion: Committed core of faculty
• Motivation: AQIP/Program Review
• Goal: Continuous Improvement
• Framework: Progression of Student Knowledge
ASSESS
ANALYZE
TEACH
ADJUST
ASSESS
™
Develop four Student Learning Outcomes, each containing
four to six components.
components
Conceptual/quantitative problem solving in chemistry
Lab techniques in chemistry
P f
Professional
i
l skills/standards
kill / t d d
Practice of chemistry as scientific endeavor
™
Develop curriculum map Æ links Student Learning
Outcomes to courses.
™
Brought
g Student Learning
g Outcomes and Curriculum
Mapping to Chemistry faculty to refine/approve .
ASSESS
p
Curriculum Map:
Biochemistry Program
R
R
R
R
R
123/ 124/
281/
121 122 125 126 270 271 282
R
361
R
R
371 372
R
R
R R
373/
Res
376 381 461 463 465 451 452 453 481 475 Coop 415 411 412 413
Students will evaluate
problems in chemistry using
conceptual and quantitative
approaches
h
Stoichiometry
*
*
*
*
*
Phys.-Chem. properties
Chemical bonding and
reactivity
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
Thermodynamics
*
*
*
*
*
*
*
*
*
*
*
Kinetics
*
*
*
*
*
*
Equilibria
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
R = Required course in General Biochemistry Curriculum
AND Professional Biochemistry program
*
*
*
Why Start with Equilibrium?
Why Start with Equilibrium?
y Teacher’s Misconceptions about the Effects of Addition of More Reactants or Products on
y
y
y
y
y
y
Chemical Equilibrium (2009) Cheung, D, Ma, H., Yang, J., Int. J. Sci. Math. Ed. 7, 1111-1133.
pp
g Chemical
Effectiveness of Instruction Based on the Constructivist Approach
on Understanding
Equilibrium Concepts (2003) Akkus, H., Kadayifci, H., Atasov, B., Geban, O., J. Res. Sci. Tech.
Ed. 21, 209-227.
Dynamic Processes of Conceptual Change: Analysis of Constructing Mental Models of Chemical
Equilibrium
q
((2002)) Chiu,, M.,, Chou,, C.,, Liu,, C.,, J. Res. Sci. Teach. 39,, 688-712.
Investigating the Relationship between Students’ Multiple Conceptions of Action and Reaction in
Cases of Static Equilibrium (2001) Palmer, D., Res. Sci. Tech. Ed. 19, 193-204.
An In-Depth Study of Misconceptions in Stoichiometry and Chemical Equilibrium at a South
African University (1996) Huddle, P., Pillay, A., J. Res. Sci. Teach. 33, 65
65-77.
77.
Students’ and Teachers’ Misapplication of Le Chatelier’s Principle: Implications for the Teaching
of Chemical Equilibrium (1995) Quilez-Pardo, J., Solaz-Portoles, J.J., J. Res. Sci. Teach. 32, 939957.
Teaching Chemical Equilibrium and Thermodynamics in Undergraduate General Chemistry
Classes (1995), Banerjee, A.C., J. Chem. Ed. 72, 879-881.
We had a good sense from teaching our own curriculum
that there were misconceptions about chemical equilibrium.
ASSESS
™ Development of an assessment project: Develop an use
instrument for one component of SLO #1: Equilibrium (F
‘08).
™ Faculty
y surveyy of equilibrium
q
exam q
questions used in
chemistry courses (F ‘08).
™ Develop set of questions from survey that can be used to
assess “knowledge
“k
l d progression”
i ” (W ‘09)
‘09).
• Variables tested: (1) Reversibility of chemical reactions; (2) Le
Chatelier’s
Chatelier
s Principle; (3) Quantitative aspects of equilibrium
equilibrium.
• Bloom’s level for each question.
K=Knowledge level, A=Applied level, S=Synthesis level
ASSESS
Distribution of Equilibrium Questions in Four Chemistry Courses
Variable
1: Reactions are reversible
2: Le Chatelier's
Principle
3: Quantitative aspects
Bloom's Level (K, A, S)
(K A S)a
Course
CHEM123
CHEM281
CHEM372 CHEM451
K
*
*
*
*
A
S
*
‐‐‐
‐‐‐
*
*
‐‐‐
*
K
‐‐‐
‐‐‐
*
A
S
‐‐
*
*
*
*
‐‐‐
*
*
K
*
*
‐‐‐
*
A
S
*
‐‐‐
*
‐‐‐
‐‐‐
‐‐‐
*
ASSESS
y instrument in courses ((Su/F ‘09))
™Verify
• Interview students: Determine if equilibrium questions
were clearly stated.
• Instructor
I t t review
i
off questions
ti
and
d rubrics
b i (F ‘09).
‘09) Thi
This
was very important for buy-in.
™Modify instrument (F’09)
(F 09)
• CHEM123/General Chemistry and CHEM451/Biochemistry I:
Did not change in question.
• CHEM372/Organic Chemistry II: Scrapped instrument,
developed new instrument.
• CHEM281/Quantitative Analysis: Make questions reflect
language used in course.
ASSESS
Original
question in
CHEM281
1. Answer the following questions about the bicarbonate buffer system. The pKa for H2CO3 is 6.1. A patient is found to have a [HCO
p
[
/
[ 2CO3]] of 2.4 3­]] of 29.5 mmol/L and a [H
mmol/L. The pH of normal blood is 7.4. Is the patient suffering from acidosis (a blood pH more acidic than 7.4) or from alkalosis (a blood pH more basic than 7.4)? SHOW your answer assuming that the bicarbonate buffer system is the only buffer system controlling blood pH. Variable: 3, Level: K, A Refined
question in
CHEM281
1. HCO3‐ (bicarbonate) is an important species in biological systems. It comes from CO2 dissolving in the blood (pH=7.42). a) Write the equilibrium reactions for CO2 dissolving in blood (Hint: since blood is mostly water, you can consider it to be water in your reactions). b) Some medicines cause people to eliminate HCO3‐ in excess. How would loss of HCO3‐ affect blood pH? c) By adding more of which of the existing species in the system would the pH return to normal (pH 7.4)? ASSESS
1. Bicarbonate buffer system question
a) CO2 + H2O ⇄ H2CO3; H2CO3 ⇄ H+ + HCO3Mark the answer as correct if there are no errors or only minor errors.
Mark the answer as incorrect if there are any significant errors, such as one or
Rubric and
scoring for
question in
CHEM281
more incorrect equations.
b) Elimination of HCO3- would cause the second equilibrium to shift to the right,
increasing H+ concentration and therefore decreasing pH.
Mark the answer as correct if the explanation is appropriate and the conclusion
(decrease in pH) is correct.
Instructor ______________________________
Number
int or
this
M k the
Mark
th answer as incorrect
i
t if th
the
conclusion
l of
i students
is
i incorrect
i
the
th section
explanation
l ________
ti iis
incorrect or incomplete.
Bicarbonate buffer system question
c) Adding
shift the second
decreasing
H+
a) equilibrium
________ to the left,students
answered
the question correctly
students answered the question incorrectly
concentration and therefore increasing
g ________
p
pH.
________
students left the question blank
Mark the answer as correct if the explanation is appropriate.
b) ________
students answered the question correctly
Mark the answer as incorrect if the explanation
is incorrect or incomplete.
________
students answered the question incorrectly
________
students left the question blank
c) ________
students answered the question correctly
________
students answered the question incorrectly
________
students left the question blank
HCO3- would
1.
ASSESS
™Administration of Instrument (W, Su, F ‘10)
Course and Number of Sections/Students in which
Equilibrium was Assessed
Course
CHEM123/Gen. Chem. II
Number of Sections Number of Students
10
339
CHEM281/Quant. Anal.
4
119
CHEM372/Org Chem.
CHEM372/Org.
Chem II
3
39
CHEM451/Biochem. I
2
66
ANALYZE
ADJUST
™ Instructors of each disciplinary group were invited to meet
with to discuss and analyze the data (W ’11)
™ Discussion lead to a number of questions.
y Evaluating the instrument
o
o
Should we change
g the q
questions for each course?
Was Bloom’s level appropriate for each question?
y Core values
o
o
How important is it that students can apply (good “A”
A level) concepts
without really understanding them (poor “K” level)?
How important is it that we address a particular concept for each course
or in the program?
y Adjustments
o
o
Should pedagogy change to incorporate more conceptual teaching of
chemistry concepts?
How willing are we to change a course, the curriculum, what we teach,
and the way we teach it?
General Observations about Assessment Across the C rric l m
Across the Curriculum
™ Personnel: A core group of motivated workers who believe in
assessing for improvement can lay the groundwork quickly!!
• Student Learning Outcomes and Curriculum Mapping
• Develop/verify assessment instruments, level of testing (K, A,S), and
variables tested.
tested
• Chemistry Education researcher – Dr. Amy Johnson
™ Time: Expect
p
to spend
p
lots of time on refinement of the
assessment instruments
• Interview students to validate questions
• Get feedback on questions by instructors to get buy-in
buy in
• Take time enough to get good data
™ Process: Develop a comfortable mechanism (assume no
release
l
titime)) th
thatt will
ill b
benefit
fit your curriculum,
i l
nott jjustt an
institutional requirement.
Michael Tew
Professor
C
Communication, Media and Theatre Arts
i i M di d Th
A
B i t
t l
t f t
Barriers to natural assessment of outcome based learning across disciplines:
y Perceived differences
y Disciplinary habits
y Language and interpretation of outcomes
y Instructional strategy
gy
y Misconceptions
y different work = incompatible data
y Common outcome = common work
y Coordination = compromised integrity
Multicourse Learning in EMU
Multicourse Learning in EMU General Education
y Quantitative Reasoning
y 12 Outcomes
y One of Eight courses in four
disciplines
y Perspectives On a Diverse World
y Global Awareness
5 Outcomes
y One of 16 courses in 12
di i li
disciplines
y US Diversity
y 5 Outcomes
y One
O off 20 courses iin 14
disciplines
y
y Arts
y 6 Outcomes
y Two of 20 courses in 5
disciplines
p
y Humanities
y 7 Outcomes
y Two of 40 courses in 12
disciplines
y Social Sciences
y 7 Outcomes
y Two of 20 courses in 11
disciplines
y Natural Sciences
y 13 Outcomes
y Two of 23 Courses in 7
di i li
disciplines
Multicourse Assessment
y Our Challenge: Multiple courses across different
disciplines + common outcomes
y Our Commitments: to disciplinary habits; to
embedded assessment; to direct assessment
y Our Question:
y Can we understand student learning in common ways
when learning is manifested in different ways/in
different performances?
An Extension of the Model
• Outcome in the discipline
• Master themes
• Evidence in the discipline
Teach
Assess
Adjust
Analyze
• Shared values and challenges
GEMAP
Faculty Focus Groups
Interpret Outcomes
Offer Exemplars
Common Language/Values/Challenges/Themes
Build Outcome Based Analytic Tool
Compile Outcome Portfolio of student work
Data Generation and Analysis
Application of Tools
Contribution to Understanding of Student C
ib i U d
di f S d Learning
d
U.S. Diversity Outcome: examine causes and consequences of social intolerance in the U.S.
y Native American
Literature Course:
y A critical
iti l essay on
selected literature
y History Course in Race,
Class, and Gender
y An essay exam
y African American
Theatre Course:
y Performance
P f
off a
scene with
interpretation
annotations
y Inter-racial/ethnic
C
Communication
i ti C
Course:
y A persuasive speech
Contributions to Process
y Common indicators of
outcome based
learning:
y Students use evidence
to defend
arguments/perspectives
y Students articulate
p p
perspectives
in
meaningful ways
y Students use language
in a way
a that
demonstrates an
understanding of
intersectionality
y Students present
moral/ethical concerns
with an awareness of
historical/cultural
context
Process Guided by Questions:
TEACH:
y
y
1. What does a
specific outcome
mean in your
discipline?
Based on naturally
occurring outcome
reflective
assignments, what
do you value and
what is
challenging?
ASSESS:
y 2. What
common
themes/language/expec
tations and
understandings do we
share?
y How does that inform
our understanding of
student learning as
reflected in student
work?
Process Guided by Questions:
ANALYZE:
y
y
Independent of
disciplinary differences,
what indicators of
student learning
emerge from our
shared understanding?
g
How can we
characterize levels of
student
t d t performance
f
against those indicators
(for a particular
outcome)?
ADJUST:
y Applying “this rubric” to a
portfolio of
multidiscipline/multi genre
multidiscipline/multi-genre
assignments,
y What contribution can we
make to our understanding
of student learning in
General Education?
Community Committed to
Community Committed to Humanized Assessment
Common values l grounding outcomes
Tolerance for divergent practices
Enlightened U d t di Understanding of Student Learning
The Added Values
y Prizing Humanized Processes
y Pedagogy and Teaching at the Center
y Mirror of natural instructional processes
y Disciplinary Habits + Common Values
y Record of ongoing dialogue provides documentation
for accountability:
y
y
y
y
What should they know?
How do we see that?
What have we learned?
Wh t are we doing
What
d i about
b t th
that?
t?
T k H
E b dd d A
t
Take Home; Embedded Assessment: ASSESS
ANALYZE
TEACH
ADJUST
Fosters professional development:
y shifts unconscious competent to
conscious reflection while
purposefully “looking for”
misconceptions;
i
ti
y As a narrative , it “captures”
improvements for
accountability;
… thus, humanizing assessment.
Unconscious Competent
Conscious Reflection
Awareness of continuous improvement of student learning
References
y
Gallagher, J. J. 1999. Improving science teaching and student achievement through embedded
assessment. MSTA Journal. http://www.msta-mich.org/index.php/publications/journalArticle/20
y
Gallagher, J. J. 2007. Teaching for Understanding: A practical guide for middle and high school
teaching. Columbus, Ohio. Pearson, Merrill Prentice Hall.
y
Liggit, P. Humanizing Academic Accountability: Embedded Assessment Gets at the Heart of
Teaching and Learning. Spiraling Upwards: The Scholarship of Teaching and Learning. Ypsilanti,
MI Bruce
MI:
B
K.
K Nelson
N l
Faculty
F
lt Development
D
l
t Center,
C t Eastern
E t
Michigan
Mi hi
U
University.
i
it IIn press.
y
Lin Ding et. al. 2009. Are we asking the right questions? Validating clicker question sequences by
student interviews American Association of Physics Teachers.
y
Kuhle, Barry X. and Jessica M. Barber, and Adam S. Bristol. "Predicting Students'
Performance in Introductory Psychology from their Psychology Misconceptions." Journal of
Instructional Psychology. 36.2 (2009): 119-24.
y
Malde, Melissa. "Mapping the Structures of Resonance." Journal of Singing. 65.5
(2009): 521-9.
C
I f
Contact Info
Peggy Liggit, Ph.D.
Director of Academic Assessment
Office of Institutional Effectiveness
234 McKenny
Eastern Michigan University
Ypsilanti, MI 48197
Peggy.Liggit@emich.edu
734-487-0199 (direct)
734-487-8288 (Office)
734-487-8290 (FAX)
Visit our Assessment Website
Steven Pernecky, Ph.D.
Professor
Undergraduate
Undergraduate Coordinator/Associate Chair
Dept. of Chemistry
Eastern Michigan University
Ypsilanti, MI 48197
Phone: 734‐487‐0383
Fax: 734‐487‐1496
www.tinyurl.com/emuassess