Session Nine
Evaluating the Curriculum
• Simply put, curriculum evaluation determines
the value of the curriculum for the students
• It also can measure if educational psychology
principles are being used
• Most evaluation measure measure
achievement, there is a call for measuring
thins like creativity, compassion, commitment
ad enthusiasm
Evaluating the Curriculum
• Those that call for measuring things like
creativity, compassion, commitment ad
enthusiasm are against the curriculum being
measured by test
• However, test provide clear evidence of what
students did
The purpose for evaluation
• Ideally there would be multiple measures
throughout the year to evaluate the
curriculum rather than just relying on just one
• The disadvantage is that you would waste
weeks testing – The PSSA take almost two
weeks and if you did that three times you
would lose over a month of school
The purpose for evaluation
• Curriculum Evaluation look at whether the
curriculum is producing the desired results
• Does it get the students to produce at the
level expected?
• Parents and teachers want to know how their
students/children compare to the rest of the
state or nationally
Activity
• What are some forms of assessment that can
be used to measure curriculum?
• Discuss in your small groups
The Purpose for Evaluation
• Means of Assessment
–
–
–
–
–
–
–
Written tests
Review of students work
Review of portfolios
Observing students
Questionnaires,
Checklists
Inventories
• Video performances
• recorded lessons
• Analyses of lesson plans
The Purpose for Evaluation
• At the classroom level, a great deal of
qualitative and informal data is used to
interpret
• Quantitative can be used as well
What do you look for?
• Four types of information
– Intrinsic
– Instrumental
– Comparative
– Decision value
What do you look for?
• Intrinsic
– Is the curriculum using best practices that are
most up to date
– Have teachers kept to the goals of the curriculum
What do you look for?
– Instrumental
• IS the curriculum reaching who it was suppose to reach
• Is it doing what it is supposed to
• Will it accomplish what it is intended to do
What do you look for?
– Comparative
– This is when it is compared to another program or
curriculum. Is it better
– what parts are weaker
What do you look for?
– Idealization what ways can we improve the
curriculum
– Are there other ways to make the program better
What do you look for?
– Decision Value- is the final part, most important
– Decide if you keep curriculum as is,
– Modify it
– Discard and get a new program
– This part is ongoing
Activity
• If you were put in charge of evaluating a
curriculum
• What are the things you would evaluate
• Think of when you were in school.
What do you look for?
– Decision Value- is the final part, most important
– Decide if you keep curriculum as is,
– Modify it
– Discard and get a new program
– This part is ongoing
What is Evaluation
• Some define it simply as gathering data to
make a decision
• The formal determination of the quality,
effectiveness, or value of a program
• Others see it as a critical inquiry, studying
phenomena so as to make informed
judgments
What is Evaluation
• How does measurement differ from
evaluation
• Define this in your groups?
Measurement vs. Evaluation
• Measurement is assigning a numbers or
numerals to something
• The score was a 20
• The score was a 95
• Which is better
Measurement vs. Evaluation
• Evaluation is assigning a value to that number
• The score was a 20 out of 20
• The score was a 95 out of a 100
• Now can you tell Which is better
Measurement vs. Evaluation
• Measurement enables educators to record
students’ degree of competency
• That is not enough, educators must bring
meaning to that number, what is enough,
what is good what is not satisfactory
• A PE teacher has a child do pull ups, how
many are satisfactory
• How many words a minute (reading) is
adequate?
Several Approaches to Evaluation
• In most cases the one method to evaluate a
curriculum can be used with other subjects,
most are not content specific
• Basically-gather data and compare it to the
goals of the curriculum. Did you meet them
Several Approaches to Evaluation
• However, how people evaluate data and
compare it to the goals of the curriculum
varies on their educational philosophy
• A behaviorist would look to see if the students
can perform the designated behaviors
established as by the curriculum
• Humanist may be looking to see if a child’s self
esteem improved
Several Approaches to Evaluation
• However, how people evaluate data and
compare it to the goals of the curriculum
varies on their educational philosophy
• A behaviorist would look to see if the students
can perform the designated behaviors
established as by the curriculum
• Humanist may be looking to see if a child’s self
esteem improved
Several Approaches to Evaluation
• In general, educators make one of three
decisions related to the curriculum
– Maintain it as is
– Revise it
– Replace it
Several Approaches to Evaluation
• In general, educators make one of three decisions
related to the curriculum
– Maintain it as is
– Revise it
– Replace it
• Assess students and teachers in terms of
instruction and learning
• Decide whether the existing managerial
organization of the school should be maintained
Several Approaches to Evaluation
• Evaluation can take place at many levels
•
•
•
•
•
•
Nationally
State
District
School
Grade
Course
Scientific or Humanistic
• Scientific evaluators favor experimental
approaches
• Very similar to qualitative design methods
– Control groups
– Standardized practices
– Random selection
Scientific or Humanistic
• Scientific evaluators focus on the learner
• They use test scores and compare students
based on standardized tests
Scientific or Humanistic
• However, many are saying that these scientific
methods are not enough and do not
accurately measure the quality of the
curriculum
• There is a call for some non-traditional
evaluation procedures
Scientific or Humanistic
• Advocates for this type of evaluation, claim
that Nontraditional evaluation procedures
paint a more complete picture of the
curriculum
• The evaluation describes the benefits, it does
not compare numbers
• They rely on observations
Scientific or Humanistic
• Nontraditional evaluations, rely on qualitative
methods
– Purposeful selection of participants rather than
Radom
– Claim there are multiple realities
– For years did not get a lot of credit, but certainly
has value
Mixed-Method Studies
• Is when both qualitative and quantitative approaches to
gathering, interpreting and reporting data are used together
• Three types
– Explanatory design
– Exploratory design
– Triangulation design
Mixed-Method Studies
• In order to be considered a mixed study it needs to contain a
significant portion of two designs and have an impact on the
work
If a qualitative design only used random sampling it still would
be considered qualitative or if a quantitative included a few
open ended questions on a survey, it is still is considered
quantitative.
Explanatory Design
• Most common type of mixed-method design
• The study is conducted first as a quantitative study, data is
gathered and interpreted as a quantitative study….Then
qualitative data is collected to shed a greater light on the
findings of the quantitative study
Exploratory Design
• This design a qualitative study is conducted
first. The purpose is to discover themes and
language. Then a quantitative study is built
around the themes or surveys are developed
with language uncovered.
Triangulation Design
• In this design, both quantitative and
qualitative studies are carried out at the same
time and the results are used to verify the
work of the other study.
Humanistic Approaches
• Five major humanistic approaches
– Interpretive
• evaluates the program in its social context, gives meaning to
the actions of the people involved
– Artistic
• The curriculum is observed and announces what is good or
bad about the curriculum, looks at the quality of the
relationships
– Systematic
• The most objective of the method, tries to be objective with
descriptions, tries to use logical reasoning, but does not rely
only on statistical data
Humanistic Approaches
• Five major humanistic approaches
– Theory driven
• Compare curriculum to a variety of philosophical
theories
– Critical-emancipatory
• Judge curriculum on how well it counters social forces
that impede an individual’s progress
Utilitarian versus Intuitionist
• Utilitarian takes the look at a curriculum based
on how efficient it is in helping a large group
of students- programs that allow the most
students to achieve would rate well
• The intuitionist approach judges a program’s
impact on a small group or on the individual
Intrinsic vs. Payoff approach
• Intrinsic approach looks at the curriculum
itself-the assumption is, if the plan is good,
then it will work
• The payoff approach looks at the difference in
scores between when students started on a
program and when they finished a program
– Pre and post testing
Formative and Summative evaluation
• Formative
– Are activities taken to improve an intended
activity
– Carried out during program development
– Provides evidence that will direct the curriculum
writers during the writing of the curriculum
– Uses sub units in a pilot or pretrial situation
– Provides information to either adjust, keep or get
rid of portions
Formative and Summative evaluation
• Formative
• In phase two, various activities are done to
determine if if they have value, does it achieve
its intended purpose ( after the pilot)
• Final phase- the curriculum is field tested- it is
evaluated as it is taught to the students
Formative and Summative evaluation
• Summative- poses the question has the
curriculum worked- done after the curriculum
has been implemented
• Has four levels of evaluation
– Level One-How did the students react, was it as
anticipated?
– Level Two- Have the students gain the knowledge
intended?
Formative and Summative evaluation
• Summative– Level Three-Can the students employ the newly
acquired skills?
– Level Two- Evaluate results- difficult- sometimes a
focus group is used for this
Evaluation Models
• There are two Scientific Models
– Congruence-contingency Model
– Context, input, process and Product model
• There are Two Humanistic models
– Connoisseurship and criticism Model
– Illuminative Model
Evaluation Models
•
•
•
•
•
Congruence-contingency Model
Focused on using objective data to make decisions
The sue of formal process are emphasized
Data was important in this model
Three are three types of data
• Antecedent Data
• Transaction
• outcome
Evaluation Models
• Congruence-contingency Model
• Antecedent Data- is any condition that existed prior the
teaching and learning took place that may have an impact
on an outcome
– Includes- aptitudes, previous achievement scores, psychological
profile scores, Discipline and attendance, teachers years of
teaching and teacher behavior rating
Evaluation Models
• Congruence-contingency Model
• Outcome – includes students achievement and sometimes
student attitudes and motor skills
• These outcomes are measured against what was
anticipated and planned for. Where they are
different explanations are sought or the curriculum is
modified
Evaluation Models
• Context, input, process, Product
• Context- study the program in the context or environment
it was used-does it matched the desired or portrayed
conditions
– Then you focus on un met or missed opportunities
• Input- provides input regarding resources
–
–
–
–
Were the stated resources allocated
Is there a way to achieve the goal with fewer resources
Are the instructional practices appropriate?
Are their other strategies that could meet the objectives?
Evaluation Models
• Context, input, process, Product
• Process- looks at implementation decisions that control or
mange the program
– Detect or predict defects
– Gain information to make decisions
• Product- occurs during implementation- it is a piloting
process that allows you to debug
– Is the final program delivering what it is intended to do?
Humanistic Models
• Does not rely on quantitative data
• More holistic
• Focuses more on human interaction than on
outcomes
Humanistic Models
• Connoisseurship and the criticism model
• Both models are designed to produce a rich
description of educational life as a
consequences of a new program
• It is a very qualitative method
Humanistic Models
• Connoisseurship
– Is a personal process. The report is either done by
or for the practitioner
– It examines, curricular design, instructional
practices, material
Humanistic Models
• criticism model– Works in a similar manner. Uses observation to
examine all parts of the curriculum, but the report
developed is reported publicly
Humanistic Models
• Illuminative model
– Sheds light on a program as it functions
– It has three steps
• Observation- observers give and overview of the
program and give a context for which it is being
delivered
• Further Inquiry-evaluators separate the significant from
the significant and seeks to determine whether the
program works and why or why not
• Explanation not a judgment process, but furnish data
about what is happening
Action research
Action Research
• Action Research is the most popular type of research
now being done with teachers in schools
• It is at the far end of the continuum, full participation
• Ideally once it is up and changes have been made,
researcher can leave, and let the changes be
implemented by the others involved that will remain
on site
Action Research
• Three main types
– Practical Participation
– Participatory Action Research
– Action research as a revolutionary Act- Not going
to cover-
Action Research
• Practical participation
– Have a setting that needs improvement, or a
problem that needs fixing.
– There is a plan that is believed can fix the problem
Action Research
• Practical Participation
– plan is developed to treat the PROBLEM
– The researcher, working with others, implements
the plan
– Data is collected to see if the plan worked.
Action Research
• Practical Research
• Example- Students are entering first grade with
poor phonetic skills
• Theory- student have poor phonemic Awareness
and are having difficulty because they are not
ready to learn phonics
• All Kindergarten students spend time on a
phonemic awareness program
• As children enter first grade, scores are measured.
Compared with the classes progress as well as to
other classes
Action Research
• Participatory Action Research
• This type of research takes a similar course as
the practical research. However, it is with a
much more serious situation
• Involves changing attitudes of teachers or
students
Action Research
• Participatory Action Research– Instead of a simple plan, research is first done to
determine the problem
– Then a plan is developed
– Then the researcher oversees the plan being
implemented
Action Research
• Finally, the plan is evaluated to measure its
effectiveness.
• Examples, Changing racial beliefs- creating
greater understanding about interacting with
minority students and minority student
interacting with non-minority teachers.
Evaluation Steps
• Focus on the curricular Phenomena to be
evaluated
• Collect the information
• Organize the information
• Analyze the information
• Report the information
• Recycle the information- it is on going so keep
doing it
High Stakes test
• Standardized so they can be compared
• Controversial
– What are some of the issues
Norm-referenced Approach
• Is when scores are compared to a reference
group or a norm of others that have taken the
test. The reference groups usually has the
same characteristics such as age, or grade
• Performance is described in relation to
position such as a percentile rank. 60
percentile mean that a child scored as well as
or better than 60 % of the norm group
Norm-referenced Approach
• Some norm-referenced test show a rank for a
national score as well as a local score.
Sometimes in very good or very poor districts,
these can be very different
Activity
• If a child had a 40 percent locally and a 85
percent nationally, is this the sign of a high or
low achieving school district
• You are in a meeting, Describe to a parent
what this means
• It means that 60% of the students in the
district scored at or above the 85th percentile
nationally.
Norm-referenced Approach
• Norm-referenced test can show a high
variation between different levels and be used
to sort for homogenous grouping ( Not in very
high achieving districts)
• Minorities are sometimes over represented .
There is a higher percentage included to make
sure the groups is represented accurately
Criterion-referenced Measures
• These measures compare a child to some
established standard or level of performance
or skill.
• Not great for separating student if a lack of
variance among scores exist
Standardized Measures
• A uniformed procedure for administration and
scoring of test has been established and must
be followed.
• Prepared commercially by experts
• Care has been given to reducing cultural bias,
validity, reliability and bias
• Not great for evaluating local curriculum in
areas of science and social studies
Standards-Based Tests
• Are based on principles of criterion referenced
test
• They are “Domain referenced- questions
drawn from a larger domain of knowledge
• Typically judged by levels such as proficient or
not-proficient.
• Used to evaluate schools
• Referred to as high stakes tests
Standardized Achievement Test
• Most are norm referenced although some are
criterion referenced
• Must select appropriate test for task
• Several types
– Diagnostic
– Achievement
– Batteries
Standardized Achievement Test
• Diagnostic tests– Identify specific strengths and weaknesses in a
particular discipline or area- IRI, QRI, Math
Inventories
• Achievement– Measure achievement in a single subject or area
• Battery
– Measure achievement in several areas
Scores from Norm -Referenced
Measures
• Grade equivalent
• Indicates how an individual compares in a
normative group in terms of grade level
• 5.0 is the average score of a norm group at the
stat of fifth grade
• 5.2 is the fifth grade Second month (
November( September -.0, October -.1
November.2, december-.3, January-.4)
Scores from Norm -Referenced
Measures
• Standard Score- Is based on o to 99.9 The closer to
99.9 the higher the score.
– Many times a standard score uses 100 as the mean, and 15
id the standard deviation
• It is based on a Z-score, with a mean and three
standard deviations on either side
• 50 Is mean +1sd=84, =2sd=98, =3sd=99.9
• -1sd=16,-2sd=2,-3sd=.1
Standard Scores
• Standards scores are used frequently and the
mean can be determine arbitrarily as well as
the standard deviation
• SAT Mean = 500, +1sd=600, +2sd=700,
+3sd=800, -1sd=400,-2=300,-3sd=200