Department of History and
Government
School of Liberal Arts
B.A. in Social Studies
Education
CIP: 13.1318
Program Code: 135
Fall 2010
Program Quality Improvement Report 2009-2010
1
I.
Student-Learning Outcomes
Fall 2010
Program Quality Improvement Report 2009-2010
2
Program Goals and Student-Learning Outcomes
Program Goal 1: Mastery of Social
Studies Content Knowledge
Program Goal 2: Gaining
Classroom Experience in Planning,
Teaching, and Assessment in the
Social Studies
Program Goal 3: Building the
Foundation for Becoming a
Professional Educator
Associated Student Learning Outcome 1
Program completers will demonstrate
acceptable levels of factual knowledge
and understanding in accordance to the
10 thematic strands as established by
the National Council for Social Studies
(NCSS), the Major Field Test in History,
and in accordance with the two following
Oklahoma Subject Area Tests (OSATs) in
social studies:
1) U.S. History/Oklahoma History/
U.S. Government/Economics
-and/or2) World History/Geography
Associated Student Learning
Outcome 2
Program completers will know how
to:
Associated Student Learning Outcome 3
Program completers will:
Fall 2010
1) compose effective daily and
long-range lesson plans
1) compile and present a professional
portfolio of all their relevant
educational achievements, reflections,
and artifacts
2) teach secondary-level social
studies content in a clear,
organized manner
2) seek certification and licensure to
teach social studies at the 7-12 grade
levels
3) assess levels of student
performance and make necessary
preparatory and instructional
adjustments to achieve optimal
student learning
3) be monitored and evaluated
regarding their post-graduate program
reflections and employment trends
Program Quality Improvement Report 2009-2010
3
Program Goals and Student-Learning Outcomes
• Student Learning Outcomes Analyzed in this PQIR
– In keeping with discussions of the Social Studies Program Assessment
Committee from Fall Term 2009, the following priority objectives were assessed
this cycle:
- Priority Outcome 1: Program completers will demonstrate acceptable levels of
factual knowledge and understanding in accordance to the 10 thematic strands as
established by the National Council for Social Studies (NCSS), the Major Field Test in
history, and in accordance with the two following Oklahoma Subject Area Tests (OSATs) in
social studies: U.S.History/Oklahoma History/U.S. Government/Economics -and/orWorld History/Geography.
- Priority Outcome 2: Program completers will know how to compose effective daily and
long-range lesson plans, teach secondary-level social studies content in a clear, organized
manner, and assess levels of student performance and make necessary preparatory and
instructional adjustments to achieve optimal student learning.
- Priority Outcome 3: Program completers will compile and present a professional portfolio
of all their relevant educational achievements, reflections, and artifacts, seek certification
and licensure to teach social studies at the 7-12 grade levels, and be monitored and
evaluated regarding their post-graduate program reflections and employment trends.
Program Quality Improvement Report 2009-2010
4
II.
Alignment of Outcomes
Fall 2010
Program Quality Improvement Report 2009-2010
5
Alignment of Goals and Objectives I
Correspondence between Social Studies Education Program Learning Outcomes and the Cameron University
Mission:
Fostering Engaged Citizenship:
– The Social Studies Education Program of Cameron University assists in improving retention and
graduation by providing majors with the development of intellectual and occupational
competencies.
Student-Centered, Experiential & Practical Learning for the Long Run:
– The practical and experiential approach to the Social Studies Education Program faculty take to
teaching the skills needed for secondary teachers ensures that the program serves student needs for
lifelong learning as well as their professional aspirations and allows for steady program
improvement.
Preparing Students for Professional Success:
– By offering both coursework and practical experiences for potential secondary teachers, the Bachelor
of Arts Degree in Social Studies Education provides the foundation for lifelong learners who will
strive to improve the quality of social studies education across southwest Oklahoma and beyond.
Contributing to the Community:
– Social Studies Education Program faculty engage with colleagues in the CU community and members
of the public to promote productive dialogue across a range of issues.
Fall 2010
Program Quality Improvement Report 2009-2010
6
Alignment of Goals and Objectives II
Correspondence between Social Studies Education Program Learning Outcomes and School of Liberal
Arts Mission:
Quality Baccalaureate programs in the Humanities:
–
Social Studies Education Program faculty provide students a rigorous training in the basic historical knowledge of all
major world regions so they may teach secondary-level social studies content in a clear, organized manner. Faculty
also assess levels of student performance and make necessary preparatory and instructional adjustments to achieve
optimal student learning. To assist students in the application of these skills in different contexts, program faculty also
work to provide students with professional experiences that will lead to their future success and enrich their
experience at CU, including placement in secondary classrooms as observers and instructors.
General Education:
–
The Social Studies Education Program plays a fundamental role in the General Education mission of the School of
Liberal Arts. It assists in improving retention and graduation by providing majors with basic analytical skills and
equipping them to be active and effective students and citizens.
Student-Centered Learning:
–
To assist in the process of achieving the program’s major student learning outcomes, Social Studies Education faculty
use a variety of creative approaches to bring students into the learning process, including group discussion, interactive
lecturing, and apprenticeship-based learning.
Community Outreach Through Faculty Scholarship, Community Partnerships, and Public Lectures,
Symposia, and Outreach efforts:
–
Fall 2010
In order to support the primary student learning outcomes, Social Studies Education Program faculty maintain their
disciplinary engagement by interacting with colleagues in the CU community and members of the public through
scholarship within and across disciplines; public lectures and academic conferences; and teacher institutes, policy
forums, and other public venues.
Program Quality Improvement Report 2009-2010
7
Alignment of Goals and Objectives III
Correspondence between Social Studies Education Program Learning Outcomes and Department of
History & Government Mission:
Baccalaureate programs in History, Social Studies Education and Political Science:
–
Social Studies Education Program faculty provide Social Studies Education Majors with rigorous training in historical
knowledge of all major world regions, gaining classroom experience in planning, teaching, and assessment in the
Social Studies.
General Education:
–
The Social Studies Education Program plays a major part in the General Education mission of the Department of
History & Government, equipping majors to be active and effective students and citizens
Community Outreach Through Faculty Scholarship, Community Partnerships, and Public Lectures, Symposia,
and Outreach efforts:
–
Fall 2010
In order to support the Social Studies Education Program’s primary student learning outcomes, Socials Studies
Education Program faculty maintain their disciplinary engagement by interacting with colleagues in the CU community
and members of the public through scholarship within and across disciplines; public lectures and academic
conferences; and teacher institutes, policy forums, and other public venues.
Program Quality Improvement Report 2009-2010
8
Alignment of Goals and Objectives IV
Correspondence between Social Studies Education Program Learning Outcomes and the Social Studies
Education Program Mission:
At the present time the Social Studies Education Program has not created a separate program mission given
the close correspondence between the departmental and program aims.
Fall 2010
Program Quality Improvement Report 2009-2010
9
Alignment of Goals and Objectives V
Correspondence between Social Studies Education Program Learning Outcomes and the
Cameron University Plan 2013:
In keeping with Cameron University’s Plan 2013, the Social Studies Education Program has
focused its Student Learning Outcomes to enrich students’ lives in terms of practical skills
and knowledge, their cultural and intellectual foundations, and their experiences with the
university community, the wider world of Southwest Oklahoma, the region, and the nation
as outlined below.
Relevant to GOAL 1, subsection 1: Maintain and enhance Cameron’s commitment to
providing programs of the highest quality in instruction, research, and service to better
meet the needs of the citizens of the region.
1) The program’s main objective is to provide local and regional school districts with
highly skilled secondary social studies teachers that have completed a nationally
recognized degree program at Cameron University.
Relevant to GOAL I, subsection 3: Ensure effective assessment of student learning
including experiential learning.
1) A main objective is to place majors in local school districts to give students on the
job experience as teachers.
Fall 2010
Program Quality Improvement Report 2009-2010
10
Alignment of Goals and Objectives V
Correspondence between History Program Learning Outcomes and the Cameron
University Plan 2013:
Relevant to GOAL I, subsection 5: Assure efficient, effective course delivery in multiple
formats.
As part of its efforts to ensure that Social Studies Education Majors and CU students
receive breadth of educational experiences to promote the achievement of the
program’s three main goals.
Relevant to GOAL II, subsection 3: Implement centralized academic advisement for
lower division students complemented by quality faculty advisement at the upper
division levels.
As an outgrowth of providing majors with the rigorous training that they require,
Social Studies Education Program faculty have increasingly emphasized quality
advisement within the Social Studies Education Program (and in particular advisement
that helps students to access opportunities here at Cameron and at the next stage of
their lives) and as part of this process we seek to maximize student exposure to
professional opportunities that bring them into contact with peers and the public
across Oklahoma and the nation.
Fall 2010
Program Quality Improvement Report 2009-2010
11
Alignment of Goals and Objectives V
Correspondence between Social Studies Education Program Learning Outcomes and the
Cameron University Plan 2013:
Relevant to GOAL IV, subsection 4: Increase educational partnerships with common
education, career technology centers, community colleges, and other Oklahoma
universities.
The Social Studies Education Program faculty regularly collaborate with
colleagues across the state to carry out research, public outreach, and
teaching missions, which either build directly on current program capacities
in order to serve our majors and CU students more effectively or assist other
programs in academia and beyond.
Relevant to GOAL IV, subsection 5: Provide student, faculty, and staff resources and
expertise to support the community.
The program’s main objective is to provide local and regional school districts
with highly skilled secondary social studies teachers that have completed a
nationally recognized degree program at Cameron University.
Fall 2010
Program Quality Improvement Report 2009-2010
12
III.
Measures of Learning and Service Outcomes
Fall 2010
Program Quality Improvement Report 2009-2010
13
Measures of Learning and Service Outcomes
• Direct Measures of Student Learning
– Program Goal 1: Mastery of Social Studies Content Knowledge
• A) Standardized Exam: OSAT Scores
- U.S. History/Oklahoma History/U.S. Government/Economics - and/or World History/Geography
• B) Standardized Exam: MFT Scores
– Program Goal 2: Gaining Classroom Experience in Planning, Teaching, and
Assessment in the Social Studies
• A) Performance Rating: Lesson-Plan Assignments
– 1) Performance Rating: Lesson-Plan Assignments
– 2) Performance Rating: Student-Teaching Evaluation Forms
– Program Goal 3: Building the Foundation for Becoming a Professional Educator
• A) Standardized Exam: Oklahoma Professional Teaching Examination (OPTE) Scores
• Indirect Measures of Student Learning
– Program Goal 1: Anonymous Post Graduate Survey
Fall 2010
Program Quality Improvement Report 2009-2010
14
Measures of Learning and Service Outcomes
• Methodologies for Program Improvement Leading to
Improvements in Student Deficits
– Given the Higher Learning Commission’s desire for corrective measures
in order to improve student deficiencies before graduation …
• in Fall Term of each AY the Social Studies Education Program Assessment
Advisory Committee evaluates the PQIR results from the last evaluative
cycle, instituting changes as needed on an annual basis.
• the Post-Graduate Survey provides student feedback that the Social
Studies Education Program acts on as needed.
Fall 2010
Program Quality Improvement Report 2009-2010
15
IV.
Program actions since last assessment
presentation
Fall 2009
Program Quality Improvement Report 2009-2010
16
Program Actions Since Last Assessment
Presentation
• Student-Learning Outcome 1: Mastery of Social Studies
Content Knowledge
– Action taken:
– The committee continued to monitor performances compared to the
History majors, national performance, and year-to-year trends to see if
program changes should be made in the future.
Fall 2009
Program Quality Improvement Report 2009-2010
17
Program Actions Since Last Assessment
Presentation
Student-Learning Outcome 2: Gaining Classroom Experience
in Planning, Teaching, and Assessment in the Social Studies
– Action taken:
– The committee discussed the utility and consistency of scoring based
upon the current scoring rubrics.
– The committee continued to monitor performances compared to the
History majors, national performance, and year-to-year trends to see if
program changes should be made in the future.
Fall 2009
Program Quality Improvement Report 2009-2010
18
Program Actions Since Last Assessment
Presentation
• Student-Learning Outcome 3: Building the Foundation for
Becoming a Professional Educator
– Action taken:
– The committee continued to monitor performances compared to the
History majors, national performance, and year-to-year trends to see if
program changes should be made in the future.
Fall 2009
Program Quality Improvement Report 2009-2010
19
V.
Priority Student-Learning
Outcomes, Assessment Data,
and Action Plans
Fall 2009
Program Quality Improvement Report 2009-2010
20
Student Learning Outcome I Measurement
Plan
PROGRAM GOAL: Mastery of Social Studies Content Knowledge
MEASUREMENT OF Student Learning Outcome
Student Learning
Outcome 1
Program
completers will
demonstrate a
satisfactory ability
to recall, apply,
and appraise the
explanatory value
of factual
knowledge related
to: U.S. History,
European History,
World History,
Oklahoma History,
U.S. Government,
Geography, and
Economics.
Fall 2010
CURRICULUM
AREA OR TARGET
AUDIENCE
Social Studies
Education Majors
and student in
HIST 4773
Methods of
Teaching Social
Studies
Measurements
U.S. History/
Oklahoma
History/ U.S.
Government/
Economics OSAT
standardized
test (direct)
-and/orWorld History/
Geography OSAT
standardized
test (direct)
-andthe Educational
Testing Service’s
Major Field Test
in History, a
standardized
test (direct).
Methods used to determine
validity of measurement
instruments
Content Validity as
Established by State Normreferenced scores generated
by the Oklahoma
Commission on Teacher
Preparation (OCTP)
Oklahoma Subject Area Test
(OSAT)
and
Content Validity as
Established by National
Norm-referenced scores
generated by the
Educational Testing Service
for its Major Field Test in
History .
Program Quality Improvement Report 2009-2010
Methods used
to determine
reliability of
measurements
Schedule for
measurements
OCTP and ETSestablished
measures
Annually, Fall
and Spring
Semester
21
Display of assessment data
Mastery of Social Studies Content Knowledge
•
Priority Outcome 1: Program completers will demonstrate a satisfactory ability to recall,
apply, and appraise the explanatory value of factual knowledge related to: U.S. History,
European History, World History, Oklahoma History, U.S. Government, Geography, and
Economics.
• Description and Rationale of Assessment Measurement:
– Oklahoma Subject Area Test (OSAT) and Major Field Test in History: Social Studies
Education Majors took the Oklahoma Commission on Teacher Preparation (OCTP)
Oklahoma Subject Area Test (OSAT) and Major Field Test (MFT) in History of the
Educational Testing Service as part of the HIST 4773 Methods of Teaching Social Studies.
Students must score above a 240 in order to pass the OSAT and we have classified an
MFT score within 5 percentile points of the national mean as a satisfactory
performance.
– Rationale: an objective testing instrument provides an excellent way to measure the
topical coverage of our curriculum which allows the Social Studies Education Program to
ensure a proper curricular balance.
Fall 2010
Program Quality Improvement Report
2009-2010
22
Display of assessment data
Mastery of Social Studies Content Knowledge
•
•
•
•
•
•
1) OSAT Scores: U.S. History/Oklahoma History/U.S. Government/Economics
- and/or - World History/Geography
Five students took the U.S. History/Oklahoma History/U.S. Government/Economics
OSAT and all five passed.
Three students took the World History/Geography OSAT and only one passed.
The NCSS requires an 80% overall pass rate for one test for accreditation purposes - a
passing score is a 240 or better out of 300.
The Scoring Rubric for Social Studies Education Candidate OSAT Results establishes the
following assessment:
•
280-300: exemplary
•
260-279: competent
•
240-259: emerging
•
239 and below: unacceptable
For assessment purposes, we will track overall pass rates and average scores annually,
and we will compare them to statewide performances and year-to-year trends.
Fall 2010
Program Quality Improvement Report 2009-2010
23
Data Table for the OSAT in U.S. History/Oklahoma History/
U.S. Government/Economics
Candidate
Total Test Score:
U.S. History/
Oklahoma History/
U.S. Government/
Economics
Sub-score 1:
United States and
Oklahoma History
Sub-score 2:
U.S. Government and
Political Science
Sub-score 3:
Economics
Sub-score 4:
Constructed
Response
1 (1x9xx1)
PASSED
257
284
277
254
148
2 (0x5xx6)
PASSED
269
261
285
282
249
3 (1x3xx4)
PASSED
259
281
270
245
199
4 (0x2xx4)
PASSED
247
257
239
227
249
5 (0x6xx2)
PASSED
266
281
277
273
199
Cameron Social Studies
Education Majors
Mean
259.6
272.8
269.6
256.2
208.8
Oklahoma State Mean
248.3
256.6
256.4
247.2
211
Scoring Rubric for Social Studies Education Candidate OSAT Results
280-300: exemplary
260-279: competent
240-259: emerging
239 and below: unacceptable
Fall 2010
Program Quality Improvement Report
2009-2010
24
Data Table for the OSAT World History/Geography
Candidate
Total Test Score:
World History/
Geography
Sub-score 1:
World History
Sub-score 2:
Geography
Sub-score 3:
Constructed
Response
1 (0x5xx6)
PASSED
249
237
267
240
2 (0x2xx4)
FAILED
236
224
248
240
3 (0x6xx2)
FAILED
231
237
243
180
Cameron Social Studies
Education Majors
Mean
238.7
232.7
252.7
220.0
Oklahoma State Mean
238.0
242.0
246.0
206.0
Scoring Rubric for Social Studies Education Candidate OSAT Results
280-300: exemplary
260-279: competent
240-259: emerging
239 and below: unacceptable
Fall 2010
Program Quality Improvement Report
2009-2010
25
Display of Assessment Data
Mastery of Historical Knowledge
• 2) Major Field Test in History: as the bulk of the teaching appointments in the
state of Oklahoma at the 7-12 grade levels are in U.S. and World History, a
second direct measure of content knowledge in history allows for a more
specific assessment of content knowledge gain over the duration of the Social
Studies Education program for our program completers
• - for assessment purposes, we have tracked overall average scores and
sub-scores , and we compared them to national performances and to scores
for Cameron History majors on the same test
• - the Major Field Test in History consists of about 160 multiple-choice
questions, which are focused on three regional areas of historical knowledge:
– Sub-score 1: U.S. History
– Sub-score 2: European History
– Sub-score 3: African, Asian, and Latin American History
Fall 2010
Program Quality Improvement Report 2009-2010
26
Display of Assessment Data
2009 Data Table for History MFT Results
Candidate
Total Test Score:
(120-200)
Sub-score 1:
United States History
(20-100)
Sub-score 2:
European History
(20-100)
Sub-score 3:
African, Asian, and Latin
American History
(20-100)
1 (1X9xx1)
155
61
48
55
2 (0x5xx6)
132
38
33
29
3 (0x1xx9)
166
68
60
69
4 (0x2xx4)
135
34
35
52
5 (0x6xx2)
134
33
36
38
6 (0x6xx8)
125
29
26
20
Cameron Social Studies
Education Majors
141.2
43.8
39.7
43.8
Cameron History Majors
(x= 4)
152.3
53.4
51.5
52.8
National
145.2
44.6
45.6
46.4
Fall 2010
Program Quality Improvement Report 2009-2010
27
Analysis of Assessment Data
Mastery of Social Studies Content Knowledge
• 1.
–
–
–
–
–
Data demonstrates satisfactory performance with all five students passing the U.S.
History/Oklahoma History/U.S. Government/Economics OSAT and four of the five
scoring above the state mean – a passing score is a 240 or better out of 300.
A) The five Cameron Social Studies Education majors on the U.S. History/Oklahoma
History/U.S. Government/Economics OSAT achieved a combined score of competent,
with three individuals achieving emerging, and two achieving competent.
B) The data for the OSAT includes a small sample size and is only for one cohort of
program completers.
C) There is a significantly higher average score (both total and for sub-scores) for
Cameron Social Studies Education majors than the state mean except for the
Constructed Response sub-score.
D) Given the small sample size and the fact that all students passed the OSAT, the
committee felt that no specific actions should be taken.
E) However, the committee will continue to monitor performances compared to the
state trends to see if program changes should be made in the future, especially in
regard to practicing writing assignments.
Fall 2010
28
Analysis of Assessment Data
Mastery of Social Studies Content Knowledge
• 2.
–
–
–
–
–
Data demonstrates unsatisfactory performance with only one of three students
passing the World History/Geography OSAT and scoring above the state mean – a
passing score is a 240 or better out of 300.
A) The three Cameron Social Studies Education majors on the World
History/Geography OSAT achieved a combined score of unacceptable, with two
individuals achieving unacceptable, and one achieving emerging.
B) The data for the OSAT includes a small sample size and is only for one cohort of
program completers.
C) There is a higher average score (both total and for sub-scores) for Cameron Social
Studies Education majors than the state mean except for the World History Subscore.
D) Given the small sample size and the fact that all students passed the U.S.
History/Oklahoma History/U.S. Government/Economics OSAT, and that the World
History/Geography OSAT was an additional and optional test the committee felt that
no specific actions should be taken.
E) However, the committee will continue to monitor performances compared to the
state trends to see if program changes should be made in the future.
Fall 2010
Program Quality Improvement Report
2009-2010
29
Analysis of Assessment Data
Mastery of Social Studies Content Knowledge
• 3)
–
–
–
–
The data for the MFT demonstrates satisfactory performance with
Cameron Social Studies Education majors scoring within five points of the
national mean.
A) The data for the MFT include a small sample size and is only for one
cohort of completers of HIST 4773, Methods of Teaching Social Studies.
B) There is a significantly lower average score (both total and for sub-scores)
for Social Studies Education majors than for History majors at Cameron
University.
C) Given the small sample size and the fact that some of the students took the
MFT before completing key content coursework in the major, the
committee felt that no specific actions needed to be taken.
D) However, the committee will continue to monitor performances compared
to the History majors and year-to-year trends to see if program changes
should be made in the future.
Fall 2010
Program Quality Improvement Report
2009-2010
30
Action plan for Student-Learning or Service Outcomes
Mastery of Social Studies Content Knowledge
StudentLearning
Outcome
Assessment Documentation and Program
Modifications
Timelines
Program completers
will demonstrate a
satisfactory ability to
recall, apply, and
appraise the
explanatory value of
factual knowledge
related to:
1) U.S. History
2) European History
3) World History
4) Oklahoma History
5) American Federal
Government
6) World Geography
7) Human Geography
Due to the unavailability of the History MFT, the History,
Political Science, and Geography faculty created a local Social
Studies test. Faculty created 165 questions from their
specific areas of expertise: U.S. History to 1865 (twenty
questions), U.S. History since 1865 (twenty questions),
Western Civilization I (fifteen questions), Western Civilization
II (fifteen questions), Early World (fifteen questions), Modern
World (fifteen questions), Oklahoma History (ten questions),
American Federal Government (twenty-five questions),
World Geography (fifteen questions), and Human Geography
(fifteen questions).
Implementation
of the Local
Social Studies
test will be at
the end of HIST
4773 Fall 2010.
Fall 2010
Resources
Current
faculty
We will continue to endeavor to create an appropriate test
by having it vetted by colleagues who teach our subject
areas at other universities, by reviewing the test ourselves
for any improvements that can be made, and by further
analyzing the data for any areas that may indicate strengths
and weaknesses in student learning. We feel we have made
a good faith effort to create a test that fairly evaluates our
students, and that they have performed well.
Program Quality Improvement Report 2009-2010
31
Student Learning Outcome 2 Measurement Plan
PROGRAM GOAL: Gaining Classroom Experience in Planning, Teaching, and Assessment in the Social Studies
MEASUREMENT OF Student Learning Outcome
Student Learning
Outcome 2
Program completers
will know how to
compose effective
daily and long-range
lesson plans, teach
secondary-level
social studies
content in a clear,
organized manner,
and assess levels of
student
performance and
make necessary
preparatory and
instructional
adjustments to
achieve optimal
student learning.
Fall 2010
Curriculum
Area or Target
Audience
Core Course:
HIST 4773,
Student
Teaching
Semester
Measurements
Locally developed
student portfolio with
longitudinal data
collected from History
4773 Methods in
Teaching Social Studies
in the form of two unit
lesson plans students
create for this core
course (direct)
and
Cameron University
Social Studies Education
Student Teacher
Evaluation rubric
adhering to the National
Council for the Social
Studies (NCSS) Ten
Thematic
Standards(direct).
Methods used to
determine validity
of measurement
instruments
Face Validity
established by all
Social Studies
Education Program
faculty
and
face validity as
established by
mentor teacher and
supervisor
generated by the
National Council for
the Social Studies
Program Quality Improvement Report 2009-2010
Methods used to determine
reliability of measurements
Schedule for
measurements
By having representatives of
each discipline taught by the
History and Government
faculty read each portfolio,
the Social Studies Education
Program employs the
philosophy of Inter-rater
reliability in evaluating
portfolios, and by having
representatives of the Social
Studies Education Program
Education department, and
mentor teachers evaluate
student performance, the
Social Studies Education
Program employs the
philosophy of Inter-rater
reliability in evaluating
professional education
seminar rubric for this Student
Learning Outcome.
Annually, Fall
and Spring
Semester
32
Display of assessment data
Gaining Classroom Experience in Planning, Teaching, and Assessment
in the Social Studies
•
Priority Outcome 2: Program completers will know how to compose effective daily and
long-range lesson plans, teach secondary-level social studies content in a clear, organized
manner, and assess levels of student performance and make necessary preparatory and
instructional adjustments to achieve optimal student learning.
• Description and Rationale of Assessment Measurement:
– Unit Lesson Plans and Student Teacher Evaluations: Social Studies Education Majors
created two unit lesson plans (8th Grade U.S. History – the Industrial Revolution and
High School Government – the American Presidency) in History 4773 Methods in
Teaching Social Studies. Social Studies Education Majors were evaluated during their
Student Teaching Assignments.
– Rationale: By having representatives of each discipline taught by the History and
Government faculty read each portfolio, and by having representatives of the Social
Studies Education Program Education department and mentor teachers evaluate
student performance, the Social Studies Education Program employs the philosophy of
Inter-rater reliability in evaluating student performance.
Fall 2010
Program Quality Improvement Report 2009-2010
33
Scoring Rubric for Lesson Plan in 8th-Grade United States History Scoring Scale:
4 = Exemplary; 3 = Competent; 2 = Emerging; 1 = Unacceptable
Category A: Topical Clarity, Appropriateness, and Sequencing
4: daily topics of instruction are always
identified and always follow a logical
sequencing of topics of study in United
States History throughout the lesson
plan
3: daily topics of instruction are mostly
identified and mostly follow a logical
sequencing of topics of study in United
States History
2: some key events and topics in United
States History are either missing or are
unclear in their description, while others
are present but lack a clear sequence of
instruction
1: the key events and topics in United
States History are poorly identified and do
not follow a logical sequence of instruction
2: key subtopics for each identified
daily topic are occasionally identified,
complete, and appropriate in terms of
the amount of material to be covered in
a single class period
1: key subtopics for each identified daily
topic are rarely identified, complete, and
appropriate in terms of the amount of
material to be covered in a single class
period
3: Learning objectives are mostly
identified for both the daily topic of
instruction and the subtopics of
instruction
2: Learning objectives may often lack
clarity in their connection to both the
daily topic of instruction and the
subtopics of instruction
1: Learning objectives are rarely or never
identified for both the daily topic of
instruction and the subtopics of instruction
3: the method or methods of instruction
are mostly stated and are mostly
appropriate for each day’s lesson plan
2: the method or methods of instruction
may often lack clarity in their
relationship to and appropriateness for
each day’s lesson plan
1: the method or methods of instruction
are rarely clear and usually lack
appropriateness for each day’s lesson plan
2: plans for classroom evaluations may
occasionally lack clarity, may
occasionally expect too little or too much
from students, and may not adequately
assess student learning of key concepts,
terminology, and ideas in United States
History
1: plans for classroom evaluations are
rarely clear in their identification, are
inappropriate in their level of student
expectations, and mostly fail to
appropriately assess student learning of
key concepts, terminology, and ideas in
United States History
Category B: Outline of Subtopics to be Covered
4: key subtopics for each identified
daily topic are always identified,
complete, and appropriate in terms of
the amount of material to be covered in
a single class period
3: key subtopics for each identified
daily topic are mostly identified,
complete, and appropriate in terms of
the amount of material to be covered in
a single class period
Category C: Identification of Learning Objectives
4: Learning objectives are always
identified for both the daily topic of
instruction and the subtopics of
instruction
Category D: Pedagogical Methodology
4: the method or methods of instruction
are always clearly stated and are
appropriate for each day’s lesson plan
Category E: Evidence of Assessment of Student Learning and Outcomes
4: any plans for classroom evaluations
(tests, quizzes, worksheets, group or
writing assignments) are always clearly
identified, are at the proper level of
student expectations, and are fully
appropriate to assess student learning
of key concepts, terminology, and ideas
in United States History
Fall 2010
3: any plans for classroom evaluations
are mostly identified and clear, are
mostly at the proper level of student
expectations, and are mostly
appropriate to assess student learning
of key concepts, terminology, and ideas
in United States History
Program Quality Improvement Report
2009-2010
34
Scoring Rubric for Lesson Plan in High School United States Government Scoring Scale:
4 = Exemplary; 3 = Competent; 2 = Emerging; 1 = Unacceptable
Category A: Topical Clarity, Appropriateness, and Sequencing
4: daily topics of instruction are always
identified and follow a logical sequencing
of topics of study in United States
Government throughout the lesson plan
3: daily topics of instruction are mostly
identified and mostly follow a logical
sequencing of topics of study in United
States Government
2: some key events and topics in United
States Government are either missing or
are unclear in their description, while
others are present but lack a clear
sequence of instruction
1: the key events and topics in United
States Government are poorly identified
and do not follow a logical sequence of
instruction
2: key subtopics for each identified daily
topic are occasionally identified,
complete, and appropriate in terms of
the amount of material to be covered in
a single class period
1: key subtopics for each identified daily
topic are rarely identified, complete, and
appropriate in terms of the amount of
material to be covered in a single class
period
3: Learning objectives are mostly
identified for both the daily topic of
instruction and the subtopics of
instruction
2: Learning objectives may often lack
clarity in their connection to both the
daily topic of instruction and the
subtopics of instruction
1: Learning objectives are rarely or
never identified for both the daily topic of
instruction and the subtopics of
instruction
3: the method or methods of instruction
are mostly stated and are mostly
appropriate for each day’s lesson plan
2: the method or methods of instruction
may sometimes lack clarity in their
relationship to and appropriateness for
each day’s lesson plan
1: the method or methods of instruction
are rarely clear and usually lack
appropriateness for each day’s lesson
plan
2: plans for classroom evaluations may
occasionally lack clarity, may occasionally
expect too little or too much from
students, and may not adequately assess
student learning of key concepts,
terminology, and ideas in United States
Government
1: plans for classroom evaluations are
rarely clear in their identification, are
inappropriate in their level of student
expectations, and mostly fail to
appropriately assess student learning of
key concepts, terminology, and ideas in
United States Government
Category B: Outline of Subtopics to be Covered
4: key subtopics for each identified daily
topic are always identified, complete, and
appropriate in terms of the amount of
material to be covered in a single class
period
3: key subtopics for each identified daily
topic are mostly identified, complete, and
appropriate in terms of the amount of
material to be covered in a single class
period
Category C: Identification of Learning Objectives
4: Learning objectives are always
identified for both the daily topic of
instruction and the subtopics of
instruction
Category D: Pedagogical Methodology
4: the method or methods of instruction
are always clearly stated and are
appropriate for each day’s lesson plan
Category E: Evidence of Assessment of Student Learning and Outcomes
4: any plans for classroom evaluations
(tests, quizzes, worksheets, group or
writing assignments) are always clearly
identified, are at the proper level of
student expectations, and are fully
appropriate to assess student learning of
key concepts, terminology, and ideas in
United States Government
Fall 2010
3: any plans for classroom evaluations
are mostly identified and clear, are
mostly at the proper level of student
expectations, and are mostly appropriate
to assess student learning of key
concepts, terminology, and ideas in
United States Government
Program Quality Improvement Report
2009-2010
35
Display of assessment data
Gaining Classroom Experience in Planning, Teaching,
and Assessment in the Social Studies
Assessment Scale:
4 = Exemplary
3 = Competent
2 = Emerging
1 = Unacceptable
*The above 4 point scale is used to determine
student performance for Program Goal II.
Fall 2010
Program Quality Improvement Report
2009-2010
36
Display of assessment data
Gaining Classroom Experience in Planning, Teaching, and
Assessment in the Social Studies
Data Table for Lesson Plan Assignment: 8th-Grade United States History
Candidate
A – Avg. Score
B – Avg. Score
C – Avg. Score
D – Avg. Score
E – Avg. Score
Total
Mean (x/5)
1 (1x9xx1)
3.5
3.5
3.5
3.5
3.5
17.5
3.5
2 (0x5xx6)
3.5
3.5
3.0
3.0
2.5
15.5
3.1
3 (0x1xx9)
3.5
3.5
3.0
2.5
3.5
16.0
3.2
4 (0x2xx4)
3.0
3.0
3.0
3.5
2.5
15.0
3.0
5 (0x6xx2)
3.0
2.5
3.0
3.5
3.5
15.5
3.1
6 (0x6xx8)
2.0
1.5
2.5
2.5
2.0
10.5
2.1
7 (1x3xx4)
3.5
3.5
4.0
3.5
3.5
18.0
3.6
Per-category Total
22.0
21.0
22.0
22.0
21.0
108
21.6
Per-category Mean
3.1
3.0
3.1
3.1
3.0
15.4
3.1
4 = Exemplary
3 = Competent
2 = Emerging
1 = Unacceptable
Fall 2010
Program Quality Improvement Report
2009-2010
37
Display of assessment data
Gaining Classroom Experience in Planning, Teaching, and
Assessment in the Social Studies
Data Table for Lesson Plan Assignment: High School United States Government
Candidate
A – Avg. Score
B – Avg. Score
C – Avg. Score
D – Avg. Score
E – Avg. Score
Total
Mean (x/5)
1 (1x9xx1)
3.0
4.0
3.5
4.0
3.5
18.0
3.6
2 (0x5xx6)
3.0
2.5
2.5
2.5
2.5
13.0
2.6
3 (0x1xx9)
3.0
2.5
2.5
3.0
2.5
13.5
2.7
4 (0x2xx4)
3.5
3.5
4.0
3.5
3.5
18.0
3.6
5 (0x6xx2)
3.0
2.5
3.0
3.0
3.5
15.0
3.0
6 (0x6xx8)
2.5
2.0
2.5
2.0
2.0
11.0
2.2
7 (1x3xx4)
3.5
4.0
3.5
3.0
3.0
17.0
3.4
Per-category Total
21.5
21.0
21.5
21.0
20.5
105.5
21.1
Per-category Mean
3.1
3.0
3.1
3.0
2.9
15.07
3.0
4 = Exemplary
3 = Competent
2 = Emerging
1 = Unacceptable
Fall 2010
Program Quality Improvement Report
2009-2010
38
Percentages of Candidates Scoring in Each Range
not acceptable*
(1-1.5 points)
emerging*
(1.51-2.5 points)
competent*
(2.51-3.5 points)
exemplary*
(3.51-4.0 points)
8th-Grade U.S. History
Lesson Plan
*performance is not
acceptable for a
prospective teacher
*performance is acceptable but
is not yet at the level expected
of a beginning teacher
*performance is at the level of
a beginning teacher in the field
*performance is at the level of
a master teacher in the field
Total
0%
14.3%
71.4%
14.3%
Category A^
0%
14.3%
85.7%
0%
Category B
14.3%
14.3%
71.4%
0%
Category C
0%
14.3%
71.4%
14.3%
Category D
0%
28.6%
71.4%
0%
Category E
0%
42.9%
57.1%
0%
Total
0%
14.3%
57.1%
28.6%
Category A
0%
14.3%
85.7%
0%
Category B
0%
57.1%
14.3%
28.6%
Category C
0%
42.9%
42.9%
14.3%
Category D
0%
28.6%
57.1%
14.3%
Category E
0%
42.9%
57.1%
0%
High School U.S.
Government Lesson Plan
^since the possible scores per candidate are whole number values of 1, 2, 3, or 4, the ranges of values for fitting percentages to each
descriptive category of not acceptable, emerging, competent, and exemplary are centered on each whole number (i.e., 1.51-2.5 points for
emerging)
^per-category totals may not equal 100% due to rounding
Fall 2010
Program Quality Improvement Report
2009-2010
39
Display of assessment data
Gaining Classroom Experience in Planning, Teaching,
and Assessment in the Social Studies
• Sample Size: 5
• For a given year the number of students being assessed
is determined by the number who complete Student
Teaching and there were 5 such students who
completed Student Teaching during the Spring 2010
semester.
Fall 2010
Program Quality Improvement Report
2009-2010
40
Display of assessment data
Gaining Classroom Experience in Planning, Teaching, and Assessment in
the Social Studies
Explanation of Ratings
4 = Exemplary—performance is at the level of a master teacher in the field
3 = Competent—performance is at the level of a beginning teacher in the field
2 = Emerging—performance is acceptable but is not yet at the level expected of a beginning teacher
1 = Not Acceptable—performance is not acceptable for a prospective teacher
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
Fall 2010
Standard
NCSS Standard 1: Culture and Cultural Diversity
Evidence Standard 1 is addressed in planning/preparation
Evidence Standard 1 is addressed in teaching/instruction
Evidence Standard 1 is achieved through student learning
NCSS Standard 2: Time, Continuity, and Change
Evidence Standard 2 is addressed in planning/preparation
Evidence Standard 2 is addressed in teaching/instruction
Evidence Standard 2 is achieved through student learning
NCSS Standard 3: People, Places, and Environments
Evidence Standard 3 is addressed in planning/preparation
Evidence Standard 3 is addressed in teaching/instruction
Evidence Standard 3 is achieved through student learning
NCSS Standard 4: Individual Development and Identity
Evidence Standard 4 is addressed in planning/preparation
Evidence Standard 4 is addressed in teaching/instruction
Evidence Standard 4 is achieved through student learning
NCSS Standard 5: Individuals, Groups, and Institutions
Evidence Standard 5 is addressed in planning/preparation
Evidence Standard 5 is addressed in teaching/instruction
Evidence Standard 5 is achieved through student learning
Mentor
Supervisor
Overall
3.60
3.50
3.40
3.00
3.00
3.00
3.29
3.24
3.19
3.50
3.50
3.30
3.00
3.00
3.00
3.25
3.25
3.15
3.40
3.50
3.40
3.00
3.00
3.00
3.24
3.29
3.24
3.60
3.60
3.50
3.00
3.00
3.00
3.46
3.46
3.38
3.50
3.60
3.40
3.00
3.00
3.00
3.28
3.33
3.25
Program Quality Improvement Report 2009-2010
41
Display of assessment data
Gaining Classroom Experience in Planning, Teaching, and Assessment in
the Social Studies
Explanation of Ratings
4 = Exemplary—performance is at the level of a master teacher in the field
3 = Competent—performance is at the level of a beginning teacher in the field
2 = Emerging—performance is acceptable but is not yet at the level expected of a beginning teacher
1 = Not Acceptable—performance is not acceptable for a prospective teacher
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
26.
27.
28.
29.
30.
Fall 2010
Standard
NCSS Standard 6: Power, Authority, and Governance
Evidence Standard 6 is addressed in planning/preparation
Evidence Standard 6 is addressed in teaching/instruction
Evidence Standard 6 is achieved through student learning
NCSS Standard 7: Production, Distribution, Consumption
Evidence Standard 7 is addressed in planning/preparation
Evidence Standard 7 is addressed in teaching/instruction
Evidence Standard 7 is achieved through student learning
NCSS Standard 8: Science, Technology, and Society
Evidence Standard 8 is addressed in planning/preparation
Evidence Standard 8 is addressed in teaching/instruction
Evidence Standard 8 is achieved through student learning
NCSS Standard 9: Global Connections
Evidence Standard 9 is addressed in planning/preparation
Evidence Standard 9 is addressed in teaching/instruction
Evidence Standard 9 is achieved through student learning
NCSS Standard 10: Civic Ideals and Practices
Evidence Standard 10 is addressed in planning/preparation
Evidence Standard 10 is addressed in teaching/instruction
Evidence Standard 10 is achieved through student learning
Overall Averages
Mentor
Supervisor
Overall
3.50
3.50
3.56
3.00
3.00
3.00
3.26
3.26
3.33
3.60
3.50
3.40
3.00
3.00
3.00
3.32
3.26
3.22
3.60
3.60
3.50
3.00
3.00
3.00
3.33
3.33
3.28
3.70
3.60
3.50
3.00
3.00
3.00
3.41
3.38
3.31
3.56
3.67
3.33
3.51
3.00
3.00
3.00
3.01
3.33
3.40
3.21
3.30
Program Quality Improvement Report 2009-2010
42
Display of assessment data
Gaining Classroom Experience in Planning, Teaching, and
Assessment in the Social Studies
• This is the first year in which we have used the
current portfolio measurement scale. We expect to
be able to provide a proper trend analysis in two
years.
• Thus, while the scores generally look promising,
there is not a large enough sampling size to show any
reliable single-sample analysis or trend-based
analysis.
Fall 2010
Program Quality Improvement Report
2009-2010
43
Analysis of Assessment Data
Gaining Classroom Experience in Planning, Teaching, and Assessment in
the Social Studies
1. Portfolio analysis of the 8th-Grade United States History Lesson Plan
indicates that six of the seven students achieved competent with
the other student achieving emerging.
2. The students mean for all five categories equaled competent.
3. The mean of means for all seven students equaled competent.
4. The data for the lesson-plan assignment also includes a small
sample size and is only for one cohort of completers of HIST 4773,
Methods of Teaching Social Studies.
5. There are concerns that the scoring rubrics for the lesson-plan
assignment may have needed further testing for closer reliability
among those doing the scoring
Fall 2010
Program Quality Improvement Report 2009-2010
44
Analysis of Assessment Data
Gaining Classroom Experience in Planning, Teaching, and
Assessment in the Social Studies
1.
2.
3.
4.
5.
Portfolio analysis of the High School United States Government
Lesson Plan indicates that four of the seven students achieved
competent with the other three students achieving emerging.
The students mean for four categories equaled competent with
Category E equaling emerging.
The mean of means for all seven students equaled competent.
The data for the lesson-plan assignment also includes a small
sample size and is only for one cohort of completers of HIST 4773,
Methods of Teaching Social Studies.
There are concerns that the scoring rubrics for the lesson-plan
assignment may have needed further testing for closer reliability
among those doing the scoring.
Fall 2010
45
Analysis of Assessment Data
Gaining Classroom Experience in Planning, Teaching, and
Assessment in the Social Studies
1.
2.
3.
Portfolio analysis of the Student Teaching Final Evaluations Rubric
indicates that students achieved an overall rating of competent performance is at the level of a beginning teacher in the field for
both the mentor teachers and Social Studies Education Program
supervisors.
Each criterion is defined by the National Council for the Social
Studies Ten Thematic Standards, and for each criterion students
achieved a rating of competent - performance is at the level of a
beginning teacher in the field for both the mentor teachers and
Social Studies Education Program supervisors.
The data for the Student Teaching Final Evaluations Rubric t also
includes a small sample size and is only for one cohort of
completers of student teaching.
Fall 2010
Program Quality Improvement Report
2009-2010
46
Action plan for Student-Learning or Service Outcomes
Gaining Classroom Experience in Planning, Teaching,
and Assessment in the Social Studies
Student-Learning
Outcome
Assessment
Documentation
and Program
Modifications
Timelines
Program completers will
know how to compose
effective daily and longrange lesson plans, teach
secondary-level social
studies content in a clear,
organized manner, and
assess levels of student
performance and make
necessary preparatory
and instructional
adjustments to achieve
optimal student learning.
The committee will
continue to monitor
performances compared
to year-to-year trends to
see if program changes
should be made in the
future.
The committee will
discuss the utility and
consistency of scoring
based upon the current
scoring rubrics.
Implementation during
the 2010-2011 academic
year.
Fall 2010
Program Quality Improvement Report 2009-2010
Resources
Current faculty
47
Student Learning Outcome 3 Measurement Plan
PROGRAM GOAL: Building the Foundation for Becoming a Professional Educator
MEASUREMENT OF Student Learning Outcome
Student Learning
Outcome 3
Program completers will
compile and present a
professional portfolio of all
their relevant educational
achievements, reflections,
and artifacts, seek
certification and licensure
to teach social studies at
the 7-12 grade levels, and
be monitored and
evaluated regarding their
post-graduate program
reflections and
employment trends.
Fall 2010
CURRICULUM
AREA OR TARGET
AUDIENCE
Measurements
Social Studies
Education Majors
and student in
HIST 4773
Methods of
Teaching Social
Studies
Oklahoma
Professional
Teaching
Examination
(OPTE)
standardized test
(direct).
Methods used to
determine validity
of measurement
instruments
Methods used
to determine
reliability of
measurements
Schedule for
measurements
Content Validity as
Established by
State Normreferenced scores
generated by the
Oklahoma
Commission on
Teacher
Preparation
(OCTP) Oklahoma
Professional
Teaching
Examination
(OPTE)
OCTP
established
measures
Annually , Fall
and Spring
Semester
Program Quality Improvement Report 2009-2010
48
Display of Assessment Data
Building the Foundation for Becoming a Professional Educator
•
Priority Outcome 3: Program completers will compile and present a professional portfolio of
all their relevant educational achievements, reflections, and artifacts, seek certification and
licensure to teach social studies at the 7-12 grade levels, and be monitored and evaluated
regarding their post-graduate program reflections and employment trends.
• Description and Rationale of Assessment Measurement:
– Oklahoma Professional Teaching Examination (OPTE): Social Studies Education Majors
took the Oklahoma Commission on Teacher Preparation (OCTP) Oklahoma Professional
Teaching Examination (OPTE) prior to their Student Teaching Semester. Students must
score above a 240 in order to pass the OPTE.
– Rationale: an objective testing instrument provides an excellent way to measure the
topical coverage of our curriculum which allows the Social Studies Education Program to
ensure a proper curricular balance.
Fall 2010
Program Quality Improvement Report 2009-2010
49
Display of assessment data
Building the Foundation for Becoming a Professional
Educator
Candidate
Total Test
Score:
OPTE
Sub-score 1:
Sub-score 2:
Sub-score 3:
Sub-score 4:
Sub-score 5:
Sub-score 6:
1 (1x9xx1)
PASSED
273
275
278
291
283
228
265
2 (0x5xx6)
PASSED
264
268
256
273
266
264
265
3 (1x3xx4)
PASSED
255
256
267
273
231
210
265
4 (0x2xx4)
PASSED
280
287
283
291
300
264
229
5 (0x6xx2)
PASSED
257
262
272
256
231
228
265
Cameron Social Studies
Education Majors
Mean
265.8
269.6
271.2
276.8
262.2
238.8
257.8
Oklahoma State Mean
262.5
264.0
268.0
271.4
258.8
243.0
252.4
Fall 2010
50
Display of assessment data
Building the Foundation for Becoming a Professional Educator
•
•
1) OPTE Scores: Five students took the OPTE and all five passed – a passing
score is a 240 or better out of 300.
For assessment purposes, we will track overall pass rates and average scores
annually, and we will compare them to statewide performances and year-toyear trends.
Fall 2010
Program Quality Improvement Report
2009-2010
51
Analysis of Assessment Data
Building the Foundation for Becoming a Professional
Educator
• 1.
–
–
–
–
Data demonstrates satisfactory performance with all five students passing the
OPTE and three of the five scoring above the state mean – a passing score is a
240 or better out of 300.
A) The data for the OPTE includes a small sample size and is only for one cohort
of program completers.
B) There is a higher average score (both total and for sub-scores) for Cameron
Social Studies Education majors than the state mean except for Sub-score 5.
C) Given the small sample size and the fact that all students passed the OPTE,
the committee felt that no specific actions should be taken.
D) However, the committee will continue to monitor performances compared
to the state trends to see if program changes should be made in the future.
Fall 2010
52
Action plan for Student-Learning or Service Outcomes
Building the Foundation for Becoming a Professional Educator
Student-Learning
Outcome
Assessment
Documentation
and Program
Modifications
Timelines
Program completers will
compile and present a
professional portfolio of all
their relevant educational
achievements, reflections,
and artifacts, seek
certification and licensure
to teach social studies at
the 7-12 grade levels, and
be monitored and
evaluated regarding their
post-graduate program
reflections and
employment trends.
The committee will
continue to monitor
performances on the
OPTE compared to yearto-year trends to see if
program changes should
be made in the future.
Implementation during
the 2010-2011 academic
year.
Program Quality Improvement Report 2009-2010
Resources
Current faculty
53
Published information on graduates
Academic Year 09-10
Fall 2010
Employed by School System
Working In Discipline
Other
5
4
1
Program Quality Improvement Report 2009-2010
54