M.S. in Education
Assessment in the Major Report
By Renee Chandler, Program Director
2009
Submitted: October 2010
Table of Contents
1.
2.
3.
4.
5.
6.
7.
8.
9.
Outcomes of Previous Assessments ............................................................................................................................................................................ 2
Questions To Be Answered From This Year’s Assessment ........................................................................................................................................ 3
Methods Used to Gather Data: .................................................................................................................................................................................... 3
Number of Students Involved . .................................................................................................................................................................................... 3
Level of Students ......................................................................................................................................................................................................... 4
Data Analysis Techniques ........................................................................................................................................................................................... 4
Analysis of Data .......................................................................................................................................................................................................... 4
How Results Were Shared with Key Instructors ....................................................................................................................................................... 14
Plans for Improvements Based on Results ................................................................................................................................................................ 14
Appendix A: MSE Benchmarks........................................................................................................................................................................................ 15
Appendix B: MSE Rubric for Portfolio ............................................................................................................................................................................ 16
Education AIM Report 2009
Page 1
1.
Outcomes of Previous Assessments
The 2008-2009 Assessment in the Major report was written at the beginning of my first year as program director of the MSE. At that time I
discovered that there was very little data regarding student performance in the MSE program. As a result of this discovery, I included the following
statement in last year’s conclusion and action plan: “To ascertain the effectiveness of the MSE program, more data needs to be collected in the form
of portfolio assessment, student ratings, and alumni and employer surveys. I will work with the instructor(s) of EDUC-790 (Professional Portfolio)
to determine a meaningful way to collect data from the portfolio that all students in the MSE program complete. A new survey will be developed to
collect anonymous ratings of the program by current students. A new follow-up survey has been developed and measures will be taken to increase
the response rates on follow-up surveys.”
As I begin my second year as program director for the MS in Education, I am in the process of implementing a new assessment system for the
program. In that plan, several new strategies have been developed to assess the growth of graduates in the program.
Three levels of benchmark criteria are outlined in the new assessment plan (see Appendix A). Assessment measures critical to the new benchmarks
include:
• ratings of application essays
• disposition ratings by recommenders identified by the student, student self-ratings, instructor ratings, and observers in the field
• e-Portfolio ratings
Some of these benchmark criteria were piloted beginning Spring semester 2010. Only limited data is currently available, however, for the purposes
of this report.
Because 2009 was a transition year for the MSE assessment system, this report will focus on data collected in the following areas:
• Portfolio data
• Follow-up survey data (including the new program-specific survey)
Education AIM Report 2009
Page 2
2.
Questions To Be Answered From This Year’s Assessment
The 2009 assessment report will focus on data which addresses the following questions:
• Portfolio data: How are students performing relative to the National Board Standards?
• Follow-up survey data: How do our graduates rank the level of knowledge, skills, and dispositions acquired in the MSE program?
3.
Methods Used to Gather Data:
•
•
4.
Portfolio: All students in the MSE program complete a required course in which they submit a Professional Portfolio. The portfolio is rated
by the instructor of EDUC-790. During the fall semester of 2009, the students were rated on the following criteria: (1) Evidence of using
web resources in portfolio development; (2) Evidence of completed professional portfolio that includes artifacts and reflections and
technology integration; and (3) Evidence within portfolio that relates to student’s program and professional goals. Beginning Spring 2010, the
program director taught EDUC-790 and rated each student’s e-portfolio according to the rubric presented in Appendix B. The five areas of
assessment align with the National Board Teaching Standards: 1. Commitment to learners and learning; 2. Content knowledge and contentspecific pedagogy; 3. Managing and monitoring student learning; 4. Systematic reflection and lifelong learning; and 5. Collaboration and
participation in learning communities. The portfolios were scored directly on the Chalk and Wire system on which they were created,
allowing for efficient data collection and analysis.
Follow-up surveys: Budget, Planning and Analysis provided summaries and reports from 1-5 year graduate follow up studies for MS in
Education.
Number of Students Involved .
Sixteen students were enrolled in EDUC-790 Professional Portfolio during the Spring semester of 2010. The data from their portfolios will
be presented.
The number of respondents to the graduate follow-up studies was small; eight graduates responded to the one-year MSE survey and four
graduates responded to the five-year MSE survey.
Education AIM Report 2009
Page 3
5.
Level of Students
Only graduate students were involved in the collection of data presented in this report.
6.
Data Analysis Techniques
Information collected will be placed in tables. Where appropriate, descriptive statistics will be used to analyze the data.
7.
Analysis of Data
Graduate follow up studies
One and five year graduate follow up studies were conducted. The Office of Budget, Planning and Analysis provided summaries of the reports to the
program director. The number of respondents to the surveys was low: There was an N of 8 total respondents to the one-year MSE program study; 4
respondents in the five-year study; no employers responded to the one-and five-year MSE follow up studies. The return rate on these surveys has
been disappointingly low such that statistically meaningful conclusions cannot be drawn. The few students who did respond gave responses that
indicated a general satisfaction with their experiences as MSE students at UW-Stout.
Results from the UW-Stout Five-Year and One-Year Follow-Up Survey and the Program-Specific survey are summarized in the tables on the
following pages.
Education AIM Report 2009
Page 4
UW-Stout Graduate Five-Year Follow-Up: MSE Program
(1=Very Dissatisfied 2= Dissatisfied 3=Satisfied 4=Very Satisfied)
1. Writing Effectively
2. Speak or Present Ideas Effectively
3. Listen Effectively
4. Utilize Computing and Digital Technology
5. Use Analytic Reasoning
6. Solve Problems Creatively
7. Critically Analyze Information
8. Appreciate and Understand Diversity
9. Develop Global Perspective
10. Organize Information
11. Make Decisions
12. Consider the Ethics of My Profession
13. Work in Teams
14. Lead Others
15. Understand Statistics
16. Identify Future Career Options
17. Develop Resume/Portfolio
18. View Community Service as My Social Responsibility
19. Mentorship by Faculty
20. Program Instruction
21. Availability of Faculty
22. Course Availability (In Terms of Program Sequence)
23. Academic Advising
Education AIM Report 2009
Year Graduated
2004
N
Mean
3
3.67
3
3.67
3
3.33
3
2.67
3
3.33
3
3.00
3
3.67
3
3.67
3
3.33
3
3.67
3
3.33
3
3.33
3
3.33
3
3.67
3
3.00
3
2.67
3
2.33
3
2.67
3
3.67
3
3.67
3
3.67
3
3.67
3
3.33
Page 5
MSE Program Specific Five-Year Follow-up Survey
(1=Strongly Disagree, 5=Strongly Agree)
Please rate the degree the program prepared you to:
Year Graduated
2004
N
Mean
3
4.67
2. Create meaningful learning experiences based on my content
knowledge.
3
4.33
3. Effectively teach language arts including phonics (PK-6, Special
Education, and Reading Teacher licenses only).
1
5.00
4. Effectively teach math skills (PK-6, Special Education and math license
only).
1
5.00
5. Provide instruction that supports student learning and their intellectual,
social and personal development.
3
4.67
6. Create instructional experiences adapted for students who learn
differently.
3
4.67
7. Use my knowledge of minority group relations to create appropriate
instruction for diverse groups.
3
4.33
3
4.33
3
4.67
3
4.67
3
4.33
3
4.33
3
4.33
3
4.33
1. Understand the content and central concepts of the discipline I teach.
8. Modify curricula when instructing students with disabilities.
9. Use a variety of learning strategies to encourage critical thinking and
problem solving.
10. Create a learning environment that encourages positive social
interaction, active engagement in learning and self-motivation.
11. Resolve conflicts between students and between students and staff.
12. Assist students in learning how to resolve conflicts.
13. Deal with crises or disruptive situations.
14. Use effective communication techniques, media and technology to
foster active inquiry, collaboration and supportive interaction in the
Education AIM Report 2009
Page 6
classroom.
15. Use instructional technology to enhance student learning.
16. Plan instruction based on knowledge of subject matter, students, the
community and curriculum goals.
17. Use formal and informal assessment strategies to evaluate student
progress.
18. Use assessment tools for students with disabilities.
19. Use developmental, multiple and measurable assessment tools to
assess student learning over time.
20. Use assessment grounded in research and based on best practices in
education.
21. Use assessment tools with identified benchmarks or levels of
proficiency.
22. Analyze students, classroom, and school performance data; make
data-driven decisions about strategies for teaching and learning.
23. Reflect on teaching and evaluate the effects of choices and actions on
students, parents, and others.
24. Assess and analyze student learning, make appropriate adjustments to
instruction, monitor student learning, and develop and implement
meaningful learning experiences to help all students learn.
25. Foster relationships with colleagues, parents and the community to
support student learning and well-being.
Education AIM Report 2009
3
4.33
3
4.67
3
4.33
3
4.33
2
4.00
3
4.33
3
4.33
3
4.33
3
4.67
3
4.67
3
4.33
Page 7
UW-Stout Graduate One-Year Follow-Up: MSE Program
(1=Very Dissatisfied 2= Dissatisfied 3=Satisfied 4=Very Satisfied)
1. Writing Effectively
2. Speak or Present Ideas Effectively
3. Listen Effectively
4. Utilize Computing and Digital Technology
5. Use Analytic Reasoning
6. Solve Problems Creatively
7. Critically Analyze Information
8. Appreciate and Understand Diversity
9. Develop Global Perspective
10. Organize Information
11. Make Decisions
12. Consider the Ethics of My Profession
13. Work in Teams
14. Lead Others
15. Understand Statistics
16. Identify Future Career Options
17. Develop Resume/Portfolio
18. View Community Service as My Social Responsibility
19. Mentorship by Faculty
20. Program Instruction
21. Availability of Faculty
22. Course Availability (In Terms of Program Sequence)
23. Academic Advising
Education AIM Report 2009
Year Graduated
2008
N
Mean
8
3.25
8
3.13
8
3.00
8
3.25
8
3.13
8
3.13
8
3.13
8
3.00
8
2.88
8
3.38
8
3.13
8
3.25
8
3.25
8
3.38
8
3.00
8
3.00
8
3.13
8
2.88
8
3.63
8
3.75
8
3.75
7
3.86
8
3.88
Page 8
MSE Program Specific Five-Year Follow-up Survey
(1=Strongly Disagree, 5=Strongly Agree)
Please rate the degree the program prepared you to:
Year Graduated
2008
N
Mean
7
4.14
2. Create meaningful learning experiences based on my content
knowledge.
8
4.50
3. Effectively teach language arts including phonics (PK-6, Special
Education, and Reading Teacher licenses only).
4
3.75
4. Effectively teach math skills (PK-6, Special Education and math license
only).
3
4.33
5. Provide instruction that supports student learning and their intellectual,
social and personal development.
7
4.71
6. Create instructional experiences adapted for students who learn
differently.
6
4.50
7. Use my knowledge of minority group relations to create appropriate
instruction for diverse groups.
6
4.33
6
4.33
8
4.63
8
4.50
6
4.17
6
4.33
7
4.29
8
4.38
1. Understand the content and central concepts of the discipline I teach.
8. Modify curricula when instructing students with disabilities.
9. Use a variety of learning strategies to encourage critical thinking and
problem solving.
10. Create a learning environment that encourages positive social
interaction, active engagement in learning and self-motivation.
11. Resolve conflicts between students and between students and staff.
12. Assist students in learning how to resolve conflicts.
13. Deal with crises or disruptive situations.
14. Use effective communication techniques, media and technology to
foster active inquiry, collaboration and supportive interaction in the
Education AIM Report 2009
Page 9
classroom.
15. Use instructional technology to enhance student learning.
16. Plan instruction based on knowledge of subject matter, students, the
community and curriculum goals.
17. Use formal and informal assessment strategies to evaluate student
progress.
18. Use assessment tools for students with disabilities.
19. Use developmental, multiple and measurable assessment tools to
assess student learning over time.
20. Use assessment grounded in research and based on best practices in
education.
21. Use assessment tools with identified benchmarks or levels of
proficiency.
22. Analyze students, classroom, and school performance data; make
data-driven decisions about strategies for teaching and learning.
23. Reflect on teaching and evaluate the effects of choices and actions on
students, parents, and others.
24. Assess and analyze student learning, make appropriate adjustments to
instruction, monitor student learning, and develop and implement
meaningful learning experiences to help all students learn.
25. Foster relationships with colleagues, parents and the community to
support student learning and well-being.
8
4.38
7
4.57
8
4.25
7
4.00
6
4.00
8
4.25
8
4.13
8
4.25
8
4.75
8
4.38
7
4.57
Some tentative generalizations can be made from examining the data from the survey results. Overall, students agreed that the MSE program
provided them with knowledge, skills and dispositions across a wide range of areas. One area that graduates indicated they felt less prepared for was:
“Use developmental, multiple and measurable assessment tools to assess student learning over time.” This issue is addressed in the upcoming
program revision.
Education AIM Report 2009
Page 10
MSE Portfolio Results
Students create an e-portfolio as part of the EDUC-790 Professional Portfolio. Starting Spring 2010, the program director taught the course and
evaluated each portfolio using the rubric included in Appendix B. There were 16 students who were evaluated during the first semester that this
rubric was utilized. Students were rated according to the following scale:
1 = Unsatisfactory/Emerging
The individual does not yet appear to have the knowledge or skills underlying the area. Fundamental knowledge or
skills are required. Performance is below expected level for a beginning teacher.
2 = Basic:
Beginning knowledge and skill development is evidenced, often due to limited education or experience in the area.
Some teaching and supervision are required.
3 = Proficient:
The individual exhibits knowledge, proficiency, and competence in most aspects of the area. The individual may
seek consultation, professional development, or assistance in some aspects of the area.
4 = Distinguished:
Advanced knowledge and skill development are evidenced in the area. Individuals with this level of skill and
knowledge can teach, mentor and/or supervise others in this domain area.
Mean scores for each of the five areas are presented in the table below:
Education AIM Report 2009
Page 11
Areas (Adapted from National Board for Teaching Standards)
1. Commitment to Learners and Learning
a. Understanding of how students develop and learn, respect for differences and belief that all students can learn is indicated by
evidence of differentiation of instruction to meet the diverse needs of learners.
b. Evidence of a thorough understanding of the school, family, and community contexts is indicated by evidence of how
contextual factors influence instruction and collaboration with the professional community.
2. Content Knowledge and Content Specific Pedagogy
a. Content knowledge if indicated by evidence of mastery of the subject(s) taught.
b. Knowledge and application of theories related to pedagogy and learning is indicated by evidence of the use of a broad
range of instructional strategies and an explanation of the choices made in practice.
c. Pedagogical expertise is indicated by evidence of the use of instructional and adaptive technologies to improve student
learning.
3. Managing and Monitoring Student Learning
a. Thorough understanding of the major concepts and theories related to assessing student learning indicated by evidence of
the use of a range of strategies to assess individual students and group.
b. Thorough understanding of student learning is evidenced by an analysis of student, classroom and school performance at
multiple points.
c. Managing and monitoring student learning is indicated by evidence of the use of assessment data to make data-driven
decisions.
4. Systematic Reflection and Lifelong Learning.
a. Systematic reflection is evidenced by continuous examination of own practice.
b. Systematic reflection and learning is evidenced by development and monitoring of professional development goals.
c. Lifelong learning is evidenced by the application of research , analysis of self performance assessment data, and reflection to
improve practice.
d. Awareness of, and the ability to critically analyze current educational research, issues and policies is indicated by evidence of
how research and policies have affected practice.
5. Collaboration and participation in learning communities
a. Professionalism is indicated by evidence of effective collaboration with families, colleagues, and communities to improve
student learning.
b. Evidence is provided of active membership in professional associations.
Education AIM Report 2009
Spring 2010
N=16
3.12
2.81
2.94
2.94
2.94
Page 12
Portfolio Ratings by Area in Spring of 2010
Education AIM Report 2009
Page 13
8.
How Results Were Shared with Key Instructors
This report, in its entirety, will be presented at program advisory committee meetings.
9.
Plans for Improvements Based on Results
Data will be shared with key instructors and the program advisory committee. Future agenda items will include a discussion of the data and
what meaningful changes can be made to improve the program.
A major program revision is under development and will hopefully be in place for the students starting the MSE program in Fall 2011. Based
on the data presented in this report and information collected from primary stakeholders, the upcoming revision will include changes to the
core requirements, additional assessment opportunities, and revisions to courses within the program that are outdated.
Education AIM Report 2009
Page 14
Appendix A: MSE Benchmarks
Benchmark I: Admission to Program
•
•
•
•
•
Philosophy of Education/Statement of Purpose Review (RUBRIC)
Official Transcript
GPA 2.75 from previous institution or transcript review
Current Resume
Recommender ratings (3)
•
•
•
Disposition self-rating EDUC-729
GPA 3.0 (transcript review)
Disposition review (RUBRIC)
Person Responsible
MSE Program Director
MSE Program Director
MSE Program Director
MSE Program Director
Benchmark II: Completion of Assessment for Learning (approx. 10 credits)
Benchmark III: Graduation
•
•
•
•
•
GPA 3.0 (transcript review)
Application for Degree Candidacy
“B” or better in EDUC-735
Disposition review by observer in the field
ePortfolio review (RUBRIC)
o Table of Contents organized by 5 Teacher Standards
o Includes reflective statements for each area of the Portfolio Assessment Rubric
o Includes at least one piece of evidence for each area of the Portfolio Assessment
Rubric
Education AIM Report 2009
MSE Program Director
Instructor of Assessment for Learning and/or Intro to
Research
MSE Program Director
MSE Program Director
EDUC-735 Instructor
Identified by candidate
EDUC-790 Instructor or PD
Page 15
Appendix B: MSE Rubric for Portfolio
Portfolio Assessment Rubric
Student Name: _______________________________________
Reviewer: ___________________________________
Unsatisfactory/Emerging:
Basic:
Proficient:
Distinguished:
Student ID #: ___________________ _____
Date: __________________
Reviewer Signature: ______________________________________________________
The individual does not yet appear to have the knowledge or skills underlying the area. Fundamental knowledge or skills are required. Performance is below expected level for a beginning teacher.
Beginning knowledge and skill development is evidenced, often due to limited education or experience in the area. Some teaching and supervision are required.
The individual exhibits knowledge, proficiency, and competence in most aspects of the area. The individual may seek consultation, professional development, or assistance in some aspects of the
area.
Advanced knowledge and skill development are evidenced in the area. Individuals with this level of skill and knowledge can teach, mentor and/or supervise others in this domain area.
Areas*
Unsatisfactory
1
Basic
2
Proficient
3
Distinguished
4
1. Commitment to Learners and Learning
c. Understanding of how students develop and learn, respect for differences
and belief that all students can learn is indicated by evidence of
differentiation of instruction to meet the diverse needs of learners.
d. Evidence of a thorough understanding of the school, family, and community
contexts is indicated by evidence of how contextual factors influence
instruction and collaboration with the professional community.
Reflection:
Yes
No
Artifact:
Artifact:
2. Content Knowledge and Content Specific Pedagogy
d. Content knowledge if indicated by evidence of mastery of the subject(s)
taught.
e. Knowledge and application of theories related to pedagogy and learning is
indicated by evidence of the use of a broad range of instructional strategies
and an explanation of the choices made in practice.
f. Pedagogical expertise is indicated by evidence of the use of instructional
and adaptive technologies to improve student learning.
Reflection:
Yes
No
Artifact:
Artifact:
3. Managing and Monitoring Student Learning
Education AIM Report 2009
Page 16
d.
Thorough understanding of the major concepts and theories related to
assessing student learning indicated by evidence of the use of a range of
strategies to assess individual students and group.
e. Thorough understanding of student learning is evidenced by an analysis of
student, classroom and school performance at multiple points.
f. Managing and monitoring student learning is indicated by evidence of the
use of assessment data to make data-driven decisions.
Reflection:
Yes
No
Artifact:
Artifact:
4. Systematic Reflection and Lifelong Learning.
e. Systematic reflection is evidenced by continuous examination of own
practice.
f. Systematic reflection and learning is evidenced by development and
monitoring of professional development goals.
g. Lifelong learning is evidenced by the application of research , analysis of self
performance assessment data, and reflection to improve practice.
h. Awareness of, and the ability to critically analyze current educational
research, issues and policies is indicated by evidence of how research and
policies have affected practice.
Reflection:
Yes
No
Artifact:
Artifact:
5. Collaboration and participation in learning communities
c. Professionalism is indicated by evidence of effective collaboration with
families, colleagues, and communities to improve student learning.
d. Evidence is provided of active membership in professional associations.
Reflection:
Yes
No
Artifact:
Artifact:
*Note: Areas 1-5 from National Board for Teaching Standards
Education AIM Report 2009
Page 17