B.S. Technology and Science Education Assessment in the Major Report By Dr. Brian McAlister & Dr. David Stricker 2014-15 Submitted: October 2015 Table of Contents Overview ..................................................................................................................................................................................................................2 Overview of the Program .........................................................................................................................................................................................2 Previous Goals and Evidence...................................................................................................................................................................................3 PRAXIS I: Pre-Professional Skills Test/CORE Tests ............................................................................................................................................5 PPST/CORE/ First Time Test Takers ......................................................................................................................................................................7 PRAXIS II: Content Test ........................................................................................................................................................................................8 Benchmark Reviews ................................................................................................................................................................................................8 Benchmark II .........................................................................................................................................................................................................10 Benchmark III ........................................................................................................................................................................................................13 Student Teaching Performance Ratings .................................................................................................................................................................15 Educational Benchmarking Inventory (EBI) .........................................................................................................................................................16 Communicating Assessment Data with Constituencies .........................................................................................................................................16 Utilization of Assessment Data to Improve Courses and the Program ..................................................................................................................16 2015-16 Goals ........................................................................................................................................................................................................16 Technology & Science Education AIM 2014-15 Page 1 Overview The University of Wisconsin-Stout School of Education (SOE) has gathered assessment data from fall semester 2003 through 2014/2015. However, the BS in Technology and Science Education officially began in the fall of 2009. In the spirit of the Assessment in the Major report, a list of assessment tools used to collect data will be presented in this report. Specifically, data gathered from these tools is used to develop program goals, inform curriculum changes, and enhance course delivery in order to improve teacher education candidate learning. This report contains data from the: PRAXIS I: Pre-Professional Skills Test (PPST) and Core Academic Skills for Educators (CORE) PRAXIS II: Content Test Benchmark I, II, and III reviews Student Teaching Performance Ratings Educational Benchmarking Inventory (EBI) 2006 and 2010 Alumni Follow-up Survey The report will also describe how assessment data is used to set programmatic goals, improve the program, program curriculum, and delivery of courses. Overview of the Program In 2013-14, the Technology & Science Education program consisted of nine Technology and Sciences Education candidates, six categorized as preeducation majors and three that were fully admitted into the program. Technology & Science Education AIM 2014-15 Page 2 Previous Goals and Evidence 1 2 3 4 5 6 2014-15 Goals Increase candidate exposure to classrooms, teachers, and students. a. School of Education staff and Technology Education faculty and staff are currently identifying schools, contemporary technology education programs, and qualified cooperating teachers that would serve as observation sites for pre-student and teacher candidates (Ongoing Goal). b. Attention will continue to be focused on developing preservice teachers’ and Technology Education Collegiate Association’s efforts to work with peers, professionals and students through competitive events, field trips, and lab activities (Ongoing Goal). Collaborate with STEM and other colleges/institutions as improving lab experiences and access for candidates is a key element for candidates to thoroughly understand their content and, as a result, it’s interdisciplinary power (Ongoing Goal). Seek input from the BSTE advisory committee that includes, among others, current candidates and alumni, STEM education professionals from other institutions and states, UW-Stout STEM college faculty, and Technology Education faculty (Ongoing Goal). Visit schools, host professional development opportunities, and increase online and mail marketing campaigns in order to increase program enrollment (Ongoing Goal). Collect data and refine planning for student candidates to begin work toward the Teacher Performance Assessment (edTPA). Work with SOE faculty and staff to create an implementation plan for edTPA through revised courses and assessment system (Ongoing Goal). Refer candidates in science, technology, and math education (STMED) courses to the Writing Center here at Stout Technology & Science Education AIM 2014-15 Evidence towards meeting goals or rationale for abandoning: We have added opportunities in STEMED 160, Intro to Technology and Science Education, for candidate to visit and interact with k-12 teachers and students in the region. In addition, candidates participate in lesson delivery at the primary level in TECED 325, Technology for Elementary School Children. This is in addition to the Pre-student Teaching experience. TEECA continues to sponsor a number of competitive events. In addition, they volunteer to serve in leadership positions with events sponsored by the STEM college. Communicated with STEM faculty regarding candidate employment and professional development opportunities that are germane to the positions they aspire to attain. Facilitated bi-annual BSTE advisory board meetings Hosted professional development opportunities, and increased online and mail marketing campaigns with the goal of increasing program enrollment. Worked with University Marketing to establish a marketing plan for BSTE. Marketed a new online option within the program. In addition, we launched a new STEMED ambassadorship program to reach out to influential teachers and leaders. All candidates completed edTPA starting in the Fall of 2014. Some were commissioned to submit their portfolios to Pearson for scoring. Results were analyzed each semester and compared against local and national scoring. Referred candidates in science, technology, and math education (STMED) courses to the Writing Center. Page 3 7 (http://www.uwstout.edu/writingcenter/) for support and feedback in order to increase PPST performance. Instructors in science, technology, and math education (STMED) courses will identify opportunities for candidates to demonstrate their philosophies of education within existing curricula in order to foster a working knowledge of their philosophy and subsequently increase candidate Benchmark II interview performance. Technology & Science Education AIM 2014-15 Instructors purposely targeted opportunities to reinforce candidates abilities to articulate details of their philosophy of education as they develop over their academic career. Page 4 PRAXIS I: Pre-Professional Skills Test/CORE Tests All education majors must pass all three sections of the PRAXIS I: Core Academic Skills for Educators (CORE). The three sections consist of reading, writing and mathematics. Prior to the fall 2013 semester, candidates would take the Pre-Professional Skills Test (PPST) which was available until the end of the 2013-14 academic year. Wisconsin state policy allows the unit to provide an exception for up to 10 % of any cohort. Also, candidates can meet their Benchmark I requirements through scores on their college entrance exam. Note that the pass rates in the table reflect attempts by all candidates prior to being formally accepted into the School of Education, all candidates are required to pass the CORE as part of Benchmark I, therefore the pass rate of candidates accepted is 100%. Table 1 - PRAXIS I: Pre-Professional Skills Test (PPST) Completion Data Teacher Education Program TSE SOE 2010 PPST Test Math Math Exception Writing Writing Exception Reading Reading Exception Math Math Exception Writing Writing Exception Reading Reading Exception 2011-12 2012-13 2013-14 # test attempts 1 # (and %) passed 1 (100%) # test attempts 0 # (and %) passed NA # test attempts 0 # (and %) passed NA # test attempts 1 # (and %) passed 1 (100%) NA NA NA NA NA NA NA NA 1 1 (100%) 0 NA 1 0 (0%) 1 1 (100%) NA NA NA NA NA NA NA NA 1 1 (100%) 0 NA 0 NA 1 1 (100%) NA NA NA NA NA NA NA NA 198 150 (76%) 210 142 (68%) 114 86 (75%) 70 53 (76%) NA NA NA NA 1 NA NA NA 213 141 (66%) 287 138 (48%) 175 95 (54%) 104 46 (44%) NA NA NA NA 1 NA 1 NA 243 138 (57%) 240 135 (56%) 157 86 (55%) 109 48 (44%) NA NA NA NA 2 NA 3 NA Technology & Science Education AIM 2014-15 Page 5 CORE Attempts and Pass Rates Teacher Education Program TSE SOE 2013-14 CORE Test CORE Math Math Exemption CORE Writing Writing Exemption CORE Reading Reading Exemption CORE Math Math Exemption CORE Writing Writing Exemption CORE Reading Reading Exemption 2014-15 # test attempts NA # (and %) passed NA # test attempts NA # (and %) passed NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA NA 15 6 (40%) 122 67 (55%) NA NA 1 NA 13 7 (54%) 136 56 (41%) 1 NA 1 NA 14 10 (71%) 113 75 (66%) 3 NA NA NA No candidates from Technology and Science Education applied for Benchmark I during the 2014-15 academic year. Technology & Science Education AIM 2014-15 Page 6 PPST/CORE/ First Time Test Takers Beginning in the 2013-14 school year, all teacher preparation programs in the state of Wisconsin are required to report on “the passage rate on the first attempt of candidates and graduates of the program on examinations administered for licensure” (Act 166, section 17). As noted in the table below, no candidates Technology and Science Education candidates attempted the exam during the 2014-15 academic year. Teacher Education Program TESE SOE Teacher Education Program TESE SOE Spring 2013 PPST Test Math Writing Reading Math Writing Reading # first time test takers NA NA NA 63 65 63 2013-14 # (and %) passed NA NA NA 52 (83%) 40 (62%) 40 (63%) # first time test takers 1 1 1 56 53 56 2013-14 CORE Test CORE Math CORE Writing CORE Reading CORE Math CORE Writing CORE Reading # first time test takers NA NA NA 13 11 13 # (and %) passed NA NA NA 6 (46%) 5 (45%) 9 (69%) Technology & Science Education AIM 2014-15 # (and %) passed 1 (100%) 1 (100%) 1 (100%) 48 (86%) 25 (47%) 28 (50%) 2014-15 # first time test takers NA NA NA 87 92 97 # (and %) passed NA NA NA 56 (64%) 41 (45%) 68 (70%) Page 7 PRAXIS II: Content Test Note that all candidates are required to pass the Praxis II to be admitted to student teaching as part of Benchmark II so the pass rate is 100% upon Benchmark II approval. All candidates seeking a Wisconsin teaching license in Science and Technology Education must pass both content tests (Praxis II) in the area for which they will be certified. Future tables will summarize Science and Technology Education candidates’ PRAXIS II results. Note that all candidates are required to pass the Praxis II in Technology Education and Broad Field Science to be admitted to student teaching as part of Benchmark II so the pass rate is 100% upon Benchmark II approval in 2009. Benchmark Reviews The UW-Stout School of Education Assessment System is designed to review candidates’ progress at three intervals during the program. Candidates are considered pre-education majors until they have passed the Benchmark I review. In this instance, the review determines a candidate’s readiness to become a teacher candidate in the Technology Education program. Beginning Spring 2015, adjustments were made to the Benchmark I Interview rubric. Later, candidates are reviewed again during the Benchmark II review and interview to determine whether they are equipped to proceed to the student teaching portion of the program. Benchmark III is completed at the end of student teaching and before a candidate is recommended for licensure. This includes the artifacts from student teaching and recommendations by cooperating teachers. TE:SE 2014-15 3 3 SOE 2014-15 156 103 Cleared for Benchmark I Review based on score from college entrance test N/A 29 Denied: No passing PPST score Denied: Low GPA Denied: Insufficient credits/course work Denied: Did not receive “C” or higher in English, Speech, Intro, or Foundation of Education courses Denied: Missing background check Denied: Other Reasons N/A N/A N/A 27 5 28 N/A 2 N/A N/A 8 N/A Benchmark I Applications Cleared for Benchmark I Review *Individual candidates who apply multiple times per academic year are counted twice or more. There might also be multiple reasons for candidates not clearing for Benchmark I review. Some candidates might also have been cleared for BM I review based on PPST/CORE scores, even though they would have been cleared based on scores from their college entrance test. Being cleared for Benchmark I review does not necessarily mean that a candidate went through with it. Technology & Science Education AIM 2014-15 Page 8 Benchmark I Rubric Results (new rubric) ePortfolio Review Rubric Foundations of Education (EDUC-326) Final Project or Program Equivalent Response Deficiency Satisfactory Deficiency Satisfactory Deficiency Satisfactory Dispositional Review Rubric Response Commitment to Learning: The candidate will demonstrate a commitment to his/her own and his/her students’ continuous learning Respect for Oneself and Others: The candidate will demonstrate respect for himself/herself and others through thoughtful and responsive communication, showing respect and collaboration Commitment to Excellence: The candidate recognizes his/her professional responsibility for engaging in and supporting appropriate professional practices for self and others Deficiency Signed Statement of Values and Dispositions Resume Technology & Science Education AIM 2014-15 Satisfactory Deficiency Satisfactory Deficiency Satisfactory TE:SE Spring 2015 N=1 0% 100% 0% 100% % 100% SOE Spring 2015 N=40 0% 100% 0% 100% 2% 98% 0% 2% 100% 98% 0% 2% 100% 98% 0% 2% 100% 98% Page 9 Benchmark I Review Results (old rubric) Artifact Name Two (2) artifacts related to the subject matter the candidate will teach Response Incomplete Complete Incomplete Complete Incomplete Complete Incomplete Complete Disposition Area Response Commitment to Learning: The candidate will demonstrate a commitment to their own and their students continuous learning Deficiency Signed Statement of Values and Dispositions Resume Philosophy Statement Respect for Oneself and Others: The candidate will demonstrate respect for others through thoughtful and responsive communication, showing respect and collaboration Commitment to Excellence: The candidate recognizes his/her professional responsibility for engaging in and supporting appropriate professional practices for self and others TE:SE 2013-14 Fall 2014 N=2 N=1 0% 0% 100% 100% 0% 0% 100% 100% 0% 0% 100% 100% 0% 0% 100% 100% SOE Fall 2014 N=39 0% 100% 3% 97% 0% 100% 3% 97% 0% 0% 0% 100% 100% 100% 0% 0% 0% No Deficiency Deficiency 100% 0% 100% 0% 100% 0% No Deficiency 100% 100% 100% No Deficiency Deficiency Benchmark II In order to be considered for a Benchmark II review, Technology and Science Education majors must have completed or be currently enrolled in a pre-student teaching field experience; have completed or be currently enrolled in core education courses and obtained at least a “C” in these same courses which includes: Education Psychology, Multiculturalism, Cross-Cultural Field Experience, Secondary Reading and Language Development, Inclusion, and Lab and Classroom Management. In addition, candidates must have a cumulative GPA of 2.75 at UW-Stout. Technology & Science Education AIM 2014-15 Page 10 Candidates are rated by two reviewers at an unsatisfactory, emerging, or basic level - with a rating of emerging or basic considered to be sufficient evidence for a candidate to be allowed to move forward in the program. No candidates attempted to complete a Benchmark II interview in 2014-15 (see Table 2). Table 2 - Benchmark II Interview Results Benchmark II Interview Results Question Describe your Philosophy of Education and how it has evolved. Describe what it means to be a "Reflective Practitioner." Describe the WI Teacher Standard and Domain you feel most competent in. Describe the WI Teacher Standard and Domain you have experienced the greatest growth. Provide Portfolio evidence (signed copy of the Instructional Technology Utilization rubric) of your competence in current instructional technology. Technology & Science Education AIM 2014-15 Response Unsatisfactory Emerging Basic n/a Unsatisfactory Emerging Basic n/a Unsatisfactory Emerging Basic n/a Unsatisfactory Emerging Basic n/a Unsatisfactory Emerging Basic n/a 20112012 N=2 0% 50% 50% 0% 0% 50% 50% 0% 0% 50% 50% 0% 0% 50% 50% 0% 0% 0% 100% 0% TE:SE SOE 2012- 2013201420142013 2014 2015 2015 N=0 N=0 N=0 N=103 N/A N/A N/A 3% N/A N/A N/A 40% N/A N/A N/A 57% N/A N/A N/A 0% N/A N/A N/A 2% N/A N/A N/A 33% N/A N/A N/A 65% N/A N/A N/A 0% N/A N/A N/A 1% N/A N/A N/A 27% N/A N/A N/A 72% N/A N/A N/A 0% N/A N/A N/A 3% N/A N/A N/A 34% N/A N/A N/A 63% N/A N/A N/A 0% N/A N/A N/A 0% N/A N/A N/A 29% N/A N/A N/A 71% N/A N/A N/A 0% Page 11 Reviewers choose 2 of the following; discuss portfolio evidence that: Unsatisfactory 0% N/A Emerging 100% N/A Demonstrates your content knowledge. Basic 0% N/A n/a 0% N/A Unsatisfactory 0% N/A 50% N/A Demonstrates your ability to create instructional opportunities Emerging adapted to diverse learners. Basic 50% N/A n/a 0% N/A Unsatisfactory 0% N/A Emerging 0% N/A Demonstrates your ability to teach effectively. Basic 0% N/A n/a 0% N/A Unsatisfactory 0% N/A Emerging 100% N/A Demonstrates your ability to assess student learning. Basic 0% N/A n/a 0% N/A Technology & Science Education AIM 2014-15 N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A N/A 3% 45% 53% 0% 1% 41% 58% 0% 18% 50% 32% 0% 3% 31% 66% 0% Page 12 Benchmark III Benchmark III is the culminating assessment for candidates in the Technology and Science Education program. Benchmark III must be completed before a pre-service teacher can be recommended for licensure. The assessment requires that all coursework is completed, that all program-specific requirements are met, a satisfactory student teaching assessment is achieved, and that a complete electronic portfolio receiving a basic or higher proficiency rating is submitted. In order to facilitate this, candidates complete a student teaching experience at both the middle school and high school level. Technology & Science Education AIM 2014-15 Page 13 Benchmark III Interview Results Question Artifacts from student teaching, reflection ratings. Final Student Teaching Assessments and Recommendations from Cooperating Teachers. Disposition ratings from student teaching from cooperating & University Supervisors. Instructional Technology Utilization Rubric. Alignment Summary of artifacts meeting all 10 Wisconsin Teaching Standards & 4 Domains/ Components & reflections/ reflection ratings. Technology & Science Education AIM 2014-15 Response Unsatisfactory Emerging Basic Proficient n/a Unsatisfactory Emerging Basic Proficient n/a Unsatisfactory Emerging Basic Proficient n/a Unsatisfactory Emerging Basic Proficient n/a Unsatisfactory Emerging Basic Proficient n/a 20112012 N=1 0% 0% 100% 0% 0% 0% 0% 100% 0% 0% 0% 0% 100% 0% 0% 0% 0% 0% 100% 0% 0% 0% 0% 100% 0% TE:SE SOE 2012- 2013201420142013 2014 2015 2015 N=1 N=0 N=0 N=117 0% NA NA 0% 0% NA NA 0% 0% NA NA 9% 100% NA NA 91% 0% NA NA 0% 0% NA NA 0% 0% NA NA 1% 0% NA NA 24% 100% NA NA 75% 0% NA NA 0% 0% NA NA 0% 0% NA NA 1% 0% NA NA 15% 100% NA NA 85% 0% NA NA 0% 0% NA NA 0% 0% NA NA 0% 0% NA NA 4% 100% NA NA 96% 0% NA NA 0% 0% NA NA 0% 0% NA NA 0% 0% NA NA 1% 100% NA NA 99% 0% NA NA 0% Page 14 Student Teaching Performance Ratings Student Teacher Evaluations Technology and Science Education Rating Scale: 1=Unsatisfactory, 2=Emerging, 3=Basic, 4=Proficient Teachers know the subjects they are teaching Teachers know how children grow Teachers understand that children learn differently Teachers know how to teach Teachers know how to manage a classroom Teachers communicate well Teachers are able to plan different kinds of lessons Teachers know how to test for student progress Teachers are able to evaluate themselves Teachers are connected with other teachers and the community Teachers make effective use of instructional technologies to enhance student learning. 2012-13 N=1 Mean 4.00 3.00 TE:SE 2013-14 N=0 Mean NA NA 2014-15 N=0 Mean NA NA SOE 2014-15 N=120 Mean 3.75 3.75 3.00 4.00 3.00 4.00 NA NA NA NA NA NA NA NA 3.75 3.75 3.58 3.77 3.00 NA NA 3.73 3.00 3.00 NA NA NA NA 3.69 3.82 3.00 NA NA 3.77 3.00 NA NA 3.76 Final student teaching evaluations are aligned with the School of Education’s conceptual framework (Danielson’s Framework for Teaching) and the 10 Wisconsin Teaching Standards. Teacher candidates finishing their student teaching experiences are evaluated by their cooperating teachers on a four point scale 1=Unsatisfactory, 2=Emerging, 3=Basic, 4=Proficient. Final student teaching evaluations are aligned with the School of Education’s conceptual framework (Danielson’s Framework for Teaching) and the 10 Wisconsin Teaching Standards. Teacher candidates finishing their student teaching experiences are evaluated by their cooperating teachers on a four point scale. A score of one indicates a lower ranking while a score of 4 indicates a higher ranking. Technology and Science Education candidates’ rankings are compared to the School of Education students’ ranks in the table above. No BSTE and Science Education candidates were involved in student teaching in 2014-15. Technology & Science Education AIM 2014-15 Page 15 Educational Benchmarking Inventory (EBI) The Educational Benchmarking Inventory (EBI) has been administered via computer to exiting teacher candidates during the fall and spring terms since 2003. Eighty-eight questions and fourteen EBI factors are collected for the purpose of unit assessment and are rated on a scale from 1 to 7 with 1= not at all, 4= moderately and 7= extremely). EBI data for candidates in this major were not available at the time of this report because there were no teacher candidates. Communicating Assessment Data with Constituencies Communicating data with faculty members, advisory board members, and candidates within the program will be done using various methods. The report will be shared with faculty members during scheduled discipline area work group meetings (DAWG) designed to support ongoing program improvement. Program faculty and staff then discuss ways to better meet the needs of candidates throughout the program. Action plans for desired change result from work group meetings. The B.S in Technology and Science Education program advisory board, which includes candidates, will receive a copy of the Assessment in the Major report during the fall advisory meeting. Their comments and recommendations for improvement will be encouraged. University supervisors will be asked to share the AIM report with cooperating teachers and solicit feedback from them regarding improvements for the program. The report will be made available to candidates within the Technology and Science Education program and an opportunity provided for feedback. Data from the AIM report will also be used in program revision processes. Utilization of Assessment Data to Improve Courses and the Program The program goals listed below were established for the 2013-2014 academic year. Many of these goals will be retained, although evidence has been provided to show progress made toward their attainment. The goals for the 2014-2015 school year are listed subsequently and reflect needs demonstrated in the data and summaries above: 2015-16 Goals 1 Continue top seek opportunities for candidate’s diverse and rich exposure to classrooms, teachers, and students. a. School of Education staff and Technology Education faculty and staff are currently identifying schools, contemporary technology education programs, and qualified cooperating teachers that would serve as observation sites for pre-student and student teacher candidates (Ongoing Goal). Attention will continue to be focused on developing pre-service teachers’ and Technology Education Collegiate Association’s efforts to work with peers, professionals and students through competitive events, field trips, and lab activities (Ongoing Goal). 2 Collaborate with STEM and other colleges/institutions as improving lab experiences and access for candidates is a key element for candidates to thoroughly understand their content and, as a result, it’s interdisciplinary power (Ongoing Goal). Technology & Science Education AIM 2014-15 Page 16 3 Seek input from the BSTE advisory committee that includes, among others, current candidates and alumni, STEM education professionals from other institutions and states, UW-Stout STEM college faculty, and Technology Education faculty (Ongoing Goal). 4 Explore ways to collaborate with STEM and CTE related program in the marketing of our programs in order to increase enrollment (Ongoing Goal). 5 Integrate edTPA data into the assessment in the major report. Use data effective to establish goals for program improvement. 6 Work with SOE faculty and staff to revise the assessment system to integrate embedded signature assessment aligned with standards and the edTPA. 7 Refer candidates in science, technology, and math education (STMED) courses to the Writing Center here at Stout (http://www.uwstout.edu/writingcenter/) for support and feedback in order to increase PPST performance. 8. Instructors in science, technology, and math education (STMED) courses will identify opportunities for candidates to demonstrate their philosophies of education within existing curricula in order to foster a working knowledge of their philosophy and subsequently increase candidate Benchmark II interview performance. Technology & Science Education AIM 2014-15 Page 17