M.S. & Ed.S. in Career and Technical Education Assessment in the Major Report By Dr. Deanna Schultz and Dr. Carol Mooney 2014-15 Submitted: October 2015 Table of Contents 1. Outcomes of Previous Assessments .................................................................................................................................................... 3 2. Questions to Be Answered From This Year’s Assessment ................................................................................................................. 3 3. Methods Used to Gather Data ............................................................................................................................................................. 3 4. Number of Students Involved .............................................................................................................................................................. 4 5. Level of Students ................................................................................................................................................................................. 4 6. Data Analysis Techniques ................................................................................................................................................................... 4 7. Analysis of Data .................................................................................................................................................................................. 4 8. How Results Were Shared with Key Instructors ................................................................................................................................. 9 9. Plans for Improvements Based on Results. ......................................................................................................................................... 9 10. Goals for 2015-16 ............................................................................................................................................................................ 9 Appendix A ............................................................................................................................................................................................... 10 Appendix B ............................................................................................................................................................................................... 12 Appendix C ............................................................................................................................................................................................... 19 MS & EDS in CTE 2014-15 Page 2 1. Outcomes of Previous Assessments Prior to 2011-12, reports were based on two sources of data: students enrolled in the CTE Coordinator certification and graduate follow up studies. Beginning in 2011, the MS and EdS CTE assessment in the major plan incorporated a self-assessment of program competencies in the form of pre- and post-tests. This data provides the program directors and teaching faculty feedback for use in course content and curriculum. 2. Questions to Be Answered From This Year’s Assessment The questions to be answered in the 2014-15 assessment report are the same as the previous year to allow for comparison of data: a. For those students completing the CTE Coordinator certification, what competencies do students perform well, above 3 (basic) rating? b. For those students completing the CTE Coordinator certification, what competencies do students average below 3 (basic) rating? c. For those students completing the master’s degree program, what competencies showed the greatest increase from the start of the program to completion? d. For those students completing the master’s, what competencies showed the least increase from the start of the program to completion? 3. Methods Used to Gather Data Upon completion of required coursework and internship, students in the CTE Coordinator licensure program compile a competency portfolio. Eleven students completed CTE Coordinator certification in 2014-15. A benchmark III rubric was used to evaluate their portfolios; the results have been compiled and presented in this AIM report. Students who entered the MS and EdS CTE programs in Fall 2014 and Spring 2015, and students who completed the MS and EdS CTE programs in December 2014, May 2015 and August 2015 were asked to complete a self assessment (pre and post test) of program competencies. The program directors sent an email to both groups of students and requested their participation in the preand post assessment online survey. New students completed the pretest during their first semester in the programs; December, May and August graduates completed the posttest prior to commencement. MS & EDS in CTE 2014-15 Page 3 4. Number of Students Involved Enrollment for 2014-2015 in the M.S. program was 56 and the Ed.S. program was 10, for a total of 66 students. Eleven students completed CTE Coordinator licensure requirements and submitted a portfolio during the 2014-15 academic year. Benchmark III rubrics were used to evaluate portfolios. Twelve students responded to the new student survey/pretest and self assessed their knowledge relative to the program competencies. Thirteen graduates completed the post-test survey and provided responses relative to their self-assessment of meeting program competencies at the time of graduation. 5. Level of Students Only graduate students were involved in the collection of data presented in this report. 6. Data Analysis Techniques Information collected from the CTE Coordinator benchmark III rubrics was entered into an Excel Spreadsheet. Frequencies and averages were calculated. Results are summarized in the report and raw data is included Appendix A. The pre- and post-tests data were analyzed using the Qualtrics data analysis and reporting functions. A Likert scale ranking, with 1 being lowest and 5 being highest, was provided for responses. Frequencies and means were reported in the analysis. Summary results along with a complete report of both pre- and post-test results can be found in the appendices. 7. Analysis of Data Competency Portfolio – CTE Coordinator- Benchmark III Eleven students completed the requirements for certification as Career and Technical Education coordinators. Highlights of their performance includes: MS & EDS in CTE 2014-15 Page 4 2014-15 Results Students demonstrated the greatest proficiency in the following standards, with a mean of 4.0/4.0 scale: 6. Developing Relationships to Support CTE Program Goals 17. Acting with Integrity, Fairness, and in an Ethical Manner These were followed by: 5. Developing and Implementing CTE Evaluation Plans for Program Improvement (3.82) 1. Planning and Advocating for CTE (3.73) 4. Knowing and Promoting of CTE Work-Based Learning Programs (3.64) 7. Developing and Managing Budgets and Grants (3.64) 12. Knowing the Relationship between Career Development and CTE (3.64) The lowest ratings of student performances were in the following standard, with a mean of 3.27/4.0 scale: 14. Integrating CTE Academic Standards into PK-12 Curricula and Assessment The next two lowest means were in two standards: 9. Using Standards to Select and Supervise Personnel (3.36) 13. Providing Leadership in Promoting Post-Secondary Options for Students (3.45) The highest and lowest ratings of student performance were above the basic level of 3.0, which is required for satisfactorily meeting certification requirements. 2013-14 Results Students demonstrated the greatest proficiency in the following standards, with a mean of 3.6/4.0 scale: 1. Planning and Advocating for CTE 2. Knowing and Promoting Comprehensive CTE Programs 4. Knowing and Promoting of CTE Work-Based Learning Programs MS & EDS in CTE 2014-15 Page 5 6. Developing Relationships to Support CTE Program Goals 9. Using Standards to Select and Supervise Personnel 10. Assessing Needs and Delivering Professional Development Opportunities 12. Knowing the Relationship between Career Development and CTE 16. Understanding of and Competency in the Ten Teacher Standards The lowest ratings of student performances were in the following standards, with a mean of 3.2/4.0 scale: 13. Providing Leadership in Promoting Post-Secondary Options for Students 14. Integrating CTE Academic Standards into PK12 Curricula and Assessment 15. Knowing the Roles and Responsibilities of Working in Profit-Making Business Environments. The highest and lowest ratings of student performance were above the basic level of 3.0. It is interesting to compare the data from the previous year’s report. The program director will share these results with the key faculty who deliver courses for the CTE Coordinator certification. The complete data set of student results are found in Appendix A. Graduate Follow up Studies Graduate follow-up studies are conducted by the UW-Stout’s PARQ (Planning, Assessment, Research and Quality) office every other year. No graduate follow-up studies were conducted in 2014-15; they will be conducted again in 2015-16. The results of the graduate follow-up surveys are analyzed by the program director when provided. The analysis guides the program director and results are shared with key faculty. Pre-test/Post-test Results Twelve students completed the pre-test/self-assessment of program competencies; thirteen graduates completed the post-test/selfassessment of program competencies. When reviewing the results, low scoring pre-test responses were compared with post-test responses. MS & EDS in CTE 2014-15 Page 6 When comparing the pre-and post-test responses, improvements were shown across all competencies (average increase of 1.43). While it is may be presumed that competency would be gained while enrolled and completing the MS CTE program, the data confirms that students advanced their knowledge and self-confidence relative to identified competencies. Table 1 follows and compares low ranked competencies by new students to post-test results. Analysis of the pre-test to post-test means indicate that the course of study in the MS and EdS CTE programs resulted in improved results; the difference between initial enrollment and graduation scores showed an increase in all competencies. It was interesting to note that the overall pre-test means of new students were higher in 2014-15 than the previous year. In 2014-2015, 16 of the 37 competencies had a mean of 3.00 or higher compared to 2013-2014, when 5 of the 37 competencies had means of 3.00 or higher on the pre-test survey. Table 1 Comparison of Low Ranked Competencies in Pretest to Post-test results Competency Pre-test mean Post-test mean Develop and implement a program of work for advisory committee(s) 2.36 4.09 Explain the legal responsibilities of various school officials 1.82 3.45 Explain state laws and state agencies as they relate to the school of CTE 2.18 4.17 Track and describe historical antecedents of contemporary social and educational thought 2.18 4.62 Coordinate the gathering, selection, and analysis of labor market data relative to program(s). 2.45 4.08 Prepare a proposal for funding career and technical education programs based on state and federal guidelines 1.45 3.44 The next two data sets present post-test results only, including results from 2013-14 for comparison (adjusted for an outlier). The lowest rated competencies on the post-test are provided in Table 2. MS & EDS in CTE 2014-15 Page 7 Table 2 Lowest rated post-test competencies Competency Use management techniques including program planning and budgeting systems. Explain the legal responsibilities of various school officials Prepare a proposal for funding career and technical education programs based on state and federal guidelines Establish and advise a professional student organization Post-test mean 2014-15 2013-14 Post-test mean (adjusted) 3.62 3.88 3.45 3.43 3.44 4.00 3.73 4.63 "Explain the legal responsibilities of various school officials” and “Prepare a proposal for funding career and technical education programs based on state and federal guidelines” also scored low on the pre-test, so the increase to 3.45 and 3.44 respectively represent an increase in competence in those areas. Also, the courses which address these competencies, Educational Grant Proposal Development and Policy and Legal Issues, are elective courses in the MS CTE program, selective courses in the Ed.S. CTE program, and are required for CTE Coordinator certification. The respondents to this years’ survey may not have taken those courses as selectives, resulting in lower post-test means. The other two low-rated competencies may reflect course taking of students in the programs. The competencies will be monitored by the program director in future program reports. The highest ranked competencies from the post-test are presented below. Table 3 Highest ranked competencies in post-test self-assessment Competency Explain the history, issues, principles and philosophy of career and technical education Use distance learning technology Apply appropriate learning theories and principles in educational practice MS & EDS in CTE 2014-15 Post-test mean 2013-14 Post-test mean 4.92 4.50 4.85 4.63 4.77 4.25 Page 8 Understand, evaluate and utilize research for decision-making and program planning Research, collect and utilize data for program solving and program planning Provide leadership to units, departments and the school relative to CTE 4.77 4.50 4.77 4.38 4.77 4.50 All six of the highest ranked competencies in 2014-15 were rated higher than in the previous year. The competency that has been consistently high over the past three years has been “use distance learning technology,” indicating the hybrid delivery of the MS CTE and EdS programs provides students opportunities to improve their competence in the use of distance learning technology. The high means for “explain the history, issues, principles and philosophy of career and technical education” and the two research competencies suggest that the foundation course (Principles of CTE) and research courses that are required of all students are achieving their learning goals. 8. How Results Were Shared with Key Instructors This report, in its entirety, will be presented at program faculty and advisory committee meetings. 9. Plans for Improvements Based on Results. Data will be shared with key instructors and the program advisory committees to identify potential areas of improvement. 10. Goals for 2015-16 Continue to monitor student competency attainment in the MS and EdS CTE programs and review alignment of competencies with core curriculum. Review the competencies with the CTE Advisory Committee and seek input on potential revisions. MS & EDS in CTE 2014-15 Page 9 Appendix A Career and Technical Education Coordinator Benchmark 3 Composites 1. Planning and Advocating for CTE 4 4 4 3 3 4 3 4 4 4 4 3.73 2. Knowing and Promoting Comprehensive CTE Programs 4 4 4 3 3 4 3 4 4 4 4 3.73 MS & EDS in CTE 2014-15 3. Knowing and Promoting CTE Co-Curricular Student Organizations 3 4 4 3 3 4 3 4 4 4 3 3.55 4. Knowing and Promoting of CTE Work-Based Learning Programs 4 4 4 3 3 4 3 4 3 4 4 3.64 5. Developing and Implementing CTE Evaluation Plans for Program Improvemnent 3 4 4 4 4 4 4 4 4 3 4 3.82 6. Developing Relationships to Support CTE Program Goals 4 4 4 4 4 4 4 4 4 4 4 4.00 7. Developing and Managing Budgets and Grants 3 4 4 4 4 3 4 4 4 3 3 3.64 8. Gathering and Analyzing CTE Data 4 4 4 3 3 4 3 4 4 4 4 3.73 9. Using Standards to Select and Supervise Personnel 4 3 3 3 3 3 3 3 4 4 4 3.36 Page 10 Career and Technical Education Coordinator Benchmark 3 Composites 10. Assessing Needs and Delivering Professional Development Opportunities 4 4 4 3 3 4 3 4 4 4 4 3.73 11. Engaging in Practices Conducive to Continuous Improvement 3 3 4 4 3 3 3 4 4 3 4 3.45 MS & EDS in CTE 2014-15 12. Knowing the Realationship between Career Development and CTE 4 4 4 3 3 4 3 4 3 4 4 3.64 13. Providing Leadership in Promoting PostSecondary Options for Students 4 4 3 3 3 4 3 3 4 3 4 3.45 14. Integrating CTE Academic Standards into PK-12 Curricula and Assessment 4 3 4 3 3 3 3 3 4 3 3 3.27 15. Knowing the Roles and Responsibilities of Working in Profit-Making Business Environments 4 3 4 3 3 4 3 4 3 3 4 3.45 16. Understanding of and Competency in the Ten Teacher Standards 4 4 3 4 3 4 3 3 3 4 4 3.55 17. Acting with Integrety, Fairness, and in an Ethical Manner 4 4 4 4 4 4 4 4 4 4 4 4.00 Average Rating 3.76 3.76 3.82 3.35 3.24 3.76 3.24 3.76 3.76 3.65 3.82 3.63 Page 11 Appendix B CTE MS and EdS Programs Pre-test/Entrance Survey - Self-Assessment Results MS & EDS in CTE 2014-15 Page 12 Initial Report: MSCTE Entrance Survey 2014-2015 Last Modified: 09/02/2015 Filter By: Report Subgroup 1. Please select your beginning semester # Answer Response % 1 Fall 2012 0 0% 2 Spring 2013 0 0% 3 Fall 2013 0 0% 4 Spring 2014 0 0% 5 Fall 2014 9 75% 6 Spring 2015 3 25% 7 8 Fall 2015 Spring 2016 0 0 0% 0% Total 12 100% Statistic Value Min Value 5 Max Value 6 Mean 5.25 Variance Standard Deviation 0.20 0.45 Total Responses MS & EDS in CTE 2014-15 12 Page 13 2. Please rate your knowledge, skill or attitude next to each statement, based at the point of entry into the CTE graduate program: MS & EDS in CTE 2014-15 Page 14 # Question Strongly Disagree 1 2 3 4 Strongly Agree 5 Total Responses Mean 1 Develop a strategic plan for program(s) operation 1 2 4 3 1 11 3.09 2 Develop a CTE program advisory committee with representation of internal and external stakeholders 1 3 4 2 1 11 2.91 3 Develop and implement a program of work for advisory committee(s) 2 5 2 2 0 11 2.36 3 2 1 4 1 11 2.82 2 2 3 4 0 11 2.82 4 5 Design an implementation plan and procedures for a program, school or department Use management techniques including program planning and budgeting systems 6 Apply continuous quality improvement practices to a program, department or school 1 1 4 4 1 11 3.27 7 Explain the legal responsibilities of various school officials 5 3 3 0 0 11 1.82 8 Explain state laws and state agencies as they relate to the school of career and technical education 3 4 3 1 0 11 2.18 0 2 1 5 3 11 3.82 3 1 5 2 0 11 2.55 2 5 4 0 0 11 2.18 9 10 11 Develop and maintain effective interpersonal relationships with internal and external stakeholders Explain the history, issues, principles and philosophy of career and technical education Track and describe historical antecedents of contemporary social and educational thought MS & EDS in CTE 2014-15 Page 15 Strongly Disagree 1 2 3 4 Strongly Agree 5 Total Responses Mean 12 Determine the philosophical implications of technology for people, human values, human knowledge and society 0 5 3 1 2 11 3.00 13 Develop and relate personal philosophy of education to the contemporary CTE program 1 2 3 3 2 11 3.27 14 Plan and maintain procedures to evaluate instruction and programs for relevance and effectiveness 1 3 4 2 1 11 2.91 15 Develop or revise a curriculum or program to meet identified needs 1 1 2 3 4 11 3.73 16 Utilize and evaluate a variety of media for instruction 1 0 3 4 2 10 3.60 17 Use distance learning technology 0 5 2 2 2 11 3.09 18 Utilize a variety of tools and systems to develop and enhance instructional materials and improve curriculum 1 1 1 5 3 11 3.73 19 Coordinate the gathering, selection and analysis of labor market data relative to program(s) 2 5 2 1 1 11 2.45 2 2 2 4 1 11 3.00 3 1 3 2 2 11 2.91 2 0 6 2 1 11 3.00 # 20 21 22 Question Establish a program that involves work-based learning Develop a plan with school personnel to offer a comprehensive CTE program offerings Devise a plan of public relations that will relate your education program to the community MS & EDS in CTE 2014-15 Page 16 Question Strongly Disagree 1 2 3 4 Strongly Agree 5 Total Responses Mean 23 Establish a career plan in cooperation with the school counseling department that will assure all students will have an opportunity to consider and review their post secondary options 1 5 3 2 0 11 2.55 24 Develop a plan to assist students in locating and securing employment 1 2 6 2 0 11 2.82 25 Integrate occupational, technical and academic competencies in courses 1 3 3 3 1 11 3.00 26 Prepare a proposal for funding career and technical education programs based on state and federal guidelines 6 5 0 0 0 11 1.45 27 Establish and advise a professional student organization 0 7 1 1 2 11 2.82 28 Articulate a career and technical education program(s) with internal and external stakeholder groups 1 3 4 2 1 11 2.91 29 Apply appropriate learning theories and principles in educational practice 1 2 2 3 2 10 3.30 30 Understand, evaluate and utilize research for decision making and program planning 1 1 4 4 1 11 3.27 31 Research, collect and utilize data for program solving and program planning 2 0 5 3 1 11 3.09 32 Plan, conduct and report surveys (graduate follow up, labor market assessment, etc.) 2 4 3 1 1 11 2.55 33 Provide leadership to units, departments and the school relative to CTE 2 2 3 4 0 11 2.82 # MS & EDS in CTE 2014-15 Page 17 # Question Strongly Disagree 1 2 3 4 Strongly Agree 5 Total Responses Mean 34 Utilize strategic planning processes within the CTE domain 1 4 5 1 0 11 2.55 35 Represent CTE programs at external and internal stakeholder groups 1 3 3 2 1 10 2.90 36 Use techniques to forecast changes and implications of emerging technology 1 4 4 1 1 11 2.73 37 Analyze needs of students' occupational interest for college and career planning 0 4 3 3 1 11 3.09 MS & EDS in CTE 2014-15 Page 18 Appendix C CTE MS and EdS Programs Post Test/Exit Survey – Self Assessment Results MS & EDS in CTE 2014-15 Page 19 Initial Report: MSCTE Exit Survey 2014-2015 Last Modified: 09/02/2015 Filter By: Report Subgroup 1. Please select your graduating semester # Answer Response % 1 Fall 2012 0 0% 2 Spring 2013 0 0% 3 Fall 2013 0 0% 4 Spring 2014 0 0% 5 Fall 2014 5 38% 6 Spring 2015 8 62% 7 Fall 2015 0 0% Total 13 100% Statistic Value Min Value 5 Max Value 6 Mean 5.62 Variance 0.26 Standard Deviation Total Responses 0.51 13 MS & EDS in CTE 2014-15 Page 20 2. My Career & Technical Education, M.S. at UW-Stout prepared me to: MS & EDS in CTE 2014-15 Page 21 Strongly Disagree 1 2 3 4 Strongly Agree 5 Total Responses Mean Develop a strategic plan for program(s) operation 0 0 2 4 7 13 4.38 2 Develop a CTE program advisory committee with representation of internal and external stakeholders 0 1 3 4 3 11 3.82 3 Develop and implement a program of work for advisory committee(s) 0 2 1 2 6 11 4.09 0 1 1 6 5 13 4.15 1 2 1 6 3 13 3.62 # Question 1 4 5 Design an implementation plan and procedures for a program, school or department Use management techniques including program planning and budgeting systems 6 Apply continuous quality improvement practices to a program, department or school 0 1 1 2 9 13 4.46 7 Explain the legal responsibilities of various school officials 0 2 4 3 2 11 3.45 8 Explain state laws and state agencies as they relate to the school of career and technical education 0 1 2 3 6 12 4.17 0 1 1 4 7 13 4.31 0 0 0 1 12 13 4.92 9 10 Develop and maintain effective interpersonal relationships with internal and external stakeholders Explain the history, issues, principles and philosophy of career and technical education 11 Track and describe historical antecedents of contemporary social and educational thought 0 0 1 3 9 13 4.62 12 Determine the philosophical implications of technology for people, human values, human knowledge and society 0 0 0 4 9 13 4.69 MS & EDS in CTE 2014-15 Page 22 13 Develop and relate personal philosophy of education to the contemporary CTE program 0 0 1 2 10 13 4.69 14 Plan and maintain procedures to evaluate instruction and programs for relevance and effectiveness 0 1 1 3 7 12 4.33 15 Develop or revise a curriculum or program to meet identified needs 0 1 1 4 6 12 4.25 16 Utilize and evaluate a variety of media for instruction 0 1 0 4 7 12 4.42 17 Use distance learning technology 0 0 1 0 12 13 4.85 18 Utilize a variety of tools and systems to develop and enhance instructional materials and improve curriculum 0 1 1 4 6 12 4.25 19 Coordinate the gathering, selection and analysis of labor market data relative to program(s) 1 0 2 3 6 12 4.08 20 Establish a program that involves work-based learning 0 0 3 2 7 12 4.33 0 0 1 5 6 12 4.42 0 0 4 2 4 10 4.00 23 Establish a career plan in cooperation with the school counseling department that will assure all students will have an opportunity to consider and review their post secondary options 0 2 2 2 6 12 4.00 24 Develop a plan to assist students in locating and securing employment 0 1 2 3 5 11 4.09 25 Integrate occupational, technical and academic competencies in courses 0 1 0 6 5 12 4.25 21 22 Develop a plan with school personnel to offer a comprehensive CTE program offerings Devise a plan of public relations that will relate your education program to the community MS & EDS in CTE 2014-15 Page 23 26 Prepare a proposal for funding career and technical education programs based on state and federal guidelines 1 1 2 3 2 9 3.44 27 Establish and advise a professional student organization 0 2 2 4 3 11 3.73 28 Articulate a career and technical education program(s) with internal and external stakeholder groups 0 1 1 5 5 12 4.17 29 Apply appropriate learning theories and principles in educational practice 0 0 0 3 10 13 4.77 30 Understand, evaluate and utilize research for decision making and program planning 0 0 0 3 10 13 4.77 31 Research, collect and utilize data for program solving and program planning 0 0 0 3 10 13 4.77 32 Plan, conduct and report surveys (graduate follow up, labor market assessment, etc.) 0 0 0 4 9 13 4.69 33 Provide leadership to units, departments and the school relative to CTE 0 0 0 3 10 13 4.77 0 0 1 7 5 13 4.31 0 0 2 5 6 13 4.31 34 35 Utilize strategic planning processes within the CTE domain Represent CTE programs at external and internal stakeholder groups 36 Use techniques to forecast changes and implications of emerging technology 0 0 1 4 8 13 4.54 37 Analyze needs of students' occupational interest for college and career planning 0 0 3 1 8 12 4.42 MS & EDS in CTE 2014-15 Page 24 3. Please list any subjects or areas of study that you feel would be of value in a graduate career and technical education program. These may be courses already included in the program that you feel deserve special emphasis or revision; however we are primarily looking for new ideas/or areas that have been neglected in the present program. Text Response I learned a lot from each of courses regarding of CTE field or other programs. career guidance across the life span adult learners in retraining programs make Psychology of Adult Learners available to on line learners provide more opportunity for face to face classes/Create a hybrid program Statistic Total Responses MS & EDS in CTE 2014-15 Value 2 Page 25 4. Please make any additional comments you feel may help to improve the graduate program. Text Response Everything was great Going through the graduate program for CTE was a bit unique for me, as I am not a teacher anywhere, I am professional in the corporate world that is looking to pursue being a professor. My advisor helped make the program align with my current work experience in training in the corporate world which I was thankful for. Some of the courses I did feel were geared towards those that were already teaching, so some of the assignments were sometime tailored to fit my situation. Not sure how to incorporate this type of student / perspective better, but I would guess there will be more students that are not currently teaching in a school, or college that enter into this graduate program - because it truly is an awesome graduate program! My responses to the survey questions that included any 2's or 3's may be explained by the nature of the classes that I chose to take during the program. I did not take many of the counseling related coursework because I was focused on another area of the MS in CTE. Great program and excellent staff!!!!!! Amazing program. A perfect fit for a working, distance learning adult! Keep up the great work. Statistic Total Responses MS & EDS in CTE 2014-15 Value 5 Page 26