WHO WE ARE https://www.youtube.com/watch?v=usQem53fDH8 A BIT MORE BACKGROUND http://www.albion.edu/academics/departments/education http://www.facebook.com/Albion.College.Teacher.Education Approximately 1300 total enrollment; 80 in TEP TEAC accredited through October 1, 2016 A BIT MORE BACKGROUND (cont’d) Elementary certifications: English language arts, integrated science, mathematics and social studies Secondary certifications: biology, chemistry, earth/space science (geology), English, French, German, history, mathematics, physics, political science, psychology, social studies (major only) and Spanish K-12 certifications: French, German, Spanish and music WHY HOLISTIC SCORING? ● Teacher Education Accreditation Council (TEAC) accreditation weakness ● change from Teacher Education Accreditation Council (TEAC) national standards to Council for the Accreditation of Educator Preparation (CAEP) standards ● need to identify what would constitute evidence and a tool for assessing the evidence ● financial constraints ● requires collaboration HISTORY OF HOLISTIC SCORING ● Gray, J. (1982). Properties of writing tasks: A study of alternative procedures for holistic writing assessment. final report. http://eric.ed.gov/?id=ED230576 ● Hunter, D. M. (1996). The use of holistic versus analytic scoring for large-scale assessment of writing.http://eric.ed.gov/?id=EJ543798 WHAT IS HOLISTIC SCORING? Holistic scoring is a process for assessing and grading the summative quality … based on professional and personal knowledge, the particular process and event, the purposes at hand, and the meanings shared by the individuals involved. An overall impression includes everything...; this cannot be broken into component parts, per se. A holistic system for scoring is legitimate and reasonable because it requires the scorers to cast a comprehensive view .... The process is synergistic because it requires iterative thinking by each participant, and invokes collaboration. It is a kind of gestalt; the whole is greater than the sum of its parts. (after Dr. Judy Nichols Mitchell) WHAT IS HOLISTIC SCORING (cont’d)? Advantages— Focuses on what writers can do, and actually did, rather than on what they didn’t do; Links function, content, and form-the acts of composing and producing; Serves as an evaluation measure and/or needs assessment; and, Is authentic to the demands of real writing tasks. WHAT IS HOLISTIC SCORING (cont’d)? Preparation— Compose/select prompt Training— Describe system Collect materials Explain prompt & anchor trial Select anchor set Tally/discuss scores Application— Read and score all materials Sort/tabulate all scores Discuss and react Select scoring scale Develop/confirm criteria WHAT IS HOLISTIC SCORING (cont’d)? WHAT IS HOLISTIC SCORING (cont’d)? Reference for four-point scoring scale— Three categories (unsatisfactory/low; basic/middle; and proficient/high); Distinguished category eliminated due to DFFT/EI conceptual framework for this category; Round down on split scores (e.g., 0+1 or 2+3); Non-adjacent and non-equivalent scores (e.g., 1, 3; 2, 4) require third scoring; and, Across three scores, count two lowest scores. See also Danielson Framework for Teaching/Evaluation Instrument. WHAT IS HOLISTIC SCORING (cont’d)? Scoring Scale— 0 (←unsatisfactory) 0, 1 (←unsatisfactory) (←proficient) 1, 2 (←basic) 3, 4 (←proficient) 2, 3, 4, 5 (←basic) 6, 7, 8 WHAT IS HOLISTIC SCORING (cont’d)? SAMPLE ANCHOR SET SCORING TABLE 4 A B C 3 2 1 0 WHAT IS HOLISTIC SCORING (cont’d)? WHAT DID WE ACHIEVE? ● TEP Curriculum Map ● Danielson Framework for Teaching Evaluation Instrument ● CAEP Standard 5 TEP Curriculum Map https://www.dropbox.com/s/q4q5lazfnktzoon/Converted%20TE P%20%26%20Danielson%20FFT%20Evaluation%20Instrumen t_correlation_v2.1.xlsx?dl=0 CAEP Standard 5: Provider Quality Assurance & Continuous Improvement The provider maintains a quality assurance system comprised of valid data from multiple measures, including evidence of candidates’ and completers’ positive impact on P-12 student learning and development. The provider supports continuous improvement that is sustained and evidence-based, and that evaluates the effectiveness of its completers. The provider uses the results of inquiry and data collection to establish priorities, enhance program elements and capacity, and test innovations to improve completers’ impact on P-12 student learning and development. CAEP Standard 5: Provider Quality Assurance & Continuous Improvement (cont’d) Quality and Strategic Evaluation 5.1 The provider’s quality assurance system is comprised of multiple measures that can monitor candidate progress, completer achievements, and provider operational effectiveness. Evidence demonstrates that the provider satisfies all CAEP standards. 5.2 The provider’s quality assurance system relies on relevant, verifiable, representative, cumulative and actionable measures, and produces empirical evidence that interpretations of data are valid and consistent. CAEP Standard 5: Provider Quality Assurance & Continuous Improvement (cont’d) Continuous Improvement 5.3. The provider regularly and systematically assesses performance against its goals and relevant standards, tracks results over time, tests innovations and the effects of selection criteria on subsequent progress and completion, and uses results to improve program elements and processes... WHAT DID WE ACHIEVE (cont’d)? ● Field Experience & Student Teaching Feedback Forms ● Lesson Plan Form ● reflection (e.g., Time I Heard Self) FIELD EXPERIENCE FEEDBACK FORM Old Form com/s/nv8yc7s72ob86p2/ASSESSMENT%20for%20NEW%20PRO GRAM%20updated%20Summer%202007.doc?dl=0 New Form https://www.dropbox.com/s/jr1adctce07a5oa/field%20experie nce_final%20evaluation__tier%201%5BR%5D.docx?dl=0 STUDENT TEACHING FEEDBACK FORM Old Form https://www.dropbox.com/s/hd297ic7jqc2gk7/finalfeedback%20form%20for%20mentor%20teachers.docx?dl=0 New Form https://www.dropbox.com/s/82mlhomshig2si7/field%20experie nce_final%20evaluation%5Bmentor_teachers%5D__tier%204 .docx?dl=0 WHAT PROBLEMS DID WE HAVE? ● Mental shift from scoring on separate attributes to scoring artifact as a whole. ● Review one artifact in relation to one DFFT element; even if two might be applicable, cumbersome to score. ● Explain the context of the artifact—course, assignment, and objectives of assignment. ● As we progressed in our understanding of the DFFT, we reassigned artifacts to different elements. ● Can’t do it all—small program, so we can score artifacts from all students without sampling; we can’t score all artifacts from all students. Choose for coverage of the framework. WHAT’S NEXT? ● Add more “tiers” to our process. ● Deepen our understanding of the DFFT to improve the validity of artifacts-to-elements match. ● Expand to actual pre-service teacher performance.