CEM CONFERENCE EXETER Interpreting Feedback from Baseline testsWhole School and Individual Student Data Day 1 Session 2 Geoff Davies 27th February 2013 Using Baseline Data for Teaching and Learning • Informing professional judgement • Begging Questions • Supporting teachers • Starting conversations • Starting a diagnostic process in some cases How able is this year group? What are the strengths and weaknesses of this cohort? How did individuals perform? Bands, percentiles, standardised scores… C B A D 0 50 60 70 80 90 100 110 120 130 140 Standardised scores Percentiles: 1 5 10 20 30 40 50 60 70 25 80 75 90 95 99 150 National Quartile Ability Bands BANDS BASELINE D C B A ALIS: AV GCSE <5.0 5.0-5.7 5.7-6.55 >6.55 Yellis % >43 43-52 52-61 >61 STANDARDISED CABT MidYIS, Yellis and Alis <90 90-100 100-110 >110 IN ANY COHORT, 25% OF THE POPULATION WILL BE IN EACH BAND. THE VALUES MAY VARY SLIGHTLY FROM YEAR TO YEAR Band Profile Graph: all MidYIS cohort A school with a ‘completely average’ intake would have 25% of pupils within each band. Band D Checking this graph each year will give you an immediate overview of your intake. Band A Independent schools only 2011-12 Overall: Percent in each Band Year 7 MidYIS Test 2011/2012 Independent Schools' Standardisation Year 7 MidYIS Test 2011/2012 Independent Schools' Standardisation 100 90 80 70 Percent 60 50 45 40 36 30 20 14 10 5 0 D C B A 7 Band Profile - MidYIS Overall Score Year 7 Academic Year 2012 35 32 Percentage 30 23 25 23 20 14 15 10 5 5 5 0 0 <= 76 76 - 88 Far Below Well Below Average Average 88 - 96 96 - 105 105 - 113 Below Average Average Above Average Category 113 - 125 >= 125 Well Above Far Above Average Average A Grammar School present year 11 (5th year) YELLIS BANDS Alis: year 12 students, two baseline profiles for the same school Nationally, 25% in each band Comments and potential implications? Vocabulary: Percent in each Band Year 7 MidYIS Test 2011/2012 100 90 90 80 80 70 70 60 60 Percent Percent 100 50 37 40 20 25 35 26 30 20 17 21 20 10 17 10 0 0 D C B A Band D D Band A 90 80 80 70 70 60 60 Percent 90 40 33 35 30 A Band A 50 40 30 29 21 20 B Skills: Percent in each Band Year 7 MidYIS Test 2011/2012 100 50 C Band D Non-Verbal: Percent in each Band Year 7 MidYIS Test 2011/2012 100 Percent 50 40 30 Maths: Percent in each Band Year 7 MidYIS Test 2011/2012 24 26 21 20 11 10 10 0 0 D C B A D Comments? C B A Proof Reading: Percent in each Band Year 7 MidYIS Test 2011/2012 100 90 80 70 Percent 60 Skills: Percent in each Band Year 7 MidYIS Test 2011/2012 100 50 40 38 26 30 19 90 20 17 80 10 70 0 D 40 30 100 29 24 26 21 C B A Perceptual Speed & Accuracy: Percent in each Band Year 7 MidYIS Test 2011/2012 50 90 20 80 10 70 0 D C B A 60 Percent Percent 60 50 40 30 30 24 23 23 D C B 20 10 0 A What do the sections of the MIDYIS test measure? Vocabulary Score The Vocabulary component of the test is generally an important element for most subjects. For English, History and some Foreign Languages it is the best. However the Vocabulary score is perhaps the most culturally linked of all the scores. Those who have not been exposed to vocabulary-rich talk or a wide variety of reading material or whose first language is not English are unlikely to have developed as high a vocabulary score as they would have developed in a different environment. Maths Score The Maths score is well correlated with most subjects but is particularly important when predicting Maths, Statistics, ICT, Design Technology and Economics. The Maths section has been designed with the emphasis on speed and fluency, rather than knowledge of Maths. Like the Vocabulary score, the Maths score is a good predictor of later academic performance. Non-Verbal Score The Non-Verbal score is composed of the three sub tests: CrossSections, Block Counting and Pictures. The Non-verbal score is important when predicting Maths, Science, Design Technology Geography, Art and Drama. It provides a measure of the pupil’s ability in 3-D visualisation, spatial aptitude, pattern recognition and logical thinking. It can give an insight in to the developed ability for pupils for whom English is a second language Skills Score In the Proof Reading section pupils are asked to spot mistakes in the spelling, punctuation and grammar of a passage of text. eg mis-spelling of words like ‘there’ and ‘their’. The PSA (Perceptual speed and accuracy) section asks pupils to look for matches between a sequence of symbols on the left and a number of possible choices on the right. Given enough time most pupils would probably get the answers correct but we are measuring how quickly pupils can find a correct match. An interesting result from our work with the Deaf and Hearing Impaired community shows that on average, Hearing Impaired pupils score well above the national average on the PSA section of the test. The PSA section allows speed to be demonstrated free from the demands of memory. The Proof Reading and PSA tests are tests for the modern world, and are designed to measure fluency and speed. They rely on a pupil’s scanning and skimming skills, skills that are desirable in examination situations. Individual Pupil Records: • Show pupils’ strengths and weakness of different sections of the baseline test • Bands A,B,C, & D (quartiles) • Scores are standardised to have a mean of 100 and a standard deviation of 15 • Tables and graphs which include error bars • Stanines and percentiles • Confidence limits 16 IPRs (Individual Pupil Record Sheets) Top of confidence limit Bottom of confidence limit An exercise for you on this later!! Scores Standardised On A Nationally Representative Sample of Schools Standardised Scores Sort Data By... Mean Score: 100 Standard Deviation: 15 Average 100.6 95.9 105.3 99.2 98.1 St. error 1.1 1.1 1.1 1.1 1.1 Band St. Score MidYIS Score Band St. Score Skills Band St. Score Non Verbal Band Maths St. Score Band St. Score Vocabulary On another day, with 95% certainty, the score of 100.6 (+/- 1.1 x 2) • would not be higher than 100.6 + 2.2 = 102.8 • would not be lower than 100.6 – 2.2 = 98.4 How might this information be useful? Some pupil data MIDYIS 2009 Vocabulary Maths Non-Verbal Skills Overall Score Band Score Band Score Band Score Band Score Band Pupil 01 122 A 125 A 116 A 107 B 126 A Pupil 02 105 B 110 A 127 A 95 C 108 B Pupil 03 105 B 93 C 110 B 89 D 99 C Pupil 04 91 C 116 A 130 A 115 A 103 B Pupil 05 111 A 144 A 122 A 103 B 129 A Pupil 06 107 B 112 A 85 D 97 C 109 B Pupil 07 115 A 106 B 100 C 86 D 112 A Pupil 08 141 A 137 A 132 A 135 A 143 A Pupil 09 104 B 92 C 105 B 109 B 98 C Pupil 10 99 C 119 A 114 A 99 C 109 B Pupil 11 108 B 126 A 130 A 140 A 118 A Pupil 12 106 B 123 A 120 A 105 B 116 A Pupil 13 103 B 96 C 103 B 104 B 99 C Pupil 14 108 B 110 B 112 A 108 B 110 A Pupil 15 95 C 104 B 103 B 122 A 99 C Stanine 1 2 4% 50 60 3 4 5 6 7 8 9 7% 12% 17% 20% 17% 12% 7% 70 80 90 100 110 120 Standardised Test Score 4% 130 140 150 Individual Pupil Record Date Of Birth: Sex: 75 Student Number: Form: 24/12/97 F Vocabulary Maths Non-Verbal Skills MidYIS Score Band Stanine Percentile Standard Score B 5 53 101 B 5 52 101 C 5 46 98 C 5 46 99 B 5 52 101 Nationally Standardised Scores with 95% Confidence Band Nationally Standardised Scores 150 140 130 120 110 100 90 80 70 60 Proof Reading PSA 106 B 96 C 50 Vocabulary Maths Non-Verbal Skills MidYIS Score A relatively lower vocabulary score might indicate a difficulty which: • could contribute to under-performance in most, if not all subjects • might lead to ‘stressful situations’ • may lead to further investigation and subsequent pupil support Individual Pupil Record Date Of Birth: Sex: RAWAD OKBI Student Number: Form: 29/08/99 M Vocabulary Maths Non-Verbal Skills MidYIS Score 8E Band Stanine Percentile Standard Score D 3 20 87 B 5 58 103 B 6 67 107 B 6 71 108 C 4 34 94 Nationally Standardised Scores with 95% Confidence Band Nationally Standardised Scores 150 140 130 120 110 100 90 80 70 60 Proof Reading PSA 88 D 118 A 50 Vocabulary Maths Non-Verbal Skills MidYIS Score A relatively low maths score might indicate potential weaknesses in subject areas which require: • numerical skills • logical thinking • skills such as sequencing CONVERSELY: a high maths score but a low vocab/reading score……. Individual Pupil Record Date Of Birth: Sex: Vocabulary Maths Non-Verbal Skills MidYIS Score RACHEL STRONACH Student Number: Form: 13/1/99 F 8E Band Stanine Percentile Standard Score B 6 63 105 D 3 15 84 A 6 76 111 B 6 67 107 C 4 37 95 Nationally Standardised Scores with 95% Confidence Band Nationally Standardised Scores 150 140 130 120 110 100 90 80 70 60 50 Vocabulary Maths Non-Verbal Skills MidYIS Score A relatively high non-verbal score might indicate potential strengths in subject areas which require: • 3d, and 3d into 2d, visualisation • Spatial awareness • Understanding images in 2d representing 3d • Extracting information from visual images • Science, D and T, Art, Geography…… and vice versa Individual Pupil Record Date Of Birth: Sex: Vocabulary Maths Non-Verbal Skills MidYIS Score SULAIMAN BAHELMI Student Number: Form: 23/2/99 M Band Stanine Percentile Standard Score A 7 81 113 B 6 66 106 D 3 18 86 B 6 68 107 A 7 77 111 Nationally Standardised Scores with 95% Confidence Band Nationally Standardised Scores 150 140 130 120 110 100 90 80 70 60 50 Vocabulary Maths Non-Verbal Skills MidYIS Score A relatively low skills score might indicate potential weaknesses such as: • Speed of processing/working • Potential underperformance in test/examination conditions • Poor written work (SPG etc.) and vice versa Skills: Proof Reading Skills: Perceptual Speed and Accuracy Date Of Birth: Sex: Student Number: Form: M Vocabulary Maths Non-Verbal Skills MidYIS Score 7Y Sex: Band Stanine Percentile Standard Score A 9 99 134 A 8 90 119 A 7 85 116 D 3 20 87 A 9 98 130 Vocabulary Maths Non-Verbal Skills MidYIS Score 150 150 140 140 130 120 110 100 90 80 70 Band Stanine Percentile Standard Score C 5 50 100 A 8 89 119 B 5 54 102 B 6 61 104 B 6 72 109 Nationally Standardised Scores with 95% Confidence Band Nationally Standardised Scores Nationally Standardised Scores Nationally Standardised Scores with 95% Confidence Band Proof Reading PSA 101 B 83 Form: M 130 120 110 100 90 80 Proof Reading PSA 70 D 60 83 D 114 A 60 50 50 Vocabulary Maths Non-Verbal Skills MidYIS Score Vocabulary Maths Non-Verbal Skills MidYIS Score SEN use of MIDYIS test results Analysis of baseline test individual skill profiles to indicate potential areas of learning difficulties e.g. If the two lowest scores are the vocabulary and skills section and are, or close to being, statistically significant (see IPR) • Then the student might be dyslexic....... • Follow-up with appropriate diagnostic tests….. Sex: Form: F Vocabulary Maths Non-Verbal Skills MidYIS Score 7C Band Stanine Percentile Standard Score C 4 27 91 B 6 68 107 B 6 75 110 D 2 10 81 C 5 45 98 Nationally Standardised Scores with 95% Confidence Band Nationally Standardised Scores 150 140 130 120 110 100 90 80 70 60 50 Vocabulary Maths Non-Verbal Skills MidYIS Score Using national baseline test scores to Identify Gifted Pupils Scores over 130 – top 2% nationally Scores over 126 – top 5% nationally Scores over 120 – top 10% nationally Scores over 110 – top 25% nationally Using MidYIS IPRs to Inform Teaching and Learning The IPR on its own simply tells us about the relative performances of the pupil on the separate sections of the test, where the pupil is strong, where performance has been significantly above or below national averages or where the pupil has significantly outperformed in one section or another. It is when the IPR is placed in the hands of a teacher who knows that pupil that it becomes a powerful tool. It is what teachers know about individual pupils: what has happened in the past, how they respond to given situations and how they work in the teacher’s specific subject that inform the interpretation of the IPR If the IPR data from MidYIS, the teacher’s personal and subject specific knowledge and experiences regarding the pupil can be shared, then there becomes a much more powerful instrument for supporting pupils’ learning needs. Some pupils will display an IPR pattern with significant differences between one or two components of the MidYIS Test. These can be the most interesting and possibly the most challenging pupils for mainstream classroom teachers. Scenarios and anecdotal findings Handout Page 3 A SELECTION OF MIDYIS SCORES FOR ‘WATERLOO ROAD’ !! Why would this be a very challenging class to teach? Vocabulary Surname Maths Non Verbal Skills MidYIS Score St. Score Band St. Score Band St. Score Band St. Score Band St. Score Band Sex A F 81 D 110 B 108 B 112 A 94 C B F 128 A 107 B 105 B 94 C 120 A C M 106 B 121 A 103 B 90 D 114 A D F 107 B 84 D 96 C 107 B 96 C E M 96 C 90 D 130 A 91 C 92 C F F 86 D 86 D 120 A 74 D 84 D G F 100 B 115 A 80 D 103 B 108 B H F 121 A 96 C 114 A 86 D 111 A I M 92 C 100 C 96 C 123 A 95 C J M 100 C 105 B 100 C 99 C 102 B K M 128 A 132 A 114 A 131 A 133 A L M 76 D 70 D 74 D 73 D 71 D What do I need to know/do to teach this (difficult) class of twelve pupils These are real anonymous scores from a number of schools around the UK 3 Vocabulary scores significantly lower than other component scores Second language? Deprived areas? Difficulty accessing curriculum.? Targeted help does work. Seen in nearly all schools. Worth further diagnosis. Could potentially affect performance in all subjects Vocabulary scores significantly higher than other component scores Good communicators. Get on. Put Maths problems in words? Mathematics significantly higher than other scores From Far East? Done entrance tests? Primary experience? Mathematics significantly lower than other scores Primary experience. Use words and diagrams? Sometimes difficult to change attitude..Difficultiess with logical thinking and skills such as sequencing Low Mathematics scores with High Non-verbal Scores Use diagrams. Confidence building often needed Pupils with non-verbal scores different from others – High Non-verbal Scores Frustration? Behaviour problems? Don’t do as well as good communicators or numerate pupils? Good at 3D and 3D to 2D visualisation and spatial awareness. Good at extracting information from visual images Pupils with non verbal scores different from others – Low Non-verbal Scores Peak at GCSE? A level ? Pupils with low Skills scores Exams a difficulty after good coursework? Suggests slow speed of processing. High Skills Scores Do well in exams compared with classwork? The Average Pupil They do exist! High scores throughout Above a score of 130 puts the pupil in the top 2% nationally Low scores throughout Below a score of 70 puts the pupil in the bottom 2% nationally 3 Sharing data with colleagues: e.g. baseline test data Last Name First Name Tutor Group Vocab + + 07S1 07Q1 07H1 07M2 Non Verbal Skills - + 117 126 106 + + 127 120 07Y1 111 + 07Y1 + 113 07S2 07S2 07T2 98 + - 137 116 + + 07Q1 07S1 07H2 Overall MidYIS 117 - 07Q2 07H1 Maths - 101 113 126 + 07H1 88 07H2 104 07M1 123 - 07M2 07S1 - 111 + + 07Q1 07Q2 7Y2 - 128 + 114 109 132 INSIGHT Pupil IPR Speed Reading Text Comprehension Passage Comprehension Overall Reading Number & Algebra Handling Data Space, Shape & Measures Overall Mathematics Biology Chemistry Physics Overall Science Vocabulary Non Verbal Skills Overall Ability Standard KS3 Score Equivalent Band Stanine Percentile B B 6 5 69 60 107 104 6c 5a C 5 45 98 5b B 5 59 103 5a D 3 19 87 4a B D 6 4 61 23 104 89 6a 5c C 4 31 93 5b A A A 7 9 8 82 96 89 114 127 118 6a 7a 7c A 8 93 122 7b B B A 6 6 8 65 70 92 106 108 121 C 4 36 94 Overall Ability Skills Maths 60 Non Verbal Vocabulary Overall Science Physics Chemistry Biology Overall Mathematics Space, Shape & Measures Handling Data Number & Algebra Overall Reading Passage Comprehension Text Comprehension Speed Reading Standardised Scores Standardised Scores With 95% Confidence Band 160 140 120 100 80 40 Overall Science Physics Chemistry Biology Overall Mathematics Space, Shape & Measures Handling Data Number & Algebra Overall Reading Passage Comprehension Text Comprehension a 8b c a 7b c a 6b c a 5b c a 4b c a 3b c 2a N Speed Reading KS3 Level IPRs in KS3 Levels KS3 Equivalent Levels with 95% Confidence Band NATIONAL AVERAGE The code: j: reduces size k: infills m: inverts Sharing the baseline Information within School Using your MIS systems Baseline test data can be useful • to indicate reasons for student learning difficulties and may go some way to explain lack of progress, flag up causes for underachievement and even behaviour problems. • for all teachers and support staff. It can help to support professional judgement and give a better understanding of the progress students make at school and their potential later performance. • to refer to for pupil reviews, writing reports, meeting parents, monitoring progress and interim assessments. HANDOUT 1 pages 1 and 2 Percentage of pupils falling into each MidYIS Band over time 2013 Current Year7 2012 Current Year 8 2011 Current Year 9 2010 Current Year 10 2009 Current Year 11 2008 Current Year 12 2007 Current Year 13 2006 Last year’s Year 13 OVERALLMIDYIS Historical Band Profiles: Overall Score (Nationally Representative Standardisation) 100% Year Band A Band B Band C Band D 2013 14 26 36 23 2012 22 21 31 26 2011 18 24 31 28 2010 24 23 28 24 2009 19 29 28 24 2008 17 34 24 25 2007 23 26 27 23 2006 18 24 26 32 90% 80% Percentage 70% 60% Band D 50% Band C 40% Band B 30% Band A 20% 10% 0% 2006 2007 2008 2009 2010 2011 2012 2013 MidYIS Baseline Testing Year VOCABULARY Historical Band Profiles: Vocabulary Score (Nationally Representative Standardisation) 100% Year Band A Band B Band C Band D 2013 19 34 28 18 2012 25 20 37 17 2011 25 24 26 25 2010 28 23 27 21 2009 23 22 29 27 2008 30 21 25 25 2007 30 26 23 20 2006 19 19 30 32 90% 80% Percentage 70% 60% Band D 50% Band C 40% Band B 30% Band A 20% 10% 0% 2006 2007 2008 2009 2010 2011 MidYIS Baseline Testing Year 2012 2013 1 MATHS Historical Band Profiles: Maths Score (Nationally Representative Standardisation) 100% 90% Year Band A Band B Band C Band D 2013 12 18 39 31 2012 17 21 26 35 2011 12 21 34 33 2010 21 20 29 29 2009 23 22 26 29 2008 16 19 36 28 2007 19 23 29 28 2006 24 18 29 29 80% Percentage 70% 60% Band D 50% Band C 40% Band B 30% Band A 20% 10% 0% 2006 2007 2008 2009 2010 2011 2012 2013 MidYIS Baseline Testing Year NON VERBAL Historical Band Profiles: Non Verbal Score (Nationally Representative Standardisation) Year Band A Band B Band C Band D 2013 29 35 22 15 2012 35 33 21 11 2011 19 35 27 19 2010 24 32 33 11 2009 34 24 31 11 2008 21 31 30 18 2007 28 37 22 13 2006 25 34 25 15 100% 90% 80% Percentage 70% 60% Band D 50% Band C 40% Band B 30% Band A 20% 10% 0% 2006 2007 2008 2009 2010 2011 MidYIS Baseline Testing Year 2012 2013 Scores Standardised on a Nationally Representative Sample of Schools 1 SKILLS Year Band A Band B Band C Band D 2013 18 27 13 42 2012 21 26 24 29 2011 24 24 19 32 2010 30 26 23 21 2009 26 26 29 19 2008 21 21 27 30 2007 23 25 31 21 2006 12 25 34 29 Historical Band Profiles: Skills Score (Nationally Representative Standardisation) 100% 90% 80% Percentage 70% 60% Band D 50% Band C 40% Band B 30% Band A 20% 10% 0% 2006 2007 2008 2009 2010 2011 2012 MidYIS Baseline Testing Year PROOF READING PERCEPTUAL SPEED AND ACCURACY Band A Band B Band C Band D 2013 14 15 19 52 2012 17 19 26 38 2011 14 20 22 44 2010 20 24 27 29 2009 21 29 18 31 2008 14 25 28 33 2007 14 35 27 24 2006 11 22 35 32 Year 2013 Year Band A Band B Band C Band D 2013 31 22 17 30 2012 30 23 23 24 2011 33 23 25 19 2010 36 22 23 19 2009 32 24 22 22 2008 23 23 24 30 2007 26 29 22 23 2006 16 24 32 29 2 1) Some teachers feel that the intake into the school has changed in the past few years. If so in which areas is the intake stronger or weaker? 2) Look at the breakdown of the individual test components for last year’s year 11. (2008 entry). What do you notice? a) Achievements in a core subject at GCSE in 2012 were below what had been achieved in previous years. Can you suggest which subject it was? b) Why would you expect better results at GCSE in this subject this year. c) This subject faces challenges in at least two further entry cohorts. Which are they? Suggest a strategy that they might follow. 3) What factors could explain the discrepancy between scores in the individual components and also in those scores from year to year? Is there anything that can be done about it? 4) Why do you think that the discrepancies between the score components can be vastly different between schools. Which scores are particularly high compared to the whole national cohort for this school. 2 HANDOUT 1 Page 3 (slides 37/38) Handout 1 Page 4 and 5 Exercise What are the Strengths and weaknesses of this A/AS level student? To use the IPR (Individual pupil record) familiarise yourself with the terms standard score, band, stanine, percentile and confidence band a)Which AS/A level subjects might be avoided? ------------------------------------b) This student chose English, Film Studies, Music Technology and Psychology. Is this a good choice? Do you foresee any problems?-------------------- You are given data relating to an institution where students completed the ALIS computer adaptive test. They are chosen because they show significant differences between the various parts of the test. Remember scores are standardised around 100. Name Overall St.Score Band Vocab St.Score Maths Band St.Score Non Verbal Band St.Score Band Average A Level subjects chosen GCSE A 78 D 49 D 99 B 92 C na biologymathsbusiness art B 94 C 115 A 85 D 104 B na biology businesspsychologyenglish C 88 D 97 C 85 D 104 B 5.6 historypsychologyenglishmedia D 101 B 107 B 97 C 80 D 5.9 businesshistoryenglishdrama E 104 B 87 D 112 A 116 A na biologyphysicsmaths business F 81 D 47 D 103 B 111 B na mathsfurthermathsbusiness G 93 C 113 A 84 D 113 A na biologybusinessfrenchgeography H 97 C 111 A 89 D 99 C 7 artenglishpsychologyreligiousst. I 87 D 68 D 100 B 109 B 5.4 mathsgeographyfrenchmusic J 105 B 67 D 124 A 85 D 6.1 mathsfurthermathspsychologyeconomi cs K 96 C 71 D 110 A 97 C na biologymathsartenglish L 92 C 60 D 111 A 97 C na mathshistoryreligiousst.english a) Are there any apparent mismatches between the subjects being followed and this data? b) What support can be given to those students who have particular weaknesses in Vocabulary or Mathematics ? c) How might predictions made for these students be tempered in the light of the inconsistencies in the test components and missing average GCSE points scores?