BOWLAND MATHS Impact analysis: Guidance Note Overall aim: to compile evidence about the teaching of problem solving and its impact, which could be used for demonstrating its effectiveness to the others Objective: simple qualitative assessment of student progress in problem solving for and by teachers Introduction: Some enthusiastic teachers have expressed interest in monitoring/measuring student progress, so that they can develop a more objective sense of what works. This note provides guidance for teachers to help them do simple impact analysis in their class/school, within a small informal network community of teachers, who could then also learn from each other. A research group in Japan, led by Keiichi Nishimura, will be conducting a parallel study with a small group of Japanese teachers in their next academic year (April 2015 – March 2016); this will provide excellent international comparative opportunities. Dr. Nishimura is an Associate Professor of Gakugei University in Japan who advises the Japanese government on maths education; he is also a founder of Bowland Japan and a core member of IMPULS. Design: Summary Use two Bowland assessment tasks as a ‘pre-test’ with a class in a given period in September/October; keep the student worksheets Make a record of all problem-solving tasks taught in the class in the year Use the same two assessment tasks as ‘post-tests’ with the same class, but in two separate periods in June/July; keep the student worksheets. Once the ‘test results’ are collected, teachers could follow up to teach the tasks Use progression grids as guides to evaluate progress made by students Assessment Tasks to be used: For lower level classes: Day Out and Tuck Shop For higher level classes: Fares not fair and Tuck Shop N.B. The above task combination is the one expected to be used in Japan. An alternative ‘similar task’ is Soft Drinks. Teachers can selected other assessment tasks, although this could limit comparative discussions with Japanese colleagues. Pre-test (September or October) - at the beginning of the year: Use two assessment tasks (A and B) in a class as ‘tests’ in which half of the students (S) do task A while the other half (S') do task B Distribute tasks A and B randomly in the class (put an equal number of the two tasks in a pile, roughly mix and distribute) Distribute worksheets – on grid paper, the responses tend to be neater Tests should be in silence with a time limit of 40 minutes, with students working alone with no intervention from teachers Observe students working and record noteworthy events Collect worksheets - without teaching the tasks Post-test (June/July) - at the end of the year: Ask all the students in the class (S and S') to do task A in one period and task B in another period; subsequently, each task can be 'taught' in class. Control group – if possible It would be ideal if the pre- and post- tests could be conducted with more than one class of similar ability (within the same school) who would be taught different levels of problem solving during the year – especially if there was a class for which no problem solving would be taught Variation across participating teachers/schools may provide useful insights – although level/type of teaching would not be easy to compare Qualitative comparisons The main focus of the analysis would be to look for progress made by students, using the task progression grids o So that teachers can ‘demonstrate’ progress with specific examples: “with ‘x’ level of teaching, students made ‘this kind’ of progress” o There should be no expectation of numerical analysis (e.g. how many students moved from one level to another within a progression grid) - but rather a qualitative analysis based on ‘checks’ Three types of comparison to ‘triangulate’, although individual teachers would not be expected to analyse all these exhaustively: o Same students doing the same tasks SA(Sept) against SA(Jul); S'B(Sept) against S'B(Jul); o Different students doing the same tasks at different times SA(Sept) against S'A(Jul), S'B(Sept) against SA(July) o Same students doing different tasks SA(Sept) against SB(Jul) S’A(Sept) against S’B(Jul) In the summer, we hope to have a one-day workshop to explore more about worksheet marking, using the progression grid, based on the students’ retained worksheets. Additional option to assess ‘pair work’ Each of above three ‘test’ periods (one for the pre-test for both tasks and one each for the two post tests) could be followed by a period in which students pair up to do the same tasks, but still as tests and without teacher intervention Pairs could be given 2/3 of the period for joint work, with the final 1/3 of the period used for individual students to write ‘reflections’ about what changes they made as a result of working in pairs (Dr. Nishimura’s team will be using this optional addition to help examine student learning through pair work)