NATIONAL ACHIEVEMENT SURVEYS (Cycle III) A presentation by Educational Survey Division NATIONAL COUNCIL OF EDUCATIONAL RESEARCH AND TRAINING Sri Aurobindo Marg, New Delhi 110016 (India) 2014 Overview Standardized tests and questionnaires(School, pupil and teacher) are used to get learning achievement data and background information Objectives • • • To study the achievement level of students in different subjects. To study the difference in achievement with respect to area, gender and social groups. To study the effect of intervening variables like home, school and teacher on students’ achievement. Pupil • NAS is sample based survey, designed to check the health of the educational system and provides information about the learning achievement of students. • • • • Number of siblings Physically challenged Language spoken at home Homework checked at home, etc. School • Conducted under the Sarva Shiksha Abhiyan (SSA) programme of Government of India in government and government-aided schools . • • • • No. of working days No. of period per day Infrastructural facilities Instructional material, etc. • Educational and professional qualification Employment status Teaching resources, etc. Teacher • • • Survey Cycles Survey Cycle Class V Class VIII Class III Cycle I 2001-05 2003-08 2003-07 Cycle II 2005-08 2007-10 2007-09 Cycle III 2009-12 2011-14 2012-14 Subject tested Mathematics, Language, Environmental Studies Mathematics, Language, Science Social Science Background Questionnaires Student, Teacher and School Mathematics, Language Steps in National Achievement Surveys I. PRE CONDUCT II. CONDUCT III. POST CONDUCT • Planning and Management • Sampling for main survey • Development of data entry manual • Curriculum Analysis • Development of manual for training of field functionaries and administration of tools • Data management: Batching, entry and cleaning* • Printing tools • Data analysis and interpretation • Item development • Development of questionnaires • Sampling (piloting) • Translation of item pool • Piloting of test items • Analysis of piloted data and finalization of tools • Training of field functionaries • Merging of cleaned files • Reporting • Administration of tools * Data Cleaning • Codes were checked (State, District, School, Student, Gender, Area, Test Form, Medium). • Cases with identical pattern of responses, were dropped. • Students appeared in more than one form of the same subject were dropped. Survey Coverage States/UTs District Classes Cycle I Cycle II Cycle III Cycle I Cycle II Cycle III Class III 29 32 34 111 254 298 Class V 30 33 31 113 244 271 Class VIII 29 32 33 103 189 285 School Teachers Classes Cycle I Cycle II Cycle III Cycle I Cycle II Cycle III Class III 5,293 7,341 7,046 8,533 10,369 14,092 Class V 4,787 6,828 6,411 10,796 14,810 10,851 Class VIII 4,378 9,239 6,722 16,612 24,071 24,486 Sampling Procedure Frame Students enrolled in government and governmentaided schools Stages 3 stage : districts, schools, students. A. District Number of districts :Using Finite Population Correction (FPC) At least 40% of the total districts in a state B. School Number of Schools The number of schools sampled from a district was determined by the total number of students required for testing and the average class size within the state/UT. Selected schools by using PPS One replacement school was assigned for each sample school. C. Student Taken all sections in consideration Students selected by using SRS maximum 40 and minimum 9 students Grade wise selection District Class III & VIII: By using FPC (Finite Population Correction) formula (Maximum 27) At least 40% of the total districts in a state Schools: In most of the states on an average 250 schools Section Class III: One section in each school selected randomly (if more than one section) Class VIII: Taken all sections in consideration Students: Students selected by using SRS Class III: maximum 20 and minimum 5 students Class VIII: maximum 40 and minimum 9 students Improvements in Cycle III • • • Framework for assessment used in development of Test items. Increased measurement points in each subject area by using multiple test forms (Classes III & VIII: 2 forms) Class III : 50 measurement points in mathematics and 36 in language, Class VIII : 90 measurement points. Scaled score were computed to know the performance of students D B Anchor Block C • • • E IRT equates the multiple booklets through common anchor items Scannable OMR sheets (in class VIII) were used to collect data from the field. RIEs faculty is involved in scanning of data. Intensive training in data collection was provided to state coordinators and district coordinator. Reporting of Results Using IRT • 2 parameter model used : item difficulty and student ability. • Results reported in scaled scores • Scaled scores are scores that have been mathematically transformed from one set of numbers (i.e., the raw score) to another set of numbers in order to make them comparable in some way Advantages • IRT places students and test items on the same numerical scale. This enables us to produce meaningful ‘maps’ of item difficulty and student ability. • In IRT, the difficulty parameter for an item does not depend on the group of test takers. This allows us to use multiple test booklets to increase measurement points in any subject and these can be linked. • IRT also allows us to compare scores from tests used in different cycles - an essential characteristic for monitoring progress over time. 100 Low achievement 200 300 Mid-point = 250 400 High achievement Current Status Class VIII • Draft Report of Class VIII has been sent to external experts for review. • State specific results will be shared in the month of February– March 2014 Class III • Some reflections are being made here without using sample weights • Weights are being calculated for finalisation of the data analysis work. Overall Score 247 (0.5) Table Overall Score Boys : 246 (0.6) Girls : 249 (0.5) Table Overall Score Rural : 247 (0.6) Girls : 252 (1) Table Reading Comprehension : Percentile Score • This graph shows the spread of achievement in each state. • • The 50th percentile shows the performance that half of the students in that state would be expected to achieve (the median). Overall In reading, Uttar Pradesh has many students who did very well (i.e. they are to the right hand side of the scale) . 50th Percentile Overall : 242 Percentile Score: Reading Comprehension State / Union Territory 10th 25th 50th 75th 90th percentile percentile percentile percentile percentile Range 75-25 Range 90-10 A & N Islands Andhra Pradesh Arunachal Pradesh Bihar Chandigarh Chhattisgarh Dadra Nagar Haveli Daman & Diu Delhi Goa Gujarat Haryana Himachal Pradesh Jammu & Kashmir Jharkhand Karnataka Kerala Madhya Pradesh Maharashtra Manipur Meghalaya Mizoram Nagaland Odisha Puducherry Punjab Rajasthan Sikkim Tamil Nadu Tripura Uttar Pradesh Uttarakhand West Bengal 185 188 182 181 217 187 193 222 187 205 189 187 191 179 181 186 204 186 194 186 183 192 189 183 181 200 185 207 179 183 178 185 192 212 218 201 205 231 215 221 240 216 226 216 217 222 192 205 217 231 212 225 212 199 217 220 209 203 226 212 225 203 205 209 217 223 232 238 225 231 268 231 240 275 237 258 237 243 266 218 229 238 282 235 270 228 223 231 237 231 225 267 229 246 229 227 242 244 262 275 273 263 275 287 274 275 305 280 281 276 283 298 229 275 273 319 279 309 269 258 272 272 279 267 293 272 271 276 272 284 283 297 309 303 294 311 319 314 303 320 316 320 312 317 331 268 312 304 339 312 336 298 283 302 306 315 295 318 308 292 314 308 319 316 325 62 55 62 70 56 60 54 65 64 55 59 66 76 36 70 56 88 67 84 56 59 55 52 71 65 68 60 46 73 68 75 66 74 124 115 112 130 101 128 110 98 128 115 123 130 140 90 132 118 135 126 141 112 100 110 117 133 114 118 124 85 136 125 141 131 133 Overall Distribution 190 215 242 279 310 63 120 • Inter quartile range is highly variable. Lower than 50th percentile of Kerala. Lower than 75th percentile of Kerala. 50th and 75th percentile of Kerala. Overall Score 245 (0.6) Table Overall Score Boys : 246 (0.7) Girls : 245 (0.6) Table Overall Score Rural : 246 (0.7) Girls : 241 (1.4) Table Mathematics : Percentile Score • • • This graph show the spread of achievement in each state. The 50th percentile shows the performance that half of the students in that state would be expected to achieve (the median). In mathematics, Arunachal Pradesh and Tamil Nadu have very similar average performance. 50th Percentile Overall : 235 Overall Percentile Score: Mathematics State / Union Territory 10th 25th 50th 75th 90th percentile percentile percentile percentile percentile Range 75-25 Range 90-10 A & N Islands Andhra Pradesh Arunachal Pradesh Bihar Chandigarh Chhattisgarh Dadra Nagar Haveli Daman & Diu Delhi Goa Gujarat Haryana Himachal Pradesh Jammu & Kashmir Jharkhand Karnataka Kerala Madhya Pradesh Maharashtra Manipur Meghalaya Mizoram Nagaland Odisha Puducherry Punjab Rajasthan Sikkim Tamil Nadu Tripura Uttar Pradesh Uttarakhand West Bengal 200 196 194 201 208 196 207 217 195 206 198 199 202 201 200 201 208 210 203 205 191 205 197 195 190 207 199 199 181 208 209 196 205 217 214 212 221 221 215 222 226 212 219 214 217 218 218 220 216 220 225 218 221 208 219 216 215 211 221 217 214 209 222 226 215 219 229 225 224 254 234 225 259 260 222 230 223 229 232 238 247 228 230 265 229 245 222 232 226 229 223 240 229 224 223 250 271 228 234 270 249 248 301 262 259 290 285 238 260 244 265 276 297 304 261 255 307 262 300 235 268 254 265 239 269 268 245 247 303 318 261 267 311 270 273 344 276 303 317 311 264 272 268 317 312 340 345 311 267 340 303 346 265 312 305 310 264 310 312 267 274 353 359 302 316 53 35 36 80 42 44 68 59 26 41 30 48 58 79 84 45 35 83 44 78 27 49 38 50 28 49 51 31 38 81 93 45 48 111 73 79 143 68 107 110 94 69 66 70 118 111 138 145 110 59 130 100 141 75 107 108 115 74 103 113 68 93 145 150 106 111 Overall Distribution 201 217 235 269 304 51 103 • Inter quartile range is highly variable. Lower than 50th percentile of Uttar Pradesh. Lower than 75th percentile of Uttar Pradesh. 50th and 75th percentile of Uttar Pradesh. Overall Score 251 (0.6) Table Overall Score Boys : 252 (0.7) Girls : 251 (0.7) Table Overall Score Rural : 252 (0.7) Girls : 249 (1.1) Table Science : Percentile Score • • • This graph shows the spread of achievement in each state. The 50th percentile shows the performance that half of the students in that state would be expected to achieve (the median). In science, Jharkhand has a very similar performance to the national average. 50th Percentile Overall : 243 Overall Percentile Score: Science State / Union Territory 10th 25th 50th 75th 90th percentile percentile percentile percentile percentile Range 75-25 Range 90-10 A & N Islands Andhra Pradesh Arunachal Pradesh Bihar Chandigarh Chhattisgarh Dadra Nagar Haveli Daman & Diu Delhi Goa Gujarat Haryana Himachal Pradesh Jammu & Kashmir Jharkhand Karnataka Kerala Madhya Pradesh Maharashtra Manipur Meghalaya Mizoram Nagaland Odisha Puducherry Punjab Rajasthan Sikkim Tamil Nadu Tripura Uttar Pradesh Uttarakhand West Bengal 208 193 189 182 208 198 222 227 189 213 200 196 202 196 189 193 211 190 202 199 195 205 200 198 185 202 193 218 180 205 184 190 211 226 214 213 210 225 219 243 247 212 230 220 219 223 218 215 214 230 217 222 220 214 222 218 223 206 223 217 234 209 226 216 214 227 254 232 232 232 243 236 275 275 230 261 239 238 246 239 242 232 262 245 242 243 226 240 234 250 228 243 237 261 232 257 247 233 249 288 260 266 272 270 268 311 311 261 298 273 274 275 295 281 265 286 298 273 296 248 273 267 284 256 272 273 281 268 301 307 269 279 331 282 306 314 294 305 335 347 287 330 301 322 313 339 323 302 317 343 307 346 274 322 315 323 278 310 322 313 296 346 352 304 318 62 46 54 61 45 49 68 64 49 68 53 56 52 77 67 51 56 81 51 76 33 51 49 61 49 49 56 47 59 75 91 55 52 122 89 118 132 85 108 113 120 98 117 101 126 111 143 133 109 106 153 104 147 80 117 115 124 93 108 129 95 116 141 168 114 107 Overall Distribution 199 221 243 279 316 58 116 • Inter quartile range is highly variable. Lower than 50th percentile of Kerala. Lower than 75th percentile of Kerala. 50th and 75th percentile of Kerala. Overall Score 251 (0.6) Table Overall Score Boys : 252 (0.7) Girls : 251 (0.7) Table Overall Score Rural : 252 (0.7) Girls : 249 (1.3) Table Social Science : Percentile Score • • • This graph shows the spread of achievement in each state. The 50th percentile shows the performance that half of the students in that state would be expected to achieve (the median). In social science, West Bengal has a very similar average performance to the national average but a narrower spread. 50th Percentile Overall : 242 Overall Percentile Score: Social Science State / Union Territory 10th percentile 25th percentile 50th percentile 75th percentile 90th percentile Range 75-25 Range 90-10 Andhra Pradesh Arunachal Pradesh Bihar Chandigarh Chhattisgarh Dadra Nagar Haveli Daman & Diu Delhi Goa Gujarat Haryana Himachal Pradesh Jammu & Kashmir Jharkhand Karnataka Kerala Madhya Pradesh Maharashtra Manipur Meghalaya Mizoram Nagaland Odisha Puducherry Punjab Rajasthan Sikkim Tamil Nadu Tripura Uttar Pradesh Uttarakhand West Bengal 188 187 183 210 201 200 221 192 205 196 193 201 184 187 197 210 196 195 186 184 190 195 188 180 211 196 205 175 192 185 189 198 210 209 213 226 220 223 248 214 224 213 217 222 204 218 220 230 223 220 209 201 208 214 209 200 232 218 226 201 218 218 215 219 227 228 243 248 241 264 274 231 252 230 239 247 226 249 239 260 265 248 229 221 228 231 229 222 259 245 246 226 254 266 235 240 255 263 285 269 272 307 311 262 279 264 273 270 272 299 270 281 307 275 270 241 265 267 261 243 277 286 270 255 298 318 270 272 275 308 324 292 305 329 339 282 309 293 319 305 312 337 311 308 343 308 313 272 305 308 285 269 307 334 300 278 336 355 304 308 45 54 71 44 52 84 63 48 54 51 56 48 69 81 50 51 85 56 61 39 57 53 51 44 45 68 45 53 80 100 55 54 86 121 141 82 104 129 118 90 104 97 125 105 128 150 114 98 147 113 127 88 116 113 97 89 95 138 96 103 144 170 114 110 Overall Distribution 194 217 242 275 309 58 114 • Inter quartile range is highly variable. Lower than 50th percentile of Uttar Pradesh. Lower than 75th percentile of Uttar Pradesh. 50th and 75th percentile of Uttar Pradesh. What Student Know and Can Do ? LANGUAGE Locate Information Overall, 54% students were able to respond correctly on items based on locate information, i.e.; simple retrieving of information from the given text. Some examples are : • • About two-third of students can identify that sports goods are mainly manufactured in Punjab. About 60% students can find out that the bark of neem tree is rough and scaly Grasp Ideas/Interpret • • • Overall students ability on Grasp Idea/Interpret is slightly lower on items tested as compared to cognitive process locate/retrieve information. Some are as under: More than half (58%) of the students can interpret the use of ‘dried leaves of neem’ in daily life Slightly higher than half (56%) of the students can grasp idea from the given advertisement that consumption of yellow peas dal is healthy and nutritious for human. What Student Know and Can Do ? LANGUAGE Infer/Evaluate • • • The variations of responses related to Infer/Evaluate on the given text was 26% to 46%. The following are the outcomes: Only 46% students could infer that why Baba Amte felt proud. Only 26% students could infer from the give story, that what qualities of Himmat Singh helped him to capture the thief. MATHEMATICS Number System • • • About 60% students can find out one’s digit of the square root of a four digit number. Nearly 44% students can find out cube root of a four digit number. Approximately 78% students are able to understand the concept of perfect square. What Student Know and Can Do ? MATHEMATICS Algebra • • • Approximately 58% students were able to find numeral co-efficient of monomial. Nearly 78% students were not able to solve linear equation in one variable. About 78% students could not be able to find the value of algebraic expression at any integer. Ratio and Proportion • • Only the item based on simple item could be responded by 31% students. Performance of students on items on ‘Ratio and Proportion’ was low, i.e.; 21% to 27%. Mensuration • • About 84% students were not able to calculate area of circular path. About 82% students were not able to calculate volume of a cylinder. What Student Know and Can Do ? MATHEMATICS Geometry • • About half of the students were able to calculate exterior angle of a quadrilateral by use of angle sum property. Nearly 77% students do not know the relationship between internal and external property of polygon. Data Handling • • Nearly half of the students can draw conclusion from given bar graph. Nearly 42% students can draw conclusion from data table. SCIENCE Biology • • • • About 65% students can reason out the effects of crop rotation and its impact on crop production. 38% students could recall the appropriate stage/time of vaccination. About 57% students can recall the name of the scientist who coined the term ‘Cell’. Nearly 43% students could identify the process of binary fission in Amoeba depicted in the diagram. What Student Know and Can Do ? SCIENCE Biology • About 41% students can reason out the consequences of loss of forests. Physics • • • • About 41% students know that why movable parts of a sewing machine are oiled. About 50% students could reason out the consequences of electric appliances handled with wet hands. About 43% students know the measuring unit of very long distance. More than 60% students could recall the biggest planet of solar system. Chemistry • • • • Nearly 54% students know that cotton is the best suited fibre to skin. More than 50% students could reason out why oxygen is essential for human life. About 52% students can recall exhaustible natural resources. About 44% students can reason out the process of chlorination of water. What Student Know and Can Do ? SOCIAL SCIENCE History • • • About 45% students know about the treatment meted out to Indians by British Official (General Dyer) in infamous episode of Jallianwala Bagh massacre. About 41% students can identify the extent of spread of the revolt of 1857 in the given map. About 48% students are able to explain the reasons of spread of European Trading Companies in India. Geography • • • • • About two-third students can identify the processes of cross production. About 56% students can identify various forms of irrigation in the given picture. Nearly 47% students know feature of economic activities. About 45% students know different rivers of India. About two-third students understand various industrial activities. What Student Know and Can Do ? SOCIAL SCIENCE Civics • • About two-third of the students know the minimum age of marriage for men and women as per the Constitution of India. Only 45% students know that Supreme Court is not a part of Parliament in India. Association of Learning Achievement with Intervening Variables Students’ Background Factors Behaviour problems such as absenteeism are associated with lower achievement but there might be other factors which contribute to lower achievement. • • • • The results suggest that coming from a larger family is associated with lower than average performance. Students with only one sibling outperform by a small, but statistically significant amount, those who have two or more brothers and sisters. The higher the parents education the more likely a student is to do well. Achievement has been found to be associated with language spoken at home is the same as the language of instruction or whether the school is in an urban or rural environment. Association of Learning Achievement with Intervening Variables contd... School Factors There are several factors associated with the school that seem to be associated with achievement. • • • • • Government aided schools have lower performance, even when the key factors such as socio-economic status, location etc. are taken into account. Schools with pucca buildings tend to do better. Co-ed schools are doing better than single sex (boys/girls) schools. Students who go to schools which who have a 6 day week do better than those at schools which have a 5 day week. Schools which have more than 180 working days a year tend to achieve better. School leadership and governance School leadership and governance seems to have a key role in achievement. • • • • Students at schools where the head teachers also teach classes tend to do better. Students at schools which have been inspected tend to do better. Students at schools who reported behaviour problems, such as late arrivals, absenteeism, skipping classes or violations of school rules, tend to do worse. Having VEC/AEC/SMC committees is associated with better performance. Association of Learning Achievement with Intervening Variables contd... The pupils’ perspective The pupils perspective is very important and anything that affects a students’ ability or desire to attend school regularly and pay more attention in various curricular activities could impact on achievement. • Incentive schemes such as offering a midday meal, free uniforms and scholarships appear to be associated with higher achievement. Teaching and learning approaches The teacher is key in mediating learning and the study looked at various factors around the teacher and their teaching practices. • • Students who had there homework checked by their teacher every day tend do better. Students who reported that they work with other students in small groups in solving mathematical problems tended to perform significantly better in mathematics. Reading Comprehension Mathematics Locate Information The bark of neem tree is 1. 2. 3. 4. Knowledge The one's digit of the square root of 2025 is : smooth and green. rough and scaly. soft and scaly. hard and green. 1. 2. 3. 4. 1 4 5 6 Percent Correct: 60.1% Percent Correct: 60.8% Application Infer/Evaluate Neem is called a “wonder tree” because 1. 2. 3. 4. its twigs are used for brushing teeth. it is a useful fertilizer. it protects newly born babies. each part of the tree is useful. Percent Correct: 31.5% Three exterior angles of a quadrilateral are 70o, 80o and 100o. The fourth exterior angle is : 1. 2. 3. 4. 70o 80o 100o 110o Percent Correct: 52.7% Science Social Science Knowledge Knowledge Chlorination is the process to purify water. It is done to 1. 2. 3. 4. remove harmful gases. separate suspended impurities. kill harmful germs. change the colour of water. What attracted European trading companies to India? 1. 2. 3. 4. Percent Correct: 43.6% Percent Correct: 47.8% Reasoning Application Oxygen exists in nature in the form of a gas. It is essential for our life because it 1. 2. 3. 4. enhances the growth of plants. is used as an antiseptic. is used for purifying water. is used during respiration. Percent Correct: 50.8% Gold and Silver Cotton, Silk and Spice Flora and Fauna Horses and Cattle A powerful caste decided to teach Rathnam a lesson. They set his hut on fire. He escaped and filed a case in a local police station. Which of the following acts rescues him? 1. 2. 3. 4. Sedition Act 1870 Civil Rights Act 1964 Prevention of Atrocities Act 1989 Minimum Wages Act 1948 Percent Correct: 33% NAS Class III Overall Score 252 (0.4) Overall Score 252 (0.4) Overall Score Boys : 251 (0.5) Girls : 253 (0.5) Overall Score Boys : 252 (0.5) Girls : 252 (0.5) Overall Score Rural : 252 (0.5) Urban: 256 (1.2) Overall Score Rural : 251 (0.5) Urban: 254 (1.3) Performance of Students in Class III 85.5 90.0 80.0 70.0 Language 65.4 58.6 60.0 50.0 40.0 30.0 20.0 10.0 0.0 Listening Recognition Comprehension 90.0 76.2 80.0 70.0 66.1 62.9 77.0 69.4 65.6 60.0 50.0 40.0 30.0 20.0 10.0 Mathematics 0.0 Geometry Number Measurement Money Data handling Patterns LANGUAGE ITEMS Look at the pictures and recognise the correct name of the picture. Then encircle the correct answer. 1. Ball 2. Doll Percent Correct: 89% 1. Table 2. Chair Percent Correct: 85.3% LANGUAGE ITEMS 19-24. Read the following passage and encircle the answers of the questions. You must have seen butterflies. Do you know where a butterfly comes from? The mother butterfly lays an egg on a leaf or plant. A small caterpillar comes out of the egg. The caterpillar eats leaves and grows bigger. Then the caterpillar attaches itself to a leaf and makes a large cocoon. This is a kind of shell that protects it from other animals. Inside the cocoon it grows wings and legs. Finally, the cocoon opens and the new beautiful butterfly comes out. It slowly opens its wings and then it flies away. What happens inside a cocoon? 1. Caterpillar changes into a butterfly. 2. Caterpillar keeps eating leaves. 3. Caterpillar grows big. Percent Correct: 45.2% MATHEMATICS ITEMS Money Which of the given three notes shows the correct amount of money to buy the toy? Rs. 40 Percent Correct: 80.2% MATHEMATICS ITEMS Number 37 boys and 26 girls are studying in a school. What is the total number of children studying in the school ? 53 63 513 Percent Correct: 74.0% MATHEMATICS ITEMS Geometry Which of the following shapes is not shown in the figure below? Rectangle Triangle Circle Percent Correct: 66.9% Way forward • Review Committee NAS Class V suggested that 3 year cycle per class is an appropriate timeframe to derive maximum benefit from the survey findings. • Efforts have been made to reduce the time period to complete the NAS. • Steps have been taken to make variables more understandable to various stake holders by developing subject wise reports. • Planning to take steps for more in-depth analysis to see the influence of various variables related to students’ background, school environment and some teachers’ related variables. Thank you For your time and attention