Achievement in Relation to
Background Factors
Background Data Gathering

During every cycle of assessment, background data are
gathered by questionnaire and analysis of fieldworkers’
reports.

Questionnaires are given to students (including a section
for parents), head teachers and teachers.

Conditions, facilities etc. in schools are observed and
recorded by test administrators.

Background data are correlated with learning
achievement to identify needs for improvement and
factors associated with achievement.
2
Background Data Gathering Framework

Background data are classified into five areas following
the Craig-Heneveld Framework:
•
•
•
•
•

The Students
Supporting Inputs from Outside the School
Teaching-Learning Process
Enabling Conditions
School Climate
Results are presented in the next section
3
The Students
Data are gathered from students about:

Socio-economic background (e.g., family size,
father’s/mother’s education and occupation)

Factors in the home environment related to achievement
(e.g., help w/ homework, space for reading, need to work
outside)

Attitudes towards education
4
For Example:
Do you work outside the home to earn money?
Scaled Scores
Variable %
Yes
No
23
77
Difference
Maths
Urdu
Science
Social Studies
383
403
370
389
455
473
481
503
Significant
Significant
Significant
Significant
Conclusion: about 23% of Grade 4 students reported having
to work outside the home. Their achievement is significantly
lower than others’.
5
Supporting Inputs from Outside
Information is gathered from different sources about:

Parent and community support (e.g., frequent schoolparent meetings, financial and other support, parental
involvement in homework)

Effective support from the education system (e.g.,
supervision, availability of teaching materials and
supplementary materials)

Adequate material support (e.g., facilities, furniture in
school, school budget for supplies)
6
For Example (1):
How many of the following six activities in the school
does the head-teacher invite parents to participate in?






Attend special events (e.g., functions, sports events)
Raise funds for the school
Participate in development planning
Volunteer for school projects, programs, and trips
Ensure that their child completes his/her homework
Serve on school committees (e.g. personnel, finance)
7
For Example (2):
Number of activities:
Activities
Maths
Language
Low (0-2 Activities)
360
373
High (3-6 Activities)
385
401
Significant
Significant
Difference
Conclusion: most head teachers reported inviting parents
to participate in a significant number of school-related
activities. This participation is positively associated with
student achievement.
8
Teaching - Learning Process
Information is gathered from different sources about:

Learning time (no. of teaching days, length of period,
time on task)

Variety of teaching strategies used in class (e.g.,
questioning techniques, investigation, practical work)

Homework (amount given, how it is assessed and what
feedback is given)

Student assessment and what feedback on their work
students receive
9
For Example:
How often does your teacher check your Maths homework?
Responses
Maths
Twice or More Weekly
401
Once a Week or Less
347
Difference
Significant
Conclusion: the great majority (95%) of students reported
that they are taught by teachers who check their
mathematics homework at least twice weekly. These
students achieve significantly better than students who
report their homework is checked less frequently.
10
Enabling Conditions
Information is gathered from different sources about:

Leadership (e.g., whether the head-teacher is primarily
an administrative or an instructional leader in the school)

Capable teaching force (e.g., teacher education,
qualifications, experience, style)

Flexibility and autonomy (e.g., decisions and discretion
of head teacher, teacher)

High level of time in school (starting/finishing time of
periods, presence/absence of teachers)
11
For Example (1):






How many of the following six indicators of academic
leadership does the head teacher have explicit policy
for?
Corporal punishment
Homework
Recording and reporting student progress
Explaining students’ mistakes
Dealing with weak students
Parental involvement
12
For Example (2):
Responses
Maths
Language
None
393
375
One
388
370
Two or More
404
403
Significant
Significant
Difference
Conclusion: about two-thirds (63%) of all students are in
schools with no explicit policy for academic matters (i.e.,
most head teachers see themselves as administrative
rather than instructional leaders). Policy for academic
matters is associated with student achievement.
13
School Climate
Information is gathered from different sources about:
 High expectations for students (school expects and
rewards success)

Teacher Attitudes (attitudes to students and school
goals, perception of school climate)

Order and Discipline (e.g., punishment policy and
methods, feeling safe at school)

Organized Curriculum available to all

Classroom Climate (e.g., student participation, teacherstudent interaction)
14
For Example (1):
Students’ Perception of School Climate

I like being in school

I think that students in my school try to do their best

I think that teachers in my school care about the
students

I think that teachers in my school want students to do
their best
15
For Example (2):
Responses
Maths
Language
Yes to all four
401
387
Yes to 3 or less
372
368
Significant
Significant
Difference
Conclusion: an overwhelming majority (92%) of students
responded “yes” to all four statements. This was
associated with higher achievement in both language and
mathematics
16
Use of Blackboard and Achievement
%
Maths
Urdu
Science
S. Studies
Yes
No
97
03
399
350
384
341
469
431
497
457
Difference


Scaled Scores
Use of
Blackboard
Significant Significant Significant Significant
Teachers’ use of black board significantly increased
students’ achievement in all the four subjects
Ninety-seven percent teachers reported they daily use
black boards
17
Use of Textbooks and Additional Resources

Majority of the subject
100
teachers used textbooks as 90
major source of knowledge. 80
Urdu Teacher
Maths Teacher
Science Teacher
Social Studies
Teacher

Around 10% teachers
reported consulting
additional resources for
teaching
Percent
70
60
50
40
30
20

Science teachers were
more likely to consult
additional resources than
other subject teachers
10
0
Textbook
Additional
sources
No use of
textbook
Back to Slide 15
18
Multi-grade Teaching and
Achievement
Multi-grade
Teaching
%
Students’ Scaled Scores
Maths
Urdu
Yes
No
21
78
442
466
Difference


Significant
495
516
Significant
Teachers’ Scaled Scores
Maths
Urdu
616
644
644
661
Significant
Significant
Lower frequency of Multi-grade Teaching is linked with increased
performance for both students and teachers
Twenty- one percent teachers reported they have to teach more than on
classes in one period
19
Rewards and Punishment in School
Scaled Scores
Reward
Yes
No
%
85
14
Difference
Punishment
Frequently
Occasionally
Never
Difference



Maths
Urdu
Science
S. Studies
403
372
390
354
474
440
504
464
Significant
Significant
Significant
Significant
351
388
405
345
376
391
444
458
476
452
490
505
Significant
Significant
Significant
Significant
%
.03
38
58
Student who Never got punished by their teachers performed significantly
better Fifty-eight percent students reported they never got punished.
Only .03 % students reported frequent punishment
Rewarding students’ performance increased achievement all the subjects
20
PTA/ SMC in School: Head Teachers’ Report
•Presence, need, funding,
or role of PTA/SMC did not
increase students
performance in any of the
NEAS tested school
•Only 39% head teachers
reported getting funds
from Govt.
•Ninety-seven percent
head teachers reported
need for SMCs in Schools
Percent
• Around 80% head
teachers reported
presence of PTA/ SMCs’ in
School
100
90
80
70
60
50
40
30
20
10
0
Yes
No
PTA/SMC
School
Need For
PTA/SMC in
Schools
Funding of
PTA/SMC
from Govt.
Copperative
Role of
PTA/SMC
21
Teaching Resources and Achievement
Scaled Scores
Teaching Resources
%
Maths
Urdu
Science
Social Studies
Yes
No
Difference
42
56
395
392
n. s
384
380
n. s
476
459
n. s
382
397
n.s
Yes
No
Difference
44
56
396
393
n. s
369
390
Significant
465
467
n. s
483
503
n.s
Yes
No
Difference
65
34
394
395
n. s
372
400
Significant
465
469
n. s
485
513
Significant
Yes
No
Difference
90
.09
396
373
n. s
380
389
n. s
468
448
n. s
496
495
n. s
Yes
No
Difference
58
41
397
396
n. s
382
379
n. s
468
465
n. s
477
483
n. s
Curriculum Document
Teaching Kit
Teachers Guide
Textbooks
AV Aids


Only 46% Head teachers reported they have the curriculum document in their school
Only 37% teachers reported using the curriculum document; use of curriculum document and other
teaching resources did not significantly increase students achievement
22
Teachers General Education in the
NEAS Sample
General Education
Rural %
Urban %
Female
Male
Female
Male
Matriculation
47
23
33
23
Intermediate
25
23
34
18
Bachelors in Arts
18
38
23
40
Bachelors in Science
.07
16
10
20
Masters
.02
-
-
-
-
-
-
-
Higher than Masters



Male teachers were better qualified than female teachers in both rural and
urban areas
There were more matriculate female than male teachers
In the NEAS selected schools there were no teachers with higher than
masters qualification
23
Head Teachers’ and Teachers’
Professional Education

About 57% teachers
were PTC qualified
60
50
There were more M. Ed
head teachers than
teachers
40
Percent

PTC
CT
B.Ed
M.Ed
Higher
than M.Ed
30
20

There were less than
1% B. Ed Teachers
and head teachers
10
0
Teachers
Head Teachers
24
Teachers Professional Training

60
Less than 50%
subject teachers
secured any kind of
training
50
Subject Content
40
Methodology

Subject curriculum
and problem solving
were highest
reported areas of
training obtained
Percent
Subject Curriculum
Analytic thinking
Problem solving
30
20
10
0
Maths
Urdu
Science
Social
Studies
25
Teachers’ Scaled Scores
Subject Teacher
Teachers
Students
Maths
620
404
Urdu
564
382
Difference

Difference
Highly
Significant
Highly Significant
Science
660
467
Social Studies
784
496
Difference

Scaled Mean
Highly
Significant
Highly Significant
Teachers scaled scores were significantly higher than students in all
four subjects
Teachers scores were well above the set mean of 500 (SD 100)
26
Assessment of Students’ Performance
in School
Method of Assessment
Scaled Scores
Urdu
Maths
Science
Social Studies
Yes
No
Diff.
398
380
n.s
416
426
n.s
476
479
n.s
513
510
n.s
Yes
No
Diff.
395
452
Significant
419
390
Significant
480
463
Significant
513
531
Significant
Yes
No
Diff.
400
335
Significant
419
382
Significant
481
418
Significant
518
460
Significant
Yes
No
Diff.
399
372
Significant
417
406
n.s
478
466
n.s
514
502
n.s
Verbal
Written
Home work
Class Performance
27
Parents Educational Profiles in the NEAS
Sample



Urban fathers were
relatively more educated
than rural fathers
Only .02% urban fathers
were graduates
Seventy-two percent rural
and 53 % urban mothers
were illiterate
Education
There were no graduate
or postgraduate mothers
in NEAS sample
Mother %
Rural
Urban
Rural
Urban
Illiterate
32
25
72
53
Primary
29
27
17
26
Matriculates
17
23
.04
13
Intermediate
04
.08
.09
.04
-
.02
-
-
01
.01
-
-
Bachelors

Father %
Masters
28
Fathers’ Education and Student
Achievement

The table shows a
trend that children of
fathers with postprimary education
performed better on
different subjects
Scaled Scores
Scaled Scores
Education
Maths
Urdu
Science S. Studies
Illiterate
389
376
462
491
Primary
399
385
469
497
Matriculates
408
394
476
506
Intermediate
391
407
472
512
Bachelors
431
414
493
569
Masters
422
399
474
536
Higher than
Masters
398
357
488
478
29
Fathers’ Occupation in the NEAS
Sample



Majority of rural fathers
were agriculture
landowners
Majority of urban fathers
were small business
owners and skilled
workers
Fifteen percent urban
and twelve percent rural
fathers in govt. jobs
Percent
Occupation
Rural
Urban
Agricultural Wage Earner
.06
.03
Agriculture Land Owner
35
.08
Skilled Workers
13
23
Small Business Owner
17
30
Government Job
12
15
Private Job
03
.07
30
Fathers’ Occupation and Students
Achievement
Fathers’ occupation




Scaled Scores
Maths
Urdu
Science
S. Studies
Agriculture wage earner
408
367
474
507
Agriculture land owner
392
375
460
491
Skilled workers
393
389
472
498
Small business owner
401
397
477
510
Government job
404
402
473
513
Private job
384
374
471
476
Fathers’ occupation did not significantly affect students achievement on Maths
and Science tests
Children of Agricultural wage earner however, scored significantly lower as
compared to govt. servants on language test
Children of private employed parents scored significantly lower on social
studies test
All other mean differences were non-significant
31
Strengthening Links between NEAS
and the Educational Planning Process

NEAS can support educational planning by:
• providing data about the actual conditions in schools; and
• indicating the possible impact of policy actions on student
achievement.

Educational planners can make NEAS more effective by:
• proposing topics to be added to background data gathering; and
• suggesting relationships to be explored.
32
Conclusion and Next Steps
33
The presentation in summary:

NEAS has assessed the achievement of Class IV
students in 4 subjects: Urdu, Mathematics, Science and
Social studies.

The assessment reveals large differences in average
performance among provinces and between students
from differing backgrounds.

These differences in average performance can be traced
back to differences in student and family characteristics,
school processes and inputs.

As illustrated in the next slide the factors that influence
performance can be divided into four types:
34
Additional work:

The lack of performance standards limits the
interpretability of the assessment results.

Additional analysis is needed to identify causal
relationships and hence where policy might yield
improvement.

The next slides illustrate how multi-level multivariate
analyses can be used to reflect on the impact of different
policy interventions.
35
Expected changes in achievement
over time:

All other things being equal one would not expect to see
large changes in achievement levels from year to year.
Evidence suggests that education reforms take at least 5
to 10 years to bear fruit.

For Pakistan one would actually expect test scores to fall
over time as rising enrollment rates draw more children
from relatively disadvantaged backgrounds into the
system.
36
Future Vision of National Assessment
in Pakistan
Proposed Recommendations by National Select Committee:
•
NEAS future planning should be till 2020 at Grade 4, 5, 8, 10
and 12 levels
•
Private schools should be included in NEAS sample on Pilot
basis in 2008 National Assessment
•
Listening and speaking competencies (skills) of student
assessment would be part of NEAS future activities
•
Pakistan can explore the possibility to participate in the next
TIMSS in 2012
37
Thank You…
38