1
Education-line document submission
Author: Chraska, Miroslav
Title: Evaluation of noncognitive areas in education
Paper presented at the European Conference on Educational Research, Edinburgh,
20-23 September 2000
Abstract:
We can evaluate education outputs (with certain simplification) with help three
principal indicators: pupil´s knowledge, pupil´s skills (practical even intellectual) and
attitudes to learn and education.
Evaluation of outputs of education in Czech republic at present is based especially
on assessment results in cognitive area, otherwise on assessment of quantity and
quality of acquire knowledge and skills. However, that evaluation will not sufficient in
future. Owing to living conditions, witch are changing very quickly, school can not
give to pupils all knowledge and skills, witch pupils will need during their life. It is
desirable, in order to school would educate individuals, which are able and which are
helpful adapt to quickly changing living conditions, individuals, witch will be able to
learn, obtain new knowledge, skills or new qualifications. In that context will be more
important, so that school would form positive attitude to learning, education, school,
particular subject etc. (we talk about attitudes to educational reality). We developed in
Faculty of education Palacky University in Olomouc (on the basis of semantic
differential) special tool ATER (questionnaire for measurement of pupil,s attitudes to
educational reality) for measurement that important effects of education.
Document type and origin:
presented at a European Conference on Educational Research, University of
Edinburgh, 20-23 September, 2000, Scotland
Suggestied key terms: education outputs, attitudes to educational reality ,semantic
differential, factor analysis
Version Number: first
2
EVALUATION OF NONCOGNITIVE AREAS IN EDUCATION
Miroslav Chraska
Palacky University Olomouc, Czech Republic
Introduction
We can evaluate educational outputs (with certain simplifications) with the help
of three principal indicators: pupils’ knowledge, pupils’ skills (practical and
intellectual) and attitudes to learning and education.
Evaluation of the outputs of education in the Czech Republic at present is
based especially on assessment results in the cognitive area, otherwise on the
assessment of quantity and quality of acquired knowledge and skills. However, that
evaluation will not be sufficient in the future. Owing to living conditions which are
changing very quickly, schools cannot give pupils all the knowledge and skills which
they will need during their lives. It is desirable for schools to educate individuals who
are able to adapt to rapidly-changing living conditions, individuals who will be able to
learn and obtain new knowledge, skills and qualifications. In that context it will be
more important for schools to form positive attitudes to learning, education, school, a
particular subject etc. (We are talking about attitudes to educational reality). We have
developed, in the Faculty of Education of Palacky University in Olomouc (on the
basis of semantic differential) a special tool known as ATER (a questionnaire for
measurement of pupil's attitudes to educational reality) to measure these important
effects of education.
Attitudes to educational reality
We define the attitudes to educational reality as pupils' or students' inclination
to react in a stable way to individual aspects and situations of educational reality and
towards oneself (comp. Hartl, 1996).
The measurement of pupils' attitudes or students to educational reality is
rather hard because we are dealing with a very tenuous variable which is hard to
cover. Traditional scales of attitudes are not quite suitable for routine diagnostics in
practice at schools and, what is more, they provide us with information where
attitudes are assessed only in one dimension (more or less expressing assessment
of the particular element of educational reality by the person researched). On the
3
basis of the research carried out we argue that it is possible to measure attitudes
towards educational reality rather precisely by means of tools created on the principle
of semantic differential.
Qualities of measurement by means of an ATER questionnaire
Measurements' credibility through semantic differential more or less depends
on the quality of the scales used for the measurements. We carried out research in
1998 - 1999 in which we appointed ourselves three essential goals:

to verify whether the suggested scales of the ATER questionnaire have adequate
construct validity,

to verify whether measurement by the ATER questionnaire is sufficiently reliable
and accurate,

to propose an acceptable method for the presentation and interpretation of the
results attained.
a) Construct validity of scales of the ATER questionnaire
We verified the validity of scales by means of a sample of 523 fifteen-year-old
pupils, who were in their last year of compulsory school attendance. The research
sample included 205 pupils who were finishing their compulsory school attendance in
the ninth class of ”basic school” (a school for the 6-15 age group) and 218 pupils of
the same age who were finishing their compulsory school attendance at a
”gymnazium”, which more or less corresponds to the traditional grammar school.
The pupils were presented with 10 term indicators (Table 1) with 10 rating
scales evaluating from one to seven. Table 2 shows an example of the term
indicator HOME STUDY and the scales which was used.
Table 1: Term indicators for measurement of attitudes to educational reality
HOME STUDY
OUR SCHOOL
STUDY IN SECONDARY SCHOOL
LEARNING
POPULAR TEACHER
UNPOPULAR TEACHER
SCHOOL REPORT
EDUCATION
EXAMINATION
4
MARKING
Table 2: Recording sheet for the ATER questionnaire
HOME STUDY
1. good
bad
2. demanding
undemanding
3. unpleasant
pleasant
4. light
dark
5. strict
lenient
6. simple
complicated
7. beautiful
ugly
8. problematic
easy going
9. sweet
sour
10. difficult
easy
For the analysis of the quality of the designed scales the results of all
respondents in all term indicators were used, i.e. 5230 ways of usage were analysed
altogether. First the correlation among the results in individual scales was calculated
and then these results were subjected to traditional factor analysis. We discovered
that the results of the measurements are influenced the most by the first two
common factors, which together clarify nearly 70% of the discovered variability. With
regard to this result of the factor analysis we decided to extract only two common
factors. Table 3 shows rotating factor matrix (rotation VARIMAX) with factor loadings
(factor weights) and communalities of individual scales.
The first factor has dominant factor loadings in the scales DEMANDING –
undemanding, STRICT – lenient, DIFFICULT – easy and PROBLEMATIC – easy
going (dominant factor loadings are showed in table 3 in bold). These scales imply
to what extent the respondents understand the term indicators as ”something”
connected with labour, difficulties, changes or activity. That is why we indicate this
factor as energy factor.
The scales GOOD – bad, PLEASANT - unpleasant, LIGHT – dark,
BEAUTIFUL – ugly, SWEET – sour express to what extent the respondents positively
5
evaluate the given term indicator (to what extent it is ”good” or ”bad” for them). Factor
delimited by the stated scales is referred to, in accordance with C. Osgood, as
assessment factor.
Table 3: Rotating factor matrix
Scale
number
1
2
3
4
5
6
7
8
9
10
Scale
good - bad
demanding – undemanding
pleasant – unpleasant
light – dark
strict – lenient
complicated - simple
beautiful - ugly
problematic – easy going
sweet - sour
difficult - easy
Energy
factor
0,37296
0,83483
0,36803
0,18782
0,77844
0,83082
0,25093
0,64433
0,21974
0,78108
Assessment Communality
factor
h2
0,74131
0,68864
0,14148
0,71696
0,74483
0,69022
0,79144
0,66165
0,27109
0,67947
0,30029
0,78043
0,83103
0,75358
0,37107
0,55285
0,76247
0,62964
0,32978
0,71884
The factor loadings of the scales are represented in picture 1 as dots in a
two-dimensional graph. It is clear from the picture that 5 of the scales have dominant
factor loadings in the energy factor while the remaining 5 scales have dominant
loadings in the evaluation factor.
By means of confirmative factor analysis we verified the hypothesis that it is
possible to describe pupils' attitudes to educational reality through two common
factors between which some correlation does exist (we talk about so-called oblique
factor rotation). The probability of this hypothesis comes out very high (P=0,98) and
the average residual correlation is very low (0,029).
b) Reliability of measurement by means of the ATER questionnaire
We were looking for an answer to the important question of to what extent the
results obtained are influenced by accidental factors and consequently what the
reliability of our measurement is. We evaluated reliability by means of methods
based on the analysis of variance. All the models of calculations used get very
similar results. It shows a rather high rate of reliability and accuracy of performed
measurement (calculated coefficients of reliability attain universally higher values
than 0,85).
c) Presentation and interpretation of attained results of measurement
In order to be able to express pupils’ attitudes to the whole of their
educational reality, we defined for both factors summary indexes.
6
Assessment of attitudes to educational reality index I assAER indicate, to what
extent the pupils perceive term indicators as "good" or "bad" for them.
I assAER 
1 n
 assi
n i 1
Energy of attitudes to educational reality index I enAER indicate to what extent
the pupils perceive term indicators as "something" connected with labour and energy.
I enAER 
1 n
 eni
n i 1
In the above formulae n is the number of term indicators, assi are assessment
factors for particular term indicators and eni are energy factors for particular term
indicators.
Some results obtained by means of the ATER questionnaire
By means of the ATER questionnaire we obtained an amount of important and
interesting data about the examined population of fifteen-year-old pupils.
For example, one interesting finding is that significant differences exist between
attitudes to educational reality at pupils in the ninth class of ”basic school” and
students in the “gymnazium”.
If we compare the results obtained in the ”basic school” and in the
”gymnazium”, we find out that there is higher assessment factor at almost term
indicators in the basic school (differences are statistically significant). This result can
be interpreted that pupils in ”basic school” (who are finishing their obligatory school
attendance) have more positive attitudes to school and all educational reality than
students of the same age in grammar school. If we compare values of energy factor
in the ”basic school” and in the ”gymnazium” we find out that are higher energy
factors in the latter. Picture 2 and 3 display the results of our research in a graphic
way.
While we expected such results in the energy factor, results in the assessment
factor are very surprising. We cannot overestimate the results of our research (mainly
owing to our small sample size) but we can raise some alarming questions e.g. "does
the Czech ”gymnazium” create suitable conditions for the development of pupils at
present?", "What is the real level of educational work in this type of school " ? etc.
7
8
Conclusion
On the basis of our research we came to the conclusion that the
measurement of attitudes to educational reality through semantic differential in the
form in which it is described by C. Osgood (e.g. Osgood at all., 1957) and by other
authors does not suit the sociocultural conditions in Czech schools and provides us
with a rather distorted picture of reality.
The differences between Osgood's findings and ours are mainly in the
interpretation of the discovered common factors. We came to the conclusion that it is
reasonable and profitable to extract only two common factors. The tool which
measures pupils' attitudes to educational reality by means of two common factors is
what we describe as the ATER questionnaire.
Bibliography
BARCHARD, K. A. - HAKSTIAN, A. R. The Effects of Sampling Model on Inference
With Coefficient Alpha. Educational and Psychological Measurement, roc. 57, 1997,
c.6, , s. 893 - 905.
BINTIG, A.: The Efficiency of Various Estimations of Reliability of Rating Scales.
Educational and Psychological Measurement, roc. 40, 1981, c.3, , s. 619 - 642.
CRONBACH, L. J.: Coefficient alpha and internal structure of tests. Psychometrika,
16, 1951, s. 297 - 334.
HARTL, P. Psychologicky slovnik. Praha: Nakladatelstvi Tercie, 1996.
CHRASKA, M. Jake jsou postoje zaku a studentu ke skole a edukacni realite ?
Pedagogika, 48, 1998, c. 1, s. 54 - 66.
CHRASKA, M. Postoje studentu ucitelstvi a ucitelu v praxi k ucitelske profesi.
(Zaverecna zprava vnitrniho grantu Univerzity Palackeho.) Olomouc: Pedagogicka
fakulta UP, 1996.
CHRASKA, M. Odhady reliability mereni semantickym diferencialem. In. Sbornik
prispevku z 6. konference Ceske asociace pedagogickeho vyzkumu. Ceske
Budejovice: Pedagogicka fakulta Jihoceske univerzity, 1998, s. 46 – 53.
KERLINGER, F. N. Zaklady vyzkumu chovani. Praha: Academia 1972.
MARES, J. Jak zjisťovat reliabilitu pozorovani ? Pedagogika, 33, 1983, c. 2, s. 169 189.
McDONALD, R .P. Faktorova analyza a pribuzne metody v psychologii. Praha:
Academia, 1991.
OSGOOD, C., SUCI, G., TANNENBAUM, P. The Measurement of Meaning. Urbana,
III, University of Illinois Press, 1957.
9
PAPICA, J. Zaklady psychometrie. Olomouc: Filozoficka fakulta Univerzity
Palackeho, 1984.
PRUCHA, J. Moderni pedagogika. Veda o edukacnich procesech. Praha: Portal,
1997.
Contact address:
Assoc Prof Miroslav Chraska
Palacky University in Olomouc
Czech Republic
phone 42068/5635154
E-mail: capv@risc.upol.cz