Research Journal of Applied Sciences, Engineering and Technology 6(19): 3520-3526, 2013
ISSN: 2040-7459; e-ISSN: 2040-7467
© Maxwell Scientific Organization, 2013
Submitted: October 17, 2012
Accepted: January 11, 2013
Published: October 20, 2013
Insights into Engineering Education Learning Outcome’s Assessment with Rasch Model
1
Azrilah Abdul Aziz, 1Mohd Saidfudin Masodi, 2Faridah Mohamed Ibrahim, 1Mohd Zaidi Omar and
1
Azami Zaharim
1
Centre for Engineering Education Research, Faculty of Engineering and Built Environment, Universiti
Kebangsaan Malaysia, 43600 Bangi, Malaysia
2
Centre for Foundation Studies, University of Malaya, 50603 Kuala Lumpur, Malaysia
Abstract: Education is a process in which people learn. Knowledge, skill or ability is gained during the learning
process to better equipped students on career opportunities. Learning process takes place in academic institutions in
which programs are developed for the various fields. Accreditation to these programs requires the academic
institutions to adopt qualification framework. The Engineering Accreditation Council of Malaysia (EAC) adopts the
American Accreditation Board of Engineering and Technology 2000 (ABET) requirements which promote Outcome
Based Education (OBE) learning process. OBE calls for the evaluation of the subjects Learning Outcomes (LO) as
specified in the Program Specification. Evaluation method has been largely dependent on students’ performance
carrying out tasks in tests, quizzes or submission of assignments. The evaluation on the students’ performance
output gives an indication on the students’ achievement of the subject’s LO. However, measurements of the
students’ achievement from the observed outcomes remain vague when raw scores are the only means of
assessment. Raw scores may not provide true conception on difficulty of tests and or the true ability of students.
This article described the measurement and analysis of the Engineering Mathematics LO’s achievement using Rasch
Measurement Model. It gives a more meaningful analysis on the students’ achievement.
Keywords: Accreditation assessment, engineering mathematics, learning outcome, Rasch measurement model,
OBE measurement.
INTRODUCTION
Learning is a complex process where it involves
cognitive, emotional and environmental influences and
experiences in acquiring knowledge or skill (Illeris,
2004; Ormrod, 1995). Assessment should reflect these
understandings by employing a diverse array of
methods, including those that call for
actual
performance, using them over time so as to reveal
change, growth, and increasing degrees of integration.
Such an approach aims for a more complete and
accurate picture of learning, and therefore firmer bases
for improving our students' educational experience
(Astin et al., 1991).
A good assessment recognizes the value of
information for the process of improvement (Deming,
2000; Mok and Wright, 2004; Wright, 1997).
Assessment approaches should produce evidence that
relevant parties will find credible, suggestive, and
applicable to decisions that need to be made. The point
of assessment is not to gather data and return "results";
it is a process that starts with the questions that involve
data gathering and subsequent analysis. It is of utmost
importance on the onset that this fundamental of
measurement must be clearly understood. Analysis
must be based on valid data and duly interpreted to
generate a reliable report with meaningful information
for prudent decision making towards continuous
improvement of teaching and learning. Rasch analysis
provides more meaningful analysis enabling better
inference be made. Rasch measurement model will be
used in this study to assess students’ achievement on
basic mathematics for engineering. The findings will
also lead to students’ classification based on their
ability-achievement hence better management in
improving their achievement towards meeting the
targeted learning outcomes (Azrilah et al., 2007).
METHOD OF ASSESSMENT
This paper explores the Learning Outcome (LO) on
an engineering mathematical subject which exposes
students to the basic concept of mathematical
engineering. The subject is expected to be able to
balance between practical applications of mathematical
equations and in-depth understanding of real situation
(Wright, 1989). The LO is the specification of what
students should learn as a result, after a specified period
Corresponding Author: Azrilah Abdul Aziz, Centre for Engineering Education Research, Faculty of Engineering and Built
Environment, Universiti Kebangsaan Malaysia, 43600 Bangi, Malaysia
3520
Res. J. Appl. Sci. Eng. Technol., 6(19): 3520-3526, 2013
Table 1: Example of students’ performance report
Students
LC008
LP001
LC007
LC003
LC005
LC004
LM002
LC006
Final test
90
88
88
85
82
77
73
73
Quiz 1
90
100
100
90
100
100
100
100
Quiz 2
100
92
100
92
100
92
75
100
Quiz 3
40
35
35
50
70
40
60
40
Quiz 4
80
80
60
80
60
60
50
70
Quiz 5
45
75
70
75
80
75
75
0
Laporan
100
82
95
76
50
71
56
64
Excel
90
90
80
90
83
80
67
83
Maths
70
75
75
80
70
70
75
70
Overall
total
87
85
85
83
78
75
71
71
Score
of time in the program. Apart from the technical
Legend
knowledge, students are introduced on report writing
1
and logical thinking. The data include several tests and
45
quizzes taken during the first semester of their
40
engineering program. Apart from the test results, the
35
assessment also includes assessment on report writing,
students’ skills on Microsoft Excel and mathematical
30
knowledge.
20
Common practice of assessment would take
15
percentages of each assessment components, including
the final examination for the semester. The summation
10
of the percentages will be the overall marks indicating
Personal estimate
5
students’ performance. Students’ with high marks are of
good performance compared to those with low marks.
-4 -3 -2 -1
5
-5
0
1
2
3
4
However, it is always a difficult situation deciding who
Location
is better than another when any two or more of the
students having similar total scores. Table 1 show
Fig. 1: Test characteristics curve showing the relationship
example of students’ performance result and decision
between total score on a test and person location
have to be made on who has better ability when it
estimates
involved similar total score.
Raw-score method of assessment provides
Probability of success = Person ability – Item difficulty
unreliable facts about students’ performances. The
Overall Total is the count of all correct answers
𝑒𝑒 𝛽𝛽𝛽𝛽 −𝛿𝛿𝛿𝛿
(1)
P(𝜃𝜃) =
provided by the students when answering the tests and
1 + 𝑒𝑒 𝛽𝛽𝛽𝛽 −𝛿𝛿𝛿𝛿
quizzes called raw scores. The scores do not take into
consideration the students’ ability on answering each
The relationship between total scores with ability
task and do not consider the difficulty of each task
estimates which is determined by the probability of
(Azrilah et al., 2007; Wright, 1989). It would not be fair
success of that person to achieve an item, reveals a nonfor those students with high marks but only able to
linear relationship. The shape of the Test
answer easy tasks compared to those with high marks
Characteristics Curve (TCC) is ogival as depicted in
on difficult tasks. Surely the next question comes to
Fig. 1.
mind is, how do we determine a task as easy or
Then Rasch model would scale the ability estimate
difficult? Rasch Model theorized (Rasch, 1960, 1961)
according to the portion of correct responses. The
that those tasks or what Rasch referred to as items, are
smaller portion of the correct responses, the higher the
of easy items when it can be answered by all students.
difficulty of an item hence the higher the item’s scale
The second theorem of Rasch Model is that, students
location. Both item and person are being scaled on the
with high ability should be able to answer all questions,
same or single scale. The higher a person's ability in
while students with less ability may have some
relation to the difficulty of an item, the higher the
problems with difficult items (Azrilah, 2009; Bond and
probability of a correct response for that particular item.
Fox, 2007). Rasch model will rearrange the pattern of
Person’s location on the latent trait is equal to the
responses according to Guttman pattern. In columndifficulty of the item, when both the person and the
wise, data are sorted from most easy to difficult items
item are located at the same location on the scale, there
and in row-wise, from most competent to less
is a probability of 0.5 chance for a correct response
competent students. The probability of answering a yes
according to Rasch model (Mok and Wright, 2004;
or a no for each task (right/wrong responses) is
Azrilah, 2009).
modeled as a logistic function of the difference between
The purpose of applying Rasch model is to obtain
the person and the item’s parameter (Mok and Wright,
measurement
from categorical response data. In their
2004; Wright, 1989; Andrich, 2004; Zamalia et al.,
raw
form,
categorical
data cannot have mathematical
2010). Refer to (1):
3521
Res. J. Appl. Sci. Eng. Technol., 6(19): 3520-3526, 2013
Table 2: Cronbach-Α and Item Reliability
Valid responses: 99.9%
Person raw score-to-measure correlation = 0.99 (approximate due to missing data)
Cronbach alpha (KR-20) person raw score reliability = 0.33(approximate due to missing data)
Summary of 8 measured items
Infit
---------------------------------Raw score
Count
Measure
Model error
MNSQ
ZSTD
Mean
726.8
242.2
0.00
0.07
1.10
0.1
S.D.
215.9
2.0
0.94
0.02
0.42
4.4
Max.
1105.0
243.0
1.82
0.10
1.91
5.9
Min.
329.0
237.0
-1.71
0.06
0.53
-7.3
Real RMSE
0.08 ADJ.SD 0.94
Separation 11.67 Item reliability 0.99
Model RMSE 0.07
ADJ.SD 0.94
Separation 13.33 Item reliability 0.99
S.E. of item mean = 0.36
Minimum extreme score: 1 items
Umean = 0.000
Uscale = 1.000
Outfit
-----------------------------------MNSQ
ZSTD
1.06
-0.2
0.34
3.7
1.06
4.3
0.85
-6.4
Table 3: Person Reliability
Summary of 243 measured persons
----------------------------------------------------------------------------------------------------------------------------------------------------------Infit
Outfit
----------------------------------------------------------------Raw score
Count
Measure
Model error
MNSQ
ZSTD
MNSQ
ZSTD
Mean
28.9
9.0
-0.03
0.36
1.03
0.0
1.06
0.0
S.D.
3.8
0.2
0.48
0.02
0.58
1.2
0.84
1.1
Max.
40.0
9.0
1.62
0.52
3.13
3.1
7.00
4.7
Min.
16.0
8.0
-1.77
0.34
0.16
-2.9
0.18
-2.1
Real RMSE
0.40 ADJ.SD 0.26
Separation 0.66
Persons reliability 0.31
Model RMSE 0.36
ADJ.SD 0.32
Separation 0.89
Persons reliability 0.44
S.E. of person mean = 0.03
operations; addition, summation, and even mean.
Therefore meaningful inference cannot be made upon
categorical data. It only allows for data summarization
using mode and median in reporting of the data (Leedy
and Ormond, 2009). Rasch Model embodies certain
criteria to be met in order to obtain measurement (Mok
and Wright, 2004; Rasch, 1961; Bond and Fox, 2007;
Andrich, 1988) and that the data should fit the model
(Andrich, 2004) and not from the perspective of
statistical modeling, model to fit the data.
The person reliability is rather low at 0.31, which
indicate a poor spread of person; indicate a high level of
inconsistencies in the students’ responses (Table 3).
This is further reveal by the Person Separation 0.66
which points out that the 243 person sample size has
almost similar abilities but the instrument does not have
enough good items to separate them into distinct group.
More items is needed to be constructed to make the
instrument better hence provide a laser cut to
discriminate the students.
The poor value of person reliability however does
not reveal that the instrument is not valid and analysis
ANALYSIS AND DISCUSSION
cannot be done. The Person Mean is located at -0.03
The Cronbach-α value is 0.33 reveals that the
logit, indicating that the students have almost the same
instrument might have some problems in assessing the
ability with the item difficulty (Item Mean = 0.00 logit)
students’ performance. Rasch analysis allows further
and the items in general are difficult to this group of
investigation, checking on the reliability of the person
students. Figure 2 depicts the location of items on the
and item (Table 2). Item reliability is at 0.99, an
measurement scale, indicating their level of difficulties.
excellent reliability value (Fisher, 2007). This indicates
Observation made from the map in Fig. 2 showed
that the item size is sufficient to provide an excellent
that 4 of the items located above the mean item and the
item spread with ranges of difficulty; there are items
other 4 items located below the mean. Item KP which
which are easy and some are difficult and some items in
refers to chemical engineering appears to be the easiest
between.
item among those eight (8) items. KP is the minimum
It also reveals that there is one (1) item with
extreme item and is shown to be located at the lowest
minimum extreme score, pointing out that there is one
position on the logit measurement ruler in Fig. 2. ES(1) very easy item located at the end most of the
Electrical system, Excel-Microsoft Excel application,
measurement scale at -1.71 logit. The item is unable to
and Laporan-Report writing, are other items that are
demonstrate the variation of person ability; all those
considered easy which is located below the item mean.
students who managed to answer the items correctly
Maths -Mathematical concepts are the easiest
cannot be discriminated between a smart and a poor
among those items that are difficult and all four (4) are
student.
located above mean item. Final-the final examination,
3522
Res. J. Appl. Sci. Eng. Technol., 6(19): 3520-3526, 2013
Persons - MAP - Items
<more>|<rare>
2
+
|T AS
|
. |
. |
|
. |
1
. +S
.# T|
.# |
.## | Q5
.#### S|
.#### |
############# | Final
0
.##### M+M MB
####### | Laporan
.##### |
.### |
.###### S|
.## | Excel
# |
-1
. T+S
|
|
. |
|
. | ES
|T
-2
+
|
|
|
|
|
|
-3
+
|
|
|
|
|
|
-4
+ KP
<less>|<frequ>
Maths
Mean Item=0.00 logit
Fig. 2: Item map
Quiz 5 and AS -Structure and civil which is the most
difficult item among the four (4).
Figure 3 is the person map where it depicts the
location of each student according to their ability
measure. In general, there is almost equal distribution
of students above and below the item mean at 0.00
logit. The initial observation made to the summary
statistics from Table 3, reveals that person separation is
0.66 near 1.00, indicating these group of students have
almost the same ability thus a very strong homogeneity.
In common practice of assessment, those with high
total marks are deemed have good command of the
subject. The data used in this paper included the total
marks of overall test percentages, and converted it into
rating of 1<60, 2>60, 3>70, 4>80, 5>90. Logically,
those with a ‘5’ and ‘4’ should be located on the upper
location, above mean item = 0.00 logit, of the
measurement scale on the map in Fig. 3. A detailed
scrutiny of the Person-Item map is shown in Fig. 4
segregated between male and female student. Figure 4
showed female students’ ability location. The
achievement overall test rated from 1 to 5 is the last
character (6th character) coded in the person
demographic. There are some students with rating ‘2’
and even ‘1’ located on the upper portion of the map;
above the mean item. Refer to person in the square in
Fig. 4. Even though by summing up all the score of the
test and quizzes gives those students low total score, to
the contrary, Rasch finds them a more able student.
In this article, the data are of the scores for each
tests and quizzes. It will be a more meaningful analysis
if further investigation can be conducted on the items
sub-construct of each of the tests and quizzes. Each
tests or quizzes would have several sub-items
(questions) within them, and usually it is based on the
learning outcome of the Table of Test Specification,
normally defined according to Bloom’s level of
knowledge. Then further findings can be explained
according to the respective Learning Outcomes (LO);
either the hierarchical achievement of the Bloom is
according to the taxonomy or the reverse and confirms
whether the students met the specified LO or not.
3523
Res. J. Appl. Sci. Eng. Technol., 6(19): 3520-3526, 2013
MAP - Persons
<rare>|<more>
+
X T|
|
|
|
|
|
S+
|T LM0724
| LM0653
X | LM0623
|S LM0174
LM1033
LM1742
| LM0103
LM1573
X | LM0192
LM0873
LM1363
LM2013
LI1534
LC1864
PC1344
PC1294
LC0114
LM0854
LC0464
LC0034
LM0524 LM0863 LC0243
LM1563 LM1683
LM1833
LM0633 LM1203 LC0074
LC1303
LM0452 LM0713
LM1012 LM1263 LC0053
LM1523 LM1752 LC0813
LM2123
LC1944
LC1904
LC1854
LC0084
LC1974
LC1964
LC1923
LC0314
LC2254
LB2353
LC0143 LC0494
LC2023
LC0163 LC0683
LC1253 LC1893
LC1953 LC2133 LC2264
XX M+M LM0662 LM0843 LM0923 LC0043 LC0093 LC0133 LC0583
LM1072 LM1702
LC2103 LC2183
X | LM0282
LM0802
LM1412
LM1792
LM2393
| LM0332
LM0483
LM1332
LM1482
LM2092
LM1492
| LM0572
LM0982
LM2381
|S LM0733
LM1042
LM1672
LM2342
X | LM0023
LM1452
| LM0541
S+T LM1441
|
|
| LM1711
|
X |
T|
+
|
|
|
|
|
|
+
|
|
|
|
|
|
X +
<frequ>|<less>
LM1762
LM0602
LM1022
LM2421
LM0881
LM1272
LM1723
LM0781 LC1143 LC1233 LC1913 LI0763
LM1402 LC2113
LM1782
LM2293
LM0433 LM1122 LC0063 LC2193
PM0963
PM0514 PM0752
PM0772 PM1323
PM1653
PC2033
PC0014 PC0264 PC1224 PC2244
PC0204 PC0504 PC0824
PM0593
PM1463
PM1822
PM0223
PM0323
PM1103
PM1192
PM1433
PM1612
PM2322
PM0383
PM0933
PM1092
PM0553
PM1622
PM2403
PC1003 PC1933 PC2064 PC2274
PM0913
PM1732
PM0303
PM0423
PM1132
PM1212
PM1543
PM1772
PM2411
PM0413
PM1052
PM1313
PM0693
PM2223
PC0273 PC0353 PC0474 PC1633 PC1813
PC1983 PC2003 PC2044 PC2074 PC2143
PM0643 PC2373
PM1152
PM1383
PM1603
PM2163
PM0293
PM0443 PC2153 PC2284
PM1082
PM1513
PM1112 PC0233 PC0943 PC2213
PM2303
PB1283
LM1882
LM0892
LM1803 LC2203
PM0363 PM0613 PM0702 PC0903 PC1372 PC1393 PC1583 PC1643 PI2173
PM2362
PM0793 PM1162 PM1662
PM0672 PM1062 PM1501 PC1422
PM1552
LM0992 LC0152 LC0182 LC2313
LM1353
LM2233
PM0122 PM0252 PM0373 PC0213 PC1993
PM0402 PM0742 PM0972
PM1172 PM1473 PM1842
PI2083
PM0342 PM0392 PM1691 PC0563
PM2052 PM2332
PM0532
LM0952 LM1241
LM1872
LM0831 LM1181
LM2431
PI1591
Fig. 3: Person - item map
CONCLUSION
The basic engineering mathematics subject
introduces the students to the basic mathematics in
engineering for all engineering students. Engineering
students are exposed to the initial and necessary
concepts for their preparation towards focused
engineering field later in their engineering program.
Besides assessing the students’ knowledge on the
subject, the assessment includes report writing, their
knowledge on the Microsoft excel application and also
on their mathematical skill.
The assessment reveals that the students find it
easy in the area of chemical engineering and electrical
engineering. The students also find it easy to handle
report writing and have no problem with Microsoft
3524
Res. J. Appl. Sci. Eng. Technol., 6(19): 3520-3526, 2013
MAP - Pers
<rare>|<m
+
X T|
|
|
|
|
|
S+
|T
| PM0963
X | PM0514
|S PM0772
PM1653
PC1344
PC1294
PM0752
PM1323
PC2033
PC0014
PC0204
PC0264
PC0504
PC1224
PC0824
PC2244
PC1003
PC1933
PC2064
PC2274
PC0273
PC1983
PC2373
PC0353
PC2003
PC0474
PC2044
PC1633
PC2074
PC2153
PC2284
PC0233
PC0943
PC2213
PC1372
PC1393
PM0593
PM1463
X | PM1822
PM0223
PM0323
PM1103
PM1192
PM1433
PM1612
PM2322
XX M+M PM0383
PM0933
PM1092
X | PM0553
PM1622
PM2403
PM0913
PM1732
PM0303
PM0423
PM1132
PM1212
PM1543
PM1772
PM2411
PM0413
PM1052
PM1313
PM0693
PM2223
PM0643
PM1152
PM1383
PM1603
PM2163
PM0293
PM0443
PM1082
PM1513
PM1112
PM2303
PM0363
PM2362
PM0793
PM0672
PM1552
PM0613
PM0702
PC0903
PM1162
PM1062
PM1662
PM1501
PC1422
|S PM0122
PM0402
PM1172
PM0252
PM0742
PM1473
PM0373
PM0972
PM1842
PC0213
|
PM0392
PM2332
PM1691
PC0563
|
|
|
X
|
S+T
|
|
|
|
X |
T|
PM0342
PM2052
PM0532
PC1813
PC2143
PB1283
Mean item = 0.00 logit
PC1993
PC1583
PC1643
PI2173
PI2083
PI1591
Fig. 4: Female distribution on person map
excel application. However, the students have some
problem in basic mathematical knowledge, material
science, Quiz 5, and find the final examination to be
difficult. The most difficult item is on the area of civil
and structural engineering, where all students are
unable to solve the tasks given under this topic. There is
definitely a need to identify the reason on why such
preference is shown by this group of students.
ACKNOWLEDGMENT
The authors wish to acknowledge the financial
support received from the Centre for Academic
Advancement, Universiti Kebangsaan Malaysia as
research grant; PTS-2012-088 and the Centre for
Engineering Education Research,
Universiti
Kebangsaan in the effort of improving the quality of
teaching and learning in engineering.
3525
Res. J. Appl. Sci. Eng. Technol., 6(19): 3520-3526, 2013
REFERENCES
Andrich, D., 1988. Rasch Models for Measurement.
Sage Publications, Beverly Hills.
Andrich, D., 2004. Controversy and the Rasch model:
A characteristic of incompatible paradigms.
Medical Care, 42(1): 1-16.
Astin, A.W., T.W. Banta, K.P. Cross, E. El_Kawas,
P.T. Ewell, P. Hutchings and B.D. Wright, 1991.
Principles of Good Practice for Assessing Student
Learning. The American Association for Higher
Education.
Azrilah, A.A., 2009. Rasch Model Fundamentals: Scale
Construct and Measurement Structure. ISBN: 978967-5772-00-9 CIP: 519.54: Azrilah.
Azrilah, A.A., M. Saidfudin, M. Azlinah, A. Nor
Habibah and Z. Sohaimi, 2007. Appraisal of course
learning outcomes using rasch measurement: A
case study in information technology education.
Int. J. Syst. Appl. Eng. Dev., 1(4): 164-172.
Bond, T.G. and C.M. Fox, 2007. Applying the Rasch
Model: Fundamental Measurement in the Human
Sciences. 2nd Edn., Lawrence Erlbaum, Mahwah,
New Jersey.
Deming, W.E., 2000. Out of Crisis. MIT Press,
Cambridge.
Fisher, W.P., 2007. Rating scale instrument quality
criteria. Rasch Measurement Trans., 21(1): 1095.
Illeris, K., 2004. Three Dimensions of Learning
Malabar. Krieger Publishing, Florida.
Leedy, P.D. and J.E. Ormond, 2009. Practical Research:
Planning and Design. 9th Edn., Pearson, ISBN-13:
9780137152421.
Mok, M. and B. Wright, 2004. Overview of Rasch
Model Families: In Intoduction to Rasch
Measurement: Theory, Models and Applications.
Jam Press, Minnesota.
Ormrod, J.E., 1995.Human Learning, Englewood
Cliffs. Prentice Hall, NJ.
Rasch, G., 1960. Probabilistic Models for Some
Intelligence and Attainment Tests. (Copenhagen,
Danish Institute for Educational Research),
Expanded Edition (1980) with Foreword and
Afterword by B.D. Wright, the University of
Chicago Press, Chicago.
Rasch, G., 1961. On general laws and the meaning of
measurement in psychology. Proceedings of the
Fourth Berkeley Symposium on Mathematical
Statistics and Probability, Berkeley, University of
California Press, California, 4: 321-334.
Wright, B.D., 1989. Rasch model from counting right
answers: Raw scores as sufficient statistics. Rasch
Measurement Transactions, 3(2): pp. 62.
Wright, B.D., 1997. Fundamental measurement for
outcome evaluation. Physical medicine and
rehabilitation: State of the Art Reviews, 11(2):
261-288.
Zamalia, M., M. Saidfudin and A.A. Azrilah, 2010.
Students Performance Evaluation for Learning
Outcomes Measurement: SPELOM Model. In:
Mastorakis, N., V. Mladenov, A. Zaharim and C.A.
Bulucea (Eds.), Recent Advances in Management,
Marketing, Finances: Recent Advances in
Electrical Engineering, a Series of Reference
Books and Textbooks. WSEAS Press, March 2010,
pp: 172-178.
3526