ASSESSMENT OF ENGLISH PROFICIENCY BY FUZZY
LOGIC APPROACH
F. Melis Cin
University of Dublin, Trinity College Dublin, School of Education, Foundation Studies, Ireland
cinf@tcd.ie
A. Fevzi Baba
Marmara University, Technical Education Faculty, Electronics and Computer Department, Turkey
fbaba@marmara.edu.tr
ABSTRACT
Assessment is one of the important tasks in the whole teaching and learning process. It has been a great influence on the students’ approach
to learning and their learning outcomes. Current education researches show that fuzzy multi-criteria assessment methods are widely used for
evaluating students’ learning. In the evaluation of English proficiency, there are several criteria to be considered. Each criterion is based on
different skills and sub skills. The English performance of the candidates is determined by total result of assessment of these criteria. It is
widely recommended to distribute equal amount of values to four main skills; reading, writing, speaking and listening. E.g. TOEFL, IELTS,
PCE…etc if overall English proficiency is aimed to be taken into consideration. In this study we developed a user friendly multiple purpose
fuzzy multi-criteria assessment software to evaluate the English performance of the students. It provides the user to change the parameters for
multiple assessment purpose. A computer assisted language assessment example of English proficiency with fuzzy multi-criteria assessment
method is given and advantages of this method are discussed.
Keywords: Expert systems, Fuzzy multi criteria assessment, English proficiency, computer assisted language assessment
INTRODUCTION
The most common and worldwide English proficiency tests
are discrete point objective and holistic rating by human
scorers of oral and written language such as TOEFL,
comprehensive English language test, Michigan Test of
English Language Proficiency, etc. The oral and written
samples of the examinees are subjected to some criteria in
these tests that indicate the level of English proficiency.
Assessment and evaluation, as applied to education, plays
a crucial role in the teaching and learning process, which
gives information about the relative goals and objectives
determined beforehand. The evaluation results indicate the
level of student understanding so that teacher can take
some preventative measures in advance, such as remedial
courses or re-planning the content and method of the
teaching (Herera,1995). While some students proceed to
the next stage some may have to take the course again.
Therefore; it helps to make a distinction between these
students (Ashworth,1982).
Each of these samples is evaluated by more than one
interviewer to prevent any inequalities to be caused by the
dominance of a single perspective and bias. It will also be
difficult, waste of time and unreliable to use a manual
approach to make an objective testing, particularly when
there are so many students to be assessed and more than
one interviewer to assess. However, computers, as an
indispensable tool of our age, are quite appropriate for this
purpose once the required data and criteria are presented to
the system.
The assessment of Foreign Languages has gained a
considerable understanding and attention during the past
decade as a necessity of expanding global economy,
science, technology and literature. English as a lingua
franca formed a reputable place in higher education
institutions. Many students should show a good level of
English proficiency to cope with the information service of
global world and to pursue their academic studies in
universities. With the constant increase in the numbers of
ESL/EFL students, there has been growing interest in
ESL/EFL assessment practices used by teachers and
university instructors in ESL/EFL classrooms. To meet the
needs of increasing number of language learners, demand
for appropriate language assessment programs increased
as well. Adequate procedures, resources are needed to
develop and implement multiple measures of assessment
for ESL students.
Since computers became a part of education world, it has
been widely come into use in English Language Teaching.
Computer assisted language learning has been developed
and facilitate the language learning. Web-based technology
and computer assisted language learning to enhance
language teaching has become a popular practice among
many foreign language/second language (FL/SL) educators
(Kern & Warschauer, 2000; Liu, 2001) Computers’ use in
English ranges from vocabulary teaching to assessment of
language. Many studies have been carried out by using
CALL (Computer assisted language learning). However, the
use of computers in assessment of English proficiency is not
widely used. Because there are not many developed
software to be used in the assessment procedures as it is a
difficult process.
The assessment decisions made by the universities’
ESL/EFL instructors involve a combination of their
knowledge, beliefs, experience, values and the interaction of
these attributes with the instructional context in which they
teach.
355
knowledge
database for
experience
Knowledge base
experience
thought
inference
decission
decission
Human expert
Expert system
Figure 1. Structure of decision making of a human expert and expert systems (Parsaye, 1988).
An example study of computer and web based technology
has been carried out by Yeh and Lo (2003) who presented a
neural network model that assesses automatically the
learner’s metacognitive knowledge level of language
learning by observing his/her online browsing behavior
without asking the student to answer any questions or filling
out any form.
Zadeh (1988) proposed a computational procedure for fuzzy
logic inference, which consists of an implication function and
inference rule. Given that A and B are both fuzzy sets
defined over U and V respectively. A fuzzy rule A→B is
first transformed into a fuzzy relation RA→B that represents
the correlation between A and B. There are several relation
methods for implication and compositional rules of inference
that we used an algebraic product (·) as a fuzzy implication
operator and the max-product as the compositional rule of
inference. Max-product relation is defined as follows:
Establishing objectivity in computer assisted programs is
also a major obstacle in developing a reliable program. One
of those programs has been developed by Lilley et all in
2004 in which they came up with computer adaptative test
to estimate the level of proficiency of English and compared
this study with computer based tests.
Given a fact is A' and a rule is A→B, Zadeh’s composition
rule says
In this study, fuzzy multiple criteria assessment has been
used for the evaluation of English proficiency of students
and a user friendly expert system is developed. Fuzzy logic
is much closer in spirit to human thinking as a natural
language. Fuzzy logic system has four principle
components; a fuzzyfication interface, a knowledge base,
decision making mechanism and a defuzzyfication. All these
parts will be explained in application as follows. During the
assessment procedure, evaluators marked each subcriterion as fuzzy linguistic variables such as; “poor”,
“unsatisfactory”, “average”, “good” and “excellent”. Each
linguistic term has an equivalent of a numerical value from 1
to 5. The use of this natural language allows flexibility to
make inferences and judgments on student’s performance.
In the conclusion, the advantage of multi criteria fuzzy logic
evaluation is discussed.
µR(x,y)=(µA(x) ·µB(y))
(1)
B' =A' ° RA→B
(2)
µB'(y)=max (µA'(x) · µRA→B (x,y))
(3)
This computation can be viewed as a vector-matrix product
with multiply. Consequently, given a rule is A→B and an
input A', the membership function of the inferred output B' is
µB'(y)=max (µA'(x) · µRA→B (x,y))
µB'(y)= max (µA'(x) ·(µA(x) ·µB(y)))
(4)
µB'(y)= (α ·µB(y))
(5)
where
α = max (µA'(x) ·µA(x))
Since value of α and the final centroid change more
smoothly depending on inputs, the inference based on this
method is more sensitive to small changes (Zadeh, 1992).
When more than one rule is enabled, the consequents of all
activated rules are combined. Supposing that B'1,B'2,…B'n
are derived results, the combined result is the individual
fuzzy results. Final step is defuzzification which converts
fuzzy results to a crisp number then it can represent final
crisp score. The most common method is the centre of area
method which finds the centre of gravity of the solution fuzzy
sets, as shown in below:
Expert Systems and Fuzzy Set Theory
Expert systems, also known as Knowledge Based Systems,
are computer programs which embody human knowledge
and understanding in a way that imitates a human expert
who can solve specific types of problems. Figure 1 shows
the structure of decision making of a human expert and
expert systems. Expert systems are composed of two parts:
an inference engine and a knowledge base. The knowledge
base contains the expert’s knowledge, and expert’s
experiences are also contained in the database for
experience (Parsaye, 1988).
Ʃ
Ʃ
N
Human beings reach a decision through a thinking process
whereas expert systems make decisions through the
inference process. Human beings make decisions in fuzzy
environments by using fuzzy variables. In order to simulate
human decision making in computer environment, fuzzy
variables should be represented in computer. This requires
the use of fuzzy set theory. Therefore, fuzzy set theory can
play a significant role in our student evaluation process.
G=
i=1
µBi wi
N
i=1
(6)
µBi
Where, wi is the value from the set that has a membership
value µi.
356
Table 1. The main and sub-criteria and weights
Main Criteria
Reading
Listening
Speaking
Writing
Weights
0.25
0.25
0.25
0.25
Sub-Criteria
Weights
sentence comprehension
0.15
word comprehension
0.15
word analysis
0.15
get the main idea
0.20
distinguish the main idea form supporting ideas
0.15
pick out key words
0.20
recognize grammatical word classes
0.20
recognize stress and rhythm patterns
0.20
recognize vocabulary
0.20
detect key words, such as those identifying topics and ideas
0.20
recognize cohesive devices
0.20
Interaction
0.20
Richness of vocabulary
0.20
pronunciation
0.20
fluency
0.20
use of grammar
0.20
support with the evidence
0.15
development of idea
0.15
use the appropriate style to the genre
0.15
use the orthography correctly (spelling)
0.15
use of contextual links to reinforce the structure
0.10
use of correct grammatical patterns
0.15
use of vocabulary
0.15
Multi Criteria Fuzzy Assessment System for English
Proficiency
sharing different perspectives of teaching, values and
coming from different cultures, education systems. This
enabled a range of diversity in determining our criteria.
The structure of multi criteria fuzzy assessment system for
English proficiency contains five main steps:
The main and sub-criteria, presented in Table.1, are
subjective and can change according to evaluator’s needs.
We determined the criteria and values according to the
system explained above. However, It should be bared in
mind that the values and criteria can be adapted according
to the needs of interviewers. Since the language learning is
holistic, we equally shared the values between main skills
and determined the values of sub-criteria as result of our
interviews with teachers. The number of sub-criteria can
also be increased and decreased depending on what you
would like to assess specifically. We can suggest that the
more sub-criteria are determined the more reliable results
are obtained. On the other hand, during the assessment
procedure it will be difficult to cope with paying attention and
following so many sub-criteria. Therefore we limited them
between five and seven in our study. While limiting the
criteria, we considered the ones that are the main evaluation
keys of that skill based on the lecturers’ list of preferences.
1.Determination of the main and sub-criteria of assessment.
2.Determination of assessment criteria weights.
3.Definition of the membership functions for fuzzyfication.
4.Fuzzy inference; finding student’s fuzzy score.
5.Defuzzification; finding student’s crisp score
Evaluation of students’ English proficiency consists of
several main criteria and sub criteria; each involving a
number of judgments. Determination of all these criteria and
weights is the most important part of the evaluation system.
Within a holistic approach; reading, writing, speaking and
listening skills are entitled as main criteria in the assessment
of English Language. Each of these skills is followed by subcriteria. The weights for primary and sub-criteria were
assigned by qualified people. The intent of the sub-criteria
is to gain in-depth information on aspects of their
assessment practices and assessment planning. These
sub-criteria might change from person to person or
evaluation to evaluation. However, in our study we
determined these sub-criteria as a result of the 6 lecturers at
Marmara University and Trinity College Dublin.
Since fuzzy logic is much closer in spirit to human thinking
as a natural language, evaluation grade must be fuzzy
variables. This process is called fuzzyfication which
converts the crisp grades to fuzzy grades. In this study,
evaluation grades are fuzzified with five membership
functions; Poor (P), Unsatisfactory (U), average (A), good
(G), excellent (E). Each linguistic term has an equivalent of
a numerical value from 1 to 5. The use of this natural
language allows flexibility to make inferences and
judgments on student’s performance. Fuzzfication process
We handed out a questionnaire form and asked them to list
the significant criteria in the assessment of English. 3
lecturers from each university joined our research. Thus, our
list composed of the sub-criteria determined by lecturers
357
is shown in Figure 3 and membership functions are chosen
as trapeze shape.
R11=
R12=
R13=
R14=
R15=
R16=
The decision making mechanism is the kernel of fuzzy logic
system, which has the capability of simulating human
decision making based on fuzzy concepts and of inferring
fuzzy decision actions employing fuzzy implication and the
rules of inference in fuzzy logic. In this study, max-product
inference method is used which the final output membership
function for each output is the union of fuzzy set assigned to
that output in a conclusion after the scaling their degree of
membership values to peak at the degree of membership for
the corresponding promise.
( 0,0
( 0,0
( 0,0
( 0,0
( 0,0
( 0,0
0,4
0,5
0,45
0,35
0,45
0,35
0,3
0,25
0,4
0,25
0,3
0,25
0,3
0,25
0,15
0,4
0,25
0,4
0,0 )
0,0 )
0,0 )
0,0 )
0,0 )
0,0 )
where, R1i; fuzzy relation of each sub-criterion of reading
main criterion and R1i [E,G,A,U,P]. The main criterion of
reading weights was determined as:
A1 = [ 0,15 0,15 0,15 0,2 0,15 0,2]
The software works in a windows environment. It has three
windows; Fuzzfication, Fuzzy Grading and Assessment and
Report. In fuzzyfication window, membership functions are
defined. The software allows the user to enter four main
criteria and eight sub criteria for each of the main criteria
and their weights. At the fuzzy grading windows, shown in
Figure 2, maximum fifteen evaluators can assess a
candidate. Fuzzy and crisp evolution scores are both shown
on assessment and report windows. The evaluated grades
are listed either according to the total score or selected main
criteria, (Baba, at all, 2008).
Total fuzzy score of reading main criterion (B1) can be found
by Eq. 3.
B1 =A1·R1=[ 0,2 0,25 0,2 0,2 0,15]
0.0
0.0
0.0
0.0
0.0
0.0
0.4 0.3 0.3 0.3 0,0
0.5 0.25 0.25 0.25 0,0
0.45 0.4 0.15 0.15 0,0
0.35 0.25 0.4 0.4 0,0
0.45 0.3 0.25 0.25 0,0
0.35 0.25 0.4 0.4 0,0
B1 = [ 0,42 0,37 0,07 0,1 0,04]
Where, B1 represented the fuzzy score of “reading” main
criterion. The crisp evaluation score is found by
defuzzification by using centre of gravity method. In the
same way, fuzzy grade of each criterion can be found and
relation matrix is written as follows:
A Sample Application
As a numerical example, suppose, student Melis Cin
gathered points, shown in Figure 2, from five evaluators who
has equal (0.2) evaluation weights.
0,42
0,16
R = 0,0
0,88
In the fuzzy multi criteria evaluation method, for fuzzyfication
each main criterion is fuzzyfied as follows:
Figure 2. Fuzzy grading Screen
358
0,37
0,4
0,04
0,12
0,07
0,28
0,28
0,0
0,1
0,16
0,48
0,0
0,04
0,0
0,2
0,0
 The application of fuzzy logic to assessment process
minimizes the subjective factors and any possible
interferences of interviewer such as prejudice and bias.
 The hierarchical structure of the evaluation enables the
interviewers to be more precise on the notions to be
assessed in each skill.
 This system operates like a human by describing the
results in linguistic terms such as “good”, “poor” and
“excellent”.
 It becomes quite easy to evaluate by utilizing computer
technology and prevents the waste of time.
 Complicated calculation will be accomplished quickly
and will prevent any errors that might be caused by
manual calculations.
 It provides a reasonable and flexible evaluation.
 Fuzzy Logic reduces the varying interpretations by
interviewers to the same data.
 It can even enable us to see the developments of
students when applied respectively.
 It improves the quality of evaluation process.
The total fuzzy grade is calculated as shown below:
B =A°R = [0,365 0.2325 0.185 0.1575 0.06 ]
The total crisp evaluation score is found by defuzzification.
The centre of gravity method is used in this study. Figure 3
shows the membership functions and total fuzzy scores of
the student, Melis Cin.
µ
Figure 3. Membership functions
REFERENCES
Each of main criterion and final crisp scores are found by
defuzzification process using Eq.6, and final crisp score are
calculated as follows:
Ashworth, A.E. (1982) Testing for continuous assessment.
Evans Brothers Limited. London
Baba, A.F., Kuşçu, D., Han, K., (2008) Developing a
software for fuzzy group decision support system: A
case study, 2nd International Symposium of
Computer
and
Instructional
Technologies,
Kusadasi,Turkey, April16-18, 2008
0,365.100+0,2325.85+0,1575.62,5+0.185.45+0.06.30
G=
0,365+0,2325+0,1575+0.185+0.06
Herera,
G=75.154 (Good)
Student’s final score is found as 75.154 which means
“good”, whereas, reading and listening are “good”, speaking
is “poor” and writing is “excellent”.
F., Herera-Viedma, E. (1995), Aggregation
operators for weighted linguistic weighted
information. Technical Report,Department of
Computer science and Artificial Intelligence,
Universidad de Granada.
Kern, R., and Warschauer, M. (2000). Theory and practice
of network-based language teaching. In M.
Warschauer & R.
Reading; 80.625 (good),
Listening;72.3 (good),
Speaking; 48,3 (poor),
Writing; 95.4 (excelent),
Final score; 75.154 (good).
Kern (Eds.), Network-based language teaching: Concepts
and practice. New York: Cambridge University
Press.
Lilley, M., Barker, T., Briton, C. (2004) The development
and evaluation of a software prototype for computer
adaptive testing. Computers & Education Journal
43(1-2), pp. 109-123.
DISCUSSION
As technology systems continue to develop, it will form a
favorable place in education to cope up with complex
systems of teaching technologies. Therefore, in the
presented study, Fuzzy Logic has been applied to
assessment of English Language Proficiency which is a
process in a way that parallels how humans think. This
Computer technology makes the assessment process easy
and quick. This hierarchical structure enables
committee to assess the different primary and sub skills of
students. Validity is also important for large-scale
assessment systems and it is often difficult to obtain an
objective evaluation because the interviewer’s bias,
personal viewpoints may interfere the evaluation process of
the time. This software provides us with a reliable system.
The following conclusions or positive implications can be
drawn from our proposed study.
 Fuzzy Logic provides objectivity and fairness to
evaluation process.
Liu, H. C. (2001). Incorporating computers into English
classrooms: Curriculum design issues. In selected
papers fromthe tenth international symposium on
English teaching, Taiwan, ROC (pp. 120–131).
Parsaye,K.,Chignell,M., (1988) Expert System for Expert,
John Willey&sons ,Inc.,Canada.
Yeh and Lo (2005) Assessing metacognitive knowledge in
web-base CALLa neural network approach.
Computers &Education Journal, Vol.44, p. 97-113.
Zadeh, L.A. (1988). Fuzy Logic, Computer, Vol.21, No.4
Apr.1988,pp.83-93
Zadeh, L.A and Kacpyrzyk, J., (1992). Fuzzy Logic for the
Management of Uncertainty, John Willey and
Sons.Inc. pp.217.
359