Assessing Confidence in
the Chinese Learner
Stephen Bruce
Napier University, Edinburgh
Assessing Confidence in the Chinese Learner

Chinese students at Napier University

Rationale for confidence-based assessment

Student responses to the test

Academic self-concept, confidence and achievement
research
English Foundation Programme

Set up in 2001 to prepare Chinese students for entry to Napier
University, both in terms of linguistic and study skills.

The needs of these students were complex, the “learning curve” for
them was extremely steep, and that the demands they faced on a
linguistically complex and culturally unfamiliar one-year Masters
would be considerable, even given the requisite IELTS pass.

Proportion of Chinese students on Napier Business Masters
programmes has increased in recent years, which has resulted in
staff and the institution facing a very different cohort with different
strengths, and also needs, to their predecessors.

Presents a major challenge to lecturers as they struggle to adapt
their teaching approaches to suit both the increased number of
overseas students and also the home students on their modules.
Dr Lesley Gourlay, Napier University (2004). Crossing Boundaries: A Case Study
Masters Level Chinese Students, LTSN (In Press).
Linguistic and Cultural Issues
Challenges for Students
LANGUAGE

Extreme difficulties in understanding / taking notes in lectures

Difficulties with tutorial participation due to lack of confidence in English

Some difficulties with tutorial tasks due to slow reading speeds

Fear of failing exams due to difficulties in writing English in a time limit
EDUCATIONAL CULTURE

Feeling “lost” in semester 1

Some students unclear about expectations in UK-style coursework &
reading

Experience extremely stressful for some
Lesley Gourlay, Napier University (l.gourlay@napier.ac.uk)
Linguistic and Cultural Issues
Challenges for Staff
LANGUAGE

Unsure if students are understanding lectures

Required to spend extra time explaining lecture content at the end

Some difficulties with tutorial participation and integration
EDUCATIONAL CULTURE

Some staff faced with numerous examples of plagiarism in coursework

Some difficulties with tasks involving critical appraisal

Some students struggled with application of theory to practice
Lesley Gourlay, Napier University (l.gourlay@napier.ac.uk)
Potential responses by the university
LECTURE
COMPREHENSION
& NOTETAKING:
TUTORIAL
PARTICIPATION
ISSUES:
PLAGIARISM AS
LANGUAGE COPING
STRATEGY:
SPECIFIC SESSIONS
FOR INTERNATIONAL
STUDENTS?
SUPPORT & IDEAS FOR
LECTURERS?
SPECIFIC SESSIONS
FOR INTERNATIONAL
STUDENTS?
MORE OPPORTUNITIES
FOR
STUDENT PREPARATION?
SPECIFIC SESSIONS
FOR INTERNATIONAL
STUDENTS?
MORE “USER-FRIENDLY”
PLAGIARISM AVOIDANCE
MATERIALS?
GROUP
ROTE-LEARNING AS
LANGUAGE COPING
STRATEGY:
STUDENT
ISOLATION &
ANXIETY:
INCREASED
WORKLOAD FOR
STAFF:
MORE “BUDDYING” AND
INTEGRATION
INITIATIVES?
MORE INTERNATIONAL
STUDENT
COUNSELLING?
EXTRA TIME GRANTED
IN WORK
ALLOCATIONS?
INTERNATIONAL
ACADEMIC
SUPPORT OFFICERS?
REDUCTION IN EXAM
PREDICTABLITY?
CLEAR POLICY ABOUT
GRAMMAR ERRORS AND
MARKS?
Lesley Gourlay, Napier University (l.gourlay@napier.ac.uk)
Confidence and achievement

Chinese learners show a preference for rote-learning and
memorisation. However their responses to learning strategy
questionnaires indicate preference for deep learning.

Learner self-esteem has been found to be positively associated with
academic achievement (Brookover et al., 1964; Prendergast & Binder, 1975; Song
& Hattie, 1984)

Comparative studies support a view that the self-esteem of young
Chinese students is lower than UK and American (Chan, 2000).
–
–
–

self-effacing and modest values in Chinese culture, strongly influenced by
the Confucian tradition of a ‘humble’ character
traditional authoritarian style of education or the highly competitive
pressures created by schools, families and society
gap in living standards in the UK and Chinese cultures
Recent advances in academic self-concept and achievement
research
Assessing Confidence in the Chinese Learner
This short pilot study is interested in two questions:
1.
Would the Chinese students responses to a confidence-based test
differ from UK students ?
•
2.
A good cultural test of this format
Would their use of confidence levels in the test relate to their academic
self-concept, or attitudes to academic life ?
•
Improving academic self-concept is often posited as mediating other
desirable attributes
Rationale for confidence-based scoring

A student’s ability in answering may fall into a number of categories
– I know it
– I’m not quite sure, but I think I know it
– Perhaps I can identify the answer by a deductive process on the distracters
– If I guess I’ve a 25% chance of being correct (for 4 answer choices)
– I really haven’t a clue
and perhaps worst of all
– I really know it …… what do you mean I’ve got it wrong !!
Davies (2002)

The standard MCQ cannot distinguish between the above
–
–
–
–

possible to pass exams with knowledge only half learned or poorly understood
the student certain of their knowledge should be rewarded
the student should not be rewarded for guesswork
encourage students to reflect on the reliability of their answers
Would the student use this knowledge to make a decision or perform an
action (usable knowledge) ?
Hassmen and Hunt (1994)
Confidence-based MCQ


Select an answer in the usual way for a multiple choice question
Indicate your confidence that you are correct.
C=1 (low)
Gardner-Medwin and Gahan (2003)
C=2 (mid)
C=3 (high)
Scoring confidence-based MCQ
What is my score if my answer is
Confidence level
C=2
(Mid)
C=3
(High)
Score when
1
2
3
Score when
0
-1
-4
< 50% >50%
>75%
Confidence level decision is governed by 2 judgements:
o
o

?
C=1
(Low)
Probability correct

or
estimated probability that the chosen answer will be correct
the impact of the reward / benefit for a right / wrong answer
Students rarely discuss their decisions in terms of explicit probabilities
Gardner-Medwin and Gahan (2003)
Encouraging good confidence judgement
Students are encouraged to reflect on the reliability of their answers.
Good confidence judgement means more marks !

If you are sure that your answer is correct ….
o
select C=2 (mid) or C=3 (high) confidence level
get the marks your confidence deserves!

If your are unsure (or guessing !) ….
o
select C=1 (low) confidence level
don’t lose marks through misplaced confidence!
Confidence-based MCQ feedback
A 100% confidence-based score is 40
answers at C=2 (mid) confidence
…. so that 40 correct at C=3 would be 150%
A summary of confidence levels….
Gardner-Medwin and Gahan (2003)
Confidence-based MCQ feedback
Is the confidence-based score higher than the % correct score?
No
1.
Check the instances of INCORRECT answers at C=3 (high)
and C=2 (mid). These can indicate areas of knowledge where
you are misinformed. A large number of these perhaps
indicates general overconfidence.
2.
Check your number of CORRECT answers at C=1 (low). A
large number indicates that you know more than you are
willing to admit and are perhaps underconfident.
Confidence-based MCQ feedback
Is the confidence-based score higher than the % correct score?
Yes
You have shown good confidence judgement!
You are willing to express an appropriate level of confidence when
considering your knowledge.
This is important when making decisions and performing actions
based on that knowledge.
The student group

31 Chinese students (54% sample)
o
o
o

Current on English as a Foreign Language (EFL) Foundation Programme
o
o

35 weeks of 20 hours of EFL study
Gateway and Starter routes dependant upon English proficiency
Students recently completed their IELTS exam (International English
Language Testing System).
o
o

18 male, 13 female
24 postgraduate and 7 undergraduate
Ages: 18-25 (19), 26-32 (8), 33-40 (4)
Entry on to a Napier Programme conditional on a student’s IELTS score
At the time the students did not know their results
Students complete a confidence-based academic vocabulary test (40 Qs)
Chinese student responses to confidence-based MCQ test
No indications that Chinese students (red) behave differently from
home based students (blue)
o
o
Questions appear quite difficult, since low marks are more represented.
930 ‘Blue’ data from UCL including 40 first time responses.
150%
Confidence-Based Mark

100%
50%
0%
20%
40%
60%
-50%
% correct
80%
100%
Chinese student responses to confidence-based MCQ test
No indications that Chinese students are reluctant to select high
confidence levels (C=3)
o

Higher test scorers showed better confidence judgement
o

on average, high confidence was selected most often
Lowest test-scorers tended to be over-confident
No significant difference between genders observed.
Proportion of confidence levels
40.0
37.7
30.0
33.4
29.0
%

20.0
10.0
0.0
C=3
C=2
C=1
Chinese student responses to confidence-based MCQ test

Instances of C=3 (high) confidence increased as the question difficulty
decreased.

‘Difficult’ language comprehension test as some small grammatical
differences separated answers and distracters.
C = 3 (high)
30
No. of students
25
20
15
10
5
0
0.0
Harder
0.2
0.4
0.6
Question difficulty
0.8
1.0
Easier
Correlating confidence-based score and final exam (IELTS)
Achievement in test scores
Confidence-based score
IELTS exam

IELTS exam
0.91 (p=0.000)
insignificant
insignificant
-
only 7 students (of 31) improved on their % correct score with good
confidence judgement.
o

% correct
6 of these students were the highest test scorers in the class.
no apparent correlation between IELTS exam and confidence-based
test
o
IELTS exam measuring larger scope of competencies
Student opinions of the test
It is appropriate for a test to measure the
confidence I have in my knowledge
18
Strongly agree
Agree
Neutral
Disagree
Strongly disagree
No. of students
1.
2.
3.
4.
5.
15
12
9
6
3
0
1
2
3
4
5
Relative agreem ent
I sometimes took a risk by selecting a higher
level of confidence than I really felt
I thought the confidence-based test gave me
valuable feedback about my knowledge
18
10
No. of students
No. of students
12
8
6
4
2
15
12
9
6
3
0
0
1
2
3
Relative agreement
4
5
1
2
3
Relative agreement
4
5
Student comments
“It is a real good interesting test and I can receive different aspects of my
knowledge. However, it will cost lots of time to finish this exam. As a result, I
suggest we can do the exam at a regular time such an once a week.”
“It is an interesting test and I would like to do it at my university. I think that is
obviously fair for students.”
“It is interesting and a little difficult to understand the result. I think I should get
a higher confidence-based score because I chose (c3) nine times, and the
number answered correct is 7. Maybe I have not understood this well.”
“I think it is quite interesting and helpful.it is also a good way to show me the
link between confidence and academic study.”
Academic self-concept and confidence judgement

Improving academic self-concept is often posited as mediating other
desirable attributes such as persistence on academic tasks, motivation,
and self-efficacy.

Is good confidence judgement a measure and a positive mediator
o
o

The locus of control refers to how people explain events that happen to
themselves and others.
o
o
o

an improved measure than the standard % correct score
informed interventions for enhancing academic self-concept can make use
of recent advances in theory Craven (1996)
internal locus - guided by his/her personal decisions and efforts.
external locus - guided by fate, luck, or other external circumstances
a more 'surface' approach to learning is associated with an external locus of
control, Watkins & Biggs (1996)
Scoring scheme & feedback helps to equate internal expectation
(probability answered correctly) with external performance (score)
o
desirable attribute that is associated with an internal locus of control
A model for self-concept

Posited to be multifaceted and hierarchical in nature
Shavelson, Hubner and Stanton (1976)

Numerous studies now based or extended from this model
Hattie (1992)
Self-concept and academic achievement
From the literature…


Support for the multifaceted nature….
o
academic self-concept and academic achievement strongly correlated
Marsh, Byrne and Shavelson (1988), Hattie (1992)
o
little correlation between social self-concepts and academic achievement
Song & Hattie (1984), Waugh (1999)
Support for the hierarchical nature less clear cut….
o
more support for a hierarchical model for adolescents and
a unitary structure for younger children Hattie (1992)
o
achievement and academic self-concept deserve special interpretations at
the level of specific subjects Marsh (1990)
o
standard self-description questionnaires based on the Shavelson model for
preadolescents, adolescents and late adolescents Marsh (1992a, b, c)
Measuring academic self-concept
1. Your perceptions of your ability and achievement
adapted from Marsh (1992), Song & Hattie (1984) and Waugh (2001)
indicate your relative agreement with each of the 20 statements:
All the time, or nearly all the time
Most of the time
Some of the time
None of the time, or almost none of the time
1st 10
general university experience
eg. I am capable of getting good marks at university
I am proud of my achievements at university
2nd 10
experience of English Language classes only
eg. I am sure of myself in English Language classes
I am achieving at a high level in English Language classes
Questionnaire feedback
For your general university experience, your responses total 18 points (30 maximum).
For your English Language classes experience, your responses total 19 points (30
maximum).
Measuring academic self-concept
2. Your attitudes to academic life
locus of control inventory adapted from Trice (1985)
select True or False to indicate your agreement with each statement:
eg. My academic marks most often reflect the effort I put into classes.
I came to university because it was expected of me
Questionnaire feedback
The closer your score is to 0, the more you
believe that your academic experience is
determined by your own abilities, efforts and
attitudes (internal factors).
The closer your score is to 28, the more you
believe that your academic experience is
determined by external factors such as
chance, other people, fate or luck.
Correlating academic self-concept, confidence and achievement
Questionnaire responses
Perception of ability & achievement
EFL
Locus of control
General
EFL
0.61 (p=0.000)
-
- 0.54 (p=0.002)
- 0.61 (p=0.000)

perception of ability and achievement scores (general and EFL
scales), increase in tandem.

as locus scores become increasingly internal (towards 0),
perception of ability and achievement scores increase accordingly
Gender observations in the questionnaire responses
The responses indicating the most internal
of locus of control scores, were male
students.
Perception of ability & achievement - Generally
10
No. of students
Scores from questionnaire:
Females tended to respond with lower
scores than males for the perception
of ability and achievement (general
and EFL) scales.
8
6
Male
4
Female
2
0
10-14
15-19
20-24
25-30
Range of scores (max is 30)
Locus of control
Perception of ability & achievement - EFL
6
8
6
Male
4
Female
2
No. of students
No. of students
10
5
4
3
Male
2
Female
1
0
0
0-7
10-14
15-19
20-24
Range of scores (max is 30)
25-30
8-11
12-14
15-18
Range of scores
(0 <------ internal : external ------> 28)
19-22
Correlating academic self-concept, confidence and achievement
Questionnaire response v. IELTS exam
Perception of ability & achievement

Locus of controls
General
EFL
insignificant
insignificant
0.43 (p=0.024)
Student’s perception of ability and achievement in EFL shows a
moderate correlation with their IELTS exam result.
o
in line with academic self-concept and achievement research – strongly
subject based: support for the hierarchical structure.

No significant gender differences in the confidence-based test scores.

Females attained higher scores in the IELTS exam.
Correlating academic self-concept, confidence and achievement
Questionnaire responses v. confidence profile
Perception of
EFL ability
Perception of
General ability
at C=3 (high)
0.35 (p=0.052)
insignificant
at C=2 (mid)
insignificant
-0.37 (p=0.038)
at C=1 (low)
Locus of
control
insignificant
insignificant

Perception of EFL ability & achievement scores are moderately
correlated with the number of correct answers at high confidence.

These figures may indicate a trend, but may be noise
o
one confidence-based test is insufficient to identify genuine correlations
Conclusions

Indications that Chinese students perform in a similar manner to
confidence-based assessment as their UK counterparts.
o

Provide a scheduled series of confidence-based tests for Chinese students
on the English Foundation programme
Examine academic self-concept and confidence-based assessment
o
o
o
potentially valuable role for the enhancement of academic self-concept, and
the development of other positive academic behaviours
Effective with younger learners who may enjoy a game perspective
The scoring scheme and test feedback seems applicable to recent research
that posits a reciprocal relationship between academic self-concept and
academic achievement (Marsh, 2003).
Acknowledgments



Lecturing staff in the Centre for Business Languages, Napier University
Nicola Beasley (Napier University) for initial software development
Tony Gardner-Medwin (UCL) for valued discussions and analysis
Bibliography
1.
Brookover, W.B., Thomas, S. and Paterson A. (1964) Self-concept of ability and
school achievement, Sociology of Education, 37, pp. 271-279.
2.
Craven, R. (1996) Enhancing Academic Self-Concept: A Large-Scale Longitudinal
Study in an Educational Setting, PhD thesis, University of Sydney
3.
Davies, P. (2002) There’s no confidence in multiple-choice testing, Proceedings
of the 6th International CAA conference, Loughborough, pp. 119-130.
4.
Echternacht, G.J. (1972) The use of confidence testing in objective tests, Review
of Educational Research, 42:2, pp. 217-237.
5.
Gardner-Medwin, A.R. and Gahan M. (2003) Formative and summative
confidence-based assessment, Proceedings of the 7th International CAA
conference, Loughborough, pp. 147-155.
6.
Gourlay, L. (2004) Crossing Boundaries: A Case Study Masters Level Chinese
Students, LTSN (In Press)
7.
Hassmen, P. and Hunt, D.P. (1994) Human self-assessment in multiple-choice
testing, Journal of Educational Measurement, 31, pp. 149-160.
Bibliography
8.
Hattie, J. (1992) Self-Concept, Lawrence Erlbaum Associates, New Jersey
9.
Khan, K.S., Davies, D.A. and Gupta, J.K. (2001) Formative self assessment
using multiple true-false questions on the Internet: feedback according to
confidence about correct knowledge. Medical Teacher, 23, pp. 158-163.
10.
Marsh, H.W (1992a) Self-Description Questionnaire (SDQ)III: A theoretical and
empirical basis for the measurement of multiple dimensions of preadolescent selfconcept: A test manual and research monograph, MacArthur, New South Wales,
Australia, University of Western Sydney, Faculty of Education.
11.
Marsh H.W., Byrne, B.M. and Shavelson, R.J. (1988) A multifaceted academic
self-concept; Its hierarchical structure and its relation to academic achievement.
Journal of Educational Psychology, 80, 366-380.
12.
Marsh H.W. (1990b) The structure of academic self-concept: The
Marsh/Shavelson model, Journal of Educational Psychology, 82, 623-636.
13.
Marsh, H.W. (2003) A reciprocal effects model of the causal ordering of academic
self-concept and achievement, Paper presented at NZARE AARE, Auckland, New
Zealand.
Bibliography
14.
Prendergast, M.A. and Binder D.M. (1975) Relationships of selected self-concept
and academic achievement measures, Measurement and Evaluation Guidance,
8, pp. 92-95.
14.
Song, I.S. and Hattie, J. (1984) Home environment, self-concept and academic
achievement measures, Journal of Educational Psychology, 76, pp. 1269-1281.
15.
Watkins, D.A. and Biggs, J.B. (Eds, 1996) The Chinese Learner: Cultural,
Psychological and Contextual Influences, CERC & ACER, (ISBN 0 86431 182 6).
16.
Waugh, R.F. (2001) Measuring ideal and real self-concept on the same scale,
based on a multifaceted, hierarchical model of self-concept, Educational and
Psychological Measurement, 61:1, pp. 85-101.