NURWAHIDAH BINTI JAMAL
M20201000289
GRU60104: RESEARCH METHODS IN EDUCATION
SEMESTER 1
ASSIGNMENT 1: CRITICAL REVIEW OF A RESEARCH ARTICLE
1. Description of the research design
1.0. Introduction
“Native and Non-native Teachers of L2 Pronunciation: Effects on Learner
Performance” by John M. Levis, Sinem Sonsaat, Stephanie Link and Taylor Anne Bariusso, as
published in TESOL Quarterly Journal, Volume 50, No. 4 in December 2016 investigated what
the effects of teachers’ first language have on the ratings of change in accentedness and
comprehensibility among students.
This essay will attempt to briefly describe the research design employed in the study,
the sampling, the methods of data collection and analysis, as well as the findings of the
aforementioned article, before discussing the strengths and weaknesses of the conducted study
and the article.
1.1. Research design
For the purpose of investigating the effectiveness of the native-English-speaking teacher
(NEST) and the non-native- English-speaking teacher (NNEST) in teaching pronunciation and
the perceptions towards them, the authors have employed an embedded mixed-methods design.
According to Creswell (2015), mixed-methods researches involve the gathering and integrating
of both quantitative and qualitative data, and later combining the strengths of both datasets to
draw the interpretations of the matter at hand. As for this study, it consisted of both quantitative
and qualitative data collection, whereby the qualitative data acted as the supplement to further
explain the quantitative results.
1
NURWAHIDAH BINTI JAMAL
M20201000289
GRU60104: RESEARCH METHODS IN EDUCATION
SEMESTER 1
ASSIGNMENT 1: CRITICAL REVIEW OF A RESEARCH ARTICLE
The quantitative analysis of the study was evident through the collection of pre-test and
post-test speech samples. Prior to the classes on pronunciation by the NEST and NNEST, the
students’ read and spontaneous speech samples were recorded during an interview session with
the first author. Similar speech samples were also collected after the seven-weeks courses have
ended. These samples were later used as the basis for the rating tasks, whereby the rating tasks
sought to answer the first research question of how the students’ accentedness and
comprehensibility are similar or different, depending on whether they are taught by either the
NEST or NNEST. Munro and Derwing’s 1-9 Likert scales were used in the rating tasks, for
assessing both accentedness and comprehensibility of the read and spontaneous speech
samples.
Meanwhile, to answer the second research question of how the students evaluate their
teacher’s
effectiveness
as a pronunciation teacher, qualitative data was collected. It was
derived from the interviews conducted at the time of both pre- and post- tests were taking place,
as well as through the questionnaires administered.
1.2. Samples/ Participants/ Respondents
There were three main parties who were involved in the study, namely ‘the teachers’, ‘the
students’ and ‘the raters’.
Both teachers were female students who were undertaking their PhD studies in applied
linguistics and technology, with different first language backgrounds. The NEST was a native
speaker of American English, while the NNEST’s native language is Turkish. In terms of their
accentedness and comprehensibility, both were rated at a high and top level of the Likert scale,
although the NNEST demonstrated a considerably lower rating in comparison to the NEST.
They also possessed a few similar traits and characteristics; both had similar educational
2
NURWAHIDAH BINTI JAMAL
M20201000289
GRU60104: RESEARCH METHODS IN EDUCATION
SEMESTER 1
ASSIGNMENT 1: CRITICAL REVIEW OF A RESEARCH ARTICLE
background, experience in teaching pronunciation, age, physical appearance, as well as
personality.
A total number of 18 students made it to the final recruitment of the sample for the study,
whereby 10 were put in the NEST’s class while the remaining was in the NNEST’s class. All
of them shared similar traits crucial to the research, which were various language backgrounds
and their proficiency in spoken English was of intermediate to advanced level.
67 selected raters were recruited from among first- and second-year college students who
had only started experiencing learning a foreign language. They also had minimal contact with
the native speakers of those foreign languages, making them inexperienced with regard to
foreign-accented English because they were exposed to foreign-accented speeches as much as
other average listeners. Since there is a negative relationship between the age of learning and
pronunciation scores (Derwing and Munro, 2015, as cited in Levis et al., 2016), it made sense
for the authors to employ inexperienced raters for the rating task. Another important similarity
among the raters was all of them were native English speakers. This is to control the
consistency of rating of the accented speeches.
1.3. Methods of data collection
The collection of quantitative data was derived from the read and spontaneous speech
samples taken during the pre- and post-test which went through the stimuli preparation, before
being rated in the rating tasks.
For the collection of the read speech samples, the students were instructed to read two
different sets of 25 sentences twice, one set each for the pre- and post-test. The students also
provided spontaneous speech samples through an informational interview and a narrative task.
For every student, only grammatically correct sentences were extracted from the interviews
3
NURWAHIDAH BINTI JAMAL
M20201000289
GRU60104: RESEARCH METHODS IN EDUCATION
SEMESTER 1
ASSIGNMENT 1: CRITICAL REVIEW OF A RESEARCH ARTICLE
and narrative tasks, with only four sentences were randomly selected to be included in the final
samples of spontaneous speech.
Both the read and spontaneous speech samples were randomized into 8 blocks, ensuring
that there is no overlapping of voice and sentences between the blocks. Care was given to
ensure the quality of the sound as to not hamper the process of rating due to technical problems.
As for the rating tasks, the raters were formed into two groups of 33 and 34 people, each
responsible for rating either the read or spontaneous speech samples. All raters went through
the process of listening to the total number of 206 sentences from either the read or spontaneous
speech samples twice, first to rate the accentedness and later the comprehensibility, or vice
versa.
Meanwhile, the collection of qualitative data was derived from the interview conducted
during the collection of speech samples for both pre- and post-tests. In addition to the interview,
the students also completed a questionnaire about their NEST or NNEST teacher.
1.4. Methods of data analysis
The analysis of quantitative data to answer the first research question involved running a
few tests such as ANOVA, Levene test and descriptive statistics. The inter-rater reliability was
put to test using iota and the result indicated a moderate level of inter-rater agreement for all
for rating categories.
The qualitative data gathered through the interviews was analysed using Nvivo10 and
coded using inductive coding methods. They ran a second coder to attest the reliability of the
coding procedures and to compare interpretations of the data as well as ensuring inter-coder
reliability. Coding queries were run to extract data related to students who improved
tremendously in all 4 categories (read speech accentedness, read speech comprehensibility,
4
NURWAHIDAH BINTI JAMAL
M20201000289
GRU60104: RESEARCH METHODS IN EDUCATION
SEMESTER 1
ASSIGNMENT 1: CRITICAL REVIEW OF A RESEARCH ARTICLE
spontaneous speech accentedness and spontaneous speech comprehensibility). The data
gathered from the questionnaire was used to explain the findings from the interview.
1.5. Findings
The rating for read speech revealed that there was no significant difference between the
two classes taught by NEST and NNEST in terms of the students’ accentedness and
comprehensibility. Similar result was shown for the spontaneous speech.
It was also found that among the students, some showed improvement in all four categories
being tested, while some did not show improvement in any category. Therefore, the students’
responses during their interviews were analysed qualitatively to figure out why such divergence
occurred. Those who improved expressed concern about being misunderstood due to their
pronunciation. Not being proficient enough in pronunciation is a common concern shared by
those who made improvements and those who did not. It was also discovered that the main key
factor that differentiate the two group of students was the absence of practice of using English
at home, whereby those who improved might use English outside of class context and those
who did not improve only use English during lessons. The level of confidence between the two
groups was also at disparity, with one of those who did not improve thought that the sevenweek course was not a sufficient timeframe for her to improve her pronunciation.
Although the result of the questionnaire demonstrated that the students rated both the NEST
and NNEST teachers equally high, the finding from the interviews still showed bias towards
the NEST as the more preferred pronunciation teacher.
5
NURWAHIDAH BINTI JAMAL
M20201000289
GRU60104: RESEARCH METHODS IN EDUCATION
SEMESTER 1
ASSIGNMENT 1: CRITICAL REVIEW OF A RESEARCH ARTICLE
2. Strengths and weakness of the study
The first strength of the study is that the authors chose to employ a mixed-methods
design, whereby the qualities of both quantitative and qualitative designs are integrated. It was
clarified in the article that the qualitative data gathered from the research acted as the
supplement to the quantitative data. It is worth noting that the supplemental role of the
qualitative data means that it adds details to the quantitative data (Bergman, 2011, as cited in
Cohen, Manion and Morrison, 2018), instead of making up for what the quantitative result
might lack in, due to the limitations of what a solely quantitative conducted research would
demonstrate. As mentioned by Creswell (2015), a quantitative research does not allow for
further probe into individual perspectives and personal stories, while a qualitative research
restricts generalisation. Therefore, it was a sensible decision to employ a mixed-methods
research because they managed to obtain a more comprehensive perspective about the rating
results, while “quantitizing” (Frankel, Wallen and Hyun, 2012) the students’ experience
working through the pronunciation class.
Creswell and Creswell (2018) also explained that the employment of embedded or
convergent mixed-methods design provides room for exploring different types of questions and
opportunities for one database to explain the other. This can be seen through how the qualitative
data is used to further explain the quantitative data, especially in the case where the students’
preference for a native teacher as the pronunciation teacher contradicted with the equally high
individual ratings given to the teachers. To explain this, conducting an interview was the best
medium for the authors to get a clearer picture as to why such contradiction occurred. This is
as supported by Creswell and Plano Clark (2011) who mentioned that one of the reasons that
calls for the use of mixed-methods design is to answer research questions which results need
explanation.
6
NURWAHIDAH BINTI JAMAL
M20201000289
GRU60104: RESEARCH METHODS IN EDUCATION
SEMESTER 1
ASSIGNMENT 1: CRITICAL REVIEW OF A RESEARCH ARTICLE
The second strength of the study is the purposive sampling. The authors recruited potential
participants based on one imperative condition; they must be non-native English speakers of
intermediate to advanced level spoken English proficiency. Although purposive sampling is
typically associated with qualitative researches, Kumar (2019) stated that one should consider
purposive sampling when there is a need to describe a phenomenon, which in this case is to
describe the effect that teacher’s native language has on students’ accentedness and
comprehensibility. By limiting the participants to only those who fulfilled the criteria, the
authors managed to have control on the data collection to answer the research questions
regarding accentedness and comprehensibility of L2 learners.
Despite the immaculately conducted study, it also posed a weakness in terms of the
manageability of the controlling variable. According to Cohen et al. (2018), the main principle
in researching causes and effects is through controlling and isolating all the variables deemed
related to the particular study. This study examined the effects of teachers’ first language (L1)
have on students’ pronunciation, with the teachers’ L1 background as the primary variable.
Therefore it is understood as to why the authors had gone to the extent of specifically choosing
the NNEST because of her ethnically indistinct appearance; to avoid the teacher from being
stereotyped of her linguistic competence based on her ethnicity. However, having
acknowledged that such bias exists, it is questionable as to why the authors let the background
identity of the NNEST as a non-native English speaker, which in this case acted as the primary
variable being examined exposed to the students while appearance-wise, she looked just as
‘native’ as the NEST. It should be the single most important piece of information that should
be controlled and isolated from the students because of the non-native speaker fallacy.
7
NURWAHIDAH BINTI JAMAL
M20201000289
GRU60104: RESEARCH METHODS IN EDUCATION
SEMESTER 1
ASSIGNMENT 1: CRITICAL REVIEW OF A RESEARCH ARTICLE
Nonetheless, I find this study an interesting one as it might as well be replicated to the
Malaysian context since English is treated as the second language but taught by all NNESTs
alike. As the authors had suggested for future researches to look into whether the findings
would be different should the NNEST has a lower rating of comprehensibility and
accentedness, a similar research could be conducted among teachers from different ethnicity
backgrounds. I believe that native or non-native speaker fallacy is not applicable in this context
because the majority of teachers in Malaysia are NNESTs, but it would be interesting to see to
what extent does their differences in rating of accentedness and comprehensibility would have
effects on students’ pronunciation.
8
NURWAHIDAH BINTI JAMAL
M20201000289
GRU60104: RESEARCH METHODS IN EDUCATION
SEMESTER 1
ASSIGNMENT 1: CRITICAL REVIEW OF A RESEARCH ARTICLE
References:
Cohen, L., Manion, L. & Morrison, K. (2018). Research methods in education (8th ed.). Oxon:
Routledge.
Creswell, J. W. (2015). A concise introduction to mixed methods research. California: Sage
Publications Inc.
Creswell, J. W. & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed
methods approaches (5th ed.). Los Angeles: Sage Publications.
Creswell, J. & Plano Clark, V. (2011). Designing and conducting mixed methods research
(2nd ed.). California: Sage Publications.
Fraenkel, J. R., Wallen, N. E. & Hyun, H. H. (2012). How to design and evaluate research in
education (8th ed.). New York: McGraw-Hill.
Kumar, R. (2019). Research methodology: A step-by-step guide for beginners (5th ed.).
London: Sage Publications Inc.
Levis, J. M., Sonsaat, S., Link, S. & Barriuso, T. A. (2016). Native and Nonnative Teachers of
L2 Pronunciation: Effects on Learner Performance. TESOL Quarterly, 50, 4, p. 894931. Retrieved from: https://onlinelibrary.wiley.com/doi/epdf/10.1002/tesq.272
9