Midterm report
NA3C0005
Comparison between “The Development and Validation of a Listening
Practice Strategy Questionnaire” and “The Metacognitive Awareness
Listening Questionnaire: Development and Validation”
“The Development and Validation of a Listening Practice Strategy Questionnaire”
(hereafter called article 1) was written by Hui-Ming Lucy Lee, Mong-Na Lo Huang,
and Wan-Ping Hung and published in the fall, 2010. The main purposes of article 1
were to develop a valid and reliable questionnaire to assess EFL students’ listening
practice strategies and to investigate listeners’ use of practice outside of class. The
other article “The Metacognitive Awareness Listening Questionnaire: Development
and Validation” (hereafter called article 2) was written by Vandergrift et al. and
published in September, 2006. The main purposes of article 2 were to develop a
valid listening questionnaire to assess second language (L2) listeners’ metacognitive
awareness and to perceive use of strategies while listening to oral texts. As reading
the titles of these 2 articles and comparing the purposes of them, there should be a
highly relevant between them. Obviously, the authors all tried to develop a valid
and reliable questionnaire to assess listening comprehension strategies.
Nevertheless, they constructed questionnaires in terms of different theories. The
design of the questionnaire in article 1 was based on theoretical models of listening
comprehension theory and practice theory. On the other hand, the design of the
questionnaire in article 2 was based on theoretical model of metacognition. Based
on their similarities, I presume that article 1 should be derived from or inspired by
article 2.
Quality of LR Comparison
In order to construct a valid and reliable questionnaire, the authors adopted theories
to develop their questionnaires. The parts of their literature review focused on
relevant theories about the directions of assessment. They also found there is a
insufficient for some assessment instruments.
In article 1, the authors pointed out that it is unclear how learners practice listening
comprehension and how often they practice listening. Moreover, there is a lack of a
reliable and valid survey instrument to assess listeners’ practice behaviors. First,
the researchers reviewed the listening comprehension theories. In this part, they
cited Lundsteen’s definition of listening. Next, they quoted Anderson’s three stages
of language comprehension: perceptual processing, parsing, and utilization.
Through analysis of comprehension process, it showed listeners processed phrases
and sentences immediately for meaningful representations.
Furthermore, they
illustrated three major kinds of listening processing: bottom-up, top-down, and
interactive processing. Second, they reviewed the practice theories. In order to
develop an expert performance, learners used extensive practice to acquire skills.
According to Anderson, there are three stages in skill acquisition: the cognitive, the
associative, and the autonomous stages. Next, they reviewed Newell and
Rosenbloom’s power law of practice, or log-log linear learning law, performance
improves as a power function of practice. In other view of expert performance,
deliberate practice is the most effective way of improving the current level of
performance. Practice is a learning strategy in SLA and second language listening
comprehension. From a conceptual standpoint, practice is classified as a cognitive
strategy. From an information-processing perspective, the routinization of cognitive
skills is characterized as a function of practice. Third, they exemplified some
empirical research on practice. The first one is that Ericsson et al. compared current
and past levels of deliberate practice in three main groups of adult violinists to
examine the effects of deliberate practice on performance. The second one is that
a seminal study was conducted by Bialystok to measure language learning strategies
and the effects of strategy use on language achievement. The third one is that Yang
reported a study to investigate 505 Taiwanese EFL university students’ beliefs about
language learning and their use of language learning strategies. The critical effects
of practice on expert performance have been supported by the reviewed empirical
research in the area of cognitive psychology. However, the authors found that in
the field of EFL listening research, there is a lack of empirical research on the role of
practice in the acquisition of EFL listening skill. They attempted to explore EFL
listeners’ practice behavior by developing a valid and reliable questionnaire and by
measuring the frequency of their listening practice strategy use.
In article 2, this questionnaire is designed to investigate the metacognitive awareness
listening of a second language learner. Therefore, the authors compared the
literatures and used Flavell’s model of metacognitive knowledge. Three categories
of metacognitive knowledge, person knowledge, task knowledge, strategy knowledge,
represented key components in the process of cognitive self-appraisal. In the field
of education and L2 learning, metacognition plays an important role in enhancing
thinking and comprehension. Learners with high degrees of metacognitive
awareness are better at processing and storing new information, finding the best
ways to practice and reinforce what they have learned. Next, the authors reviewed
the research of metacognitive knowledge about listening. To elicit learners’
metacognitive knowledge about listening, various procedures have been used, most
commonly diaries, interviews, and questionnaires. The hypothesized relationships
among metacognition, motivation, and listening proficiency were further explored by
Vandergrift. However, none of these earlier self-report measures had been
subjected to rigorous validation procedures.
Research Methods Comparison
Both research had recruited a large sample of respondents. Article 1 had recruited
306 university students in Taiwan. Article 2 had recruited 1,478 university and high
school students in various countries. In order to validate the questionnaires, they
administered a pilot study and a formal study. During data analysis, they used an
exploratory factor analysis and a confirmatory factor analysis to test the validity and
reliability of the factors. However, in article 1, the authors focused on analysis the
items of each factor. In article 2, the authors focused on analysis how many factors
model is suitable.
Participants
In article 1, 306 university students in Taiwan were recruited. The development of
the questionnaire involved 3 phases of a pilot study and a formal study. Since this
study was conducted in Taiwan, the authors used 2 language versions to verify the
validity of the questionnaire. This research method is more robust than the study in
article 2. During the first phase of the pilot study, 16 English major students were
recruited for exploratory interviews. The second phase of the pilot study attempted
to assess the clarity between English version and Chinese version of the LPSQ. 10
students from the previous interview were chosen. The third phase of the pilot
study, 94 students were recruited and encouraged to ask questions about the
questionnaire during the administration process. The formal study was conducted
in 2001 (Is it a mistype? Or this article was published in 2010, using the result in
2001 seems far away from the reality.) Another question is the authors did not
mention the total participants in formal students. If they recruited overlapped
students in this research, is it adequate for the research result?
In article 2, the draft version of the MALQ was field tested with 966 respondents in
various countries, including Canada, Singapore, and The Netherlands. During this
phase, instructors administer the questionnaire after the class of listening activity
and all respondents completed the questionnaire in English. Respondents were
encouraged to comment on any item that was unclear to them. However, not all of
the respondents’ mother languages are English. Is it adequate to use English
version questionnaire in this phase? Next, a confirmatory phase of the study was
conducted with another large sample of respondents. In order to test whether
there was a relationship between the listening behaviors reported in the MALQ and
actual listening performance, a listening comprehension test was administered.
This resulted in a reduced, but still relatively large sample of 341 students, including
115 EFL learners from Iran and 236 FSL learners from Canada. The questionnaire
used in confirmatory phase was translated for the Iranian participants.
Data Collection and Analysis
In article 1, all the quantitative pilot data analyses were computed using version 16.0
of SPSS. First, an intraclass correlation was employed to examine the relationships
between the English and Chinese versions of the questionnaires. Second, the
Pearson product-moment correlation was employed to assess the validity of the LPSQ.
Third, Cronbach’s alpha was used to examine the LPSQ’s internal consistency
reliability. Exploratory factor analysis using principal component analysis (PCA) with
oblique Promax rotation with Kaiser Normalization was performed using SPSS to
identify underlying constructs for the fifty-one practice strategy items on the LPSQ
for formal study. One unsatisfactory variable was removed at a time. First, a
corrected item-total correlation analysis was used to remove items with correlation
coefficients of 0.1 or less. Second, items with main loadings had absolute values of
less than∣.30∣were deleted. Third, crossloading items with differences in factor
loading of less than 0.1 were dropped. Cronbach’s alpha was used to examine the
reliability of the scale. Finally, descriptive statistics were computed using SPSS to
summarize the students’ responses to the questionnaires.
In article 2, during exploratory factor analysis, an investigation of the MALQ factor
structure was conducted through SPSS to determine whether there was empirical
support for separate factors pertaining to metacognitive awareness of L2 listening
and to identify any items that might be removed from the questionnaire. To
estimate the maximum number of factors that might be interesting began with a
principal component analysis (PCA) on the 51 items of the first version of the MALQ.
For reasons of parsimony and meaningfulness, the scree test of eigenvalues plotted
against factors was examined. Principal-axis factor analysis (PFA) was chosen to find
the optimal number of factors. Promax rotation with Kaiser Normalization allows
for intercorrelation among factors. The 51 items then underwent two PFA runs
specifying five and six factors. A five-factor solution met the goals of interpretability,
and was preferable in terms of comprehensibility. Next, to examine whether there
were any unsatisfactory items. All items were evaluated for any observed low
factor loadings, complex loadings, and/or reduced reliabilities. Using an iterative PFA
analyses with Promax rotation, 39 items yielded five factors. A confirmatory factor
analysis was conducted based on data collected from the second sample, using
LISREL 8.72. Based on the results of the exploratory factor analysis, the authors
thought there should be a further comparison of the goodness-if-fit among the four-,
five-, and six-factor model. They employed maximum likelihood (ML) estimation.
Several widely accepted goodness-of-fit indexes were computed, including CFI, TLI,
PNFI, RMSEA. Based on the goodness-of-fit indexes of the confirmatory factor
analysis (CFA), combined with the results of the exploratory factor analysis, the
five-factor model was selected as the model of choice. Finally, in order to establish
a relationship between the listening behaviors reported in the MALQ and actual
listening behaviors, the authors correlated the data from the MALQ with the
participants’ scores on a listening comprehension test. The correlation coefficient
confirmed the relationship between listening comprehension ability and
metacognitive awareness of the processes underlying successful L2 listening. To
further reinforce the strength of this relationship, they regressed the scores of
listening test on the MALQ scores. That indicated that about 13% of the variance in
listening performance could be explained by metacognition.
Findings Comparison
According to the data analysis and discussion, the authors of 2 articles believed they
constructed valid and reliable questionnaires as useful tools to assess their desired
focal points.
In article 1, the authors constructed a listening practice strategy questionnaire.
There were 5 practice factors: problem-solving, language processing, deliberate
practice, aural immersion, and English-comprehension practice in this questionnaire.
The questionnaire used a 10-point scale. The five-factor model accounted for 46.9%
of the total variance. A Cronbach’s alpha of .92 indicated high reliability.
Regarding the frequency of strategy use, the aural immersion factor was sometimes
used and ranked first among the five factors, followed by the problem-solving factor,
which was also sometimes used. English-comprehension practice was least used
among the five factors. Finally, the adequate validity and reliability of the
questionnaire could make it a useful tool for listening strategy training.
In article 2, the authors constructed a metacognitive awareness listening
questionnaire. There were 5 factors in this questionnaire. However, they were
different from factors in article 1. The 5 factors were problem-solving, planning and
evaluation, translation, person knowledge, and directed attention. Not only was
the significant relationship between the behaviors that listeners report using MALQ
scores and actual listening performance listening test scores, but the regression
analysis further verified a meaningful relationship between metacognition and
listening comprehension success. Correlation between the scores accounts for
about 13% of the variability in listening performance. However, this questionnaire
is a self-report instrument.
Instructors should consider the MALQ as one source of
information about the current level of student awareness of L2 listening and strategic
competence in processing L2 oral texts. Using the MALQ can enable L2 learners to
become self-regulated listeners who can better capitalize on the aural input. By
increasing their awareness of the listening process, students can learn how to
become better listeners which will enable them to learn another language more
quickly and more efficiently.