3 Standards for Sound Applied
Linguistics Research

The qual–quant not only provides an interactive continuum useful
for characterizing applied linguistics research but can also serve
as a basis for understanding the standards that researchers use in
judging the soundness of such research. This discussion will again
focus on primary research, leaving secondary research out because
it involves a separate set of issues. Because I consistently started
with the qualitative-exploratory end of the continuum in the
previous section, out of fairness, I will start with the quantitativeexperimental end in this section.
3.1 The quantitative-experimental end of
the continuum


Researchers at the quantitative-experimental end of the
continuum value the concepts of reliability, replicability,
validity, and generalizability.
Generally speaking, those are the standards quantitative
researchers hold up in judging the soundness of their
research.
3.1.1 Reliability



The standard of reliability in quantitative research requires
researchers to demonstrate both (1) the reliability of the
instruments used in their studies and (2) the reliability of the
results of their studies.
Reliability of instruments is concerned with the degree to which
the results of a questionnaire, test, or other measuring instrument
are consistent. Addressing this issue typically means answering the
question: To what degree would the results be the same if the
instrument were administered repeatedly?
Similarly, the reliability of the results of a study is concerned with
the degree to which the results would be likely to reappear if the
study were replicated under the same conditions.
3.1.2 Replicability




The standard of replicability in quantitative research requires
researchers to provide enough information about a study to allow
other researchers to replicate or repeat the study exactly as it
was originally conducted.
The replicability of a quantitative study can be improved by thorough
and complete descriptions of: (1) the participants in the study and
how they were selected,
(2) the instruments used in the study as well as arguments for their
reliability and validity, and
(3) the procedures followed in collecting the data, scoring or coding
the instruments, and analyzing the results (see Brown, 1988, ch. 5 for a
more complete discussion of what should be included to make a
study adequately replicable).
3.1.3 Validity



The standard of validity in quantitative research requires
researchers to demonstrate both internal and external validity.
Internal validity is the degree to which the results of a study can be
accurately interpreted as meaning what they appear to mean.
External validity is the degree to which the results of a study are
contrived or artificial, or put another way, the degree to which the
results apply to the outside world. Listed below are a number of
issues identified by Campbell and Stanley (1963) as potential threats
to the internal and external validity of a study:

The internal and external validity of a quantitative study can be
improved by consciously guarding against all of these threats when
planning and conducting research (for more on controlling these
potential threats, see Campbell & Stanley, 1963; Brown, 1988, 1997; or
Hatch & Lazaraton, 1991).
3.1.4 Generalizability


As mentioned above, the standard of generalizability in
quantitative research requires researchers to show the degree to
which the results of a study can justifiably be generalized, or
applied, to a larger population or to other similar groups.
A study can be extremely well designed, controlled, and internally
valid, but lack external validity. The problem is that controls,
when introduced, may make the study artificial and thus limit the
external validity.
3.2 The qualitative-exploratory end of the
continuum


Researchers at the qualitative-exploratory end of the continuum
value the concepts of dependability, confirmability, credibility, and
transferability.
Generally speaking, those are the standards qualitative
researchers hold up in judging the soundness of their research.
3.2.1 Dependability




The standard of dependability in qualitative research requires that
researchers account for (1) any shifting conditions directly related to
the people and things they are studying and
(2) any modifications they have made in the design of their study as
it has progressed.
The purpose of such accounting is to help researchers and their
readers get a more exact understanding of the context.
Dependability is roughly analogous to the concept of reliability
(described above) in quantitative studies. The dependability of a
qualitative study can be improved by using such techniques as
stepwise replications, overlapping methods, and/or inquiry audits (for
more on these concepts, see Davis, 1992, 1995; or Brown, 2001).
3.2.2 Confirmability




The standard of confirmability in qualitative research requires that
researchers fully reveal the data they are basing their interpretations
on, or at least make those data available.
The point is that, whether or not anybody actually takes the
researchers up on it, they should make their data available so that
other researchers could examine them and confirm, reject, or
modify the original interpretations.
The confirmability of a qualitative study is approximately analogous
to the concept of replicability (described above) in quantitative
studies.
Confirmability can be improved by using audit trails (as described
briefly in Davis, 1992, 1995; Brown, 2001; or in more detail in Lincoln &
Guba, 1985).
3.2.3 Credibility



The standard of credibility in qualitative research requires researchers
to show that they maximized the accuracy of their definitions and
their characterizations of the people or things under investigation –
especially as the various participants in the study judged those
interpretations.
Credibility is more or less analogous to the concept of internal validity
(described above) in quantitative studies.
The credibility of qualitative studies can be enhanced by using
such techniques as member checking, negative case analysis, peer
debriefing, persistent observations, prolonged engagement,
referential analysis, and/or triangulation (for more on these
techniques, see Davis, 1992, 1995; or Brown, 2001).
3.2.4 Transferability



The standard of transferability in qualitative research requires
researchers to describe the research design, context, and
conditions so well that the readers can decide for themselves if
the interpretations apply to another context with which they are
familiar.
Transferability in qualitative research is approximately analogous to
the concept of generalizability (described above) in quantitative
studies.
Transferability can be enhanced by using thick description (for more on
this notion, see Davis, 1992, 1995; Lazaraton, 1995; or Brown, 2001).
3.3 How interactive are the standards of
sound research?



Figure 19.6 is my attempt to represent the interactive relationships
among the various standards continua. Recall that earlier, I
defined interactive as all possible combinations of the points
along the continua.
In other words, for each of the standards continua in Figure 19.6,
a study may need to focus entirely on the standard at the
qualitative-exploratory end of the continuum or the standard at the
quantitative-experimental end.
In other cases, researchers may need to balance their concern for
standards at both ends of the continuum to varying degrees.
3.4 Ethical considerations



General social sciences research ethics have been discussed from
many points of view (for an overview of this work, see Kimmel, 1988).
Periodically over the years various national and international
organizations have even attempted to provide guidelines for their
memberships (e.g., the American Psychological Association, which
has provided various sets of guidelines for the ethical conduct
of research: 1953, 1982, 1994).
Kimmel (1988) discussed some of the sorts of ethical problems that
arise in social sciences research:

Instead of dwelling on the potential problems that unethical behavior
can cause in research, I would prefer to simply delineate some of the
steps that can be taken to avoid ethical pitfalls in applied linguistics
research anywhere along the qual–quant continuum. Some of the
most important ethical and professional responsibilities fall into
three categories (adapted from Brown, 1997): participant issues,
analysis responsibilities, and concerns for the audience of a study:



Since ethics is an area where all research methods and
techniques come together and tend to agree, I will end here. However,
I would like to mention one further set of considerations that is seldom
listed in discussions of research ethics.
In my view, all researchers in applied linguistics have two overriding
ethical responsibilities: (1) to continue reading, learning, and
growing as researchers in order to better serve the field, and
(2) to design research that is effective and fits well into the
particular institutional contexts involved by selecting those
characteristics and standards along the qual–quant continuum that will
best interact to provide systematic and principled answers to the
many important questions that remain to be answered in applied
linguistics.
Thanks!