How Research Methods Textbooks
Mislead Psychologists About
Scientific Method
Brian Haig
University of Canterbury
5 April 2013
Introduction
1. Textbooks are the major source of formal learning in
undergraduate education, and they are prominent in
graduate education as well.
2. In psychology, students take more “methods” courses
than any other type of course (e.g., University of
Canterbury: Psyc206; Psyc344; Psyc460/Psyc461 are all
compulsory for graduate research students).
3. All undergraduate students take a generic second- or
third-year course on research methods.
4. There are dozens of textbooks available for adoption in
such courses.
4a A Sample of Textbooks
Gravetter, F. J., & Forzano, L-A. B. (2009). Research
methods for the behavioral sciences (3rd ed.). Belmont:
Wadsworth.
Leary, M. R. (2012). Introduction to behavioral research methods
(6th ed.). Boston: Allyn & Bacon.
Mitchell. M. L., & Jolley, J. M. (2013). Research design
explained (8th ed.). Belmont: Wadsworth.
White, T. L., & McBurney. D. H (2013). Research methods
(9th ed.). Belmont: Wadsworth.
5. Thomas Kuhn (1970) argued that textbooks in the
various sciences contribute in a major way to the
dogmatic initiation of learners into established research
traditions.
6. Psychology’s research methods textbooks are largely
uncritical in the presentation of their subject matter.
7. It follows that they cannot be a source for a genuine
education in research methods.
8. I identify a number of deficiencies in the way
psychology’s research methods textbooks deal with
scientific method.
9. Method is central to science, and scientific method is,
therefore, important. Most of what we know in science
is acquired through application of its methods.
The Typical Research Methods
Textbook
1. The standard textbook possesses a common structure
(psychology as a science, research questions, ethics,
writing, validity, experimental and nonexperimental
research, data exploration and inference, … .).
2. The first chapter (approx. 35 pages) is often dedicated
to scientific method, broadly understood.
3. Cameron Ellis and I did a content analysis of a
representative sample of 16 current undergraduate
research methods textbooks (Haig & Ellis, 2013).
4. All textbooks have been through multiple editions
(range=2-11; median=5.5; latest edition=10/16).
A Bill of Indictment
1. Most textbook authors are not methodologists by
speciality (only 6 of 28 in the sample, with
‘methodologist’ construed generously).
2. The most serious limitation of textbooks on research
methods is that the methods dealt with are not
informed by their accompanying methodology
(Methodology is the interdisciplinary field that studies
methods. It includes statistics, philosophy of science,
and cognitive science).
This results in an impoverished understanding of the
methods.
3. Rarely are methodological references on scientific
method provided.
i) few citations of professional methodologists and
their work (including philosophers of science)
ii) heavy reliance on secondary sources (when citations
provided). Example: Stanovich (1998), How to think
straight about psychology (5th ed.)
iii) methodological points about scientific method often
made by appealing to substantive research
(particularly in social psychology), which is
referenced much more than methodological
research.
4. The changes in successive editions of books are often
minimal.
a) revisions are driven more by attempts to capture,
or retain, market share than present new
methodological developments.
b) publishers encourage authors to produce new
editions regularly in order to encourage sales of new
books (the “problem” of second-hand book sales).
Theories of Scientific Method Ignored
An Argument:
P1 Credentialed theories are the major vehicles of
scientific knowledge.
P2 This holds for method as well as matter.
P3 Major theories of scientific method are not
presented in textbooks.
Cl Texts books do not present genuine knowledge about
scientific method.
This state of affairs is both embarrassing and
unacceptable.
Major Theories of Scientific Method
1. Inductive Method
- 10/16 texts mention induction, but:
- Most characterize inductive inference without relating it to
inductive method (e.g., radical behaviorist method,
Bayesian confirmation theory).
- Induction characterized as enumerative (or generalizing)
induction. But there are different types of induction
(e.g., enumerative, eliminative, probabilistic). These are
not mentioned or distinguished in any of the 16 texts.
- Inductive nature of constructive replication not
explicitly stated in any text.
2. Hypothetico-Deductive Method
- The most popular account of scientific method
- Claims a hypothesis or theory is evaluated in terms of its
test predictions
- 9/16 texts neither mention nor describe it.
- 5/16 vaguely describe it without naming it.
- 2/16 name it and describe it.
- 1/16 says it is controversial.
- No text talks about its limitations or its proper scope of
application.
3. Inference to the Best Explanation
- Concerned with explanatory reasoning. In brief, it
captures the idea that theories are evaluated according
to their explanatory worth.
- Frequently used in the evaluation of scientific theories
(e.g., Darwin’s theory of natural selection)
- Not mentioned or referenced in any of the 16 texts
- Methods of inference to the best explanation are
available (e.g., the theory of explanatory coherence).
- Structural equation modeling that combines model fit
with model parsimony can be regarded as a hybrid
account of inference to the best explanation.
4. Bayesian Method
- Widely regarded as the leading theory of scientific
confirmation in contemporary philosophy of science
- Not mentioned, referred to, or indexed in any of the 16
textbooks (as ‘Thomas Bayes’, ‘Bayes’ theorem’,
‘Bayes factors’, ‘Bayesian statistics’, or ‘Bayesianism’)
- Note that Bayesian statistics is beginning to be promoted
by a small group of behavioral science methodologists.
Conclusion: One cannot learn much of worth about
scientific method from methods textbooks that claim
to deal with scientific method! (Additionally, one can
learn a good number of things that are not so).
Psychologists on Scientific Method
A few psychologists have written about scientific method
in a serious vein:
Cattell, R. B. (1966). Psychological theory and scientific method. In R. B. Cattell
(Ed.), Handbook of multivariate experimental psychology (pp. 1–18). Chicago, IL:
Rand McNally.
Haig, B. D. (2005a). An abductive theory of scientific method. Psychological
Methods, 10, 371-388.
O’Donohue, W., & Buchanan, J. A. (2001). The weaknesses of strong inference.
Behavior and Philosophy, 29, 1-20.
Rozeboom, W. W. (1997). Good science is abuctive, not hypothetico-deductive.
In L. L. Harlow, S. A. Mulaik, & J. H. Steiger (Eds.), What if there were no
significance tests? (pp. 335-391). Hillsdale, NJ: Erlbaum.
Skinner, B. F. (1956). A case history in scientific method. American Psychologist, 11,
221-233.
None of these references are cited in any of the 16 texts.
What Do Methods Textbooks Say
About Scientific Method?
1. “Scientific Method is the Scientific Method”
- 11/16 texts spoke of the scientific method
(with no agreement on what it is!).
- 3/16 spoke about a scientific method.
- 2/16 didn’t mention scientific method at all.
There is no such thing as the scientific method.
Instead, we have a plurality of scientific methods.
2. “Scientific Method is a Series of Steps”
- 9/16 texts described scientific inquiry as a series of
steps (as few as 4 steps, as many as 11)
- the steps listed vary greatly:
Example 1: Gravetter & Forzano, Research methods for
the behavioral sciences (2009).
Step 1: Observe behavior or other phenomena
Step 2: Formulate tentative explanatory hypothesis
Step 3: Use the hypothesis to make a testable prediction
Step 4: Evaluate the prediction via planned observations
Step 5: Support, refute, or refine the original hypothesis
… and so on in a circle or spiral
Example 2:
‘HOMER’: An acronym for the five steps of the scientific
method
“ Each letter of Homer stands for one of the five steps in
the scientific method : hypothesize, operationalize,
measure, evaluate, and replicate/revise/report.”
(Lakin, et al., 2007)
The propensity to characterize scientific method as a
series of fixed steps is probably due to John Dewey’s
(1910) five-step account of scientific method, which
had an enormous influence on American education
(Rudolph, 2005). Dewey later revised his formulation to
do away with fixity of order and spoke of phases, not
steps.
3. “Scientific Method is a Process of Falsification”
a) Falsification is the process of showing a claim to be
false.
b) 13/16 texts mention ‘falsification.’
c) All texts fail to distinguish between the process of
falsification and Popper’s theory of falsificationism
with its attendant ideas of conjectures and
refutations, corroboration, and demarcation.
d) 3/13 texts mention that falsification(ism) is
controversial, but none of them deal with the wellknown criticisms of it in the philosophy of science.
Phenomena Detection and Theory
Construction
1. The methodological distinction between phenomena
detection (e.g., the discovery of empirical regularities)
and theory construction (the generation, development,
and appraisal of explanatory theories) is of
fundamental importance to science.
2. These are two quite different sorts of undertaking and
they employ different research methods (Haig, 2013).
3. 1/16 texts explicitly draws the distinction (Leary, 2012).
7/16 almost draw it (e.g., ‘law’/‘theory’ distinction).
8/16 fail to draw the distinction.
4. No text systematically discusses different research
methods in relation to the distinction.
5. At best, textbook treatments of theory construction
focus on hypothetico-deductive theory testing for
empirical adequacy.
6. Methods specifically tailored to theory generation (e.g.,
exploratory factor analysis), theory development (e.g.,
analogical modeling), and theory appraisal (e.g.,
inference to the best explanation) all deserve a proper
place in research methods textbooks (Haig, 2005).
7. Methods textbooks in psychology emphasize data
analysis at the expense of theory construction.
The Neglect of Philosophy of Science
The philosophy of science is an important part of
scientific methodology.
It is a major resource for leaning about the conduct of
scientific research.
Yet, the 16 texts make little or no use of it (3/16 mention
it; none use it).
Popper (1959) and Kuhn (1962), are sometimes
mentioned; Lakatos (1970) and Laudan (1977) are
occasionally mentioned. None of them are used. The
last 25 years of developments in the philosophy of
science have been completely ignored (see Blachowicz,
2009).
As a consequence of this general neglect, these texts have
failed to capture the profound changes in our
understanding of science that philosophy of science has
brought about in the last 50 years (e.g., Proctor &
Capaldi, 2001).
Early editions of the McGuigan and Leary texts stressed
the importance of philosophy of science in
understanding research methods.
“An understanding of the philosophy of science is important to an
understanding of what science is, how the scientific method is used,
and particularly of where experimentation fits into the more
general framework of scientific methodology”. (McGuigan,
1960, p. 111)
Interestingly, both McGuigan and Leary dropped
reference to philosophy of science in later editions
(publishers’ pressure to conform?).
In his section on philosophy of science, Leary (2001)
claimed:
1. Many philosophers of science are behavioral scientists.
2. Many researchers take courses in the philosophy of
science.
I think both claims are false. It would be good for both
disciplines if they were true.
Concluding Thoughts
1. Textbooks should present inductive, hypotheticodeductive, and Bayesian methods, as well as inference
to the best explanation, as major local theories of
scientific method with domain specific applications.
2. Knowledge of these scientific methods would help
learners better understand the process of phenomena
detection, the complexities of theory testing, and the
nature of explanatory reasoning in science.
3. The dominant “twofer” model (Aiken, West, & Millsap,
2008) (where researchers with expertise in a
substantive area of psychology, teach statistics and
research methods courses) extends to writing research
methods textbooks.
This two-tiered twofer model poses serious problems
for psychology because it contributes to a decrease in
the quality of instruction about scientific method
generally (and research methods more specifically).
4. The content of these four major theories of method
cannot be taught without making substantial use of the
relevant philosophy of science literature (this use of
philosophy of science should extend to instruction
about methodological processes more generally).
5. Textbooks on research methods should be written by
professional research methodologists, thus ensuring
that they are appropriately methodologically informed.
6. These texts should deal with the full range of behavioral
research methods and strategies.
7. Psychology needs just a few up-to-date general research
methods textbook (say, 5, not 50) written by
professional methodologists.
8. These texts would likely vary depending on the
methodological predilections of the authors.
Example: Rosnow and Rosenthal (2013), Beginning
Behavioral Research: A Conceptual Primer (emphasis on
meta-analysis, effect sizes, and the binomial effect-size
display, reflecting Rosenthal’s own methodological
work).
9. This variation in textbook content would be
appropriate for an education in behavioral research
methods, where it is genuinely controversial as to what
we should learn.
References
Aiken, L. S., West, S. G., & Millsap, R. E. (2008). Doctoral training
in statistics, measurement, and methodology in psychology.
Replication and extension of Aiken, West, Sechrest, and Reno’s
(1990) survey of PhD programs in North America. American
Psychologist, 63, 32-50.
Blachowicz, J. (2009). How science textbooks treat scientific
method: A philosopher’s perspective. British Journal for the
Philosophy of Science, 60, 303-344.
Dewey, J. (1910). How we think. Boston: Heath.
Haig, B. D. (2005). An abductive theory of scientific method.
Psychological Methods, 10, 371-388.
Haig, B. D. (2013). Detecting psychological phenomena: Taking
bottom-up research seriously. American Journal of Psychology, 126,
135-153.
Haig, B. D. , & Ellis, C. (2013). How research methods textbooks
mislead psychologists about scientific method (in preparation).
Kuhn, T. S. (1962/1970). The structure of scientific revolutions (2nd ed.)
Chicago: University of Chicago Press.
Lakatos, I. (1970). Falsification and the methodology of scientific
research programmes. In I. Lakatos & A. Musgrave (Eds.),
Criticism and the growth of knowledge (pp. 91-196). Cambridge:
Cambridge University Press.
Lakin, J. L., Giesler, R. B., Moros, K. A., & Vosmik, J. R. (2007).
HOMER as an acronym for the scientific method. Teaching of
Psychology, 34, 94-96.
Laudan, L. (1977). Progress and its problems: Towards a theory of scientific
growth. Berkeley: University of California Press.
Nickles, T. (1987). Methodology, heuristics, and rationality. In J. C.
Pitt & M. Pera (Eds.), Rational changes in science (pp. 103-132).
Dordrecht: Reidel.
Popper, K. R. (1959). The logic of scientific discovery. London:
Hutchinson.
Proctor, R. W., & Capaldi, E. J. (2001). Improving the science
education of psychology students: Better teaching of
methodology. Teaching of Psychology, 28, 173-181.
Rudolph, J. L. (2005). Epistemology for the masses: The origins of
“the scientific method” in American schools. History of Education
Quarterly, 45, 341-376.