Complementary Research
Methods
Michele Jacobsen, PhD
Postdoctoral Fellow, SERN
University of Calgary
dmjacobs@ucalgary.ca
http://www.ucalgary.ca/~dmjacobs/phd/methods/
Jacobsen, D. M.
Overview
• Outline relative merits of Triangulation - the
integration of both qualitative and quantitative
research methodologies
• Review a variety of research methods:
– Experimental design, ethnography, case study,
survey methods.
• Discuss Nowaczyk and Underwood’s (1995) paper.
Jacobsen, D. M.
Fundamental Goals...
• Of Science:
– To Understand, To Predict, To Control
• Of Scientists:
– To communicate discoveries and
findings to a community of peers
Jacobsen, D. M.
Designing Research
Dimensions of Analysis
• Research Purposes - theoretical or applied?
• Research Problems - what questions are asked?
• Research Settings - simulated or natural?
• Research Investigators - background and training
• Research Methods - a continuum
– Experimental, Ethnography, Case study, Survey
Jacobsen, D. M.
Evaluating Research
• Validity
– A concern for most social scientists is the complex nature of the
phenomena under study: human behavior.
– Multiple perspectives are required in order to adequately reflect the
richness of these complexities.
• Reliability
– Consistency, Replicability
• Usefulness or Value of Investigation
– Contribution to knowledge
– Advance THEORY and PRACTICE in discipline
Jacobsen, D. M.
Research Methodologies
A continuum rather than “either/or”
• Qualitative
• Quantitative
– Goal: To Understand,
Predict
• Descriptive accounts
• Similarities and
Contrasts
– Applied and Theoretical
– Research Questions
– Field study
• Natural conditions
– Goal: To Predict and
Control
• Measure and Evaluate
• Generalize to population,
reproduction
– Basic and Theoretical
– Hypothesis testing
– Lab study
• Controlled, contrived
Jacobsen, D. M.
Validity and Reliability
• Both Quantitative and Qualitative research designs
seek reliable and valid results. For example:
– Quantitative Reliability: Data that are consistent or
stable as indicated by the researcher's ability to
replicate the findings.
– Qualitative: Validity of findings are paramount so that
data are representative of a true and full picture of
constructs under investigation.
Jacobsen, D. M.
Part Versus Whole
• “Whole” is often greater than “Parts”
• It is a non-trivial matter to infer the behavior of the whole
from the behavior of its parts
– Quantitative research designs strive to identify and
isolate specific variables within the context (seeking
correlation, relationships, causality) of the study.
– Qualitative design focuses on a holistic view of what is
being studied (via documents, case histories,
observations and interviews).
Jacobsen, D. M.
Data Collection
• Quantitative
– Emphasis on numerical data, measurable variables
– Data is collected under controlled conditions in order to
rule out the possibility that variables other than the one
under study can account for the relationships identified
• Qualitative
– Emphasis on observation and interpretation.
– Data are collected within the context of their natural
occurrence.
Jacobsen, D. M.
Static and Dynamic
• Quantitative
– The accumulation of facts and causes of behavior
through careful isolation, measurement and evaluation
of variables.
– Predictability and Control over time.
• Qualitative
– Concerned with the changing and dynamic nature of
reality.
– Understanding a Point in time
Jacobsen, D. M.
Triangulation
• Combines independent yet complementary
research methods.
– Simultaneous triangulation:
• Use of both qualitative and quantitative methods at the same time
• e.g., Survey methods and Case study
– Sequential triangulation:
• Results of one method are essential for planning the next method
• e.g., Exploratory Pilot study precedes Experimental design
• Also known as “mixed methods”
Jacobsen, D. M.
Benefits of Triangulation
• Advantages of each complement the other
– resulting in a stronger research design, and
– more valid and reliable findings.
• Inadequacies of individual methods are minimized
– threats to Internal Validity are realized and addressed
• Example
– Quantitative design strives to control for bias so that facts,
instances, phenomena can be understood in an objective way.
– Qualitative approach strives to understand the perspective of
participants or a situation by looking at firsthand experience to
provide meaningful data.
Jacobsen, D. M.
Additional Benefits
• Triangulation offers a balance between logic and stories.
• Qualititative research, which emphasizes exploration,
understanding, contextualizing, introspection, and theory
construction, provides a strong base for wider quantitative
measures, scaling, and generalization.
• Quantitative research, which emphasizes large samples,
can provide an overview of an area that can reveal patterns,
inconsistencies, and so forth, that can be further
investigated with qualitative methods.
Jacobsen, D. M.
Effectiveness of Triangulation
• In order for Triangulation to be used effectively, four
principles must be adhered to:
– 1. research question(s) must be clearly focused;
– 2. strengths and weaknesses of each chosen method must
complement each other;
– 3. data collection methods should be selected according to their
relevance to the nature of the phenomenon being studied;
– 4. a continual evaluation of the approach should be under-taken
during the study.
Corner (1990)
Jacobsen, D. M.
Range of Research Methods
•
•
•
•
Experimental design
Ethnography
Case study
Survey
Jacobsen, D. M.
Experimental Design
• Hypothesis testing
• Independent and Dependent Variables
– For example - Predictor: method of instruction, Resulting
differences: math performance
• Sampling of Population
• Experimental and Controlled Conditions
• Random assignment
Jacobsen, D. M.
Experimental Research
• The researcher does something to the subjects or
objects or research, and then attempts to determine
the effects of these actions
• Reporting
– Careful description of sampling procedure
– Inferential statistics, effect size, and so on.
Jacobsen, D. M.
Ethnography
• Defined: a picture of the “way of life” of some identifiable
group of people
• Anthropology - “doing fieldwork”, “going native”
• Preoccupied with culture, and how people interact with
each other
• Qualitative Methodology - Both a research process and a
product
– Outcome: an ethnographic account
Jacobsen, D. M.
Ethnographic Process
• The ethnographer is the primary research instrument
• One year or more in the field setting
– long enough to see a full cycle of activity
– For example, a full school year
• Tension and balance between involvement and
detachment
– Outsider’s broad and analytical perspective on group studied
– Insider view, familiarity, empathy, identification with group
Jacobsen, D. M.
Field Research Techniques
• An Inquiry Process of multiple methods:
– Participant observation
• privileged, active participant
• passive observer
– Interviewing
• key informants, structured, unstructured
• groups, surveys and questionnaires
– Making and using records
• historical documents, archives, written records
Jacobsen, D. M.
Validity and Reliability of the
Ethnographic Account
• “The satisfactoriness of the explanation is
what counts, not the power of the method for
deriving it”.
• Significance is derived socially, not
statistically
Jacobsen, D. M.
Case Study
• Understanding the intricate complexity,
idiosyncrasy of one particular case
– investigation of a “bounded system”
– Some entity deemed worthy of close watch
• a single child, a single classroom, a single school, a single
national program…
• Goals
– Understand and report the uniqueness of individual cases
(both commonalities and differences)
– Usually no attempt to represent case by single or multiple
“scores”
Jacobsen, D. M.
Case Study Methods
• Similar to ethnographic field methods
– ASKING - Interviews
• Gather narrative and testimony
– WATCHING - Observations
– SEARCHING - Written records and artifacts
• Reporting
– Develop a conceptual structure, look for patterns,
consistencies, repetitions, and manifestations pertinent
to your research question(s)
Jacobsen, D. M.
Validity and Reliability
• There are many different stories to be told
– Different researchers have different questions to
answer, different conceptualizations of the situation,
and set different boundaries for the case
• Generalizability: What is true of one case is often
true about other cases
– Consistencies can be found - predictability
– How many cases are needed before patterns emerge? It
depends...
Jacobsen, D. M.
Survey Research Methods
• Purpose and Goal
– Describe specific characteristics of a large group of
persons, objects, or institutions
– Understand present conditions, rather than the effects
of particular intervention (as in experimental research)
• Sample of Population
– Groups of interest are well defined and chosen using
well defined rules
– Representativeness
Jacobsen, D. M.
Survey Methods
• Mail
– postage and printing costs, participation rate
• Telephone
– sampling, wage and time costs, participation rates
• Face-to-Face
– wage and time costs, participation rates, like structured interview
• Web-based
– anytime, anywhere, cost effective
Jacobsen, D. M.
Issues in Survey Construction
•
•
•
•
•
Item (question) and scale construction
Pilot Testing and revision
Sampling procedures
Analysis and reporting of results
Generalizability
– Drawing conclusions about the conditions, attitudes,
opinions, or status of a population of persons, objects,
institutions, or other entities.
Jacobsen, D. M.
Nowaczyk and Underwood (1995)
http://epaa.asu.edu/epaa/v3n20.html
• "Possible Indicators of Research Quality for
Colleges and Universities"
– Used qualitative method, “focus group”, to
investigate research quality in higher
education
Jacobsen, D. M.
Results
• Academics promoted the use of both quantitative
and qualitative measures to report on “quality”
– QUANTITY OF:
• Journal publications, conference presentations, books and book
chapters, awards, grants, budget, and so on…
– QUALITY OF:
• Reputation of publication, reputation of granting agency,
quality of conference, peer reviews of research programs,…
• Quality of institutions that hire graduate students
• Societal benefit of research
Jacobsen, D. M.
Resources
• Jaeger, R. M. (1997). Complementary Research Methods for Research in
Education, (2nd ed). American Educational Research Association:
Washington, DC.
• Edyburn, D. L. (1998). The Electronic Scholar: Enhancing Research
Productivity with Technology. Prentice-Hall: Columbus, OH.
• Nowaczyk, R. H., & Underwood, D. G. (1995). Possible indicators of
research quality for colleges and universities. Education Policy Analysis
Archives, 3(20). [On-line]. Available: http://epaa.asu.edu/epaa/v3n20.html
• Bowen, K. A. (1996). The Sin of Omission -Punishable by Death to
Internal Validity: An Argument for Integration of Qualitative and
Quantitative Research Methods to Strengthen Internal Validity. [On-line].
Available: http://trochim.human.cornell.edu/gallery/bowen/hss691.htm
Jacobsen, D. M.