ISSUES IN ADOPTING MIXED METHOD IN A RESEARCH ~ EXAMPLES FROM RESEARCH ARTICLES
1. PURPOSE - is the purpose of adopting mixed methods made clear by the researcher?
* When it’s not mentioned, the researcher does not give recognition to the full contribution of each method
Article no
Title
1
Asan, A. (2007). Concept Mapping in Science Class: A Case Study of fifth grade
students. Educational Technology & Society, 10 (1), 186 – 195.
2
Wesely, P. M. (2010). Language Learning Motivation in Early Adolescents Using
Mixed methods Research to Explore Contradiction. Journal of Mixed Method, 4(4),
295-311.
Mitchell, S. N., Reilly, R., Bramwell, F. G., Solnosky, A., & Lily, F. (2004). Friendship
and Choosing Groupmates: Preferences for Teacher-Selected vs. Student-Selected,
Journal of Instructional Psychology, 31(1), 20-32.
Hilton, A., Nichols, K., & Gitsaki C. (2008). Scaffolding Chemistry Learning Within
the Context of Emerging Scientific Research Themes Through Laboratory Inquiry.
Papery, presented at AARE Conference, Queensland University of Technology,
Brisbane, Australia.
3
4
5
Hsieh, Y-C. J., & Cifuentes, L. (2006). Student-Generated Visualization as a Study
Strategy for Science Concept Learning. Educational Technology & Society, 9(3), 137148.
Purpose in adopting mixed
methods
Not mentioned
Assumed to be for
corroboration purpose.
Not stated.
Assumed to be for
expansion purpose.
Not mentioned.
Assumed to be for
corroboration purposes.
To gain information that
would lead to an
understanding of how
approaches and resources
used supported the
development of students’
knowledge and
representational
competence.
Not stated.
Assumed to be for
complementarity purpose.
2. METHOD – Flexibility & Appropriateness
* When mixed without careful consideration of the assumptions / rules and expectation regarding their conduct, corruption
of those methods can occur. Therefore, the findings are subject to question.
Articl
e no
1
Hypothesis / Research
Questions
(simplified)
1. There will be a statistically
significant difference between
the experimental group
(exposed to concept map
program) with the control
group (traditional teaching
method) with respect to
academic achievement.
2. Concept mapping as an
instructional tool has a positive
effect on students’ attitudes.
Methods used to answer
research questions
Method Limitation
How could the limitation be
addressed?
H1: Quantitative method
- Pre-test MCQs
- Experimental group:
exposed to ‘Inspiration’
concept mapping program
- Control group : traditional
teaching
- Post-test MCQs to both
groups
- CM scoring rubrics
Weakness in
instrumentation
- limited number of
students
- fixed on attitude
only
- metrics for attitude
assessment
Longitudinal studies
- conduct a longitudinal study
to explore students attitudes
towards the use of concept
map
- increase sample size
- diversify, for example:
- gender
- motivation
- cultural background
H2: Qualitative method
- Interview for
Experimental group
students
- One open-ended
question, duration is not
specified
2
3
1. What does an examination of
the second language (L2)
learning motivation of early
adolescent immersion
graduates reveal about
important issues in immersion
education?
2. How do the data collected
through student interviews
compare with the data
generated by the results of the
Attitude/Motivation Test
Battery?
RQ1: Qualitative method
- Semi-structure
interview
1. Choosing one’s groupmates
would have a negative effect
on subsequent attitudes
toward choice.
H1: Quantitative
- Teacher as a researcher
- Pre – post test nonequivalent group design
- Questionnaire
2. Students would be forced to
choose their friends when
given the chance to choose
their groupmates.
RQ2: Quantitative method
- 40 items Likert scale
type of questions was
used to measure
opinions, belief,
attitudes.
H2: Qualitative
- Focus groups interview
Weakness in
instrumentation
- poor response rate in
interview
- lost contact with
parents through mail,
handout and telephone
for interview request
Quantitative
-Likert-type
scale questions to
measure opinions,
beliefs, and attitudes is
well documented in
the behavioral
sciences and in L2
research
Qualitative
-transcribing the 33
interviews
Assumed that attitude
is objective.
social effects
4
5
1. Is the use of digital
technologies supports
students’ understanding of
chemistry?
2. Is the use of inquiry-based
learning support students’
understanding of chemistry?
3. What are the perceptions of
the students’ of the
scaffolding approaches used?
1. Did students trained to
visualize on paper or
computer perform better on
comprehension test than
those not trained to do so?
2. Was there any difference on
comprehension posttest
scores between students who
trained to visualize on
computers and those who
trained to visualize on papers?
RQ1 – RQ3: Quantitative method
- Pre-test and Post- test: to test for knowledge
and conceptual
understanding (KCU)
and representational
competence (RC).
- Questionnaire
pertaining to interest
RQ3: Qualitative method
- Semi –structured
interview questions to
gauge their level of
interest.
RQ 1 & 2: Quantitative
method
-
Post-test
-
Comprehension test
items validated by 3
content experts.
Interviewer
Allow equal interval duration
unknown,
between phases.
Instructor who
intervene the
lesson is unknown
unequal duration
between the phases
No actual evaluation of Participants asked to report the
prior knowledge on the extent of prior knowledge with
experimental material. a Likert scale.
Assumed that prior knowledge
did not confound their test
scores.
3. What was the difference
among groups on their level of
engagement during study
time?
4. What were the elements in
the school setting and
learners’ behaviors that might
contribute to the
effectiveness or lack of
effectiveness of the use of
software to generate
visualizations during study
time?
5. What were students’ attitudes
toward generating
visualizations during study
time?
6. Had students ever been
exposed to the visualization
skills that were identified in
the provide visualization
workshops?
RQ 3 – 6: Qualitative
method
RQ 3 & 4 – Observation
RQ 5 & 6 – Interview
No clear description of
how the observation
and interview was
conducted.
No description of
whether verification
was conducted
through member
checking and peer
debriefer.
3. SAMPLING
* Sampling issue must be resolved with respect to purpose of the research and in particular how are the results to be
generalized to a population beyond the sample.
Article
no
1
2
Hypothesis / Research
Questions
(simplified)
1. There will be a
statistically significant
difference between the
experimental group
(exposed to concept
map program) with the
control group
(traditional teaching
method) with respect to
academic achievement.
2. Concept mapping as an
instructional tool has a
positive effect on
students’ attitudes.
1. What does an
examination of the
second language (L2)
learning motivation of
early adolescent
immersion graduates
reveal about important
issues in immersion
Sampling Size
&
Method of Sampling
Non-equivalent control
group design
Limitations
Can the results be
generalized?
Quantitative:
They are by no means
- for comparison between two conclusive for generalization
groups: minimum number of purpose
th
5 grade students ( N=23 ) samples must be 10
- control group (n=10)
- experimental group
(n=13)
-
33 students were
interviewed
Random sampling of 4
students from whole
population of 136.
(questionnaire)
- 2 students from
They are by no means
conclusive for generalization
purpose
Number of subjects is too
small
education?
3
continuing programme
group
2 students from
discontinued
programme group
2. How do the data
collected through
student interviews
compare with the data
generated by the results
of the
Attitude/Motivation Test
Battery?
1. Choosing one’s
139 students
groupmates would have 5 science classes
a negative effect on
subsequent attitudes
1. Questionnaire
toward choice.
LA
NA
HO
TS SS TS SS TS/SS
23 27 22 25 24
2. Students would be
forced to choose their
friends when given the
chance to choose their
groupmates.
Assumes benefits of
heterogeneity can
construct groups in a
more equitable manner.
How participants were
allocated to groups was
not described.
2. Focus groups involved
3 groups with students
who had selected their
own groups.
Quantitative:
Missing values where N =121
How was the missing values
treated?
Interviewer was unknown.
-
4
5
1. Is the use of digital
technologies supports
students’ understanding
of chemistry?
2. Is the use of inquirybased learning support
students’ understanding
of chemistry?
3. What are the
perceptions of the
students’ of the
scaffolding approaches
used?
1. Did students trained to
visualize on paper or
computer perform
better on
comprehension test
than those not trained
to do so?
2. Was there any
difference on
comprehension
posttest scores
between students who
trained to visualize on
computers and those
who trained to visualize
on papers?
N = 49 students
- Control group (n=22)
- Experimental group
(n=27)
N = 92
Random assignment of 2
classes to be:
- Control group (n = 28)
- Visualization-paper
group (VP) (n = 30)
- Visualization-computer
group (VC) (n = 34)
Group equivalence was
emphasize; but there was
significantly more male in
VC group
Not conclusive
No description of whether
interview was conducted with
the whole sample or subsample.
Lack of random selection of
individuals to groups.
Findings from the
quantitative analysis may not
be generalized.
It has poor population,
ecological and operational
transferability.
The significant difference in
the VC group having more
male than female could have
affected the score
distribution of the VC group,
which may contribute to the
group having no difference
in performance with the VP
group.
3. What was the
difference among
groups on their level of
engagement during
study time?
4. What were the
elements in the school
setting and learners’
behaviors that might
contribute to the
effectiveness or lack of
effectiveness of the use
of software to generate
visualizations during
study time?
5. What were students’
attitudes toward
generating
visualizations during
study time?
6. Had students ever been
exposed to the
visualization skills that
were identified in the
provide visualization
workshops?
4. DATA ANALYSIS
Which method to be used must be judged on how well it inform the research purposes (answer the research
questions)
* Difficulty arises in terms of generating a need to trade off between the intensive detail of hermeneutic analysis
(interpretative / explanatory) and the extensiveness of statistical (quantitative) inference.
a. Qualitative part
- no standard coding
- different researchers have different way to determine the coding system across data type
- multidirectional: possibility that analysis will reveal different dimensions for each code
- quantitizing code: meaning become fixed and single dimensional
b. Quantitative part
- different approaches to data analysis in treating variations
Article
no
1
Hypothesis / Research
Questions
(simplified)
1. There will be a
statistically significant
difference between the
experimental group
(exposed to concept
map program) with the
control group
(traditional teaching
method) with respect to
academic achievement.
Methods used to
answer research
questions
Analysis Method
H1:
- Pre-test MCQs
- Experimental grp:
exposed
to ‘Inspiration’ concept
mapping program
- Control grp : traditional
teaching
- Post-test MCQs to both
grps
1. T-test: Paired-Sample
Assumptions Made in Selecting
the Methods
1. t-test:
- Assumed that subjects
2. Concept Map Scoring
comprising the sample have
Rubric
been randomly selected, thereby
it can be generalized to a
3. Concept Map and
population.
MCQs Correlation
- Assumed normality where the
- Pearson r correlation
scores of the population from
coefficient
which the sample is taken are
normally distributed.
2. Concept mapping as an
instructional tool has a
positive effect on
students’ attitudes.
- CM scoring rubrics
4. Analysis of interview
was not described
H2:
- Interview for Exp grp
5. Quantitize codes
students
pertaining to
- One open-ended
attitudes.
question,
- duration is not specified
-
Assumed independent
observations where each score
within the sample is independent
of all other scores.
2. CM Scoring Rubric:
Assumed concepts are objective,
and be counted.
3. Assumes a relationship between
performance in MCQs and concept
map.
4. Variation in views & exceptional
view can be used to develop theory.
2
1. What does an
examination of the
second language (L2)
learning motivation of
early adolescent
immersion graduates
reveal about important
issues in immersion
education?
2. How do the data
collected through
student interviews
RQ1: Qualitative method Analyzed the interview
- Semi-structure
by identifying moments.
interview
RQ2:
Quantitative method
- 40 items Likert scale
type of questions was
used to measure
Research question 2:
- t-test
- Effect size
- Cronbach’s α
5. Assumed data from qualitative
coding could be scaled.
1. t-test:
- Assumed that subjects
comprising the sample have
been randomly selected, thereby
it can be generalized to a
population.
- Assumed normality where the
scores of the population from
which the sample is taken are
normally distributed.
- Assumed independent
observations where each score
compare with the data
generated by the results
of the
Attitude/Motivation Test
Battery?
3
opinions, belief,
attitudes.
within the sample is independent
of all other scores.
1. Choosing one’s
H1:
groupmates would have - Pretest - Posttest non a negative effect on
equivalent control
subsequent attitudes
group
toward choice.
- Questionnaire ( CLS)
2. Students would be
forced to choose their
friends when given the
chance to choose their
groupmates.
H2:
- Focus groups
involved students
who had selected
their groups only
- Use 3 structured
questions
-
Research question 2:
Data analyzed statistically and
hermeneustic analyzed data could
be integrated.
Manova:
3 (group) x 2 (test)
repeated measures
multivariate analysis
of variance
(MANOVA)
-
-
-
2 (gender) x3 (group)
(ANOVA)
-
5 (class) x2 ( test)
repeated measures
multivariate analysis
of variance
-
Categorizing the
responses of the
participants on a flip
chart according to
themes.
Assumed that samples are drawn
from normally distributed
population.
Samples are independent and
randomly drawn from the
population.
Data from the sample group is
homogenous.
Anova:
-
-
Assumed that the population
from the samples is normally
distributed.
The scores from each group have
homogeneous variance.
4
5
1. Is the use of digital
technologies supports
students’ understanding
of chemistry?
2. Is the use of inquirybased learning support
students’ understanding
of chemistry?
3. What are the
perceptions of the
students’ of the
scaffolding approaches
used?
1. Did students trained to
visualize on paper or
computer perform
better on
comprehension test
than those not trained
to do so?
2. Was there any
difference on
comprehension
posttest scores
between students who
RQ1 – RQ3: Quantitative - Paired sample t-test
method
- Pre-test and Posttest: to test for
knowledge and
conceptual
understanding (KCU)
and representational
competence (RC).
- Questionnaire
pertaining to interest
1. t-test
- Assumed that subjects
comprising the sample have
been randomly selected, thereby
it can be generalized to a
population.
- Assumed normality where the
scores of the population from
which the sample is taken are
normally distributed.
- Assumed independent
observations where each score
Transcript was
within the sample is independent
systematically coded.
of all other scores.
Code was then
2. Assumed data from qualitative
quantitized into
coding could be scaled.
percentage.
RQ3: Qualitative method Semi –structured
interview questions to
gauge their level of
interest.
RQ 1 & 2: Quantitative
Statistical analysis
method
1. One-way analysis of
variance.
Post-test
2. Tukey HSD post hoc
test.
Comprehension test
items validated by 3
content experts.
RQ 3 – 6: Qualitative
method
Qualitative analysis
3. How data collected
One-way analysis of variance
assumes
Each sample is drawn from a
population that is normally
distributed.
Variances of each population
distribution are the same.
Each observation within a group is
independent of all other
observations.
Post-hoc analysis assumes to seek
3.
4.
5.
6.
trained to visualize on
computers and those
RQ 3 & 4 – Observation
who trained to visualize
on papers?
RQ 5 & 6 – Interview
What was the
difference among
groups on their level of
engagement during
study time?
What were the
elements in the school
setting and learners’
behaviors that might
contribute to the
effectiveness or lack of
effectiveness of the use
of software to generate
visualizations during
study time?
What were students’
attitudes toward
generating
visualizations during
study time?
Had students ever been
exposed to the
visualization skills that
were identified in the
provide visualization
workshops?
by qualitative
method were
analyzed was not
described.
for significance to prevent Type 1
error (H0 is true but rejected).