Issues in Study Design
Petri Nokelainen
petri.nokelainen@uta.fi
School of Education
University of Tampere, Finland
CONTENTS
Issues in Study Design
Writing Scientific Reports
Ten Questions About YOUR Research
Issues in Study Design
Scientific research
Theoretical
T1: Research body
AND
Empirical
AND OR
T2: Innovation
E1: Numerical data
AND
OR
E2: Textual /
nominal data
Issues in Study Design
Qual: {T1,T2,E2} {T1,T2}
Quan: {E1} !
A: {T1,T2}
B: {T1,T2,E1}
C: {T1,T2,E2}
D: {T1,T2,E1,E2}
Theoretical research
Empirical research
Issues in Study Design
B: {T1,T2,E2}
Description of person’s attitudes, feelings, meanings,
knowledge, etc. about S.
Research question
C: {T1,T2,E1}
How many cases in the data D have certain attribution x?
How many cases in the data D have certain attribution y?
Is there a relationship between x and y?
Is that relationship causal?
Issues in Study Design
(Nokelainen, 2008, p. 119)
Issues in Study Design
Issues in Study Design
• The left-hand side of the
figure shows two main
categories of data collection:
– Probability sample (PS) and
– Non-probability sample (NPS).
• Both methods aim to produce
a scientific, representative
sample from the target
population.
Issues in Study Design
• According to Jackson (2006), a representative sample is
“like” the population.
– Thus, we can be confident that the results we find based on
the sample also hold for the population.
• This is not a problem with PS, which is based on
random, stratified or cluster sampling.
– In random sampling each member of the population has an
equal likelihood of being selected into the sample.
– Stratified random sampling allows taking into account
different subgroups in the population.
– If the population is too large for random sampling of any sort,
cluster sampling is applied.
Issues in Study Design
• Problems arise with NPS as the individual members of
the population do not have an equal likelihood of being
selected to be a member of the sample.
• The most commonly applied NPS technique is
convenience sampling (CS) in which participants are
obtained wherever they can be found and wherever is
convenient for the researcher (Hair, Anderson, Tatham
& Black, 1998).
Issues in Study Design
• Why, then, educational scientists use NPS, typically CS?
– Simply because it “tends to be less expensive [than RS] and it
is easier to generate samples using this technique” (Jackson,
2006, p. 84).
Issues in Study Design
• However, on the lower lefthand part of the figure, it is
shown that when researcher
ensures that the CS is like the
population on certain
characteristics (location and
dispersion descriptive
statistics about, for example,
age and job title), it becomes
a quota sample (QS).
– A quota sample is better than a
CS as it allows us to ensure that
the results we find based on the
sample also hold for the
population.
Issues in Study Design
• The upper part of the figure
contains two sections, namely
“parametric” and “non-parametric”
divided into eight sub-sections
(“DNIMMOCS OLD”).
• Parametric approach is viable only
if
– 1) Both the phenomenon modeled
and the sample follow normal
distribution.
– 2) Sample size is large enough (at
least 30 observations).
– 3) Continuous indicators are used.
– 4) Dependencies between the
observed variables are linear.
• Otherwise non-parametric
techniques should be applied.
Issues in Study Design
• First, study design (D) is
made on the basis of the
research question and major
goal.
• According to de Vaus (2004,
p. 9), “… research design is to
ensure that the evidence
obtained enables us to
answer the initial question as
unambiguously as possible.”
Issues in Study Design
• In order to obtain relevant evidence, we need to specify
the type of evidence needed to answer the research
question.
• More specifically, we need to ask: Given this research
question, what type of evidence (data) is needed to
answer the question in a convincing way?
Issues in Study Design
• Sometimes we proceed with the so-called qualitative
designs, sometimes a quantitative orientation is more
appropriate, and sometimes we work both qualitatively
and quantitatively (mixed-methods research, for a
thorough discussion, see Brannen, 2004).
• Methodological, conceptual etc. triangulation.
• Design research is quite new approach, see BannanRitland (2003).
Issues in Study Design
• Experimental design (a.k.a. ‘pretest post-test randomized
experiment’) is the most recommended approach, but only
possible with a random sample (a.k.a ‘probability sample’) and
random assignment (participants are randomly selected for the
experimental and control groups).
• Research is conducted in a controlled environment (e.g.,
laboratory) with experiment and control groups (threat to
external validity due to artificial environment).
• Using experimental design, both reliability and validity are
maximized via random sampling and control in the given
experiment (de Vaus, 2004).
Issues in Study Design
Random assignment
Exp.
Pre
I
Post
Contr.
Pre
-
Post
Random sample
Issues in Study Design
Random
assignment to
groups
Pretest
Intervention
Post-test
Experimental
group
Measurement (X)
Treatment
Measurement (Y)
Control
group
Measurement (X)
No treatment
Measurement (Y)
Issues in Study Design
• Quasi-experimental design (a.k.a. non-equivalent groups design)
resembles experimental design but lacks random assignment
(sometimes also random sampling) and controlled research
environment.
• This type of design is sometimes the only way to do research in
certain populations as it minimizes the threats to external validity
(natural environments instead of artificial ones).
Exp.
Pre
I
Post
Contr.
Pre
-
Post
Random / convenience
sample
Issues in Study Design
• The most popular quantitative approach in educational research,
correlational design (a.k.a. ‘descriptive study’ or ‘observational
study’), allows the use of non-probability sample (a.k.a
‘convenience sample’).
• Most correlational designs are missing control, and thus loose
some of their scientific power (Jackson, 2006).
– Some research journals accept factorial analysis (main and interaction
effects, e.g., MANOVA) based on quasi-experimental design.
Convenience
sample
Exp.
Pre
I
Post
Issues in Study Design
– Observational studies can further be classified into
cross-sectional and longitudinal studies (see Caskie &
Willis, 2006).
• Longitudinal design includes series of measurements over
time.
– Change over time, age effect.
• Cross-sectional study involves usually one measurement
and is thus considerably cheaper and faster to conduct
(although producing less controllable and less powerful
results).
– If there are several measurements, individual participants answers
are not connected over time (e.g., due to anonymity).
– Causal conclusions are usually out of scope of this research type
(ibid.).
Issues in Study Design
•
Longitudinal design
–
–
One sample that remains the same throughout the study.
Longitudinal study produces more convincing results as it allows the
understanding of change in a construct over time and variability and
predictors of such change over time (ibid.).
– However, it takes naturally more time to carry out and suffers
from participant drop-out.
Sample
Pretest
Intervention
Post-test
Random
sample
Measurement (X)
Treatment
Measurement (Y)
Issues in Study Design
•
Cross-sectional design
–
Measurement is conducted once (or several times) and the sample varies
throughout the study.
Sample
Pretest
Intervention
Post-test
Convenience or
random
sample
Treatment
Measurement (Y)
Convenience or
random
sample
No treatment
Measurement (Y)
Issues in Study Design
RANDOM
SAMPLING
RANDOM
SELECTION
PRETEST-POSTTEST RANDOMIZED EXPERIMENT
TEST
RS
CONTROL
Pre
I
Post
Pre
-
Post
NON-EQUIVALENT GROUPS DESIGN
TEST
Pre
I
Post
CONTROL
Pre
-
Post
I
Post
RS
CORRELATIONAL DESIGN
CS
TEST
Pre
Issues in Study Design
• Why do, then, educational scholars use correlational designs over
controlled experiments?
• The first answer is simple: Correlational designs are far easier,
faster and inexpensive to conduct than experimental designs.
• The second answer is more complex as we need to ask if the
controlled experiment approach is at all viable method to study
educational research questions.
Issues in Study Design
• In science and psychology, most areas of interest are
quite easily quantifiable and replicable (like, for
example, freezing point of chocolate or systolic blood
pressure).
• However, in educational research we study, for
example, topics like ‘pedagogical aspects of digital
learning material’ (Nokelainen, 2006) or compare preexisting characteristics of interest (e.g., gender, age,
educational level).
– In such situations researchers do apply correlational designs,
but still aim to employ different types of data in the analysis
with a complementary way (quasi-experimental study).
Issues in Study Design
•
Case study design is applied in qualitative research.
–
–
The aim is to collect information from one or more cases and stydy,
describe and explain them through how and why questions.
Cases are represented, for example, by individuals, their communication
and experiences. (For thorough discussion, see Flyvbjerg, 2004.)
Issues in Study Design
– As a conclusion, Abelson’s (1995) concept of
statistics as principled argument becomes useful:
• Data analysis should not be pointlessly formal, but instead
“ ... it should make an interesting claim; it should tell a
story that an informed audience will care about and it
should do so by intelligent interpretation of appropriate
evidence from empirical measurements or observations”
(p. 2).
Issues in Study Design
• Second, optimal sample size
(N) is divided into two
sections in the figure:
– Samples that operate in the
optimal area (n  30 – 250) for
traditional parametric
frequentistic techniques (Black,
1993; Tabachnick & Fidell,
1996), such as t-test or
exploratory factor analysis, and
the samples that fail to do so (n
< 30 or n > 250).
Estimation of sample size
• N
– Population size.
• n
– Estimated sample size.
• Sampling error (e)
– Difference between the true
(unknown) value and observed
values, if the survey were
repeated (=sample collected)
numerous times.
• Confidence interval
– Spread of the observed values
that would be seen if the survey
were repeated numerous times.
• Confidence level
– How often the observed values
would be within sampling error of
the true value if the survey were
repeated numerous times.
(Murphy & Myors, 1998)
Issues in Study Design
• Traditional non-parametric techniques, such as MannWhitney U-test, are considered to operate robustly, also
with small samples (-> lack of power?).
– Bayesian approach, however, is free of such restrictions.
Issues in Study Design
• Third, independent
observations (IO) are
always expected, also in
time series analysis.
Issues in Study Design
• Controlled experiment designs,
when conducted properly, rule
out IO violations quite
effectively (Martin, 2004), but
correlational designs usually
lack such control (e.g., to rule
out employee’s co-operation
when they respond to the
survey questions).
– On the other hand, some
qualitative techniques, like focus
group analysis (Macnaghten &
Myers, 2004), are heavily based
on non-independent
observations as informants are
asked to talk to each other as an
important part of the data
collection.
Issues in Study Design
• Fourth, parametric
techniques assume
continuous (c)
measurement level (ML)
of indicators (i.e., so
called ‘quantitative’
variables).
Issues in Study Design
PHENOMENON
Discrete
0 1 2, ..
OBSERVATION
Discrete
0 1 2, ..
Issues in Study Design
PHENOMENON
Continuous
0
∞
OBSERVATION
Discrete
0 1 2, ..
Continuous
0
∞
Issues in Study Design
Measurements
Qualitative
Quantitative
Discrete
Nominal
Ordinal
Continuous
Ordinal
Interval
Ratio
Issues in Study Design
• Non-parametric analysis is
based on ordering of values
and thus discrete (d) or, when
applicable, nominal (n) values
are expected (i.e., so called
‘qualitative’ variables).
– A respondent’s income level
(euros) or age (years or months)
is a representative example of
the first indicator type.
– A Likert scale from 1 to 5 is an
example of the second indicator
type (ordered discrete values).
– Respondent’s gender is an
example of the third indicator
type (nominal discrete values).
Issues in Study Design
– It is important to note that the central limit theorem,
discovered by Pierre-Simon Laplace (1749 - 1827),
assures an approximate normal distribution for
practically all sums of independent random variables.
• For example, it allows the use of parametric t-test with
binomial or ordinal indicators (as the sample of normally
distributed group means are compared, not the indicator
values themselves).
– Bayesian analysis is based on discrete values, and
thus, continuous values must be disceticized
(automatically or manually) before the analysis.
Issues in Study Design
• Fifth, parametric techniques
are technically based on the
assumption of the
multivariate distribution (MD)
that is normal (n) by nature.
• Non-parametric techniques
expect any shaped similar
distributions (s).
– This is a great news to anyone
who has collected real-life
educational science empirical
data and checked both
univariate and multivariate
variable distributions as usually
almost all variables violate quite
heavily against the normal
distribution assumption with
small sample sizes
(e.g., below n = 100).
Issues in Study Design
• Some researchers try to force their indicators to follow
multivariate normal distribution by applying various
transformation techniques (e.g., logarithmic, square),
but with varying success.
– The motivation for transformations lies behind the fact that in
order to enable parametric analysis (i.e., based on, e.g.,
normal distribution) the bivariate or multivariate statistical
dependencies (S) must be linear (l).
– It is important to note that this assumption does not hold for
the Bayesian techniques.
Issues in Study Design
Non-parametric statistics
Chi-square 2
Multiway Frequency Analysis 2
Spearman Rank Order Correlation rS
Mann-Whitney U
Wilcoxon Signed Rank
Kruskal-Wallis H
Friedman
Bayesian dependency modeling (B-Course)
Logit analysis, Logistic regression
Bayesian classification modeling (B-Course)
Categorical variable modeling (Mplus)
Parametric statistics
Pearson Product Moment Correlation rP
Independent-samples t
Paired-samples t
One-way between-groups ANOVA F
Two-way repeated-measures ANOVA F
ANCOVA, MANOVA
Regression analysis R
Exploratory factor analysis
Principal component analysis
Cluster analysis
Discriminant analysis
Classification analysis
Confirmatory factor analysis
Issues in Study Design
• Sixth, extreme values, outliers
(O), affect the results and,
thus, the conclusions, of some
parametric techniques
severely (e.g., regression and
discriminant analysis) and
should be recognized and
removed (see, e.g.,
Tabachnick & Fidell, 1996).
– Non-parametric analysis
techniques are not affected by
such values as their analysis is
not based on multivariate
normal assumption (i.e., linear
dependencies between
variables).
Issues in Study Design
• Seventh, when calculating
correlations (C), Pearson
product moment
correlation (rP) should be
applied with continuous
indicators, and Spearman
rank-order correlation (rS)
with ordinal indicators.
– Both techniques are valid
to detect linear
dependencies.
Issues in Study Design
• The last point is to discuss
about the two types of
statistical dependencies
(S) among the variables
under analysis, namely
linear (l) and non-linear
(nl).
Issues in Study Design
• It is natural to assume,
that both parametric and
non-parametric
techniques designed to
detect linear
dependencies work best
with samples that contain
linear dependencies.
– However, there are nonlinear techniques, such as
Bayesian analysis and
neural networks that also
allow the investigation of
both dependency types.
Issues in Study Design
• The figure contains a
reference to the
qualitative analysis
techniques, referring here
to the empirical textual
evidence based approach
(e.g., individual or focus
group interviews, narrative
stories).
Issues in Study Design
• Firstly, it is obvious that qualitative research operates
with small samples (usually n < 30).
– There is nothing suspicious working with small samples:
Bartlett, Pavlov, Piaget and Skinner did that too!
Issues in Study Design
• Secondly, probability samples could also be used by
qualitative researchers (as stated in the figure), but not
as the only way to produce scientifically important
findings.
• Gobo (2004) illustrates this by listing important
qualitative research studies based solely on nonprobability samples:
–
–
–
–
–
Alvin Gouldner (1920-1980)
Howard Becker (1928-)
Ernest De Martino (1908-1965)
David Sudnow (1938-2007)
Aaron Cicourel (1928-).
Issues in Study Design
• Gobo (2004) defines a new concept of generalizability for
qualitative research by arguing that the concept of
generalizability is based on the idea of social
representativeness, which allows the generalizability to
become a function of the invariance (regularities) of the
phenomenon.
– Thus, “The ethnographer does not generalize one case or event
… but its main structural aspects that can be noticed in other
cases or events of the same kind or class.” (id., p. 453.)
Issues in Study Design
• Thirdly, both qualitative and Bayesian analysis
techniques allow researcher to apply a priori input to
the modeling process and update the model on the
basis of increased level of knowledge.
Issues in Study Design
Writing Scientific Reports
Ten Questions About YOUR Research
http://www.uta.fi/aktkk/lectures/sw
Writing scientific reports
Original
idea for the
research
Database of
scientific
knowledge
Publication of
the report
Literature
review
Research
questions /
hypotheses
Design of the
study
Sample
Measurements
Theory
RQ’s
Methodology
Results
Conclusions
Discussion
Methodology
Link between RQ’s
and statistical
analyses
NO TURNING BACK!
Peer Review
Writing
scientific report
Data analysis
Data collection
Writing scientific reports
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
Title
Author(s) name(s) and affiliation(s)
Abstract and keywords
Introduction / Goals or aims of the study (periodicals: research
questions)
Theoretical framework / literature review (periodicals: research
questions)
Research questions (dissertation)
Method
7.1 Sample, participants
7.2 Measures / instruments
7.3 Procedure
7.4 Statistical analyses
Results
Conclusion(s) and/or Summary
Discussion
Acknowledgements / credits
References
Appendix(es)
Writing scientific reports
• 1. Introduction
• School leadership is currently one of the most widely studied and
published areas in social sciences. However, leadership as a social
process, affecting both end products and personnel emotions, is
seldom studied (Nokelainen & Ruohotie, 2006). In this sense one
interesting direction to look at is Emotional Intelligence (EI)
research that has recently become one of the most important
constructs in modern psychological research. EI refers to “the
competence to identify, express and understand emotions,
assimilate emotions in thought, and regulate both positive and
negative emotions in one and others” (Matthews, Zeidner, &
Roberts, 2002, p. 123).
Writing scientific reports
• 2. Theoretical Framework
• The theory as formulated by Salovey and Mayer (1990; Mayer &
Salovey, 1997) framed EI within a model of intelligence.
Goleman’s model formulates EI in terms of a theory of
performance (1998b). Goleman argues (2001) that an EI-based
theory of performance has direct applicability to the domain of
work and organizational effectiveness, particularly predicting
excellence in jobs of all kinds, from sales to leadership. Goleman,
Boyatzis and McKee further state (2002, p. 38) that EI
characteristics are not innate talents, but learned abilities.
Writing scientific reports
• 2. Theoretical Framework
• Theoretical framework is summarized in Figure 1. Figure
represents self-regulation (Zimmerman, 1998, 2000) as a system
concept (Boekaerts & Niemivirta, 2000) managing leadership
behavior through interactive processes between motivation,
volition, emotion, attention, metacognition and action control
systems. As Markku Hannula (2006) points out, self-regulation
should be seen to be much more than mere metacognition.
Writing scientific reports
Figure 1. Self-regulation as a system concept managing leadership
competence through interactive processes between different control
systems. (Adapted from Zimmerman, 2000, p. 15-16.)
Writing scientific reports
• 2. Theoretical Framework
• Daniel Goleman popularized the term emotional intelligence and
claimed that EI was “as powerful and at times more powerful than
IQ” in predicting life success (1995, p. 34). He aimed to show in his
studies that emotional and social factors are important (1995,
1998a), but his “views on EI often went far beyond the evidence
available” (Brackett et al., 2004). A recent study showed that most
popular EI and ability measures are only related at r < .22, i.e.
about five per cent of common variance (Brackett & Mayer, 2003).
Brackett, M. A., Lopes, P., Ivcevic, Z., Pizarro, D., Mayer, J. D., & Salovey, P. (2004). Integrating
emotion and cognition: The role of emotional intelligence. In D. Dai, & R. J. Sternberg (Eds.),
Motivation, emotion, and cognition: Integrating perspectives on intellectual functioning (pp. 175194). Mahawah, NJ: Lawrence Erlbaum Associates.
Writing scientific reports
• 3. Method
• 3.1 Sample
• The non-probability sample consists of 124 Finnish teachers from
four comprehensive (n = 84) and two upper secondary (n = 40)
schools. All the schools were located in Helsinki, capital of Finland
(about 560 000 inhabitants, 9.3% of total population 5 223 442).
Each respondent was personally invited to complete a paper and
pencil version of the questionnaire. Participants were asked to
evaluate their attitude towards the statements measuring
emotional leadership.
Writing scientific reports
• 3. Method
• 3.1 Sample
• The respondents’ age was classified into four categories: (1) 21 to
30 years old (n = 18, 14.5%); (2) 31 to 40 years old (n = 25, 20.2%);
(3) 41 to 50 years old (n = 34, 27.4%); (4) over 50 years old (n = 39,
31.5%). Seventy per cent of the respondents were females (n = 87,
70.2%), the rest were males (n = 29, 23.4%).
Writing scientific reports
• 3. Method
• 3.2 Instrument
• Emotional Leadership Questionnaire operationalises Goleman and
his colleagues (2002) four domains of emotional intelligence
characteristics with 51 items: (1) self-awareness, (2) selfmanagement, (3) social awareness and (4) relationship
management. Table 1 depicts four EL domains and the eighteen
associated characteristics (see Appendix for item level details).
Writing scientific reports
• 4. Results
• Next, we examine with descriptive statistics how subordinates’
evaluated their superior’s emotional leadership. Table 1 shows
that the school principals had quite strong self-awareness (M = 3.7
– 3.8, SD = 0.8 – 1.0). This finding is natural, as especially selfconfidence is an important characteristic of a good leader. On the
other hand, we suspect that this result is partly a self-fulfilling
prophecy as teachers expect to see those atypical Finnish
mentality characteristics strongly present in their leaders.
Writing scientific reports
• 4. Results
Writing scientific reports
Figure 2. Comparison of disagreement (SD) between the
three age groups on the IRSSQ dimensions.
Writing scientific reports
Figure 2. Comparison of disagreement (SD) between the
three age groups on the IRSSQ dimensions.
Writing scientific reports
Figure 3. Bayesian network of Finnish school principals
Emotional Leadership competencies.
Writing scientific reports
Figure 3. Bayesian network of Finnish school principals
Emotional Leadership competencies.
Writing scientific reports
Writing scientific reports
Writing scientific reports
Box-plot
Writing scientific reports
• 5. Conclusions
• In this paper, we presented a 51 item self-rating Likert-scale
Emotional Leadership Questionnaire (ELQ) that operationalises
Goleman’s et al. (2002) four domains of emotional intelligence.
Our goal in this paper was to study with an empirical sample the
construct validity of the four-domain model (Goleman et al., 2002)
of EL. The non-probability sample consisted of 124 Finnish school
teachers from six different capital area schools.
Writing scientific reports
• 6. Discussion
• We asked teacher’s to evaluate their superiors according to our
fixed, person-related questions. In the next version of the ELQ, we
will add an additional scale measuring the importance of each
question in a five-point Likert scale. This allows us to compare
personal level EL factors to other measures, for example, the
Multiple Intelligences Profiling Questionnaire (MIPQ), an
operationalization of Howard Gardner’s’ MI theory, (Tirri, K.,
Komulainen, Nokelainen, & Tirri, H., 2002).
Writing scientific reports
• References (APA style, http://www.apa.org)
• Bar-On, R., Tranel, D., Denburg, N. L., & Bechara, A. (2003). Exploring the
neurological substrate of emotional and social intelligence, Brain, 126(3), 17901800.
• Boekaerts, M., & Niemivirta, M. (2000). Self-regulation in learning: finding a
balance between learning and ego-protective goals. In M. Boekaerts, P. R.
Pintrich, & M. Zeidner (Eds.), Handbook of Self-regulation (pp. 417-450). San
Diego, CA: Academic Press.
• Carmines, E. G., & Zeller, R. A. (1979). Reliability and Validity. Beverly Hills, CA:
Sage Publications.
• EQ Symposium (2004). About Reuven BarOn’s Involvement in Emotional
Intelligence. Retrieved April 13, 2007, from
http://www.cgrowth.com/rb_biolrg.html
Writing scientific reports
Test your APA knowledge: What is WRONG with the following reference list?
Cohen, J. (1988) Statistical power analysis for the behavioral sciences. Second Edition. Hillsdale, NJ,
Lawrence Erlbaum Associates.
Hair, J. F., Anderson, R. E., Tatham R. L. & Black, W. C. (1995). Multivariate data analysis. Fourth edition.
Englewood Cliffs: Prentice Hall.
Howell, David (1997). Statistical Methods for Psychology. Belmont, CA: Wadsworth Publishing
Company.
Nokelainen, P., & Tirri, H. (2007). The Essential Benefits of Using Bayesian Modeling in Professional
Growth Research. In S. Saari & T. Varis (Eds.), Professional Growth, 413-423). Hämeenlinna, FI:
RCVE.
Kerlinger, F. 1986. Foundations of Behavioral Research. Third Edition. New York: CBS College Publishing.
Tirri, K., and Nokelainen, P. (2008). Identification of multiple intelligences with the Multiple Intelligence
Profiling Questionnaire III. Psychology Science Quarterly, 50(2), 206-221.
Issues in Study Design
Writing Scientific Reports
Ten Questions About YOUR Research
Ten questions about YOUR research
Critical Appraisal Skills Programme (CASP)
http://www.phru.nhs.uk/casp/casp.htm
• Rigor
• Credibility
• Relevance
(Abelson, 1995.)
Ten questions about YOUR research
Screening
1. Was there a clear statement of the aims of the study?
2. Is a qualitative/quantitative methodology appropriate?
Research Design
3. Was the research design appropriate to address the aims of the
research?
4. Was the sampling technique/recruitment strategy appropriate
to the aims of the research?
Ten questions about YOUR research
Data collection
5. Were the data collected in a way that addressed the research
issue?
•
•
•
•
•
•
Justification of the setting for data collection.
Clarification how data were collected (e.g., questionnaire, focus group,
semi-structured interview,..).
Justification of the chosen methods.
Explicit report on methods (e.g., how the instruments were delivered,
what were the instructions, how interviews were conducted, was there
a topic guide, how data was stored, ..).
Explicit report on modifications to the methods during the study.
Discussion of the sample size (effect size, power)/saturation of data.
Ten questions about YOUR research
Reflexivity
6. Has the relationship between researcher and participants been
adequately considered?
•
Critical examination of researchers own role, potential bias and
influence during
•
•
•
formulation of research questions
data collection, including sample recruitment and choice of location.
How researcher responded to events during the study.
Ten questions about YOUR research
Ethical issues
7. Have ethical issues been taken into consideration?
•
•
Detailed description how the research was explained to participants – so
that reader is able to assess whether ethical standards were maintained.
Discussion of the issues raised by the study.
Ten questions about YOUR research
Data analysis
8. Was the data analysis sufficiently rigorous?
•
•
•
•
•
•
In-depth description of the analysis process.
Selection of the statistical techniques/use of thematic analysis (e.g., how
the categories/themes were derived from the data?)
Qualitative: How the data presented was selected from the original
sample to demonstrate the analysis process?
Is a sufficient data presented to support the findings?
To what extent contradictory data are taken into account?
Whether the researcher critically examined their own role, potential
bias and influence during analysis (qualitative: selection of data for
presentation.
Ten questions about YOUR research
Findings
9. Is there a clear statement of findings?
•
•
•
•
Explicit findings.
Adequate discussion of the evidence both for and against the
researchers arguments.
Discussion of the credibility of the findings (triangulation, respondent
validation, more than one analyst,..).
Discussion of the findings in relation to the original research questions.
Ten questions about YOUR research
Value of the research
10. How valuable is the research?
•
•
•
•
Contribution to existing knowledge and understanding.
Identification of new areas where research is necessary.
Transferability (generalizability/representativeness) of findings to other
populations.
Consideration of other ways how the research may be used.
References
• Abelson, R. P. (1995). Statistics as Principled Argument. Hillsdale, NJ: Lawrence
Erlbaum Associates.
• Anderson, J. (1995). Cognitive Psychology and Its Implications. Freeman: New
York.
• Bannan-Ritland, B. (2003). The Role of Design in Research: The Integrative
Learning Design Framework. Educational Researcher, 32(1), 21-24.
• Brannen, J. (2004). Working qualitatively and quantitatively. In C. Seale, G.
Gobo, J. Gubrium, & D. Silverman (Eds.), Qualitative Research Practice (pp.
312-326). London: Sage.
• Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Second
edition. Hillsdale, NJ: Lawrence Erlbaum Associates.
• Fisher, R. (1935). The design of experiments. Edinburgh: Oliver & Boyd.
• Flyvbjerg, B. (2004). Five misunderstandings about case-study research. In C.
Seale, J. F. Gubrium, G. Gobo, & D. Silverman (Eds.), Qualitative Research
Practice (pp. 420-434). London: Sage.
References
• Gigerenzer, G. (2000). Adaptive thinking. New York: Oxford University Press.
• Gigerenzer, G., Krauss, S., & Vitouch, O. (2004). The null ritual: What you
always wanted to know about significance testing but were afraid to ask. In D.
Kaplan (Ed.), The SAGE handbook of quantitative methodology for the social
sciences (pp. 391-408). Thousand Oaks: Sage.
• Gobo, G. (2004). Sampling, representativeness and generalizability. In C. Seale,
J. F. Gubrium, G. Gobo, & D. Silverman (Eds.), Qualitative Research Practice
(pp. 435-456). London: Sage.
• Hair, J. F., Anderson, R. E., Tatham, R. L., & Black, W. C. (1998). Multivariate
Data Analysis. Fifth edition. Englewood Cliffs, NJ: Prentice Hall.
References
• Jackson, S. (2006). Research Methods and Statistics. A Critical Thinking
Approach. Second edition. Belmont, CS: Thomson.
• Lavine, M. L. (1999). What is Bayesian Statistics and Why Everything Else is
Wrong. The Journal of Undergraduate Mathematics and Its Applications, 20,
165-174.
• Luoma, M., Nokelainen, P., & Ruohotie, P. (2003, April). Learning Strategies for
Police Organization - Modeling Organizational Learning Prerequisites. Paper
presented at the Annual Meeting of American Educational Research
Association (AERA 2002). New Orleans, USA.
• Nokelainen, P. (2006). An Empirical Assessment of Pedagogical Usability
Criteria for Digital Learning Material with Elementary School Students. Journal
of Educational Technology & Society, 9(2), 178-197.
References
• Nokelainen, P. (2008). Modeling of Professional Growth and Learning: Bayesian
Approach. Tampere: Tampere University Press.
• Nokelainen, P., & Ruohotie, P. (2005). Investigating the Construct Validity of the
Leadership Competence and Characteristics Scale. In the Proceedings of
International Research on Work and Learning 2005 Conference, Sydney,
Australia.
• Nokelainen, P., & Ruohotie, P. (2009). Non-linear Modeling of Growth
Prerequisites in a Finnish Polytechnic Institution of Higher Education. Journal of
Workplace Learning, 21(1), 36-57.
• Thompson, B. (1994). Guidelines for authors. Educational and Psychological
Measurement, 54(4), 837-847.
• de Vaus, D. A. (2004). Research Design in Social Research. Third edition.
London: Sage.