Appendix
Validating of Quality Appraisal Across Study Methods (QASM) tool
Preliminary validation of QASM was undertaken by seeking comments from experienced
researchers. Inter-rater reliability testing which led to refining of the wording of three of the ten
items. Studies included in this review were independently rated by two reviewers. For 95%,
ratings differed by no more than two points; the exceptions differed by three points. Raters agreed
on 84% of all items. Disagreement was attributable to the different disciplinary backgrounds of
the two raters. Discussion led to reconciliation and an agreed rating.
1
Appendix Figure 1
Quality appraisal across study methods (QASM)
Yes
No
Not
able to
determine
1pt
0pt
0pt
0-4 pts weak*; 5-7 pts moderate**; 8-10 pts strong***
Qualitative1
Method
specific
Reliability of
findings
Validity of method and analysis
Validity of method
and analysis (continued)
Appropriate
study
design
Reporting
1. Stated a qualitative objective or question
Quantitative descriptive2
Quantitative comparative3
Mixed Methods4
2. Gave description of demographic
characteristics of interviewed participants,
unless it would compromise anonymity
1. Stated a quantitative objective or
hypothesis
2. Gave description of demographic
characteristics of participants completing
questionnaires/ surveys
1. Stated a quantitative objective or
hypothesis
2. Gave description of demographic
characteristics of participants completing
questionnaires/ surveys
1. Stated objectives, hypotheses or research questions
fitting to mixed methods research
2. Gave description of demographic characteristics of
participants unless it would compromise anonymity
3. Gave clear description of qualitative
methods
3. Gave clear description of quantitative
methods
3. Gave clear description of quantitative
methods
3. Gave clear description of mixed methods
4. Used appropriate sampling for stated
qualitative objective or question
4. Used appropriate sampling for stated
quantitative objective or hypothesis
4. Used appropriate sampling for stated
quantitative objective or hypothesis
4. Used appropriate sampling for stated mixed methods
objective, question or hypothesis
5. Qualitative methods were appropriate for 5. Quantitative methods were appropriate
stated objective or question
for stated objective or hypothesis
5. Quantitative methods were appropriate for 5. Mixed methods were appropriate for stated objective,
stated objective or hypothesis
question or hypothesis
6. Justified the qualitative data collection
6. Used appropriate outcome measures
6. Used appropriate outcome measures (e.g. 6. Justified qualitative data collection approach and
(e.g. including appropriate framework used (e.g. validated measures or measures with
validated measures or measures with face
used appropriate quantitative outcome measures
to structure questions or prompts) or
face validity)
validity)
generation (e.g. of ethnographic data)
7. Addressed researcher’s reflexivity,
7. Addressed bias (e.g. by including
7. Addressed bias (e.g. by concealing
7. Addressed bias relative to research design
critically examining the research process
discussion of sample representativeness)
allocation and blinding if appropriate)
appropriate to qualitative and quantitative methods
(e.g. discussed how factors within the
that were used
research process might have shaped its
outcome)
8. Used analytic strategies that represented 8. Reported controlling of confounding
8. Reported comparability of study groups,
8. Considered qualitative themes from a minority of
and explored both minority and majority
variables
and where not comparable, controlled for
participants alongside dominant themes as well as
views (e.g. probing minority views or the
confounding variables
quantitative variables that could lead to alternative
‘dissenting voice’)
explanations of the data
9. Used appropriate method to examine the 9. Used appropriate method to examine
9. Used appropriate method to examine the
reliability of findings extracted from
the reliability of findings, using data
reliability of findings, using data collected
interview/ethnographic data (e.g. checking
collected by questionnaire/survey (e.g.
by questionnaire/survey (e.g. used two
reliability of thematic coding by
examined the same research question with sets of experiments to test the same
independent assessment of transcripts)
two different sets of data; including ‘open’
research question)
survey questions)
10. Gives clear description of context in
10. Achieved complete or near complete
10. Achieved complete or near complete
which qualitative data is being generated
outcome data and non responses. If not,
outcome data and non responses. If not,
described characteristics of nondescribed characteristics of nonrespondents or drop outs
respondents or drop outs
9. Used triangulation of qualitative
interview/ethnographic data and quantitative
questionnaire/survey data to discuss reliability of
findings
10. Integrated qualitative or quantitative methods or
results
1) data generated by interview/ethnographic methods; 2) data generated by quantitative methods; no comparative experimental group; 3) data generated by quantitative
methods; with comparative group; 4) data generated by quantitative questionnaire/survey methods & qualitative interview/ethnographic methods
2