Cochrane or cock-eyed? how should we conduct systematic reviews
of qualitative research?
Andrew Booth, Senior Lecturer in Evidence Based Healthcare Information, School of Health
and Related Research, University of Sheffield, S1 4DA Email: A.Booth@sheffield.ac.uk
Paper presented at the Qualitative Evidence-based Practice Conference, Taking a Critical
Stance. Coventry University, May 14-16 2001
Abstract
The quantitative versus qualitative debate has taken significant steps towards reconciliation
within the wider field of evidence based practice. Nevertheless, far more insidious discrimination
remains. Systematic review methodology exhibits all the characteristics of "institutionalised
quantitativism" in that criteria for a "good" review are almost entirely determined by the
quantitative methods promoted and perpetuated by the Cochrane Collaboration. Nobody who
understands qualitative research would insist that its primary studies demonstrate alien concepts
such as "sample size" or "statistical power". Yet comparably fundamental absurdities persist with
regard to qualitative syntheses. Why should systematic reviewers of qualitative research pursue a
"gold standard" comprehensive literature search when concepts such as "data saturation" have an
established pedigree? Why should they apologise for an absence of meta-analysis when littleknown techniques such as meta-ethnography could be included in a reviewer's toolbox? Why
shouldn't they apply systematic, explicit and reproducible principles of thematic or concept
analysis to create syntheses that advance our understanding of qualitative issues and highlight
research gaps? The author draws on experience of a dozen systematic reviews, a third qualitative,
to suggest how systematic reviews of qualitative research might acquire a methodology that is
more sympathetic to the paradigm within which they are conducted.
Introduction
Meta-synthesis, the "science of summing up"(Light & Pillemer 1984), is a building block of
evidence based practice. It may be either quantitative (meta-analysis) or qualitative (metaethnography). Its value lies in the recognition that busy practitioners find it almost impossible to
make decisions based upon the massive and increasing volume of research evidence (Sleep &
Clark, 1999). They therefore require an overview of work in a specific area that has been
compiled by "finding, evaluating and synthesising evidence from research studies" (Hek,
Langton & Blunden, 2000). Historical precedent for the value of such an enterprise dates back at
least to Aristotle:
"it is necessary, while formulating the problems of which in our further advance we are
to find the solutions, to call into council the views of those of our predecessors who have
declared an opinion on the subject, in order that we may profit by whatever is sound in
their suggestions and avoid their errors". (Aristotle, De Anima quoted in Bland et al,
1995)
Early experiences of evidence based practice, particularly in the context of critical appraisal,
were invariably accompanied by the postscript "But what about qualitative research?". To a
certain extent this polarisation into quantitative and qualitative camps has been headed off at
every level of the evidence based practice paradigm. Sackett, one of the founding fathers of
evidence based healthcare, issued a message of reconciliation through a BMJ Editorial (Sackett
& Wennberg, 1997). Similarly Greenhalgh, another leading proponent of the fledgling
movement, made a persuasive appeal that "How evidence based is my practice" should be
answered in qualitative as well as quantitative terms (Greenhalgh, 1996) and also advocated the
importance of Narrative Based Medicine (Greenhalgh, 1999). Within the specific technique of
critical appraisal the Critical Appraisal Skills Programme (CASP) in Oxford, among many
others, has developed a checklist for appraising qualitative research studies. At the producer, as
opposed to consumer, end of the spectrum, recognition by the Cochrane Collaboration of
qualitative research was signalled by the formation of a Qualitative Research Methods Network.
Notwithstanding such advances systematic review methodology exhibits all the characteristics of
"institutionalised quantitativism" in that criteria for a "good" review are almost entirely
determined by the quantitative methods promoted and perpetuated by the Cochrane
Collaboration. This appears to be not so much a conscious effort by the Collaboration to
discriminate against this particular paradigm of research methodology as a profound lack of
awareness of the potential contribution of qualitative methods of meta-synthesis to our
knowledge of healthcare interventions. A search of the Cochrane Systematic Review
Methodology database in early 1997 yielded only one reference on qualitative systematic
reviews - the seminal work Meta-ethnography by Noblit and Hare (1988).
Methods
A citation search for the Noblit and Hare reference on the Science Citation and Social Science
Citation Indexes via the Web of Science yielded 15 relevant articles. These comprised both
methodological discussions and case studies of the use of meta-ethnographic approaches. This
led, in turn, to a subject search using the keywords "meta-ethnograph$", "metaethnograph$",
"meta-synthesis", and "metasynthesis" on the MEDLINE, CINAHL and Science and Social
Science Citation Indexes to retrieve additional articles. This cursory literature search was
designed to provide an external frame of reference against which the experience of the author, in
conducting four collaborative systematic literature reviews, could be compared. These four
systematic reviews, underpinned to varying degrees by the methods described by Noblit and
Hare, were as follows:
 A systematic review for the NHS Health Technology Assessment Programme on the Use of
Modelling in the Planning and Prioritisation of Clinical Trials (Chilcott et al, In press)
 A systematic review for the NHS Health Technology Assessment Programme on the Use of
Health-related Quality of Life Measures in Economic Evaluation (Brazier et al, 1999)
 A systematic review for the English National Board for Nursing, Midwifery and Health
Visiting on the nursing contribution to rehabilitation (Nolan et al, 1997)
 A systematic review for the American Association for Retired Persons on coping with
multiple chronic health conditions in older persons (Dowzer et al, 2000)
Literature Review
Techniques in meta-ethnography originated within the field of education as a means of
synthesising the findings from school inspection reports (Noblit and Hare, 1988). Following
identification of a definable set of studies an iterative process of analysis takes place, typically
using the using the constant comparative method (Glaser & Strauss, 1967; Kearney 1998a). This
leads to the identification of themes or metaphors that permeate the literature (Beck, 2001). The
potential of meta-ethnography as a tool for knowledge synthesis was identified at an early stage
of its development (Klein, 1989). From education it was a comparatively short migration to the
contiguous field of sociology where Zhao (1991) placed such techniques within the broad
categorisation of meta-study and its associated branches of metatheory, metamethod and
metadata-analysis.
Within the healthcare literature meta-synthesis appears most frequently in nursing research
studies (Sherwood, 1999). Occasionally meta-ethnography's origins in education resurface
intriguingly in a nursing context (Beck, 2001).
It is interesting to observe clusters of studies around a small number of broad topics:
 Chronic disease (Barroso and Powell-Cope, 2000; Thorne & Paterson, 1998)
 Coping (Burke et al, 1998; Paterson et al, 1998)
 Caring (Beck, 2001; Frederiksson, 1999; Sherwood, 1997)
Ironically the burgeoning interest in the application of the Cochrane Collaboration's systematic
reviews methods to the fields of social work, law and education has resulted in the formation of a
parallel organisation, the Campbell Collaboration. This in turn has led to a resurgence for metaethnography in the education literature, alongside other methods of synthesis such as narrative
reviews, vote-counting reviews, meta-analyses and best evidence synthesis (Davies, 2000).
Applications of meta-synthesis within evidence based practice
Qualitative meta-synthesis can make a number of possible contributions to evidence based
practice. It can be used:
(1) For the generation of models and theories (Estabrooks et al, 1994; Finfgeld, 1999; Thorne
and Paterson, 1998).
(2) For the subsequent validation and gauging of the empirical support for theories (Forte, 1998).
(3) In methodological reviews as a means of determining the existence of certain bodies or
"schools" of thought.
(4) To chart the development of concepts longitudinally in the literature over a period of time
(Thorne and Paterson, 1998).
(5) To complement the findings and interpretation of quantitative systematic reviews, for
example in looking at issues around patient acceptance or compliance.
(6) As a means of identifying significant domains or attributes to enable the development of
prototype instruments or scales.
Unlike a quantitative systematic review or meta-analysis where the review process is conducted
linearly in a series of discrete, sequential steps and where deviation from an a priori protocol
poses threats to validity, the corresponding method in a qualitative review involves continual
iteration and retracing of steps. In fact it is arguable whether an a priori review protocol is
appropriate to an interpretative process such as meta-ethnography (itself more akin to the process
of discovery initiated by a grounded theory approach in primary qualitative research). However
for ease of presentation it will suffice to examine each of the stages of the review process for a
qualitative review in turn.
Study Identification
Why should systematic reviewers of qualitative research pursue a "gold standard" comprehensive
literature search when concepts such as "data saturation" have an established pedigree?
According to NHSCRD guidelines for conducting systematic reviews of effectiveness (2000):
"The aim of the search is to generate as comprehensive a list as possible of primary
studies, both published and unpublished, which may be suitable for answering the
questions posed in the review. Identification of relevant literature by a thorough unbiased
search strategy is crucial. This is because the validity of the findings of a review is
directly related to the comprehensiveness of the literature search that is used to capture
the relevant studies."
Here is an example of an instance where the "big is beautiful " approach to systematic reviews
required by quantitative reviews is implicitly extended to cover qualitative reviews. Many of the
techniques for checking the validity of systematic reviews are underwritten by the fundamental
assumption that one is attempting to identify, if not the whole population of studies, as many
relevant articles as possible. The funnel plot, for example, is an attempt to quantify the effect that
missing studies from this optimal population might have on the results of the review. The intent
of the literature search for a quantitative review is to identify a statistically representative sample
of the published literature.
Methodological reviews, such as those produced for the NHS Health Technology Assessment
Programme, make much of the virtue of the comprehensiveness of the data sources that have
been searched. However, such reviews belong more appropriately within a qualitative paradigm.
Is it necessary for a qualitative review to be equally comprehensive in its coverage? At this stage
it will be appropriate to draw a parallel, not with quantitative systematic reviews, but with
primary qualitative research. Statistical representativeness is not a primary requirement for
qualitative data collection where systematic non-probabilistic sampling is more commonly used
(Mays and Pope, 1996). If we think of the papers in a qualitative review as being 'informants'
then we seek to identify specific groups of papers that possess characteristics that are relevant to
the phenomenon being studied. We aim to include a wide range of types of papers (i.e. reflecting
as many of the themes or schools of thought as possible) and, preferably to select 'key
informants' (i.e. papers that lead us to important sources of knowledge in the form of additional
citations). The concept of 'data saturation' is also of relevance here. Once a particular theme has
been identified further occurrences of this theme are only of interest in strictly quantitative terms
unless they expand on or modify an already-identified theme. Interestingly quantitative reviewers
are currently seeking methods to define a 'law of diminishing returns' beyond which further
literature searching has little justification in order to manage the inordinate expense of the
searching process. For qualitative reviews the answer to this problem already exists in the
principles of data saturation used in primary studies.
Case study : Systematic review on the Use of Modelling in the Planning and Prioritisation
of Clinical Trials
Although the proposal originally submitted to the NHS HTA Programme focused on traditional
methods of comprehensive literature searching we also piloted methods of 'citation snowballing'
to investigate their potential for methodological reviews. A team of subject experts identified
seven key papers representing a number of perspectives on the use of modelling. References at
the end of these papers were examined in context and a further 34 papers were identified in this
manner. References at the end of these 34 papers led in turn to a further 200 papers.
Unfortunately the resources for this project did not allow more than two 'rounds' of this approach
- we were unable to follow up the next 'round' of references (n=425). In an ideal world this
process would be continued until a point of data saturation is reached i.e. no new references are
identified.
Within this context the technique of citation searching is of particular interest. Citation
searching, although necessarily complementary to comprehensive literature searching, is viewed
with some caution by quantitative reviewers. It can lead to reference or citation bias whereby
supporting articles are more likely to be cited and conflicting articles are more likely to be
omitted. However, in contrast to the effectiveness literature, where such dangers are likely to be
critical, there is little reason to believe that this is equally problematic in the literature of
methodology where techniques of argument and counter-argument are essential for the
development of ideas and schools of thought. Using the analogy of the 'key informant', however,
one might propose that a 'key' article in this context is one that references as broad a selection of
seminal articles as possible rather than one which is itself cited by numerous articles! In practice
triangulation of both approaches with citation searching (i.e. forward and backwards) will be
required.
So does the following statement from the NHS CRD guidelines apply equally to qualitative
reviews? :
"The thoroughness of the literature search is one factor that distinguishes systematic
reviews from traditional reviews. It is also important to ensure that the process is as
thorough and unbiased as possible".
It might be argued that, while it is certainly important for the search process to be free from bias,
it is more important that the search process be systematic, explicit and reproducible rather than
comprehensive. Thoroughness in this context should apply to the rigour of the search process not
to its comprehensiveness. To argue otherwise would most definitely be "cock-eyed".
Literature searching for qualitative systematic reviews should exhibit the following
characteristics:
a) Identifying major "schools of thought" in a particular area whilst being alert to the
identification of variants, minority views and dissenters. It is particularly important to
identify negative or disconfirming cases (Paterson et al, 1998)
b) Searching within a broad range of disciplines so as to bring different views (e.g. clinician,
consumer, manager, health economist, statistician, research commissioner etceteras) to bear
on the topic in hand.
c) Using complementary electronic and manual search techniques to ensure that materials are
not missed either through the inadequacies of indexing or through selective coverage of
databases
Study Appraisal
Techniques in study appraisal are also dominated by the quantitative paradigm. Quantitative
systematic reviews run the danger of underestimating the evidence from relevant literature if
they only include that of a certain methodological quality (Edwards et al, 1998). This might
result in missing or distorting the true message that the review is trying to identify. Perversely,
using 'weaker' study designs tends to have the effect of amplifying the strength of a demonstrable
effect and so quantitative reviewers are usually willing to trade the loss of additional perspectives
against improved precision of findings. Clearly, in a qualitative review, if research is rejected on
the basis of design alone there is a very real risk of denying valuable insights that contribute to
our interpretation of a phenomenon. Qualitative research, with its relativist perspective,
acknowledges the existence of multiple views of equal validity (Popay et al, 1998) and is not
readily amenable to the use of the hierarchies of evidence promulgated by evidence based
medicine (Barbour, 1998; Barbour, 2001).
Edwards and colleagues (1998) recognise the importance of assessing the message or 'signal'
within each individual piece of research. They advocate balancing an assessment of
methodological quality against the weight of its message, rather than rejecting studies that fall
below a certain quality threshold. They argue that fundamentally flawed research will still be
rejected, but other papers will have the importance attached to their signal tempered by the
amount of 'noise' around that signal. The balance of these two elements, termed the 'signal to
noise ratio', is of great relevance to techniques of meta-ethnography.
Barbour (2001) has recently introduced an important dissenting voice to the stampede to adopt
critical appraisal techniques in order for qualitative research to "pass muster" in the new regime
of evidence based practice. She describes a number of "technical fixes" (such as purposive
sampling, grounded theory, multiple coding, triangulation, and respondent validation) that simply
may serve to reassure those that question the contribution that qualitative research might make
and yet not advance a broader understanding of the rationale and assumptions behind qualitative
research.
Study Analysis
With the cautions of Barbour (2001) about the cosmetic overuse of multiple coding still fresh in
mind it is perhaps imprudent to assert the need for two observers in qualitative reviews when
data extraction and, more importantly, coding of themes takes place (Compare Paterson et al,
1998). Barbour (2001) correctly identifies that "multiple coding concerns the same issue as the
quantitative equivalent "inter-rater reliability" and is a response to the charge of subjectivity
sometimes levelled at the process of qualitative data analysis". On the surface it appears another
example of qualitative reviewers having to play by the rules pioneered by quantitative reviewers.
However to what extent is the use of multiple coders a property of the material being reviewed
and how much is it a function of essential systematic review methodology?
Here the answer to our question does not seem to lie in the fact that both quantitative and
qualitative reviewers have arrived at a common device of multiple observers. Rather its
importance lies in why it is being done. In quantitative reviews the aim is to achieve an
acceptable degree of concordance between researchers, thereby increasing confidence in the
validity of the review. This is usually assessed using the device of a kappa statistic. However as
Barbour (2001) again observes, admittedly in the context of primary qualitative research:
"the degree of concordance between researchers is not really important; what is
ultimately of value is the content of disagreements and the insights that discussion can
provide for refining coding frames. The greatest potential of multiple coding lies in its
capacity to furnish alternative interpretations and thereby……in alerting researchers to
all potentially competing explanations. Such exercises encourage thoroughness, both in
interrogating the data at hand and in providing an account of how an analysis was
developed. Whether this is carried out by a conscientious lone researcher, by a team, or
by involving independent experts is immaterial: what matters is that a systematic process
is followed and that this is rendered transparent in the written research project".
The use of multiple observers in study analysis for qualitative reviews, such as was the case in
our comorbidity review, should be clearly defined as being for the unique requirements of
qualitative research. It should not be misinterpreted as an attempt to adopt uncritically the model
promulgated by quantitative reviews of effectiveness.
The 'signal to noise' metaphor from the field of communication (mentioned above) is also
pertinent to debates concerning whether one should conduct meta-ethnography of authors' own
interpretations of data or only of verbatim comments reported by authors in their texts. It is
possible to see the two sorts of data, that is:
 Primary data from transcripts, and
 Secondary data from published articles
as being complementary sources of a 'signal'. Primary data has less potential for noise but is of
such large quantities that it is very difficult to pinpoint the exact nature of a 'signal' while
secondary data is much more selective, is amplified by the authors' interpretations but therefore
runs the risk of losing integrity. Interestingly such issues are analogous to debates about
individual patient data versus literature (Stewart & Parmar, 1993) in the meta-analytic domain.
Thematic analysis can be characterised within two approaches:
a) An approach analogous to grounded theory (Wuest 2000) is used whereby the constant
comparative method produces insights from the literature that are 'discovered' rather than
being a preconceived framework used by the researcher.
b) An existing organising construct is used to provide meaning to the disparate themes
identified from the analysis (Frederiksson, 1999).
The reviews outlined in this paper include both approaches to thematic analysis; the comorbidity
review used a grounded theory approach to discover approaches to coping as documented in the
literature while the rehabilitation review used a conceptual framework by Corbin and Strauss
(1988) as a means of structuring themes according to the characteristics of the target conditions.
Study Synthesis
According to Sandowski and colleagues (1997) the accumulation of qualitative studies in recent
years has led to little cumulation of the understandings gained from them. Techniques used have
included reciprocal translations of key metaphors and concepts and qualitative and quantitative
comparative analyses to produce narrative and theoretical integrations. The major problem, yet to
be resolved, is developing usable and communicable systematic approaches that synthesise
composite findings and yet maintain the integrity of individual studies.
Once the synthesis has been completed follow-up validation strategies may be required to
safeguard the validity of the findings (Mays and Pope, 1996). Several techniques are available
and some of these have been employed in the reviews outlined in this paper:
a) "Triangulation", comparing oral testimony from subjects with the written record of the
systematic review, was used in the review for the American Association for Retired Persons.
Phase 2 of this project involved eliciting data from those experiencing multiple chronic
health conditions to see if their perceptions matched the framework derived from the
literature.
b) The framework developed from the methodological literature from the review on the use of
modelling in trials was validated by the random selection of case studies to be assessed
against this framework
c) A checklist derived from the literature of the use of health-related quality of life measures in
economic evaluation was validated by sampling from a single year of the health economics
literature.
d) Findings from the general literature on the role of the nurse in rehabilitation were then
examined using theoretical sampling in conditions that were characterised according to a
two-by-two matrix of acute versus chronic and young-onset versus old-onset.
Other NHS Health Technology Assessment reviews have attempted to validate constructs from
the literature in interviews or with focus groups. From the broader literature Khunti (1999)
describes how multiple methods (systematic literature searching, brainstorming with health
professionals and patients, a focus group and interviews with key informants) can be used for
hypothesis generation. It is interesting in this context to note that the literature review identified
37 out of 54 putative factors (68.5%) affecting the quality of care of patients with diabetes. A
combination of brainstorming and literature review identified 51 out of 54 factors (an impressive
94.4%).
Meta-study permits critical analysis of the location of current inquiry within the larger context of
systematic patterns in knowledge development (Thorne and Paterson, 1998). It is important to
note, however, that the sheer quantities of data involved may result in certain arbitrary decisions
being made. For example Thorne and Paterson (1998) only examine in depth one of a large
number of themes identified while it is not unusual (in common with primary qualitative
research) for researchers to report only the "top themes" identified from the literature which may
result in the loss of important concepts.
In addition to detailed exposition of multiple themes and in-depth analysis of those regarded as
being specifically important it is essential that the qualitative reviewer indicates the existence of
negative or disconfirming cases (Paterson et al, 1998). This is particularly the case in
methodological reviews where a minority approach may have tremendous potential for
development but be otherwise overshadowed by the existence of a substantive corpus of work.
This is analogous to the importance of investigating sources of homogeneity in quantitative
meta-analyses (Thompson, 1994).
Summary
From the brief exploration above it is possible to identify some of the defining characteristics of
qualitative systematic reviews:
(1) Their intention is not to identify all literature on a particular topic. Whilst the aim is to
identify specific groups of papers that possess characteristics that are relevant to the
phenomenon being studied this need not imply statistical representativeness.
(2) Particular attention is focused on negative or disconfirming cases. This adds to the richness
of the insight that the review provides on the phenomenon of interest.
(3) Such reviews acknowledge the existence of multiple views. Caution must therefore be
exercised so that valuable insights are not lost by application of some a priori definition of
quality through critical appraisal.
(4) Although use of multiple reviewers may be in common with quantitative reviews the
essential contribution of this to a qualitative review lies in its capacity to furnish alternative
interpretations, thereby alerting researchers to potentially competing explanations.
(5) Whereas quantitative reviews seek to handle "noise" by exclusion (through the application of
a predetermined quality threshold) qualitative reviews seek to optimise the "signal to noise"
ratio.
(6) Following on from (5) it will often be necessary to validate the findings from a qualitative
review with reference to a subset of case studies, use of key informants or focus groups or
other methods of triangulation.
The Way Forward?
As qualitative methods become popular and qualitative reports abundant, it is a sign of
methodological maturity when researchers discuss techniques for synthesising findings about
related phenomena from diverse samples (Kearney, 1998b). Advocates of evidence based
practice are starting to look beyond the more established models of research synthesis for
innovative approaches (Greenhalgh, 1998; Sleep & Clark, 1999). Qualitative metasynthesis is
one response to concerns about the relevance and utility (Sandelowski, 1997; Sandelowski et al,
1997) of qualitative research, and much work remains to be done in developing and harnessing
appropriate methodologies. Given the pioneering role of the Cochrane Collaboration in
developing review methodology it is doubly unfortunate that opportunities to develop qualitative
systematic reviews are constrained by:
(1) the implicit assumption that systematic review methods that are conceived and developed in
the context of quantitative reviews should be translated uncritically for use in qualitative
reviews.
(2) the apparent willingness of the Cochrane Collaboration Qualitative Research Network to
adopt a "handmaiden" rather than "partnership" role in contributing to methodological
developments.
This latter constraint is best exemplified by an extract from the objectives from the draft module
of the proposed Qualitative Research Methods Group:

Develop and disseminate appropriate methodological standards for combining data from
qualitative studies with other data within the context of a systematic review.
There is currently no forum for the development of qualitative systematic reviews per se (as
opposed to the use of qualitative data within systematic reviews). This deficiency is shared by
methodological reviews that seek to embody existing schools of thought or isolated viewpoints
in synthetical documents that advance the state of our knowledge. Paradoxically this includes
documents, such as those produced by the NHS Health Technology Assessment Programme, that
inform the conduct of quantitative systematic reviews! Now that is what I call cock-eyed!
References
Barbour RS (1998) Mixing qualitative methods: quality assurance or qualitative quagmire?
Qualitative Health Research 8: 352-361
Barbour RS (2001) Checklists for improving rigour in qualitative research: a case of the tail
wagging the dog? BMJ 322: 1115-1117.
Barroso J & Powell-Cope GM (2000) Metasynthesis of qualitative research on living with HIV
infection. Qualitative Health Research 10 (3): 340-353.
Beck CT (2001) Caring within nursing education: A metasynthesis. Journal of Nursing
Education 40 (3): 101-109.
Bland CJ, Meurer LN & Maldonado G (1995) A systematic approach to conducting a nonstatistical meta-analysis of research literature. Academic Medicine 70 (7): 642-653.
Brazier, J., Deverill, M., Green, C., Harper, R. & Booth, A. (1999). A review of the use of health
status measures in economic evaluation. Health Technology Assessment 3 (9).
Burke SO, Kauffman E, Costello E, Wiskin N & Harrison MB (1998) Stressors in families with
a child with a chronic condition: an analysis of qualitative studies and a framework. Canadian
Journal of Nursing Research 30(1): 71-95.
Chilcott J, Brennan A & Booth A (In press) The Use of Modelling in the Planning and
Prioritisation of Clinical Trials. Health Technology Assessment
Corbin JM and Strauss A (1988) Unending work and care: managing chronic illness at home.
Jossey-Bass, San Francisco.
Davies P, Wolf FM & Holmes L (2000) An Organisational Framework for Preparing and
Maintaining Systematic Reviews in Education: A Discussion Document.
http://campbell.gse.upenn.edu/papers/9a_framework.html [accessed 01/05/2001]
Davies P (2000) The relevance of systematic reviews to educational policy and practice. Oxford
Review of Education 26 (3-4): 365-378.
Dowzer C, Basford L, Booth A & Poon, LW (2000) Living with Multiple Health Conditions: a
systematic search and literature review. [A report for the American Association of Retired
Persons]. Sheffield: School of Health and Related Research (ScHARR), University of Sheffield.
Edwards AG, Russell LT & Stott NCH (1998) Signal versus noise in the evidence base for
medicine: an alternative to hierarchies of evidence? Family Practice 15 (4): 319-322
Estabrooks CA, Field PA & Morse JM (1994). Aggregating qualitative findings: An approach to
theory development. Qualitative Health Research 4: 503-511.
Finfgeld DL (1999) Courage as a process of pushing beyond the struggle. Qualitative Health
Research 9 (6): 803-814.
Forte JA (1998) Power and role-taking: a review of theory, research, and practice. Journal of
Human Behavior in the Social Environment 1(4): 27-56.
Fredriksson L (1999) Modes of relating in a caring conversation: a research synthesis on
presence, touch and listening. Journal of Advanced Nursing 30(5): 1167-76.
Glaser B & Strauss A (1967) The discovery of grounded theory. Chicago: Aldine.
Greenhalgh, T (1996) "Is my practice evidence-based?" BMJ 313: 957-958.
Greenhalgh T (1998) Meta-analysis and beyond: applying secondary research methods to
primary care. British Journal of General Practice 48 (433): 1540-1541.
Greenhalgh, T (1999) Narrative based medicine in an evidence based world. BMJ 318: 323-325.
Hek G, Langton H & Blunden G (2000) Systematically searching and reviewing literature. Nurse
Researcher 7 (3): 40-57.
Jensen LA & Allen MN (1994) A synthesis of qualitative research on wellness-illness.
Qualitative Health Research 4: 349-69.
Jensen L & Allen M (1996) Meta-synthesis of qualitative findings. Qualitative Health Research
6: 553-560.
Kearney MH (1998a) Truthful self-nurturing: a grounded formal theory of women's addiction
recovery. Qualitative Health Research 8 (4): 495-512.
Kearney MH (1998b) Ready-to-wear: Discovering grounded formal theory. Research in Nursing
& Health 21 (2): 179-186.
Klein SS (1989) Research and Practice - Implications for Knowledge Synthesis in Education.
Knowledge-Creation Diffusion Utilization 11(1): 58-78.
Khunti K (1999) Use of multiple methods to determine factors affecting quality of care of
patients with diabetes. Family Practice 16(5): 489-94
Lemmer B, Grellier R & Steven J (1999) Systematic review of nonrandom and qualitative
research literature: exploring and uncovering an evidence base for health visiting and decision
making. Qualitative Health Research 9(3): 315-28.
Light RJ & Pillemer DB (1984). Summing up: the science of reviewing research. Cambridge,
MA: Harvard University Press.
Mays N & Pope C (1996) Qualitative research in health care. London: BMJ Publishing Group.
NHS Centre for Reviews and Dissemination (2000). Undertaking Systematic Reviews of
Research Effectiveness: Guidelines for those carrying out or commissioning reviews. Report 4.
2nd edition. York: NHS Centre for Reviews and Dissemination, University of York.
Noblit GW & Hare RD (1988) Meta-ethnography: synthesising qualitative studies. (Qualitative
Research Methods, Volume 11) London: Sage.
Nolan M., Booth A. & Nolan J (1997) New directions in rehabilitation: exploring the nursing
contribution. Research Reports series No. 6. London: English National Board for Nursing,
Midwifery and Health Visiting.
Paterson BL, Thorne S & Dewis M (1998) State of the science. Adapting to and managing
diabetes. Image - the Journal of Nursing Scholarship 30(1): 57-62.
Popay J, Rogers A & Williams G (1998). Rationale and Standards for the Systematic Review of
Qualitative Literature in Health Services Research. Qualitative Health Research 8 (3): 341-351.
Sackett DL and Wennberg JE (1997) Choosing the best research design for each question. BMJ
315: 1636.
Sandelowski M (1997) To be of use: Enhancing the utility of qualitative research. Nursing
Outlook 45: 125-132.
Sandelowski M, Docherty S & Emden C (1997) Qualitative meta-synthesis: Issues and
techniques. Research in Nursing & Health 20: 365-371.
Sherwood GD (1997) Meta-synthesis of qualitative analyses of caring: defining a therapeutic
model of nursing. Advanced Practice Nursing Quarterly 3(1):32-42.
Sherwood G (1999) Meta-synthesis: merging qualitative studies to develop nursing knowledge.
International Journal for Human Caring 3 (1): 37-42.
Sleep J & Clark E (1999) Weighing up the evidence: the contribution of critical literature
reviews to the development of practice. NT Research 4(4): 306-14.
Smith M & Stullenbarger E (1991) A prototype for integrative review and meta-analysis of
nursing research. Journal of Advanced Nursing 16(11): 1272-83.
Stewart L & Parmar MKB (1993) Meta-analysis of the literature or of individual patient data: Is
there a difference? Lancet 341: 418-422.
Thompson SG (1994) Why sources of heterogeneity in meta-analysis should be investigated.
British Medical Journal 309: 1351-1355.
Thorne S & Paterson B (1998) Shifting images of chronic illness. Image - the Journal of Nursing
Scholarship 30(2):173-8
Wuest J (2000) Negotiating with helping systems: An example of grounded theory evolving
through emergent fit. Qualitative Health Research 10 (1): 51-70
Zhao SY (1991) Metatheory, Metamethod, Meta-Data-Analysis - What, Why, and How.
Sociological Perspectives 34 (3): 377-390.