The Qualitative Assessment and
Review Instrument (QARI)
PART 3:
META-AGGREGATION
Its history
Its philosophy
A worked example
Meta-aggregation
• A structured, and process driven approach to
systematic review drawing on the classical
understandings and methods associated with
systematic review of quantitative literature as practiced
by the Cochrane and Campbell Collaboration
• Based on an a-priori protocol
– Established, answerable question
– Explicit criteria for inclusion
– Documented review methods for searching, appraisal,
extraction and synthesis of data
Meta-aggregation: History
• Participative consensus project involving
Australian academics
• The outcome = an aggregative approach to the
synthesis of qualitative evidence that:
– emphasised the complexity of interpretive and
critical understandings of phenomena; recognised
the need to ensure that the process is practical and
usable;
– balanced utility of the outcomes with the
complexity of the material;
– was grounded in pragmatism.
Meta-aggregation: A pragmatic
approach
• The philosophy of pragmatism - Charles Peirce, William James, John
Dewey
– the value of any thought lies in its practical use and consequences. It is
the focus on practical consequences that characterizes meta-aggregation
as a synthesis approach
– Central to pragmatism were communities of inquiry or groups of people
with a shared interest, problem or issue trying to resolve doubt through
critical reasoning.
Research articles
– Actions are assessed in the light of their practical consequences.
Aggregation concentrates on the original researchers’ findings (processed
data) and summarizes common and competing findings to produce crossstudy generalizations that lead to recommendations for action  The
utility of an outcome is important.
Meta-aggregation: a worked example
• Review Topic:
Obstacles to the implementation of Evidence-Based
Practice in Belgium: a context-specific study
• Question formulation:
– Setting: Belgium (Belgian health care system)
– Population: Health Care Practitioners
– (Phenomena of) Interest: Obstacles toward the
implementation of Evidence-Based Practice
– Comparison: (additional literature search on strategies
developed by countries with other systems)
– Evaluation: Experiences and perceptions
Meta-aggregation: a worked example
• Search strategy:
– Major databases: Medline, CINAHL, Psychinfo, Embase, Social Sciences
abstracts and ERIC (1990-May 2008).
– Where possible we used a methodological filter for qualitative
research.
– Other information sources:
• ‘Federal research actions’-database from the Belgian governmental
department of science
• Consultation of Belgian experts in qualitative research methods and/or
Evidence-Based Practice to check on any other published material that could
be of use for the synthesis.
• Screening of references
– Keywords:
• ‘Evidence-Based Practice (Medical Subject Heading (MeSH) term)’, including
Evidence-Based Dentistry, Nursing and Medicine.
• Additional keywords: evidence-based combined with the geographical notion
‘Belgian’, ‘Flemish/Flanders’ or ‘Walloon’.
Meta-aggregation: a worked example
• Inclusion criteria:
– Study type: Qualitative, empirical research papers. Opinion
pieces and descriptive articles were excluded.
– Study participants: All types of health care practitioners e.g.
physicians, dentists, nurses, physiotherapists, psychologists
– Topic of interest: Obstacles to the implementation of EBP.
– Context: the Belgian health care system.
– Outcome of interest: Experiences and/or perceptions from
participants.
8 studies met the inclusion criteria
Meta-aggregation: a worked example
• Critical appraisal of selected studies:
• No criteria based quality appraisal, although the
software QARI provides a standard critical appraisal
checklist.
– limited amount of studies found
– the majority of the selected studies were written by the lead
reviewer, who would also be one of the appraisers.
• An overall judgment approach was used instead. This
approach has been proven to deliver the same outcome
(Dixon-Woods, 2007). However, it tends to be less explicit
about potential reasons for exclusion.
• An evaluation based on the JBI-critical appraisal
instrument is recommended for other review teams
opting for a meta-aggregative approach to synthesis.
Meta-aggregation: a worked example
• 85 different findings were found
• The findings were classified in 9 major categories:
–
–
–
–
–
–
(1) evidence is hard to implement
(2) decision making processes are influenced by practitioner variables
(3) decision-making processes are influenced by practitioner variables
(4) commercial/financial interests affect EBP
(5) governmental regulations influence the process of implementation,
(6) EBP is more feasible for practitioners working in a scientific setting
compared to those working in other settings
– (7) physicians’ display of power hampers other professionals to put
evidence-based practice into effect
– (8) a lack of knowledge and skills hinders the implementation of evidencebased practice
– (9) attitudes of health care practitioners hinder the implementation of
EBP.
Meta-aggregation: a worked example
• These categories were further analysed to
produce 4 synthesized statements
– Synthesis 1: Evidence might have a limited role in decision-making processes in
daily practice, if the importance of the scientific component is not stressed
(categories 1, 2 and 3).
– Synthesis 2: Aspects other than quality of care will steer the EBP agenda, if
governmental regulations and economic interests are contra-productive for
delivering the best possible care (categories 4 and 5).
– Synthesis 3: Although EBP is intended to serve all practitioners, some health
care providers will benefit less from EBP than others, if inequity issues between
practitioners are not solved and support for field workers is not established
(categories 6 and 7).
– Synthesis 4: A lack of competences will hinder the implementation of EBP, if
gaps in knowledge and skills are not being filled and efforts to change contraproductive attitudes are not undertaken (categories 8 and 9).
Meta-aggregation: a worked example
Meta-aggregation: a worked example
•
Implications for practice and policy
– Consider educating patients and exploring potential information
channels that influence patients’ opinions to create space for
well-informed decisions (synthesis 1).
– Provide easy and free access to well-structured, compact and
relevant information targeted to a particular discipline and
consider helpdesks. Screen information, control its quality and
translate it to the field (synthesis 1).
– Consider updating the Belgian nomenclature and reimbursement
system to bring it in line with the latest evidence (synthesis 2).
– Consider incentives for those practitioners who are engaging
themselves for the implementation of EBP in daily practice, to
keep them motivated and to prevent from creating a negative
spiral in which practitioners tend to believe that efforts to
improve practice automatically lead to a loss of income
(synthesis 2).
Meta-aggregation: a worked example
• Implications for practice and policy
– Enhance communication and cooperation between physicians
and other partners in health care, via their professional
groups and journals (synthesis 3)
– Consider programs tailored to the needs of specific disciplines
and stimulate multidisciplinary education should be
stimulated to create mutual understanding across disciplines
(synthesis 3, 4).
– Consider direct access to allied health services to increase
autonomy (synthesis 3)
– Integrate EBP in the basic curricula (synthesis 4).
The Qualitative Assessment and
Review Instrument (QARI)
Designed to provide a systematic
process mirroring that taken for
systematic reviews of quantitative
research, whilst being sensitive to the
nature of qualitative data.
Starting a synthesis project
(or review of qualitative evidence)
Critical Appraisal of Evidence
arising out of
qualitative research
Data Extraction
Reduce
•The findings of many studies into a single
document
Summarise
•Methods
•Phenomena
•Findings
The units of extraction in this process are specific findings and
illustrations from the text that demonstrate the origins of the
findings.
In QARI a finding is defined as: A conclusion reached by the
researcher(s) and often presented as themes or metaphors.
Levels of credibility
Unequivocal - relates to evidence beyond reasonable doubt which may include findings that are
matter of fact, directly reported/observed and not open to challenge
Credible - those that are, albeit interpretations, plausible in light of data and theoretical
framework. They can be logically inferred from the data. Because the findings are interpretive
they can be challenged.
Not Supported - when 1 nor 2 apply and when most notably findings are not supported by the
data
Meta-Synthesis
The aim of meta-synthesis is to assemble findings; categorise
these findings into groups on the basis of similarity in meaning;
and to aggregate these to generate a set of statements that
adequately represent that aggregation. These statements are
referred to as synthesised findings - and they can be used as a
basis for evidence based practice.
In QARI, a synthesised finding is defined as an overarching description of a group of
categorised findings that allow for the generation of recommendations for practice.
Can be stated propositionally as “if-then” statements - for example: “If students
are advised reconsider their choice of study , their relatives will sometimes feel as
if they are not involved”. (a somewhat awkward and eccentric form)
We prefer the declamatory form that emphasizes the probability of the claim:
“Relatives of students that are advised to reconsider their choice of study may
feel as if they are not involved if strategies to include them are not pursued”.