Appraisal, Extraction and Pooling of
Qualitative Data and Text
- Evidence from qualitative studies,
narrative and text.
(JBI/CSRTP/2012/0003)
Introduction
• Recap of Introductory
module
–
–
–
–
Developing a question (PICo)
Inclusion Criteria
Search Strategy
Selecting Studies for Retrieval
• This module considers how
to appraise, extract and
synthesize evidence from
qualitative studies and text.
Program Overview
Day 1
Time
0900
0930
1030
1100
Session
Introductions and overview of Introductory
Module
Session 1: Introduction to Qualitative
Evidence Synthesis
1230
1330
Morning Tea
Session 2: The Systematic Review of
Qualitative Evidence
Session 3: Critical Appraisal of Qualitative
Evidence
Lunch
Session 4: Data Extraction
1400
Session 5: Data Analysis and Meta-synthesis
1430
1500
1700
Afternoon tea
Session 6: QARI Trial
End
11.30
Group Work
Group Work 1: Paradigm, methodology and
research question
Group Work 2: Critical Appraisal using JBI-QARI
Program Overview
Day 2
Time
0900
0915
Session
1030
1100
Overview of Day 1
Session 7: Systematic review of Textual
Opinion Papers
Morning Tea
Session 8: NOTARI Trial
1200
1230
1330
1645
1700
Session 9: Protocol Preparation
Lunch
Session 10: Protocol Presentations
Session 11: Assessment and Summation
End
Group Work
Group Work 3: Critical Appraisal
Session 1: Introduction to Qualitative
Evidence Synthesis
Qualitative Research is…
• “…a situated activity that locates the observer in the world.
It considers a set of interpretive, material practices that
make the world visible…[it] involves an interpretive,
naturalistic approach to the world. This means qualitative
researchers study things in their natural settings, attempting
to make sense of, or to interpret, phenomena in terms of the
meanings people bring to them”
(Denzin & Lincoln, 2000)
Qualitative research and health care
• Health care is a people centered process concerned with
health and healing
– Health care derives much of it’s knowledge from a range of
biomedical sciences
• However this does not supply all the knowledge that is
necessary to provide holistic care for the patients and
clients
• Qualitative researchers use humanistic frameworks to
examine ordinary activities of everyday life
Qualitative Research Findings as Evidence
for Practice
• Qualitative evidence is of increasing importance in health
services policy, planning and delivery.
• It can play a significant role in:
– understanding how individuals / communities perceive health,
manage their own health and make decisions related to health
service usage;
– increasing our understandings of the culture of communities and
of health units;
– Informing planners and providers;
– evaluating components and activities of health services that
cannot be measured in quantitative outcomes.
Application of Qualitative Research to
Practice
• As with quantitative research, results from a single
study only should not be used to guide practice
• The findings of qualitative research should be
synthesized in order to develop recommendations for
practice
Research Paradigms
• Serve as a model/world view or philosophy that
structures knowledge and understanding
• Define what is to be observed and scrutinised
• Prioritise questions
• Inform the interpretation of results
Dominant Paradigm 1:
Positivist/Empirical-Analytical
• Assumptions
– There is an external, real world of objects apart from
people, which is comprehensible to us
– Researchers can compare their claims against this
objective reality
• Seeks to predict and control
Dominant Paradigm 2: Interpretive
• Assumptions
– Reality as we know it is constructed/produced intersubjectively (between people).
– Meaning and understanding are developed socially and
experientially.
– Researchers’ values are inherent in research.
– Findings of knowledge claims are created
– All interpretations are located in a particular context,
setting and moment.
• Seeks to understand
Dominant Paradigm 3: Critical
• Assumptions
– Examines issues of power and envisions new opportunities
– Main task social critique, whereby the restrictive and alienating
conditions of the status quo are brought to light
– Common assumptions with Interpretivism; however, moves past
describing meaning to emancipation and change.
• Seeks to critique and emancipate
Qualitative Methodologies
Phenomenology
• Assumes that people come to know a phenomena
through their experience of it
• Focuses on INDIVIDUAL meaning
• Emphasizes a focus on people's subjective
experiences and interpretations of the world.
• Understand how the world appears to others.
Ethnography
• Study of Culture/specific cultural group.
– What is the way of life of this group of people?
– Everyday life is worth of study.
• Focuses on SOCIAL meaning.
• Researcher immerses self in cultural group.
– Field Work - primarily participant observation and interview data
• Focuses on the context of communities
• Used increasingly in healthcare to explore the relationship
between health and culture.
Grounded theory
• Develop theory grounded in real world;
• Searches and generates theoretical explanations
from observations of the world (induction);
• The constant collection and iterative analysis of data
to enable theory to emerge; and
• Data are not constrained by predetermined
theoretical framework, but define boundaries of
inquiry.
Action Research
• Basis in critical social science;
• Researchers interact with the
participants to achieve change;
• Often community-based;
– used with implementation studies in
healthcare; and
• Treats the individual as an autonomous
being, capable of exercising agency
Discourse analysis
• Discourse – to talk, converse; hold forth in speech or
writing on a subject;
• Discourse as patterns of ways of representing
phenomena in language;
• Application of critical thought to social situations and
the unveiling of (hidden) politics within socially
dominant or marginalised discourses.
Qualitative data analysis
• Is based on principles of thematic analysis for most
methodologies
– A process of identifying and presenting meaning in the
form of themes, metaphors or concepts;
• there is wide variability in the terms used and how they are
defined;
– Not suited to all methodologies (i.e. discourse analysis)
Qualitative data analysis
• Three strategies:
– Data analysis as a separate step following data
collection
– Data analysis occurs simultaneously with data collection
– Data collection and analysis are ‘staged’
Congruity between Paradigm,
Methodology and Methods
Paradigm
POSITIVIST
Methodology
Methods
Clinical Trials
Measurement of
outcomes
Cohort Studies
Other Methodologies
using numerical data
INTERPRETIVE
CRITICAL
Phenomenology
Interviews
Ethnography
Observation
Grounded Theory
Field Work
Action Research
Group Processes
Reflective Journaling
Paradigm
Methodologies
Methods
Randomized Controlled Trial Measurement of variables
Empirico-Analytical Cohort Study
Correlational Study
Time Series Study
Case Series Study
Positivist/
Interpretivist
Phenomenology
Ethnography
Grounded Theory
Critical
Action research
Discourse Analysis
Interviews
Focus groups
Observations
Field work
Field observations Purposeful
interviews
Textual analysis
Participative group work
Reflective Journals
In-depth interviews
Focus Groups
Analysis of communications, written
text and policies
Group Work 1
• In pairs, work through the Group Work 1 task in the
Workbook.
• Reporting back
Session 2: The Systematic Review of
Qualitative Evidence
Trends and Debates within the International Literature
Meta-synthesis: approaches
Approach
Purpose
Evidence
of
Interest
Search
Strategy
Critical Appraisal
Data
Extraction
Method of
Synthesis
Outcome
Software
Available
Narrative
Synthesis
To summarise
two or more
papers in
narrative form
Generic
Unspecified/
Selective
Not specified
Not
Specified
Unspecified
Review
articletype report
NO
Metaethnography
The generate
new
knowledge
/theory use
processes of
interpretation
Findings
of
qualitative
research
studies
Not
comprehensive or
exhaustive;
seeks
saturation –
theoretical
sampling
Opposed; all
studies included
as each may
provide insight
into the
phenomena of
interest
Extraction of
key
concepts
Refutational
synthesis;
Reciprocal
translation;
Line of
argument
synthesis.
Higher
order
interpretation of
study
findings
NO
(Although
QARI can
be used)
Approach
Purpose
Evidence of
Interest
Search
Strategy
Critical
Appraisal
Data Extraction
Method of
Synthesis
Outcome
Software
Available
Realist
Synthesis
To develop
and test
program
theories
Generic; but
more suited
to qualitative
research
studies
Not
comprehensive or
exhaustive;
seeks
saturation
Not specified
Extracts positive
and negative
mechanisms/stra
tegies that
influence change
Tacit testing of
implicit
theories;
building theory.
A transferable theory
on “what
works, for
whom, in
which
circumstances”
NO
Thematic
analysis
To
aggregate
findings of
2 or more
studies
Findings of
qualitative
research
studies
Not
comprehensive or
exhaustive;
seeks
saturation –
theoretical
sampling
Not specified
Extracts of
major/recurrent
themes in
literature
Aggregation of
themes/
metaphors/
categories
A
summary
of findings
of primary
studies
under
thematic
headings
NO
(Although
QARI can
be used
Approach
Purpose
Evidence
of Interest
Search
Strategy
Critical
Appraisal
Data
Extraction
Method of
Synthesis
Outcome
Software
Available
Content
analysis
To analyse
then
summarise
content of
papers.;
occurrences
of each
theme
counted and
tabulated
Generic
Usually
comprehensive with
predetermined search
strategy
Not
specified
Content
extracted then
coded using
extraction tool
designed to
aid
reproducibility
coded data
categorised
under
thematic
headings; also
counted and
tabulated
A summary
of findings,
and their
rate of
occurrence,
of primary
studies
under
thematic
headings
YES
Metasynthesis/
Metaaggregation
To aggregate
the findings
of included
studies
Findings of
qualitative
research
studies
Comprehensive;
detailed
search
strategy at
protocol
stage
required
Required,
using
standardised critical
appraisal
instrument
Extraction of
findings PLUS
data that gives
rise to finding
using data
extraction
instrument
Aggregation of
findings into
categories;
and of
categories into
synthesised
findings
Synthesised
findings that
inform
practice or
policy in the
form of a
standardise
d chart
YES
Meta-synthesis: worked examples
Meta aggregation
Meta ethnography
Meta aggregation
• A structured and process driven approach to
systematic review drawing on the classical
understandings and methods associated with
systematic review of quantitative literature as practiced
by the Cochrane Collaboration
Meta aggregation
• Based on an a-priori protocol
– Established, answerable question
– Explicit criteria for inclusion
– Documented review methods for searching, appraisal,
extraction and synthesis of data
Meta aggregation
• Explicitly aligned with:
– Philosophy of pragmatism
• Delivers readily useable findings
• Informs decision making at the clinical or policy level
– Transcendental phenomenology
• Looks for common or “universal” essences of meaning
• Attempts to “bracket” pre-understandings of the reviewer
Transcendental Phenomenology
• Based on the philosophic traditions of Husserlian
phenomenology:
– the intuitive examination of essences that have immediate
validity;
– Seeks to avoid undue influence of the reviewer on the text;
– Seeks to generate practice level theory that has explanatory
power for policy or practice
– Seeks to preserve the intended meaning of text
The needs and experiences of people
with a diagnosis of skin cancer: a
systematic review
Janet Barker, Arun Kumar, Wendy Stanton and Fiona Bath-Hextall, JBI Library of
Systematic Reviews, 2011; 9(4):104-121
Inclusion Criteria
• Participants
– Adults with a diagnosis of skin cancer.
• Phenomena of Interest
– The needs and experiences of people with skin cancer.
• Context
– Hospital or community-located dwellers.
• Types of studies
– All qualitative studies that described or analyzed the needs and experiences
with people who had been diagnosed with skin cancer.
Methods
• Each paper was assessed independently by two
reviewers for methodological quality. The internal
validity (quality) of research papers was assessed
using JBI-QARI.
• There were no disagreements between the two
reviewers and therefore a third reviewer was not
required.
Results
Results ( cont/…)
• A total of 12 findings were extracted from the 2
included qualitative papers
• These findings were aggregated into 4 categories
on the basis of similarity of meaning.
• The categories were synthesized to generate 2
synthesized findings.
Results/cont..
• The 2 studies used a qualitative approach but
neither specified a particular methodological
approach.
• Both studies were conducted in the United Kingdom
(UK), using semi structured interviews.
Recommendations arising
• There is a real need to increase knowledge of skin cancer so
that people do not delay in seeking medical help as early
diagnosis can dramatically improve both prognosis and the
patient experience since early lesions are treated more simply
compared with larger or neglected lesions.
• Health professionals caring for these patients need to
understand the psychosocial concerns of this patient group in
order to design services appropriately and to provide patients
with the support they need and information that they can
easily understand.
Meta-Ethnography
• Explicitly aligned with philosophy of interpretivism
• Searches for new meaning
• Focuses on multiple realities
Hermeneutic Interpretivism
• Is the basis for meta-ethnography
– Interpretation of text for its inner meaning
– Focuses on the reviewers interpretive skills
– Seeks to re-interpret the published literature
– Seeks to generate new, mid level theoretical
explanations
– Seeks engagement between reviewer and text
Meta ethnography
• From within the social sciences to develop theories
from existing ethnographic data,
• Iterative development of emic interpretations,
• Incorporates 7 phases, these can be aligned to the
systematic review process, but there is no
requirement to do so.
Meta ethnography
•
•
•
•
•
•
•
Identify the research interest
Decide which studies to include
Read the studies
Determine how the studies are related
Translate the studies in to each other
Synthesize the translations
Express the synthesis
Meta ethnography: 3 stages of synthesis
• First order interpretations
– Themes, metaphors or concepts identified
• Second order interpretations
– The researchers interpretation of how the identified
concepts relate to each other
• Third order interpretations
– Seeks to encompass themes within each other
Interpretation
Second Order Interpretation:
• Reciprocal
– Like interpretations are brought together “this one is like that
one..”
• Refutational
– Competing discourses; must also have a relationship that can be
explored
Third Order Interpretation:
• Line of argument
– Asks what do the parts infer about the whole
Patient adherence to Tuberculosis
treatment
Salla A. Munro, Simon A. Lewin, Helen J. Smith, Mark E. Engel1, Atle Fretheim, Jimmy Volmink.
‘Patient Adherence to Tuberculosis Treatment: A Systematic Review of Qualitative Research’.
PLoS Medicine July 2007.
Inclusion criteria
• Patients, carers or health professionals delivering
DOTs
• Perceptions of adherence was the phenomena of
interest
• Context was patients undergoing DOTs therapy for TB
• Qualitative studies on the perception of adherence
Methods
• Studies were assessed using a checklist
• Data extracted using a standardized form
• Synthesis was reciprocal and line of argument
Results
•
•
•
•
•
Following screening, 44 papers were included
First order interpretations 8
Second order interpretations 6
Third order interpretations 4
Expression of the synthesis: visual model
Recommendations
• Increase visibility of TB programs in the community, which may
increase knowledge and improve attitudes towards TB
• Provide more information about the disease and treatment to patients
and communities
• Increase support from family, peers, and social networks
• Minimize costs and unpleasantness related to clinic visits. Increase
flexibility/ patient autonomy
META ETHNOGRAPHY
QARI
METAGGREGATION
FIRST ORDER ANALYSIS 
SECOND ORDER INTERPRETATION
THIRD ORDER INTERPRETATION
 STEP 2: CATEGORIES
 STEP 3: SYNTHESISED FINDINGS
Session 3: Critical Appraisal of
Qualitative Evidence
1004 references
172 duplicates
832 references
Scanned Ti/Ab
715 do not meet
Incl. criteria
117 studies
retrieved
82 do not meet
Incl. criteria
35 studies for
Critical Appraisal
Why Critically
Appraise?
• Combining results of
poor quality research
may lead to misleading
understandings of
issues explored
The Critical Appraisal Process
• Every JBI review must set out to use an explicit
appraisal process. Essentially:
– A good understanding of philosophical perspectives,
methodologies, methods is required in appraisers; and
– An agreed instrument/checklist should be used.
Validity, Reliability and Generalizability
• Judging the quality of qualitative research remains
deeply contested
Analogous criteria for paradigmatic
assumptions
Quantitative
Qualitative
Reliability
Dependability
Internal Validity
Credibility
External Validity
Transferability
Reliability/dependability
• Appropriateness of methodology, methods and
implementation of the research methods, regardless
of paradigm
• The focus of dependability is on achieving
consistent quality rather than repeatability.
Internal validity/credibility
• Credibility addresses whether a finding has been
represented correctly
– Assessment of credibility is multi-dimensional, including
goodness of fit and representativeness,
– Credibility is auditable - the process may be based
upon researcher confirmation, member checks, peer
checks, second researcher analysis, or observation.
External validity/transferability
• Findings are not generalizable in the quantitative sense of
the word
– generalization is “narrowly conceived in terms of sampling and
statistical significance.”
– “qualitative research is directed toward naturalistic or idiographic
generalizations, or the kind of generalizations made about
particulars”
– Schofield (1990) describe qualitative metasynthesis as “cross-case
generalizations created from the generalizations made from, and
about, individual cases.”
Sandelowski et al(1997)
Critical Appraisal
• General acceptance of need for quality
• Ongoing debate around the role of appraisal
• Particular focus of debate on role of scales and sum
scores in appraisal
• In practice, appraisal instruments for qualitative
research tend to focus on establishing the degree to
which the evidence applies to practice (transferability)
rather than internal validity (credibility)
The premise
• Methods applied during the systematic review of
qualitative evidence should firstly be congruent with
the universally accepted process of systematic
review.
• The characteristics of a systematic review might be
debated in terms of the detail, but there is general
acceptance of a series of steps, stages or processes
The Joanna Briggs Approach
• Establishing credibility in qualitative research requires:
– A standardized approach that is clearly defined, articulated,
and applied, and
• Takes in to consideration the relationship between:
– Methodology and methods
– Methods and data
– Data and conclusions
The Qualitative Assessment and Review
Instrument (QARI)
• QARI provides a systematic process mirroring that of
systematic reviews of quantitative research whilst sensitive to
the nature of qualitative data.
• a significant proportion of qualitative work is not designed to
address questions or issues of an immediate practical nature
• this is consistent with the traditions of many qualitative
methodologies
• JBI focuses on evidence to improve global health, reviews with
a clinical or policy focus are encouraged.
Critical Appraisal in JBI-QARI
Critical Appraisal of Interpretive Research
• In interpretive and critical inquiry validity relates to
the rigor of the process of inquiry
– limiting bias to establish validity in the appraisal of
quantitative studies is antithetical to qualitative
approaches to inquiry
• QARI incorporates a checklist to appraise rigor
generically
Group Work 2
• Read the 2 papers provided and then undertake
critical appraisal of the paper using the JBI Critical
Appraisal Checklist for Interpretive and Critical
Research.
• Report Back
• Enter the details into JBI QARI
Session 4: Data Extraction
Study data and Data Extraction

1004 references
172 duplicates
832 references
Scanned Ti/Ab
117 studies
retrieved
35 studies for
Critical Appraisal
26 studies incl.
in review
715 do not meet
Incl. criteria
82 do not meet
Incl. criteria
Data most
frequently
extracted
Considerations in Data Extraction
•
•
•
•
•
•
•
•
•
•
Source - citation and contact details
Eligibility - confirm eligibility for review
Methodology - approach to the inquiry
Methods - how data were collected
Phenomena of Interest - the described experience or activity
Setting - specific context of study eg, hospital
Geographical location - region or country
Culture - cultural characteristics of setting or participants
Participants - demographic data such as gender, age etc
Data analysis - analytic approach taken, eg thematic analysis etc
Extracting Findings
• The units of extraction in this process are specific findings (and illustrations from the
text that demonstrate the origins of the findings);
• In Meta-aggregation a finding is defined as: A conclusion reached by the
researcher(s) and often presented as themes, metaphors, findings, concepts,
conclusions etc;
• Some papers may present themes and sub themes (or findings of the qualitative
analysis at different levels); for such papers the reviewer will need to decide which
of these levels represent authentic findings; however, only findings of one level
should be extracted from a specific paper.
• No matter what level is decided, the actual verbatim words of the researcher
must be extracted.
Extracting Illustrations
• Meta-aggregation requires reviewers to also extract an
illustrative excerpt that the researcher presents in support of
that particular finding;
• It is only necessary to extract one such supporting illustration;
• The actual verbatim words the researcher uses as the
illustration must be extracted.
Assigning a Level of Credibility to
Qualitative Evidence
Levels of Credibility- Qualitative
• Unequivocal - relates to evidence beyond reasonable doubt
• Credible - those that are, albeit interpretations, plausible in light of
data and theoretical framework.
• Not Supported - when 1 nor 2 apply and when most notably findings
are not supported by the data
– Should not be included in synthesis to inform practice
Day 2
Session 5: Data Analysis and
Meta-synthesis


General Analysis - What Can be
Reported and How
– What phenomena of interest has been evaluated
– The meaningfulness/appropriateness/feasibility of the
experience/activity
– Contradictory findings and conflicts
– Limitations of study methods
– Issues related to study quality
– The use of inappropriate definitions
– Specific populations excluded from studies
– Future research needs
Meta-Synthesis
• Analysis and synthesis of qualitative studies
• Based on processed data
• Aim of meta-synthesis
– is to assemble findings;
– categorize these findings into groups on the basis of similarity in
meaning;
– aggregate these to generate a set of statements that adequately
represent that aggregation.
Synthesis of the Results of Qualitative
Studies
• Differing methodologies, such as phenomenology,
ethnography or grounded theory, can be mixed in a single
synthesis of qualitative studies because the synthesis is of
findings and not the data.
• This is a critical assumption of meta-aggregation.
Meta Synthesis vs Meta Analysis
 As qualitative studies differ from RCT, meta-synthesis utilizes a different
approach from that used during meta-analysis.
 Both approaches provide an understanding based on populations,
settings and circumstances
 Unlike meta-analysis, meta-synthesis deals in multiple realities and so
provides but one interpretation of the phenomenon.
Executing Meta Synthesis using
QARI
Assembling the findings
• Once findings are extracted from all included papers;
– The reviewer transitions from a focus on papers to a
focus on assembling all of the findings
– This is executed inside JBI-QARI
Creating Categories
• Categorization involves repeated, detailed examination of the assembled findings
• The reviewer identifies groups of findings on the basis of similarity in meaning to
create categories
– A category “Name” is the element of analysis that appears in the synthesis
and is therefore a full description that conveys the whole, inclusive meaning of
the category; that comprehensively represents the meanings embodied in the
findings it encompasses
– A Category “Summary” is a brief explanatory statement created by the
reviewer to assist interpretation and auditabililty.
Allocating Findings to Categories
Synthesized Findings
• In meta-aggregation a synthesized finding is an overarching
description of a group of categorized findings that allow for the
generation of recommendations for practice.
Creating Synthesized findings
• Meta-aggregation of categories involves repeated, detailed examination of the
assembled categories
• The reviewer identifies groups of categories on the basis of similarity in meaning
– A Synthesized Finding “Name” is the element of analysis that conveys the whole,
inclusive meaning of the categories; and that comprehensively represents the meanings
embodied in the findings it encompasses
– A synthesized finding “Name” must also draw a conclusion or claim and communicate
the probability of that conclusion or claim
– A Synthesized Finding “Summary” is a brief explanatory statement created by the
reviewer to assist interpretation and auditabililty.
Creating Synthesized Findings
– Can be stated propositionally as “if-then” statements
• e.g. “If a patient is awaiting a final diagnosis, their relatives
will sometimes feel as if they are not involved”
– Preference for the indicatory form (i.e. suggesting or
demonstrating the expedience or advisability of action emphasizing the probability of the claim)
• “Relatives of patients awaiting a final diagnosis of brain death
may feel as if they are not involved if strategies to include
them are not pursued”.
Allocating categories to synthesized findings
Session 6: QARI Trial
Program Overview
Time
0900
0915
1030
1100
Overview of Day 1
Session 7: The systematic review of text and opinion
Morning Tea
Session 8: NOTARI Trial
1200
1230
1330
1500
1515
1615
1700
Session 9: Protocol Preparation
Lunch
Session 10: Protocol Presentations
Afternoon tea
Session 10: Protocol Presentations/cont…
Session 11: Assessment and Summation
End
Session
Overview
• This module considers
how to appraise,
extract and synthesize
evidence from
qualitative studies and
text and opinion
papers.
Session 7: The systematic review
of text and opinion
Text, Expert Opinion and Discourse
– Narrative and text (e.g. review papers, discussion papers and policy
documents) present information that arises out of experience or
observation. Much of the resources drawn on to develop policy are
narrative and textual in nature rather than the outcomes of formal
research
– The opinions of leaders, professional organizations and learned
bodies draw on the collective wisdom of experienced experts and
are often referred to for guidance in relation to policy making and
professional practice
• Expertise is highly regarded and is linked to the ability of an expert to
“have to hand” information in a given area and is associated with
the possession of large amounts of knowledge and fluency in
applying this knowledge
• Discourse refers the verbal interchange of ideas that is grounded in
language and in the context within which it occurs. Discourse in the
professional and public domains is a source of knowledge that can
be used to inform policy and clinical decision making.
Text, Expert Opinion and Discourse as
Evidence for Policy and Practice
•
•
Narrative, opinion, expertise and discourse often represent the best
available evidence in areas where research is limited, or where the
knowledge that is needed is generally generated through policymaking or other processes rather than through formal research
This kind of knowledge cannot be ignored as legitimate sources of
evidence for policy and practice
• While this kind of evidence is not a product of “good”
science it is empirically derived and mediated through the
cognitive processes of health professionals or policy
makers who have typically been trained to be analytical
• The superior quality of evidence derived from research is
not denied rather, in its absence, it is not appropriate to
discount opinion as non-evidence
The Narrative, Opinion and Text Assessment
and Review Instrument (NOTARI)
Critical Appraisal of Evidence
arising out of Opinion and Text
Critical Appraisal of Text and Opinion
This focus on limiting bias to establish validity in the
appraisal of quantitative studies is not possible when
dealing with text and opinion
• The assessment of validity within the JBI-NOTARI process
focuses on:
•




examining the opinion;
identifying the credibility of the source of the opinion;
establishing the motives that underlie the opinion; and
locating alternative opinions that give credence to it or, conversely,
question it.
Critical Appraisal of Evidence
arising out of Text and Opinion
Report Back
Introduction to Data Extraction and
Meta-synthesis
Data Extraction
Aims
• Reduce
–The findings of many publications into a
single document
• Summarise
–Messages
–Conclusions
Extracting Conclusions
• The units of extraction in this process are specific
conclusion stated by the author/speaker and the
text that demonstrate the argument or basis of the
conclusion
Assigning a Level of Credibility
to Text and Opinion
Levels of Credibility- Text and Opinion
• Unequivocal - relates to evidence beyond reasonable doubt
which may include conclusions that are matter of fact, directly
reported/observed and not open to challenge
• Credible - those that are plausible in light of an argument and
theoretical framework.
• Not Supported - when 1 nor 2 apply and when most notably
conclusions are not supported by sound argument
Meta-Synthesis
• The aim of meta-synthesis is to assemble conclusions; categorise these
conclusions into groups on the basis of similarity in meaning; and to aggregate
these to generate a set of statements that adequately represent that aggregation.
These statements are referred to as synthesised findings - and they can be used as
a basis for evidence based practice.
Session 8: NOTARI Trial
Session 9: Protocol Development
Session 10: Protocol Presentations
Session 11: Assessment