Issues and Challenges around
Searching the Literature
Overview
• What is Already Known on Searching the
Qualitative Research Literature
• Overview of Methodological Issues/Challenges
• Recent Developments (with focus on literature
of last two years)
• Outstanding Issues/Challenges
What is already known
What is Already Known? – 1 (Gallacher
et al, 2013)
•
Finding relevant qualitative studies arduous due to inadequate indexing (Dixon
Woods et al, 2007; Ring et al 2011)
•
Papers often lack abstracts or include non-informative titles, making it difficult to
establish relevance (Dixon Woods et al, 2007).
Several papers outline strategies (filters) for searching for qualitative
studies (Walters et al, 2006; Wilczynski et al, 2007; Wong et al, 2004; McKibbon et al, 2006)
Helpful techniques involve electronic or hand searching (Greenhalgh et al 2005;
Bates, 1989):
•
•
–
–
–
–
–
–
•
•
Reference or footnote tracking (looking backwards at references in articles found).
Citation tracking (tracking forward subsequently citing articles).
Personal knowledge and personal contacts.
Contacting the authors of known papers or experts in the field.
Hand searching relevant journals.
Internet browsing such as berry picking (where one search leads to another and ‘clusters’ of
papers are often found together).
Greenhalgh et al (2005) found only 30% of primary sources from predefined
search strategy. 51% found by other predefined methods (i.e. reference,
footnote and citation tracking).
However this may be topic specific and may reflect a priori decisions to
concentrate on non-electronic sources
What is Already Known? - 2
• “Literature searches…open-ended iterative processes
where the topic or research question of interest is
honed over time as the nature of the evidence
becomes more apparent” (Finfgeld-Connett & Johnson
2013)
• Number of articles not only critical factor – reports
“may lack enough thick description to fully develop
concepts and the interrelationships among them”
(Finfgeld-Connett & Johnson 2013)
• Unpublished studies may contain rich, thick description
• Goal may not be aggregative – theoretical saturation
may play a part – selection of sample is crucial
References for What is Known Already - 1
• Bates MJ: The design of browsing and berrypicking techniques for
on-line search interface. Online Rev 1989, 13:407-424.
• Dixon-Woods M, Bonas S, Booth A et al.: How can systematic
reviews incorporate qualitative research? A critical perspective.
Qual Res 2006, 6:27-44.
• Finfgeld‐Connett, D., & Johnson, E. D. (2013). Literature search
strategies for conducting knowledge‐building and theory‐generating
qualitative systematic reviews. Journal of Advanced Nursing, 69(1),
194-204.
• Greenhalgh T, Peacock R: Effectiveness and efficiency of search
methods in systematic reviews of complex evidence: audit of
primary sources. BMJ 2005, 331:1064-1065.
• Ring N, Jepson R, Ritchie K: Methods of synthesising qualitative
research for health technology assessment. Int J Technol Assess
Health Care 2011, 27:384-390
References for What is Known
Already - 2
• Walters LA, Wilczynski NL, Haynes RB: Developing optimal search
strategies for retrieving clinically relevant qualitative studies in
EMBASE. Qual Health Res 2006, 16:162168. PubMed Abstract | Publisher Full Text
• Wilczynski NL, Marks S, Haynes RB: Search strategies for identifying
qualitative studies in CINAHL. Qual Health Res 2007, 17:705710. PubMed Abstract | Publisher Full Text
• Wong SL, Wilczynski NL, Haynes RB: Developing optimal search
strategies for detecting clinically relevant qualitative studies in
MEDLINE. In Medinfo 2004: Proceedings of the 11th World Congress on
Medical Informatics; San Francisco. Edited by Fieschi M, Coiera E, Jack
Li YC. Amsterdam: IOS Press; 2004:311-314.
• McKibbon KA, Wilczynski NL, Haynes RB: Developing optimal search
strategies for retrieving qualitative studies in PsycINFO. Eval Health
Prof 2006, 29:440-454. PubMed Abstract | Publisher Full Text
Overview of Methodological
Issues/Challenges
• Bias towards/Predominance of quantitative
research and publication of resultant reports
• Non-optimal indexing of qualitative studies
(CINAHL more evolved than MEDLINE)
• Qualitative research represents various research
methodologies, including ethnography,
phenomenology, grounded theory and narrative
analysis, which may hinder retrieval
• Lack of informative manuscript titles and
abstracts
Overcoming barriers:
• Searches should be as transparent as possible
without jeopardizing the creativity and
complexity of the process (Finfgeld-Connett &
Johnson 2013)
• Key “test” for knowledge-building/theory
generating review – would small amounts of
conflicting information substantially change the
findings? [Theoretical Saturation/ Qualitative
Sensitivity Analysis]
• Weaknesses in indexing mean that sensitivity of
searches may need to be reduced to allow time
for other search strategies (Pearson et al, 2011)
Simple search strategies vs complex
ones
• Three broad-based terms (i.e. “qualitative”, “findings”
and “interviews”) as effective as more complex search
strategies in identifying relevant qualitative research
reports (Flemming & Briggs 2007).
• Search strategies with broad search terms (“qualitative
research” or “qualitative studies” or “interviews”)
[combined with CAM terms] had highest recall and
precision (Franzel, 2013).
• Within time-limited context, protocol-driven, targeted,
and reference-checking search strategies most effective
(Pearson et al, 2011)
Intervention Searching vs Condition
Searching (Lorenc et al, 2012)
• Tying search terms of SR of qualitative
evidence too closely to interventions may
compromise consistency of the review.
• Dilemma: Performing condition-wide searches
(with no other change to strategies) would
become highly over-inclusive and volumes of
records impracticably large. (Suggests need for
alternative sampling stategies)
Is More Necessarily Better?
• For knowledge-building and theory-generating
systematic reviews, “more is better only when it helps
to fully explicate a concept, substantiate an
interconnection between or among concepts, or build
a line of argument. Simply, more of the same does not
necessarily help to achieve these objectives. In fact,
collecting more of the same may merely escalate the
cost of a study, clutter the database and obfuscate
important inferences. Concepts and interrelationships
among them can only be more fully explicated based
on data that adds depth, breadth, meaning and
understanding to a phenomenon” (Finfgeld-Connett &
Johnson 2013).
Searching for religion and mental health
studies required health, social science, and grey
literature databases (Wright et al, 2014)
• PsycINFO = best performing database
• ArabPsyNet, CINAHL, Dissertations and Theses, EMBASE,
Global Health, Health Management Information
Consortium, MEDLINE, PsycINFO, and Sociological Abstracts
essential to retrieve included references.
• Citation tracking and personal library of one of the research
teams made significant contributions of unique, relevant
references.
• Religion studies databases (Am Theo Lib Assoc, FRANCIS)
did not provide unique, relevant references.
• Literature searches for reviews/evidence syntheses of
religion and health studies should include social science,
grey literature, non-Western databases, personal libraries,
and citation tracking activities.
Exhaustive versus Expansive
That is the Question!
Exhaustive versus Expansive (FinfgeldConnett & Johnson 2013)
• Exhaustive searches when conducting
summative and aggregative systematic
reviews
• Expansive searches when conducting
knowledge-building and theory-generating
systematic reviews
Exhaustive versus Expansive
•
•
•
•
Multiple databases
Comprehensive list of terms
Assumption of homogeneity
Other approaches are
“supplementary”
“Combing the Area”
• Databases to reflect
contributing disciplines
• Terminology may
characterise “narrative” of
each discipline
• Likelihood of divergent
cases
• Sibling studies (shared
context)
• Cluster searching
• Citation searching
• “Supplementary”
techniques may be more
important
“Following Up Leads”
Outstanding challenges (Gallacher et
al, 2013)
• Creating an appropriately sensitive and specific
search strategy was a significant challenge….
• Adding ‘qualitative methods’ made search
strategy considerably more specific while
retaining sensitivity, as demonstrated by return of
all key papers identified in scoping search.
• Our final results showed that 94% of papers were
identified by our predefined database search.
Recent Developments
• Importance of Context/Theory
• Database Coverage
• Importance of Supplementary Search
Methods
• Appropriate Selection of Sampling
Methods
• Reporting Standards (e.g. ENTREQ)
Searching for Contextual Richness
Requires identification of related (sibling)
reports i.e. cluster searching (cluster becomes
unit of analysis, not study) (Booth et al, 2013)
Cluster searching for “Siblings” (Booth
et al, 2013)
Element
Procedural Steps
Citations
Backwards reference chaining
Lead Authors
Author searching; Backwards reference chaining
Unpublished materials
Web searches; repositories
Scholar searches
Citations
Theories
Backwards reference chaining
Early Examples
Cited works (Forward reference chaining)
Related Projects
Co-citations
Searching for Theories
• Theory not typically reported in Abstracts
• Theoretical base differs by discipline cp. HSR vs
Public Health vs Nursing vs Psychology vs Sociology
• Reporting of Theory differs by discipline
• Level of Theory may vary e.g. Individual versus
Society (Psychology vs Sociology)
• When is a “theory” a Theory? – labelling (model,
framework, concepts ) and naming (“Health Belief
Model”)
Database Coverage
• CAM search - PubMed yielded 87% of relevant
included qualitative studies (Franzel et al, 2013).
• Five different QES PubMed coverage values 35/44
(79.5%); 9/10 (90%); 10/11 (91%); 9/9 (100%);
7/28 (25% - Grey literature) (Booth 2012
[Unpublished])
• But 5/28 of studies located only through
supplementary searches of three sources
(Stansfield et al, 2012). 21 search sources
required to locate all studies. [Explanation - Role
of Grey Literature and Geographical focus – UK
only]
Supplementary Strategies
More diffuse topic, move beyond
electronic searching
Supplementary Strategies - 1
• Multiple search strategy more likely to identify
relevant QR than sole reliance on electronic
searching.
• Purpose of synthesis determines appropriate
sampling/search strategy. E.g. mapping out key
conceptual developments – if aim not aggregative,
omission of papers unlikely to have dramatic effect
on results.
• Suggests max. circa 40 papers - difficult to maintain
sufficient familiarity with > 40 papers (Campbell et
al, 2011)
Supplementary Strategies - 2
• Need ‘belt and braces’ (hand-searching; consultation
with experts)
• Searching for books/theses particularly challenging
(not indexed in same way as journal papers) (Campbell
et al, 2011)
• Snowballing and consultation with experts for a realist
review (Pawson et al, 2004)
• Obtaining authors' suggestions - resource-intensive
process with negligible results (Pearson et al, 2011)
• Additional search techniques essential to locate
further high quality references (Papaioannou et al,
2010)
Sampling: Appropriate ≠ Comprehensive (Suri,
2011)
• 16 strategies for sampling in QES
• E.g. Snowball sampling - seeking information from key
informants about other ‘information-rich cases’.
– ‘The chain of recommended informants would typically
diverge initially as many possible sources are
recommended, then converge as a few key names get
mentioned over and over’ (Patton, 2002, p. 237).
• Identify most cited primary research reports by ‘footnote
chasing’ (searching citation indices, browsing through
bibliographies, previous research syntheses, primary research
reports, policy documents, papers written by practitioners
and papers written for practitioners).
Combination or Mixed Purposeful
Sampling (Suri, 2011)
• Employ two or more sampling strategies to select evidence to
adequately address purpose.
• Mixed purposeful sampling can facilitate triangulation and
flexibility in meeting the needs of multiple stakeholders (e.g.
extensive sampling for generalisations at higher level of
abstraction. Typical case sampling to provide readers with
immediacy of typical studies that contributed towards
informing more abstract generalisations).
• When selecting combination of sampling strategies,
synthesists must reflect on how strategies complement each
other.
Footnote chasing (Suri, 2011)
• cp. footnote chasing for exhaustive sampling,
footnote chasing for snowball sampling
involves locating most cited papers.
• However, may reinforce confirmatory bias (i.e.
studies agreeing with prevalent wisdom more
likely to be published and cited, studies that
contest conventional wisdom less likely to be
published or cited)
ENTREQ & Searching (Tong et al, 2012)
3
4
5
6
Approach to
searching
Indicate whether search was pre-planned (comprehensive
search strategies to seek all available studies) or iterative (to
seek all available concepts until they theoretical saturation is
achieved).
Inclusion criteria
Specify inclusion/exclusion criteria (e.g. in terms of
population, language, year limits, type of publication, study
type).
Data sources
Describe information sources used (e.g.electronic
databases (MEDLINE, EMBASE, CINAHL, psycINFO,
Econlit), grey literature databases (digital thesis, policy
reports), relevant organisational websites, experts,
information specialists, generic web searches (Google
Scholar) hand searching, reference lists) and when searches
conducted; provide rationale for using data sources.
Electronic Search
strategy
Describe literature search (e.g. provide electronic search
strategies with population terms, clinical or health topic
terms, experiential or social phenomena related terms, filters
for qualitative research, and search limits).
Improvement in Search Reporting
1988-2004
2005-2008
(Hannes & Macaitis, 2012)
Outstanding Issues/Challenges
• How to understand publication bias in qualitative
research (not around positive/negative findings)?
• How to systematise (and document) more
intuitive search approaches e.g. cluster searching
and searching for theory?
• How does sampling strategy translate into search
strategy?
• How to construct sampling frames for studies?
• How to sample for diversity?
• How many sources are enough?
• How to retrieve rich data? How to retrieve data
on theory and context?
Publication Bias in Qualitative
Research? – Part One
• “This does not mean that publication biases do not
exist in….qualitative research….a bias of potentially
greater proportions may threaten searches for
qualitative research reports…..in some circles,
qualitative research is perceived…of lesser quality and
value than quantitative research……qualitative studies
may be less frequently conducted, submitted for
publication and/or published in high quality and
easily accessible journals….raw data (i.e. research
findings)….needed to conduct a qualitative systematic
review may not be readily available” (Finfgeld-Connett
& Johnson 2013)
Publication Bias in Qualitative
Research? – Part Two
• “ignoring grey literature, such as dissertations/theses,
government reports, monographs and books, on the
basis that it may be of lesser quality (as with
quantitative research) is empirically and logically
invalid…..[such] documents may be particularly rich
sources of qualitative data as page limits are not
generally imposed. Also, although lengthy report
formats are relatively uncommon in the health
sciences, they tend to be the norm in disciplines, such
as anthropology….., where context-rich data are likely
to be found”. (Finfgeld-Connett & Johnson 2013)
• Qualitative researchers often choose to publish in book
form (“Truncation bias”)
Conclusions
• Increasing Importance of Explanatory
Sources (e.g. Context and Theory)
• Need for Ongoing Investigation of
Database Coverage and Supplementary
Search Techniques
• Requires Exploration/Selection of
Appropriate Sampling Methods
• Bottomline: Value versus Effort TradeOff
References - 1
• Booth A, Harris J, Croot E, Springett J, Campbell F, Wilkins E.
Towards a methodology for cluster searching to provide conceptual
and contextual "richness" for systematic reviews of complex
interventions: case study (CLUSTER). BMC Med Res Methodol. 2013
Sep 28;13:118.
• Campbell R, Pound P, Morgan M, Daker-White G, Britten N, Pill
R, et al. Evaluating meta-ethnography: systematic analysis and
synthesis of qualitative research. Health Technol
Assess 2011;15(43).
• Flemming, K., & Briggs, M. (2007). Electronic searching to locate
qualitative research: evaluation of three strategies. Journal of
Advanced Nursing, 57(1), 95-100.
• Gallacher K, Jani B, Morrison D, Macdonald S, Blane D, Erwin P, May
CR, Montori VM, Eton DT, Smith F, Batty GD, Mair FS; International
Minimally Disruptive Medicine Workgroup. Qualitative systematic
reviews of treatment burden in stroke, heart failure and diabetes methodological challenges and solutions. BMC Med Res Methodol.
2013 Jan 28;13:10.
References - 2
• Franzel, B., Schwiegershausen, M., Heusser, P., & Berger, B. (2013).
How to locate and appraise qualitative research in complementary
and alternative medicine. BMC complementary and alternative
medicine, 13(1), 125.
• Guise, J. M., Chang, C., Viswanathan, M., Glick, S., Treadwell, J.,
Umscheid, C. A., ... & Trikalinos, T. (2014). Systematic Reviews of
Complex Multicomponent Health Care Interventions [Internet].
http://www.ncbi.nlm.nih.gov/books/NBK194851/
• Hannes, K., & Macaitis, K. (2012). A move to more systematic and
transparent approaches in qualitative evidence synthesis: update on
a review of published papers. Qualitative Research, 12(4), 402-442.
• Papaioannou, D., Sutton, A., Carroll, C., Booth, A., & Wong, R.
(2010). Literature searching for social science systematic reviews:
consideration of a range of search techniques. Health Information &
Libraries Journal, 27(2), 114-122.
References - 3
• Pearson, M., Moxham, T., & Ashton, K. (2011). Effectiveness of search strategies
for qualitative research about barriers and facilitators of program delivery.
Evaluation & the Health Professions, 34(3), 297-308.
• Stansfield, C., Kavanagh, J., Rees, R., Gomersall, A., & Thomas, J. (2012). The
selection of search sources influences the findings of a systematic review of
people’s views: a case study in public health. BMC Medical Research
Methodology, 12(1), 55.
• Suri, H. (2011). Purposeful sampling in qualitative research synthesis.
Qualitative Research Journal, 11(2), 63-75.
• Tong, A., Flemming, K., McInnes, E., Oliver, S., & Craig, J. (2012). Enhancing
transparency in reporting the synthesis of qualitative research: ENTREQ. BMC
medical research methodology, 12(1), 181.
• Suri, H. (2013). Towards Methodologically Inclusive Research Syntheses:
Expanding Possibilities. Routledge.
• Wright JM, Cottrell DJ, Mir G. Searching for religion and mental health studies
required health, social science, and grey literature databases. J Clin Epidemiol.
2014 Jul;67(7):800-10.