How to Write a Systematic
Review of the Literature
Health Environments Research
& Design Journal
1-16
ª The Author(s) 2017
Reprints and permission:
sagepub.com/journalsPermissions.nav
DOI: 10.1177/1937586717747384
journals.sagepub.com/home/her
Debajyoti Pati, PhD, FIIA, IDEC, LEED AP1,
and Lesa N. Lorusso, MBA, MS Arch2
Abstract
This article provides a step-by-step approach to conducting and reporting systematic literature reviews
(SLRs) in the domain of healthcare design and discusses some of the key quality issues associated with
SLRs. SLR, as the name implies, is a systematic way of collecting, critically evaluating, integrating, and
presenting findings from across multiple research studies on a research question or topic of interest. SLR
provides a way to assess the quality level and magnitude of existing evidence on a question or topic of
interest. It offers a broader and more accurate level of understanding than a traditional literature review.
A systematic review adheres to standardized methodologies/guidelines in systematic searching, filtering,
reviewing, critiquing, interpreting, synthesizing, and reporting of findings from multiple publications on a
topic/domain of interest. The Cochrane Collaboration is the most well-known and widely respected
global organization producing SLRs within the healthcare field and a standard to follow for any researcher
seeking to write a transparent and methodologically sound SLR. Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA), like the Cochrane Collaboration, was created by an
international network of health-based collaborators and provides the framework for SLR to ensure
methodological rigor and quality. The PRISMA statement is an evidence-based guide consisting of a
checklist and flowchart intended to be used as tools for authors seeking to write SLR and meta-analyses.
Keywords
systematic literature review, healthcare design, evidence based design
As the name suggests, systematic literature
review (SLR) is a systematic way of collecting,
critically evaluating, integrating, and presenting
findings from across multiple research studies on
a research question or topic of interest. It is
“systematic” since it adopts a consistent, widely
accepted, methodology. The methodology should
address concerns regarding quality issues, such as
bias, replicability, credibility, et cetera.
SLR provides a way to assess the quality level
and magnitude of existing evidence on a question
or topic of interest. It offers a broader and more
accurate level of understanding than a traditional
literature review. Imagine a single cross-sectional
study as a single snapshot in time and a longitudinal study as a single video. As the number of
snapshots and videos available on a question or
topic increases, it offers a better and more accurate grasp on the phenomena of interest. Since a
single research is just one snapshot or video,
1
2
Texas Tech University, Lubbock, TX, USA
University of Florida, Gainesville, FL, USA
Corresponding Author:
Debajyoti Pati, PhD, FIIA, IDEC, LEED AP, Texas Tech
University, Lubbock, TX 79409, USA.
Email: d.pati@ttu.edu
2
Health Environments Research & Design Journal XX(X)
basing design or policy decisions on a single
research is not as robust as basing decisions on
findings from multiple studies. There are, of
course, issues related to the quality of snapshots
or videos, and the viewpoint from which those
were taken. Poor quality could result in less clarity
and, hence, less confidence. Changes in viewpoint
could offer a different perspective of a phenomena
or introduce bias. Even with appropriate measures
for quality, findings from individual studies may
vary owing to differences in population, setting,
instruments, random chance, and so on. A properly
conducted SLR considers the quality of issues and
incorporates multiple viewpoints, providing a
broader and more complete picture.
SLRs are particularly gaining importance in
the healthcare design field. Starting from about
600 published studies in 2004 (Ulrich, Zimring,
Quan, Joseph, & Choudhary, 2004), the number
of studies in the field has increased to a level that
facilitates SLRs on important questions. This
growing body of knowledge is making SLRs in
the healthcare design field both possible and critically important. From single-patient rooms to
decentralization, there is a wide range of topics
of importance to efficiency, safety, care quality,
cost, patient/family/staff experience, and so on,
which could benefit from SLRs. Articles on SLRs
are available in other fields of studies. The purpose of this article is to provide a step-by-step
approach to conducting and reporting SLRs in the
domain of healthcare design and discuss some of
the key quality issues associated with SLRs.
The purpose of this article is to provide a
step-by-step approach to conducting and
reporting SLRs in the domain of
healthcare design and discuss some of the
key quality issues associated with SLRs.
Types of SLRs
SLR began within the realm of healthcare and can
be traced back to the work of Scottish naval surgeon
James Lind who, in 1753, conducted the first randomized controlled trial (RCT) and recognized systematic methods as a key component in avoiding
bias, which remains a pivotal goal of systematic
research (Dunn, 1997). By the 1970s, metaanalysis was formalized as a process of research
synthesis (Glass, 1976) and evidence-based policy
and practice gained momentum in the 1990s.
SLRs are only one type of review of literature.
The type of review one needs to conduct depends
on the objective of the review. Broadly speaking,
the two most commonly known types of literature
reviews are the “traditional” or narrative review
and the “gold standard” or SLR (Booth, Sutton, &
Papaioannou, 2016). Traditional reviews, also
known as conventional or nonsystematic reviews
(Greenhalgh, 2014), are generally faster and easier to conduct and are sometimes appropriate due
to a short time frame or lack of resources. Grant
and Booth (2009) provide a more comprehensive
classification of 14 different types of reviews in
the fields of medicine and library sciences. In general, reviews differ in the question being asked,
including, among others: What is the current theoretical framework/understanding on a topic/
domain—based on either just peer-reviewed publications or all sources? What are some gaps, in
theory or evidence, in existing published literature? Is there sufficient justification for conducting
a study? What is the most appropriate study question/topic? Which research question/topic will provide the best return on investment? What is the
effect size of an intervention, based on quantitative
findings from previous studies? How did a certain
past intervention (design, policy, practice, etc.)
work? What is the volume of literature available
on a topic/domain of interest?
The key variations among these review types are
in the primary audience group being targeted
(intended end use of the review), scope, level of
interpretation, level of evaluation of individual
study quality, type of information included (published literature vs. other information sources), type
of data included (qualitative, quantitative, or both),
depth of analysis of published data/findings, level
of synthesis, level of investigator bias, level of publication bias, extent of time commitment by the
review team, and the type and amount of resources
needed to conduct a review, among others.
Among these 14 different review types articulated by Grant and Booth (2009), 3 are of relevance
to the current discussion, namely: (1) meta-analysis,
(2) qualitative systematic review or qualitative
Pati and Lorusso
evidence synthesis, and (3) systematic review.
These three types of reviews share several common
attributes: need for systematic search, need for comprehensiveness, need for study quality assessment,
need for interpretation, and, most importantly, the
need for transparency of the methodology such that
the study can be replicated. The variations between
the three types of review originate from the long
history and maturation of science and publication
in medicine and library sciences, resulting in more
targeted review types.
Meta-analysis uses statistical analysis procedures on quantitative data to estimate the effect
size of a specific intervention, where the input
data are the end products of the statistical procedures of a number of studies with vastly similar
questions, setting, population, intervention, and
time period. The authors believe that the healthcare design research field does not currently
enjoy sufficient volume of quantitative studies
with the above similarities, which may, however,
change as the field matures. The intent of a metaanalysis is similar to those of a systematic review.
The objective of a qualitative systematic review
or a qualitative evidence synthesis is similar to
meta-analysis, except that these reviews deal primarily with qualitative studies, and hence the outcomes and resulting end uses are different.
A systematic review adheres to standardized
methodologies/guidelines in systematic searching,
filtering, reviewing, critiquing, interpreting, synthesizing, and reporting of findings from multiple publications on a topic/domain of interest. It attempts to
capture the broadest set of available literature on the
topic of interest. After assessing the quality of individual studies, SLRs may eliminate low-quality
studies from further consideration. Moreover,
owing to the extensive documentation and reporting
of the steps and assumptions, SLRs renders itself
amenable for replication. It demands a team effort
(at least two) to eliminate bias, among other issues.
SLRs are inherently time intensive. Owing to the
aforementioned quality demands, SLRs is regarded
as the most robust level of evidence available on a
topic or question of interest.
SLRs is regarded as the most robust level
of evidence available on a topic or
question of interest.
3
Key Steps in Conducting SLRs
The first and most crucial step in conducting
an SLR is to familiarize oneself with generally
accepted standards on the subject. Today, the
Cochrane Collaboration is the most wellknown and widely respected global organization
producing SLRs within the healthcare field and a
standard to follow for any researcher seeking to
write a transparent and methodologically sound
SLR. Founded in 1993, Cochrane consists of
over 37,000 contributors from more than 130
countries who make up over 50 subject review
groups (Cochrane, n.d.). Together, this network
of contributors has transformed the healthcare
industry by publishing SLRs that gather and
summarize high-quality evidence to aide
researchers, physicians, caregivers, and others
in making well-informed decisions that positively
impact human health globally. Cochrane provides
guidance on methodological standards which are
available at methods.cochrane.org/mecir and are
compliant with the Preferred Reporting Items for
Systematic Reviews and Meta-Analysis
(PRISMA) which provides the framework for
SLR (Moher, 2009; PRISMA, n.d.).
PRISMA, like the Cochrane Collaboration,
was created by an international network of
healthcare-based collaborators seeking to
strengthen and streamline the methodological
rigor and quality of SLRs. First published
in 2009 as the PRISMA statement, it is an
evidence-based guide consisting of a checklist
and flowchart intended to be used as tools for
authors seeking to write SLR and meta-analyses
(Moher, 2009; PRISMA, n.d.). Several extensions to the original document were published
in 2015 and can all be found at www.prisma-sta
tement.org/extensions/default.aspx. Although
originally created for the review of randomized
trials, PRISMA is broadly applicable to myriad
types of research. The actual SLR then proceeds
systematically through various sequential steps,
including question formulation, team formulation, identification of search domains and publication sources, systematic search, systematic
critical analysis, systematic interpretation, and
systematic reporting. Table 1 outlines key steps
in the SLR process.
4
Health Environments Research & Design Journal XX(X)
Table 1. Key Steps in Conducting an SLR.
Step Description
1
2
3
4
5
6
7
8
9
10
Familiarize yourself with the PRISMA framework.
Develop your study question. It should be clear and unambiguous.
Develop your study team. SLRs work best when the team has at least three
members, where disagreement between two members is resolved by a
third member. Make sure that each member has the necessary
qualification and experience to conduct your specific SLR.
Identify the concepts to be included in your search. For SLRs focused on
interventional and/or comparative effectiveness studies, complete the
PICO search model. For other study questions, in general, one should
articulate at least three concepts of interest: (1) population(s) of interest,
(2) setting(s) of interest, and (3) issue(s) of interest.
Define each term in the study question and the identified concepts of
interest. Do not assume that any term is unequivocally understood either
by the team members or by the readers.
Select databases relevant to your topical area to conduct your search (at
least one, typically no more than five). Typical healthcare-based databases
are listed on the right. You may also include other databases as well,
depending on your topic, such as ERIC, JSTOR, etc. It is a good idea to
enlist the expertise of a reference librarian at this point.
Guided by the terms identified and defined in Step 5, now finalize your plan
for the systematic search. The first substep is to develop a series of key
words to be used for the search.
Next, decide the inclusion and exclusion criteria. All team members must
understand these criteria. Terms in the inclusion/exclusion criteria must
also be defined. Define the limits you will use for your systematic search.
Decide whether or not you will include gray literature (non-peerreviewed publications, dissertations, thesis, conference proceedings, etc.)
and whether or not you will incorporate reference list searching
(snowballing) or hand searches. See Appendix for example.
Next, go into each database individually to tailor your key words and search
terms specifically to each search engine. Each database will have its own
system for terms. Some use MeSH and truncated terms and some do not.
MeSH are Medical Subject Headings and is a vocabulary thesaurus
managed by the National Library of Medicine and are tied to key words
embedded within published manuscripts. When used, they allow the
researcher to search very specifically. This step has an ebb and flow to it
as you develop your search terms and key words. Too specific and your
search will yield nothing, too broad and your search will yield will be
unmanageable. Use the key word generators within the database to
create a formula of your search terms, using “or,” “and” as appropriate
between terms. Document in a table for transparency and replicability.
See Appendix for example.
Title review: Now conduct your search for all databases. Your initial review
can be done using titles within the actual database to screen out
irrelevant articles. Be generous in your inclusion at this point, making sure
to keep your yield manageable based on your team, time, and resources.
Export the article yield into a reference citation manager and then from
the citation manager into an excel spreadsheet as a .csv file. This will
become your journal citation report (JCR) which is a vital tool for the
screening process and overall search documentation.
Helpful Hints
If definitions already exist in
published literature, one
could adopt those.
PubMed, CINAHL,
PsycINFO, MEDLINE,
Web of Science
Limits may include Specific
date ranges, human
subjects, participant
descriptive, publication
type, language, etc.
The overall yield amount
considered manageable will
depend on the size of your
analytic team. Aim for
initial yields no more than
500 per database as a rule
of thumb.
Reference citation managers:
Zotero, EndNote,
RefWorks, etc.
(continued)
Pati and Lorusso
5
Table 1. (continued)
Step Description
Helpful Hints
11
You can use viewing panes
within Excel to help
condense the cells to only
what is relevant during this
stage (abstract and
screening questions).
12
13
13
14
15
Within the JCR, create a tab for each database, including snowballing, hand
searching, and gray literature on their own tabs, if included in the search.
Organize headers for relevant information including author name, abstract,
publication journal, DOI, volume, year of publication, etc. Include columns at
the right pertaining to your exclusion criteria that can be answered by yes or
no. Remove any duplicates by striking through and hiding the cells. This way if
you make a mistake, the information is still there and you can reincorporate
it into your JCR if needed. In case you are not sure whether to include or
exclude an article based on the title, either retain it or consult the other
team members.
Abstract review: Up to this point you have screened only titles, so now you
will begin to read through the abstracts of each article and screen them
against the exclusion criteria. Instead of inserting “yes” or “no,” it is helpful
to create separate columns for the two and insert a number “1” (for yes or
for no, as applicable) so that you can run a sum total at the bottom of each
spreadsheet to keep track of how many duplicates were removed, how
many were excluded and how many were included per database. Once you
have completed this for each separate database, create a tab for all included
articles. Copy and paste all the included articles into this sheet and sort by
title to quickly find and remove any remaining duplicates. This information
will be important data for the PRIMSA flowchart.
Abstract phase screening quality check: To strengthen validity and reliability
for your search results, work with at least one other researcher to
conduct an inter-rater reliability assessment. For instance, you can give
the second reviewer 20% of the total number of abstracts you reviewed,
randomly chosen, and leave your exclusion fields blank so that the second
reviewer can run a parallel screening process on the subset of abstracts.
Compare your findings and aim for at least 90% agreement. Discuss any
areas of disagreement with the third team member reviewer until you
reach an inclusion/exclusion agreement.
On the last tab in the Excel spreadsheet, organize your process totals in the
JCR and update your PRISMA flowchart with the information. Keep the
JCR as a tool and record of your process.
Critical review of full articles: Once you have the list of final included articles,
conduct in-depth critical review of each article. Extract all relevant
information from each article and insert in the appropriate place in the Excel
document. As part of the critical review, rate the level of quality of evidence
for each article using the framework you have adopted for your study.
Critical review assessment quality check: To strengthen validity and reliability
for your critical review findings, work with at least one other researcher to
conduct an interrater reliability assessment. For instance, you can give the
secondary reviewer 20% of the total number of articles you critically
reviewed, randomly chosen, and leave your assessment fields blank, so that
the second reviewer can run a parallel critical evaluation process on the
subset of articles. Compare your findings and aim for complete agreement.
Closer the agreement to 100%, more robust is the critical review process.
A level of at least 90% agreement may be acceptable. Discuss any areas of
disagreement with the third team member.
Note, once you have deleted
articles from inclusion in
the JCR you can remove
them from your reference
citation manager for
consistency.
You may review the research
question, exclusion
criteria, and definitions
again to ensure uniformity
of understanding.
PRISMA flowchart can be
found at http://www.prismastatement.org/
You may review the research
question, the quality of
evidence scale, quality
issues, and definitions again
to ensure uniformity of
understanding.
(continued)
6
Health Environments Research & Design Journal XX(X)
Table 1. (continued)
Step Description
16
Helpful Hints
You can also calculate
Update your PRISMA flowchart with any changes from the inter-rater
interrater reliability with
reliability analysis. Now you will create a new, clean spreadsheet that will
the level of quality of
become your matrix of included studies. This spreadsheet will be simpler
evidence to further
than the JCR with headings that are tailored specifically to the purpose of
strengthen validity and
your overall SLR. These might include first author, year, level of evidence,
reliability of your SLR.
methods, participants, environmental intervention, outcomes, setting,
assessments, results, etc. This will likely be published with your article
and should be well organized. Now you are ready to compose your
report, following the guidelines from the Cochrane Collaboration and the
PRISMA checklist, and using Table 2.
Note. SLRs ¼ systematic literature reviews; PRISMA ¼ Preferred Reporting Items for Systematic Reviews and Meta-Analysis.
Contents within the 27 PRISMA checklist items
include guidelines regarding key steps in the process and are broken into title, abstract, methods,
results, discussion, and funding related to the SLR
(Moher, 2009; PRISMA, n.d.). According to the
PRISMA checklist, authors should first clearly
identify the report as a systematic review, metaanalysis, or both. This is the only aspect of PRISMA
not required by the Cochrane Collaboration but is a
good rule to follow for healthcare design researchers who wish to make the nature of their work
easily identifiable for publishers and readers. The
following is a summary of the PRISMA checklist,
which can be found and downloaded at http://
www.prisma-statement.org/ for use as a guide.
Introduction. The two critical elements within the
introduction are the author’s rationale and objectives. The rationale for the review should be
written in a way that explains the topic in light
of what is currently understood in the existing literature. The objectives should follow and clearly
identify the question(s) to be addressed and how
they relate to the various concepts of interest. In
interventional studies, a typical articulation of concepts of interest is participants/interventions/comparisons/outcomes (PICO) and study design
(Figure 1). The PICO process comes from the field
of evidence-based medicine (EBM) to address a
healthcare-based question. In SLRs, PICO can be
used to frame the search terms and key words to
the specific topic being studied (Centre for
Evidence-Based Medicine, 2014; Centre for
Reviews and Dissemination, 2009; Lorusso &
Bosch, 2016). Investigators should identify or
develop their specific list of concepts of interest,
such as population/setting/issue, based on the
study objective and report it in the introduction.
Abstract. The abstract of the report should consist
of a detailed summary including the following elements, as applicable to each report: background,
objectives, data sources, study eligibility criteria,
participants and interventions, study appraisal and
synthesis methods, results, limitations, conclusions
and implications of key findings, and inclusion of
a systematic review registration number if applicable. The overall goal is to be clear and concise in
the abstract so as to transparently communicate the
contents of the paper to readers.
Methods. The methods section should outline the
overall process followed within the SLR. It is
important to maintain clarity and transparency
throughout the description of the process followed in the SLR in order to reduce bias and
improve the translation of the overall findings.
This section will include information on the protocol and registration, eligibility criteria, information sources, search strategy, study selection
and data collection process, data items studied,
risk of bias in individual studies, a summary of
A Suggested Model for
Systematically Composing
an SLR Report
Key Sections in Writing an SLR Following the
PRISMA Checklist
Pati and Lorusso
7
P
Patient, Population or Problem
e.g. Dementia
I
Intervention
e.g. Multisensory Environments
C
Comparison of Intervention (if appropriate)
NA
O
Outcome to Measure or Achieve
e.g. Patient Behavior
Figure 1. Explanation of participants/interventions/comparisons/outcomes (PICO) search model (Center for
Evidence-Based Medicine).
measures, synthesis of results, risk of bias across
studies, and any additional analysis. This section
provides an overview explanation preceding the
results section which will provide information
culminating from a deeper dive into many of the
same topics for the studies included in the final
review. Protocol and registration, if it exists and
where it can be found, should be reported. Eligibility criteria should be provided of included
studies such as study characteristics, time frame
of studies, language, publication, and so on, and a
rationale for why the author used these specific
eligibility criteria. Information sources included
in the review should be clearly described. These
are typically the databases targeted for the review
with an explanation of the dates used within the
review and an explanation of any contact with
authors for additional included studies. The
search strategy should be clearly defined for one
or more databases. This should be provided in
great detail to allow for future updates or replication and should include any limits used by the
author. The study selection process should be outlined regarding screening of findings, ratings of
eligibility, and a breakdown of what articles were
included in the review. This is supported by the
PRISMA flowchart, which provides a clear, graphic representation of the process from included
articles through to the final selection of included
articles (Figure 2). A template is available at
http://www.prisma-statement.org/PRISMAState
ment/FlowDiagram.aspx
The data collection process and precise
method used by the authors to gather information
from the literature should be clearly provided.
This is often done through an excel document
known as the journal citation report (JCR) that
collates information extracted from reference
managers in an organized way (author name,
publication, publication date, volume, abstract,
etc.). It is often used as a tool by the author or
team writing the SLR to rate the found articles
against the inclusion criteria and keeps a record of
duplicates, excluded and included articles. Data
items are listed within the methods section and
include variables and all assumptions or simplifications made by the SLR author(s). Risk of bias
in individual studies is provided and discussed in
terms of how the risk was assessed for each article
and how it will be used in the data synthesis of the
SLR. A summary of measures should be outlined in
terms of the primary measures used within the articles included in the SLR. This helps to clarify the
overall analysis and helps readers understand how
outcomes were achieved. A synthesis of results is
explained by a description of the methods used by
the SLR author(s) to review data and evaluate the
level of evidence. Risk of bias across studies is
discussed that may impact the overall findings. This
is different from the risk of bias within individual
studies because it discusses a broader range of risk
including publication bias or limited reporting of
data throughout studies. Additional analyses are
important particularly in SLRs, which might
include sensitivity or subgroup analyses, metaregression, and so on. It may also include any other
additional analysis conducted by the SLR author or
team relevant to the synthesis of findings.
Results. The results section of the SLR is presented
in descriptive, graphic, and detailed tabular formats. Combined, these representations should
carefully describe the study selection, characteristics, risk of bias within and of individual studies,
synthesis of results, risk of bias across studies, and
any additional analysis of findings. Whereas the
methods section provides an introduction, and
outline of how these topics will be discussed, the
8
Health Environments Research & Design Journal XX(X)
Figure 2. Example of Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) flow
diagram.
results section provides great detail specifically on
the articles included in the final review. Study
selection, supported by the PRISMA flowchart,
provides detailed information on the number of
studies screened, assessed for eligibility, and
excluded or included for review with an explanation for exclusion throughout the process. Study
characteristics are provided for each included article along with information on the data that were
extracted. This could include participant or site
descriptions, intervention, method, time frames,
and so on. This information is supported graphically in a matrix of the included studies. Risk of
bias within studies is identified with outcome-level
assessments. Results of individual studies are provided for all outcomes including information on
the intervention and any group differences. This
can be done in written format, separated into common themes or areas of affinity, and supported
graphically by the SLR in a matrix of the included
studies. A synthesis of results will provide information on meta-analysis of included articles and
Pati and Lorusso
assessed levels of evidence. Risk of bias across
studies and additional analysis provides information on broad risk of bias across all included studies and detail any additional analysis conducted on
included literature.
Discussion. Lastly, the discussion section of the
article will bring all the information together into
a final summary of the evidence, listing limitations of the SLR and offering conclusions stemming
from the review. In healthcare design SLRs, this
section may also include a description of possible
implications for practice that can be used by healthcare designers, researchers, and administrators in
future applications. The summary of evidence will
restate the main findings and discuss the overall
quality level and magnitude of evidence and how
each finding relates to key user groups within the
healthcare industry. A discussion of limitations is
an important component of the SLR to overcome
bias. All limitations including any risk at any level
of the review process should be identified. All
reviews will have some form of risk of bias and so
this is not a section that should be avoided or seen as
a weakness to the overall report. To the contrary, an
open and straightforward discussion of any risk of
bias serves to increase the credibility of an SLR.
Conclusions. The concluding section of the discussion will include the author(s) perspective on the
findings and their role related to other reported findings and how they may impact future research or
healthcare design overall. Any sources of funding
should be reported including a description of any
other support from sources supplying data and the
subsequent role of funding sources related to the SLR.
Table 2 provides a suggested step-by-step,
generic model for composing an SLR report. It
includes important questions that need to be
responded to in each section of an SLR report.
One may ignore questions that do not apply to
one’s specific SLR study.
Explanation of Key Elements Within
the Review
As discussed earlier, critical factors separating
SLR from traditional reviews are clarity, validity,
and auditability (Booth et al., 2016). These
9
elements are best achieved by following the
Cochrane Collaboration’s guidelines and the
PRISMA statement checklist closely. Adhering
to these frameworks helps ensure a level of uniformity in the SLRs, supporting transparency, and
reducing overall risk of bias. This uniformity
incorporates key elements including an assessment
of the evidence, the concepts of interest and
PRISMA flow diagrams, a JCR, and a matrix of
included studies.
Both the quality level and magnitude of evidence need to be analyzed in an SLR. The quality
level of evidence pertains to the quality of an individual study. The magnitude of evidence pertains
to the volume of quality evidence available on a
topic or question of interest. Referring back to our
analogy of snapshots in the introduction, the quality level of evidence can be viewed as the quality
of each photograph whereas the magnitude of evidence can be viewed as the number of good quality
photos available for stitching together.
The purpose of critically assessing the evidence
in an SLR is important because it provides the
reader with a description of its quality. This assessment is often tied directly to the inclusion/exclusion
criteria when an author will state that only certain
quality of evidence will be included. This may be
done in an SLR of a topic where the author(s) goal is
to report only the highest quality of evidence and
there is enough available literature to be selective.
Sometimes, however, this is not possible particularly in the healthcare design domain, currently,
because the body of knowledge regarding a topic
may not yet be so extensive as to allow it. Whatever
the decision of the author or team conducting the
SLR, it is important to rate the quality level of evidence. Examples can be found in EBM (Balshem
et al., 2011) and also in healthcare design literature
(Hamilton, 2011; Marquardt, Bueter, & Motzek,
2014; Pati, 2011; Stichler, 2010) and are commonly
described in a simple table format outlining the
descriptions of each quality level.
Other key elements within the review are the
concepts of interest (such as PICO) and PRISMA
flowchart both of which relate to the overall
search strategy of the SLR. It is important to note
that when designing the specific search strategy
for each SLR, it is incredibly valuable to enlist the
help of a research librarian with expertise within
10
Health Environments Research & Design Journal XX(X)
Table 2. A Step-by-Step Model for Composing an SLR Report.
Main
Sections
Subtopics
Abstract
Introduction
Method
Team composition
Search preparation
Search process
Critical review
Results
Description of
final list
Content of final list
Content of each
thematic area
Key Questions to Respond to
[as required by the targeted journal]
What is the SLR study question?
Background: What was already known on the topic/question of interest?
What drove the need for conducting this specific SLR?
What is the objective of this SLR?
What is the significance of this SLR study?
How many investigators were on the team? What were their qualifications,
expertise, and levels of experience?
What databases were targeted?
What key words/phrases were used in each database search?
What were the inclusion and exclusion criteria?
How and when were the database searches conducted? How many phases of
search were involved?
Was a hand search conducted? How was it conducted?
How was it determined whether to include or exclude a specific publication?
How many steps were involved in arriving at the final list of publications?
What was involved in each step?
How was interrater reliability ensured during the abstract review phase?
How was the critical review conducted?
What information was extracted and recorded?
How was the quality level of each study determined? What quality
framework was used? How were qualitative, quantitative, and mixedmethod studies evaluated?
What method was used to develop thematic areas of relevance? How were
studies within each thematic area synthesized?
Were any statistical analysis/meta-analysis conducted on quantitative
findings? If not, why?
How was interrater reliability ensured during the critical review phase?
How many articles were found in each sequential search step? How many
were retained in each step and how many were discarded? What was the
final number of articles on which critical review was performed?
How many studies were empirical?
Of the empirical studies, how many were causal/experimental, how many
were relational, and how many were descriptive?
Of the empirical studies, how many were qualitative, how many were
quantitative, and how many used mixed methods?
In which countries were the studies conducted?
How many studies were classified at each level on the quality of evidence
scale?
What thematic areas were identified?
For each thematic area (separately), what are the key descriptions of the
contents (respond to the same questions in the “content of final list,” but
for each thematic area)?
What were the key findings in each thematic area? What are some patterns
observable? Are there any converging/diverging findings?
What is the overall confidence (magnitude of evidence) within each thematic
area?
Within each thematic area, how many studies and what proportion of
studies fall under each level of the quality of evidence scale?
(continued)
Pati and Lorusso
11
Table 2. (continued)
Main
Sections
Subtopics
Key Questions to Respond to
Discussions
Summarization
What new information was generated from this systematic review?
What is the general level (quality and magnitude) of evidence in various
thematic areas? Which of the thematic areas enjoy higher quality and
magnitude of evidence? Which areas currently lack quality and/or
magnitude of evidence?
What are the limitations of this SLR?
What topics/questions should be examine in future studies?
What are the key take-away messages of this SLR for your target readers?
[include complete list of articles reviewed and/or cited in the SLR, denote
articles included in the SLR matrix with an asterisk*]
[as appropriate]
Limitations
Future studies
Conclusions
References
Appendix
the specific topic to be reviewed. Research librarians with a focus on healthcare, for example, have
extensive knowledge of the broad range of literature and are adept in assisting in the development of the PICO model serving as a guide for the
search terms and key words relating the patient,
population or problem, intervention, and outcome
to be evaluated in the SLR, if the SLR is focused
on interventional studies. Once established, the
PICO search model for the SLR is often illustrated with a table and it is helpful to document
the exact key words and terms used in a separate
appendix for future revision or replication of the
search. In case of SLRs not restricted to interventional studies, librarians can also assist in developing the right search terms for population,
settings, and issues of interest. The PRISMA flow
diagram is also used to document the search strategy. A template is available from the PRISMA
statement’s website that can be edited and
inserted directly into a Word document. The
flowchart is simple and easy to follow and provides a consistent way to communicate the results
of the search process from the initial findings,
inclusion of gray or snowballed literature, exclusions, and final inclusion counts for the SLR.
One of the most important elements within the
SLR is the matrix of included studies. The matrix is
a tabular representation of the results section of the
paper made in a spreadsheet document that summarizes the details of each included study and the
overall findings of the SLR. It is important to note
that the articles included in the final review are
included in the reference list of the paper and are
marked with an asterisk to clearly identify them for
the reader. The matrix will be tailored specifically to
each SLR based on the unique objectives of the
paper. Typical components of the matrix include the
last name of the first author and year of publication,
assigned quality level of evidence, methods, participants’ description, intervention summary, list of
outcomes, description of setting and assessments,
and a brief recap of the results of each study. The
intention of the matrix is to serve as a tool providing
an easily digestible format for readers to understand
the purpose and outcomes of the SLR.
Important Issues in
Conducting SLRs
There are several issues that warrant close attention while conducting and reporting an SLR. The
key question one should ask while conducting an
SLR is whether the intended audience will consider the findings credible (irrespective of the
actual nature of SLR findings). If trust is not generated among the readers, then findings may not be
perceived as credible, which will be counterproductive to the objectives of the SLR study. The
following subsections outline some of the issues
that investigators need to accord special attention
to in order to generate trust and confidence.
The key question one should ask while
conducting an SLR is whether the intended
audience will consider the findings
credible.
12
Systematicity
One distinguishing feature of SLRs that separates it
from most other types of reviews is systematicity.
Being systematic includes a systematic process of
searching for literature, filtering results of literature
searches, critical analysis of the studies, evaluating
quality of identified studies, evaluating the level of
evidence, interpretive analysis of findings across multiple studies, and reporting for the intended audience.
Documentation
One simple strategy for generating trust and confidence is documentation. Each step and results
should be documented and reported in a clear,
unambiguous manner.
Study Question
The most critical aspect of any research study is
the study question. It should be noted that SLR,
owing to its interpretive component, could be
deemed as a research study. The study questions
impact every subsequent step in a study. It should
be clear and precise.
Definitions
One way of achieving clarity and precision in the
study question is developing clear, unambiguous,
and documented definitions of all terms associated with the study. Definitions are important in
bringing all the investigators and the audience on
the same page. Table 3 provides the definitions of
key terms used in this article.
Search Strategy
The objective of a typical SLR is to include all
literature associated with a study question, including literature in peer-reviewed publications and
other sources, such as dissertations. Numerous
non-peer-reviewed sources, also referred to as
“gray literature” should also be brought into scrutiny, such as white papers and industry/trade articles. Search may (or should) include manual
review of the reference section of key articles.
In some fields, quantitative studies that do not
result in a statistically significant findings face
Health Environments Research & Design Journal XX(X)
greater difficulties in getting published in peerreviewed journals (also known as “publication
bias”), whereas from a theoretical perspective and
in numerous real-life situations knowledge
regarding nonsignificant findings could also be valuable. As a result, it is essential to search and retrieve
associated literature from all types of sources.
How a search is conducted could lead to bias,
whether intentional or otherwise. The appropriate
way to develop trust in the audience of an SLR is
to demonstrate that a broad, complete, and thorough search was conducted, wherein the probability of missing any important work related to
the study question is minimized.
Inclusion and Exclusion Criteria
Another source of potential bias and ambiguity is
in the way literature found in the searchers are
included or excluded from further analysis. Clear
documentation of criteria used for inclusion or
exclusion of a study will result in a higher trust
and credibility in the SLR findings.
Critical Review
Producing a good SLR is dependent on critical
review of the quality of each study. There are two
approaches to assessing study quality. One
approach focuses purely on the research design,
wherein studies with higher internal validity are
considered higher in quality. Quantitative studies
have typically been considered higher in quality
than qualitative studies since assessing internal
validity is easier in quantitative study design. For
instance, an experimental study design would be
considered higher in quality than a relational or
descriptive study or a qualitative study design.
However, research design is not the only factor
(and should not be) to assess in determining study
quality. Perhaps, when time is limited, a classification based just on research design is justified. It
should never be assumed that a good research
design necessarily results in a good quality study.
Ideally, each study should be assessed deeper on all
quality issues—construct, internal, external, and
conclusion validity as well as reliability. Issues
such as definitions, study setting, tools and measures, sampling strategy, sample characteristics,
Pati and Lorusso
13
Table 3. Definition of Terms Found in Article.
Term
Definition
Areas of affinity
Auditability
Bias
Clarity
Cochrane Collaboration
Conformity
Credibility
Cross-sectional Study
Databases
Dependability
Dissertations
Effect size
Eligibility criteria
Gray literature
Industry/trade articles
Instruments
Intervention
Investigator bias
Journal citation report
Limits
Longitudinal study
Magnitude
MeSH
Meta-analysis
Mixed-methods Studies
Peer-reviewed
publications
Population
PRISMA
Protocol
Publication bias
Qualitative
Quality level
Quantitative
Randomized
controlled trial
Reference manager
Replicability
Research question
Sample characteristics
Sample size
Sampling strategy
Setting
Snowballed literature
Statistically significant
Subgroup analyses
Synthesis
Themes
Theoretical framework
Topic of interest
Transferability
Common themes found in the SLR literature.
Ability for the SLR to be externally reviewed and replicated.
Prejudice, predisposition, or partiality to a topic or viewpoint.
Related to the level of transparency, openly displaying the process.
Global healthcare research organization producing SLRs.
Consistency in design a method or process.
Reliability or trustworthiness of a method or process.
Study that analyzes data at a specific point in time.
Online archives (e.g., PubMed, CINAHL, MEDLINE, etc.)
Related to the soundness of the findings.
A five-chapter report written as a requirement for a PhD degree.
Statistical measure of strength of relationship between variables.
Criteria or boundaries that must be shared by all articles included.
Unpublished or published noncommercially, not peer reviewed.
Articles published in a commercial magazine, targeting industry.
Measurement tools to gather data (e.g., survey, observations, etc.)
That which is being tested in an experimental study.
Occurs when investigators influence results due to predisposition.
Document used in an SLR to organize and analyze literature found.
Boundaries within which search is conducted (e.g., date range, species, etc.)
Data is gathered from the same subjects over a long period of time.
Relating to the size or extent of existing, published literature.
Medical Subject Headings, thesaurus terms embedded in literature.
Statistical approach combining results from multiple quantitative studies.
Studies that employ both qualitative and quantitative methods.
Evaluation prior to publication by others within the same field.
Describes the people at the focus of a study.
Preferred Reporting Items for Systematic Review and Meta-Analyses
A document outlining rationale and detailed procedure of a study.
Occurs when a publication has a known partiality to a topic or findings.
Research involving nonnumeric data.
Relating to the value and perceived excellence of a study.
Research involving numeric data.
Studies where subjects are assigned randomly to receive one of several interventions
where one is a base-line of comparison.
Online software used to organize and manage literature findings.
The ability for a study to be repeated with similar results.
The question that a researcher attempts to find an answer.
The defining features of a study sample (e.g., gender, race, age, etc.)
The number of participants in a study.
The researcher’s approach to finding participants for a study.
The location of a particular study (e.g., hospital, nursing home, etc.)
Literature found through the reference list of included articles.
A result or finding between variables not attributed to chance.
Exploration of how people respond differently to an intervention.
Combination of findings.
Similarities or commonalities found between studies.
Interrelated concepts.
Subject matter that a researcher seeks to find deeper understanding on.
How easily a finding can be applied to other settings.
(continued)
14
Health Environments Research & Design Journal XX(X)
Table 3. (continued)
Term
Definition
Transparency
Truncated terms
Validity
Relates to the clarity and openness in the design of a study.
Search technique that broadens to various spellings with an asterisk
Closeness to the truth.
sample size, analytical procedures, measures
adopted to remove bias, among many others, should
be evaluated in order to accurately, systematically,
and transparently gauge the quality of a study.
One fundamental requirement for conducting
critical quality review of a study at this depth is to
develop a team with appropriate qualifications to
enable it. One should carefully select the team of
a proposed SLR to achieve a robust level of critical review. It should also be noted that qualitative methods also have quality issues similar to
those in quantitative methods, such as credibility,
transferability, dependability, and conformity
(Healey, 2014). In case of qualitative or mixedmethod studies, team members need to have sufficient knowledge and experience to evaluate the
deeper quality issues in both types of methods.
Reporting
Reporting findings from SLRs should not be a
simple additive task, where findings from various
studies are reported as a summative narrative.
One of the strengths of SLRs is in the interpretive
analysis of findings from multiple studies. This
requires in-depth domain knowledge and expertise of the investigating team.
A critical issue in reporting is investigator bias.
As mentioned previously, any hint of investigator
bias could lead to an erosion of trust and credibility
of SLR findings. Since every individual has some
degree of bias, it is necessary to take precautionary
measures to eliminate any. Having a team (instead
of one investigator) is one of the strategies to eliminate or reduce bias in the literature search and
selection phases. The ideal solution is to include
team members who have little bias on a topical area.
In the reporting phase, if one’s strong bias is known,
it may be advisable to exclude that individual from
the interpretation and reporting phases, or subject
the interpretation and reporting to scrutiny by multiple team members or external experts.
Conclusion
With a growing body of evidence pertaining to the
influence of physical design on clinical, behavioral, operational, and business processes outcomes, there will be a growing demand for SLRs
on healthcare design topics. Key factors supporting SLRs are clarity, validity, and auditability
(Booth et al., 2016). It is important to note that
although bias is significantly reduced in SLRs, its
existence is not eradicated. Also, despite their merits, SLRs are the most demanding types of literature reviews to conduct and involve a structured
process for conducting and reporting. Although the
process is significantly more involved than a traditional literature review, there is a great deal of
support available through organizations like the
Cochrane Collaborative and the PRISMA statement. As with any literature review, SLRs are
undoubtedly strengthened through the additional
support of a research librarian with expertise in
your field of study. Overall, adhering to the
requirements described in this article, and/or other
similar SLR guides available in medical literature,
will ensure a credible, high-quality SLR and contribute vital information to the ever-growing
healthcare design knowledge base.
Implications for Practice
SLRs provide more comprehensive and holistic
evidences than single research studies. Use SLRs
to inform decision making when available.
Use SLRs published in peer-reviewed journals
only. SLRs not subjected to peer-review may
or may not have investigators’ biases.
Always check that your definition/understanding of terms used in practice are identical to
those used in the SLR. Frequently, differences
in the way terms are defined lead to erroneous
conclusions, with potentially undesirable
downstream outcomes.
Pati and Lorusso
15
Appendix
Table A1. Search Strategy Template.
Research Question(s)
Databases, Resources, and Limits
Core Databases
þReference List Searching
þHand Searching Journals
Table B1. Primary Literature Searching Template.
Population Intervention Outcome
Concept:
Concept
Concept
Mesh/
subheadings
Text words
Note. Database: (Do this for each of your databases and gray
literature).
PubMed Search Strategies
POPULATION (Example below, do the same
for intervention/outcome as well)
(“Dementia/drug therapy”[Mesh] OR “Dementia/
rehabilitation”[Mesh]
INTERVENTION
OUTCOME
Yield:
Duplicates Removed:
Total Added
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of
interest with respect to the research, authorship,
and/or publication of this article.
Funding
The author(s) received no financial support for
the research, authorship, and/or publication of
this article.
References
Balshem, H., Helfand, M., Schunemann, H., Oxman, A.,
Kunz, R., Brozek, J. . . . Guyatt, G. (2011). GRADE
Gray Literature
-Clinical Trial Registries
-National Clearinghouse
-Google Scholar
-Dissertation Sources
-Conference Abstracts
Limits
1. Language:
2. Years:
3. Publication Types:
4. Species:
guidelines: 3. Rating the quality of evidence. Journal
of Clinical Epidemiology, 64, 401–406.
Booth, A., Sutton, A., & Papaioannou, D. (2016). Systematic approaches to a successful literature review
(2nd ed.). Los Angeles, CA: Sage.
Centre for Evidence-Based Medicine. (2014, June 10).
Asking focused questions. Retrieved February 22,
2016, from http://www.cebm.net/asking-focusedquestions/
Centre for Reviews and Dissemination. (Ed.). (2009).
CRD’s guidance for undertaking reviews in healthcare
(3rd ed.). York, England: York Publishing Services.
Cochrane. (n.d.). Cochrane Collaboration. Retrieved
October 10, 2017, from www.cochrane.org/about-us
Dunn, P. M. (1997). James Lind (1716-94) of Edinburgh and the treatment of scurvy. Archives of Disease in Childhood—Fetal and Neonatal Edition, 76,
F64–F65. Retrieved from https://doi.org/10.1136/
fn.76.1.F64
Glass, G. V. (1976). Primary, secondary, and metaanalysis of research. Educational Researcher, 5,
3–8. Retrieved from https://doi.org/10.3102/
0013189X005010003
Grant, M., & Booth, A. (2009). A typology of reviews:
An analysis of 14 review types and associated methodologies. Health Information and Libraries Journal, 26, 91–108.
Greenhalgh, T. (2014). How to read a paper: The
basics of evidence-based medicine (5th ed.). Chichester, England: John Wiley.
Hamilton, D. K. (2011). What constitutes best practice
in healthcare design? Health Environments
Research & Design Journal, 4, 121–126. Retrieved
from https://doi.org/10.1177/193758671100400210
Healey, J. (2014). Statistics: A tool for social research
(10th ed.). Belmont, CA: Wadsworth.
16
Lorusso, L., & Bosch, S. (2016). Impact of multisensory environments on behavior for people with
dementia: A systematic literature review. The Gerontologist. Retrieved from https://doi.org/10.1093/
geront/gnw168
Marquardt, G., Bueter, K., & Motzek, T. (2014).
Impact of the design of the built environment on
people with dementia: An evidence-based review.
Health Environments Research & Design Journal,
8, 127–157.
Moher, D. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA
statement. Annals of Internal Medicine, 151, 264.
Retrieved from https://doi.org/10.7326/0003-4819151-4-200908180-00135
Pati, D. (2011). A framework for evaluating evidence
in evidence-based design. Health Environments
Research & Design Journal, 4, 50–71.
PRISMA. (n.d.). PRISMA transparent reporting of systematic reviews and meta-analyses. Retrieved
October 10, 2017, from www.prisma-statement.org
Stichler, J. (2010). Weighing the evidence. Health
Environments Research & Design Journal, 3, 3–7.
Ulrich, R., Zimring, C., Quan, X., Joseph, A., &
Choudhary, R. (2004). The role of the physical environment in the hospital of the 21st century: A oncein-a-lifetime opportunity. Concord, CA: The Center
for Health Design.
Health Environments Research & Design Journal XX(X)
Author Biographies
Debajyoti Pati is currently a professor of environmental design and Rockwell Endowment Chair at Texas
Tech University. He has over 28 years of experience
in research, practice, and teaching in the United States,
Canada, and India. He was twice voted among the 25
most influential people in healthcare design and has
received over ten awards for research excellence. His
areas of interest include health, healthcare, healthy
environments, sustainability, public architecture, judicial facilities, performance-based design, organizational learning, and developing indicators of facility
performance. He is on the editorial board of HERD and
on the board of directors of the NIBS-AIA BRIK
program.
Lesa Lorusso is a licensed interior designer, with over
15 years of experience in both academic and professional practice. Her educational background includes a
BS in interior design, MBA and MS in architecture.
She practices in Florida and is currently a doctoral
candidate and research assistant within the College of
Design, Construction and Planning at the University of
Florida and researcher with the Geriatric Research
Education and Clinical Center (GRECC) at the
Malcolm Randall Veteran’s Administration Medical
Center. The focus of her doctoral research is a collaboration between UF and the VA and investigates the
impact of multisensory environments on behavior for
Veterans with dementia.