Systematic Review: Approach
to using Science in Riparian
Rule Analysis
Systematic Review Conference
Corvallis, Oregon
May, 2013
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Outline
• Background: review purpose
• ODF & SR: ODF history & use of Systematic
Review (SR)
• SR Protocol: How ODF conducts this review
• Lessons: Results & improving the process
• Wrap up: Next steps
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Purpose of science review
Review needs to address:
1. ORS 527.714 (5)(c) “The proposed rule reflects
available scientific information, the results of
relevant monitoring and, as appropriate, adequate
field evaluation at representative locations in
Oregon;”
2. Rule analysis objective:
“Establish riparian protection measures for small and
medium fish-bearing streams that maintain and
promote shade conditions that ensure, to the
maximum extent practicable, the achievement of the
Protecting Cold Water criterion.”
3. 16 rule alternatives
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Purpose of science review
Review needs to address:
How do scientific studies inform consideration
of alternatives with respect to the rule analysis
objective?
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Previous ODF experience with
Systematic Review (SR)
• 2004 former Gov. Kitzhaber presented to
Board
• 2005 INR “SR Opportunities & Challenges”
report for ODF
• 2008 Pilot SR + report on SR process
• Recent conversations with ‘08 participants
Background
ODF & SR
SR Protocol
Lessons
Wrap up
ODF use of Systematic Review
• Develop SR process (protocol)
– ODF developed draft protocol
– Stakeholders, RipStream External Review Team
provided input on question and protocol
– Librarian helped refine structured literature search
• Conduct SR
– ODF conducts lit. search and inclusion/exclusion
– External reviewers
– extract & synthesize information
– write report with feedback from ODF, tech. experts,
stakeholders
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Step 1. Review Question
• Answerable in scientific terms
• Generated in collaboration with relevant
decision-makers, stakeholders, tech. experts
• Structural elements: generate literature search
terms, determine inclusion criteria
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Step 1. Review Question
Structural
elements
Population
Element details for this review
Comparator
Small and medium streams in forests in western
PNW
Forest management with riparian protection
measures
Lack of forest management
Outcome
Change in stream temperature or riparian shade
Intervention
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Step 1. Review Question:
For small and medium streams in the western Pacific
Northwest, in or adjacent to forest harvest operations,
what are the effects of near-stream forest management
on stream temperature and/or shade?
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Step 2. Literature search
• Extensive, rigorous, pre-defined
• Search in:
–
–
–
–
–
Databases
Agency websites
Libraries
Discipline experts
References, citations
• Search for:
– Government reports
– Peer-reviewed and in review
– Graduate theses
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Step 3. Study inclusion criteria
• Directly related to SR question
• Pre-defined, rigorous
• Consistently applied
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Step 3. Study inclusion criteria
– Measure stream temperature, riparian shade
– Small & medium streams in recently-harvested
forests
– Riparian area managed to protect cold water or
retain shade
– Controls exist (pre-treatment, reference sites)
– Peer-reviewed, government reports, in review,
theses
– Geography: Forest ecosystems similar to W.
Oregon
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Step 4. Data Extraction
Note: Steps 4-6 External reviewers complete
• Info and analysis of each study placed in tables
• Mostly, methods & results (Evidence)
• Enables rigorous, transparent analysis of
evidence
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Step 5. Study quality & relevance
• Consistent metrics of study quality and
relevance to review question
• Relevance, effectiveness re: rule alternatives
• In tables for ease
of comparison
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Step 6. Data synthesis
For each alternative
– # relevant studies
– Degree of effectiveness of intervention(s) (e.g.,
buffers)
– Range of variation (buffer metrics, outcomes,
effectiveness)
– Gaps in understanding
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Lessons Learned
Stakeholder input
• Structured & throughout process
– Protocol & question
– Which studies and reviewers to consider
– List of included & excluded studies
• Miss any relevant studies?
• Misapply inclusion criteria?
– Input on draft report
– Comment on final report at presentation to Board
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Lessons Learned
Finding qualified reviewers
• Qualifications
– PhD
– >5 years experience: physical aspects of Oregon
streams
– Peer-reviewed publication record
• Academics hard to interest (“only” $10k,
abnormal peer-review situation)
->Consultants
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Lessons Learned
Search & inclusion
• Search found ~1450 articles
• 27 studies passed all inclusion filters
• Inclusion/exclusion fate documented in
spreadsheet
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Lessons Learned
Improve search & inclusion process
• ~ 1-1.5 week search, 3-4 weeks assess
inclusion
– ~1250 found via databases = big time sink
• Tested: Found all included studies without
using databases (citation & reference searches)
– 4 studies (in review/never published) found only
via contacting researchers
– 5 studies found only via agency web search
(Alaska DEQ, WA DNR, ODF)
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Lessons Learned
Study Assessment
• Calibration (4 same articles) + discussion w/
reviewers
• Consistent, rigorous assessment of study
statistical analysis = challenging
• Rule alternatives: relevance, effectiveness –
not typically part of Science
• Reviewers currently reviewing studies
– > protocol modifications documented
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Next Steps
• External reviewers draft SR report – June, 2013
• Feedback on draft SR report from ODF,
stakeholders, technical experts – late June, 2013
• External reviewers address feedback in final SR
report - July, 2013
• Present final report of Systematic Review to
Board - November, 2013
– Use report-> develop rule prescriptions (March ‘13)
Background
ODF & SR
SR Protocol
Lessons
Wrap up
Next Steps
• Present at AWRA, SAF national conferences
• Submit for peer-review
Thank you
http://www.oregon.gov/odf/BOARD/docs/2013_
March/BOFATTCH_20130306_09_01.pdf
TFrueh@ODF.state.or.us
Rule alternatives
Category
Alternative
Current FPA: Maintain current standard target prescriptions (no action)
State Forests Standards: State Forest Management Plan riparian protection standards. Three
(3) zone RMAs with varying retention standards established for multiple purposes including
but not limited to maintaining shade.
Derived variable retention: Use collected data or literature to explore a variety of stand metrics
versus shade outcomes (including but not limited to RipStream data).
Scientific Principle(s)
Include no-action as comparison
Shade known key driver of
stream temperature increases.
Utilize RipStream and any other
relevant monitoring or research
Variable
data that links riparian stand
retention
conditions to shade.
buffer
Large tree variable retention – Emphasizes retention of largest trees close to the stream in
Increased riparian basal area
addition to a no-cut and variable retention buffer.
linked to higher shade levels.
Minimize gaps: Maintain existing standard targets but modify current spatial retention
Strategic RMA tree retention to
standards (basal area/1000 ft) to minimize shade gaps
minimize shade gaps.
Basal-area retention by aspect: Increase basal area density retention targets on south-sided Aspect known to influence
RMAs only
stream temperature and shade.
Field-based shade standard: Establish process to implement field-measured shade standard Shade known key driver of
rather than managing for RMA stand metrics
stream temperature increases.
Shade approach from Washington DNR method: Three (3) zone RMAs based on a
Shade known key driver of
combination of site index, shade and basal area retention standards with linkages to applicable stream temperature increases.
temperature standards. Uses channel migration zones (CMZs) versus high water level in
Shade
standard current Oregon FPA.
Shrub shade: Develop alternative to account for shade contribution from shrubs or other non- Sites with significant shrubs or
woody plants.
other non-woody plants may
allow greater tree removal while
maintaining shade.
Hardwood sites: Consider alternatives to account for conifer vs. hardwood-dominated
Hardwoods provide shade but
riparian stands.
conifer retention emphasis in
FPA, does not account for sites
Hardwood
poorly suited for conifers.
contributio
Hardwood shade: Include hardwoods in basal area standard target.
Hardwoods provide shade but
n
Study
Inclusion criteria
Geographic extent6
Study type5
Study design4
Intervention3
Setting2
Outcome1
•
•
•
•
•
•
1 Reported
(via at least one figure or table) primary measurements of stream
temperature, riparian shade (or a proxy thereof), or insolation.
2 Small and medium streams (i.e., with contributing areas or average annual
flow less than 1.5 times the upper limit of medium stream defined in
[Oregon Department of Forestry, 1994] ( 11250 ac.(45 km2) and 15 cfs,
respectively), or wetted width or bankfull width less than 1.1 times the
maximum wetted width or bankfull width from [Groom et al., 2011b] (4.0 m
and 8.7 m, respectively) in mountainous terrain with forests harvested less
than five years before data collection of a study .
3 Near-stream area managed for protection of cold water and/or riparian
shade. Management prescription is clearly quantified (e.g., buffer width,
basal area retention).
4 Controls exist (either pre-treatment data, control sites, or reference sites)
5 Peer-reviewed papers, government reports, manuscripts in review, and
graduate theses, all of which must be primary studies that describe methods
and contain primary data.
6 A portion of the study must have been conducted in any of the following
locations: parts of Oregon, Washington, and British Columbia west of the
crest of the Cascades, the Siskyous of northern California, northwest British
Columbia, southeast Alaska, or the coastal range of northern California
Publication title and principal investigator(s)
Study dates and study duration (# of years, dates within a
year)
Study location (watersheds, region/state, country), settings
where riparian buffers were applied
Ecosystem type; plant association group; type of forest
Stream size (avg. annual flow, contributing area, HUC, avg.
wetted width, etc.)
Research question(s), hypotheses, objectives
Study design1
Pretreatment data (yes/no), # of years of pretreatment data
Details on management action(s) (e.g., sizes and types of
buffers; clearcut or thin on both or single sides of streams)
Replications (if applicable)
Nature of the outcome measures used, their relative
importance and robustness2
Sample sizes and results with estimates of variation3
Notes concerning study quality with evidence or reasoning
behind the notes4
Potential sources of bias or error
Effects modifiers5
Notes6
Method references7
•
•
•
•
•
•
•
1 Brief
description of study design, e.g., BACI, # of sites, types of controls
(pre-treatment, reference, upstream), site layout.
2 Examples of outcome measures: stream temperature, basal area, riparian
shade; relative importance refers to instances where a proxy is used (e.g.,
canopy cover for shade at stream surface), how representative is the proxy?
Robustness refers to how well outcomes were measured (e.g., accuracy of
measurements, frequency, sound method for measuring)
3For sample size, list with respect to particular results (e.g., “warming of X
degrees (n=4)”); Llist specific results that are most pertinent to answering
the question and help inform the rule objective, referencing a figure or table
where appropriate; include confidence limits or standard deviations.
4 Address study quality questions such as: Did authors adequately address
fundamental processes? How well did they conduct their statistical
analyses? Were biases addressed?
5 Discuss how well, and which, effects modifiers were considered (see list
of effects modifiers in Section 3.3)
6 Notes allows for additional insight reviewer may provide on study quality
7refers to references that are essential to understanding methods of an
article.
Relevance
Study
Quality
Question/Objective10
Stream size9
Mountains8
Geography7
Statistically robust6
Nsample5
Ncontrol4
Nreplicate3
Study design2
Pre-treatment1 (yrs)
Duration (years)
1Data
collected before treatment with the number of years of pre-treatment data in parentheses
(X=yes, blank=no)
2H=high=Replicated sampling, replicated controls, sampling before and after treatment;
M=medium=unreplicated, controlled, sampling before and after treatment; L=low=unreplicated,
uncontrolled, sampling before and after treatment or unreplicated, controlled, sampling after
treatment (modified from [Fazey and Salisbury, 2002]).
3Number of treatment replicates; add succinct description (e.g., “9 sites, 3 yrs. Post-treatment,
treatment X”), knowing that greater detail is captured in Table A.6.2.
4Number of control replicates;
5Number of samples
6 Were statistical analyses chosen appropriate for data? Did authors adequately explore the data
to answer the questions? (note: not concerned with study design). H = yes to both question, M =
yes to one question; L= no to both questions.
7H=high= west of crest of Cascades in OR, WA, BC plus the Siskyous (i.e., sites most similar to
those in western Oregon); L=low=Coast Range of N. CA, Vancouver Island, NW BC, SE Alaska
(i.e., sites somewhat similar to those in western Oregon).
8In mountainous terrain (X=yes, blank=no)
9H=high=small or medium streams as defined in either of [Groom et al., 2011b; Oregon
Department of Forestry, 1994] (i.e., with contributing areas < 7500 ac.(30 km2), or average annual
flow < 10 cfs, or wetted width <3.7, or bankfull width < 7.9 m); L= low = “near” medium size
stream (i.e., contributing areas 7500 - 11250 ac. (30-45 km2), or 10 - 15 cfs average annual flow,
or 3.7 - 4.0 m wetted width, or 7.9 - 8.7 m bankfull width).
10H=high=study objectives or questions directly relate to review question; L=low= study has
relatevant data even though study objectives orquestions are not directly related to review
question. This metric is not applicable to shade, and we will continue to cogitate on how to
objectively and consistently state whether a study is statistically robust.
One-sided buffer
Plan for alternative
2
practice
Oregon Plan
No-cut aspect buffers
Derived no-cut buffer
Hardwood shade
Hardwood sites
Shrub shade
Shade approach from
WA DNR method
Field-based
shade
standard
Basal
area retention by
aspect
Minimize
gaps
Effectiveness3
Large tree variable
retentionvariable
Derived
retention
State
Forests
1
Standards
Current
FPA1
Study
Relevance
1
Standards are summarized in [Groom et al., 2011b].
2 Any other type of treatment that may have been studied.
3 Effective at preventing stream from warming or
maintaining shade: ++= prevented warming or maintained
shade; += reduced warming or had a smaller decrease in
shade relative to another treatment , 0= no change; I=
inconclusive; - =resulted in maximal warming or largest
decrease in shade compared with other treatments.
Background
ODF & SR
SR?
SR Protocol
Wrap up
1. Review question:
Secondary question:
-Addresses causes of differences between studies’
outcomes
For small and medium streams in the western Pacific Northwest,
in or adjacent to forest harvest operations, how do effects
modifiers (e.g., discharge, substrate characteristics, length of
buffers, stream aspect), in combination with near-stream forest
management, change stream temperatures or riparian shade?
Background
ODF & SR
SR?
SR Protocol
Wrap up
Step 3. Study inclusion criteria
Geography
• Exact geographic extent of rule analysis TBD
• Degradation finding based
on RipStream study (Coast
Range, Interior Geographic
Regions)
• Thus, include studies with
similar hydrology, climate,
vegetation as RipStream sites
Background
ODF & SR
SR?
SR Protocol
Wrap up
3. Study inclusion criteria
Geography: Relevant forest types and landscape
– OR, WA, BC west of crest of Cascades
– Siskiyous in S. OR & N. CA
– SE Alaska, Vancouver Is., NW BC
– Coast Range of northern CA
Background
ODF & SR
SR?
SR Protocol
Wrap up
What is systematic review?
Purpose
Participation
Systematic Review
Traditional Review
Focused question or hypothesis General or focused
Experts, stakeholders, authors
Authors only
Pub. search, inclusion
Structured, rigorous, extensive Authors’ choice
Quality assessment
Consistent analysis of methods, Authors’ choice
biases, additional factors
Basis for synthesis
Evidence, relevance, sound
methodologies
Evidence, interpretation,
maybe analysis of
methodology
Yes – record of all decisions
no
Transparent?
Background
ODF & SR
SR?
SR Protocol
Wrap up
What is systematic review?
• Pre-established protocol
1.
2.
3.
4.
5.
6.
Focused question
Structured search strategy
Inclusion/exclusion criteria
Study quality & relevance assessment
Data/meta-data extraction & comparison table(s)
Data synthesis strategy
• Include stakeholders, technical experts
• Relatively new technique for natural resources
sciences