Modeling biological-composition time series in integrated stock assessments:
data weighting considerations and impact on estimates of stock status
Fishery 1
Fishery 2
P. R. Crone
Southwest Fisheries Science Center (NOAA)
Center for the Advancement of Population Assessment Methodology (CAPAM)
8901 La Jolla Shores Dr., La Jolla, CA 92037, USA
Presentation outline
• Study description
• Results
• Conclusions
• Further work
Study description
• Motivation and expectations
o Better
understanding of impact that data weighting considerations
in typical assessments have on baseline management statistics …
contribute to good practices for stock assessment development
is based on a limited pool of assessments …
is able to provide quantitative results for particular statistical
comparisons, is not a substitute for simulation-based tests
o Meta-analysis
Study description
General
•Assessment archive
o Pool
of recently conducted fish stock (species) assessments used for management
o Assessments for small pelagic (3), large pelagic (7), and groundfish (19) species
o Assessments based on the Stock Synthesis model
o Majority of assessments conducted in 2015, some 2011-14
•Biological-composition time series
(‘marginal’, e.g., no./pct. by length bin and time step)
o Age (marginal)
o Conditional age-at-length (‘random at length’, age-length key format)
o Size (marginal, e.g., weight, biological compositions based on different bin structure)
o Weight (unfitted empirical weight-at-age data)
o Various ways of using/combining biological-composition time series in assessments
o Length
Study description
General (continued)
•Data weighting of biological compositions ‘outside’ the model
o Initial
(input) sample sizes for biological compositions are assessment/analyst-specific
o Sometimes
o More
based on actual number of fish (e.g., sport fishery compositions, CAAL)
often based on number of boat trips, hauls, sets, wells, sample adjustment formula, etc.
o Can
be based generally on variance estimates determined from sample/survey programs
o Can
be based generally on variance estimates from simulation analysis (e.g., bootstrap methods)
o Often
caps (thresholds) are used for input sample sizes (e.g., 100-200)
o Input
sample size determination was not addressed in this evaluation
Study description
General (continued)
•Data weighting of biological compositions ‘inside’ the model
of biological-composition time series is based initially on input sample size … subsequently,
adjusted internally based on comparing observed and expected values from fits to the time series
o Variability
data weighting approaches for composition time series in integrated assessment models …
McCallister and Ianelli (1997) and Francis (2011) methods often considered in practice
o Various
o ‘Effective’ sample
size in Stock Synthesis model (McCallister and Ianelli methods) reflects number of random
samples (drawn from multinomial distribution) needed to produce fit as precise as model’s predicted fit
o Actual
weighting values (scalars) for composition data reflect various mean estimates calculated from ratios of
effective to input sample sizes (multiplicative based)
o Francis
method basis is variation of mean length/age of the composition time series, accounts for correlation
among length or age groups, results in greater variation surrounding composition time series
o In
practice, ad hoc caps (thresholds) are implemented for estimated scalars >1
o Internally
implemented data weighting methods for composition time series were addressed in this evaluation
Study description
Assessment models
Data weighting methods
• Baseline (Final)
o Assessment
model for advising management
• Unweighted (UW)
o Final
model that includes no (internally) weighted composition time series
o All scalars (‘weighting values, variance adjustments, lambdas’) = 1
• McCallister-Ianelli (AM)
o Scalar
estimate reflects arithmetic mean from model fits to composition time series (based on
ratios of effective sample size to input sample sizes)
• McCallister-Ianelli (HM)
o Scalar
estimate reflects harmonic mean from model fits to composition time series (based on
ratios of effective sample size to input sample sizes)
• Francis (F0)
o Assessments
that included only length- and/or age-composition time series and no CAAL time
series (based on FA)
• Francis-Method A (FA)
o Assessments
that included CAAL time series along with length and/or age-composition time
series (mean estimates indexed by year)
• Francis-Method B (FB)
o Assessments
that included CAAL time series along with length and/or age-composition time
series (mean estimates indexed by year/length bin)
Study description
General (continued)
•Model development/estimation
o For
each species, final assessment model re-configured according to recommended scalars from
respective data weighting method (cap=100 and single iteration)
o For
a species, from 3-5 alternative models were developed for overall study, depending on the
biological compositions, SS version, convergence issues
o Data
weighting addressed only biological compositions included in the model, i.e., no weighting
applicable to other input data (e.g., index of abundance time series) or parameter assumptions
(e.g., σR of stock-recruit relationship)
o Data
weighting methods described in McCallister and Ianelli (1997), Francis (2011), Methot and
Wetzel (2013), Punt (in press)
•Output
o Management
quantities of interest: MSY, FMSY, Bcurrent, Depletion (SSBcurrent / SSB0)
o Comparisons based on means/CVs and medians/REs
Data weighting methods – Example (SS effective sample size)
Fishery_Survey Year Input N (sample size) Effective N (sample size) Effective N / Input N
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Means
Arithmetic
Harmonic
1983
1984
1985
1986
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
106.72
91.64
104.24
120.00
165.16
179.08
143.32
84.56
66.20
79.76
107.52
124.56
108.24
87.56
108.56
90.20
66.64
76.40
84.44
85.80
62.80
101.16
91.96
95.72
64.36
28.92
16.88
19.88
30.84
47.80
71.72
16.00
10.51
16.97
5.64
15.79
15.56
57.87
3.31
17.06
10.33
49.40
38.04
26.31
9.51
110.99
30.41
50.16
49.55
10.37
3.82
2.84
21.12
3.59
14.13
18.89
29.81
21.16
3.43
20.18
8.61
6.68
8.41
22.21
0.10
0.19
0.05
0.13
0.09
0.32
0.02
0.20
0.16
0.62
0.35
0.21
0.09
1.27
0.28
0.56
0.74
0.14
0.05
0.03
0.34
0.04
0.15
0.20
0.46
0.73
0.20
1.01
0.28
0.14
0.12
1.39
85.27
22.27
9.95
0.33
0.12
Data weighting method (scalar calculation)
McCallister and Ianelli (AM)
Mean (Input N) / Mean (Effective N) = 22.27 / 85.27 = 0.26
Mean (Input N / Effective N) = 0.33
McCallister and Ianelli (HM)
Mean (Input N) / Mean (Effective N) = 9.95 / 85.27 = 0.12
Mean (Input N / Effective N) = 0.12
Mean length (cm)
Data weighting methods – Example (FA/FB/F0 diagnostic plot)
5
Year
Study description
Analysis flow chart
Species
Assessment model
(Final)
Assessment model
(Data weighting method)
Unweighted (All scalars=1)
UW
McCallister/Ianelli (Arithmetic mean)
AM
P. sardine
.
.
.
.
.
.
.
N = 29 species
Baseline
McCallister/Ianelli (Harmonic mean)
Final
HM
Francis (No CAAL)
F0
Francis (CAAL, Method A)
FA
Francis (CAAL, Method B)
FB
Output
(Management quantities)
MSY, FMSY, Bcurrent, DEP
MSY, FMSY, Bcurrent, DEP
MSY, FMSY, Bcurrent, DEP
MSY, FMSY, Bcurrent, DEP
MSY, FMSY, Bcurrent, DEP
MSY, FMSY, Bcurrent, DEP
Results
Data Weighting Methods
Scalar Ranges by Biological Data Type
Data weighting method
Length
Low High
McCallister-Ianelli (AM) 0.03 53.91
McCallister-Ianelli (HM) 0.01 11.38
Francis (F0)
0.004 10.13
FA
FB
Biological data
Age
CAAL
Low High Low
High
0.21 8.35
0.12 5.32
0.04 >1000
0.65 >1000
0.03 >1000
0.02 >1000
0.13 >1000
Assessment (species) examples
Species
Data weighting method
Composition time series
Management quantity
Length
Age
CAAL MSY (mt) Fmsy Bcurrent (mt) Depletion
P. sardine (S3)
P. sardine (S3)
P. sardine (S3)
P. sardine (S3)
P. sardine (S3)
P. sardine (S3)
P. sardine (S3)
Final (F)
Weight_None (UW)
Weight_McCallister-Ianelli (AM)
Weight_McCallister-Ianelli (HM)
Weight_Francis (F0)
Weight_Francis (FA)
Weight_Francis (FB)
X
N.P. albacore (L1)
N.P. albacore (L1)
N.P. albacore (L1)
N.P. albacore (L1)
N.P. albacore (L1)
N.P. albacore (L1)
N.P. albacore (L1)
Final (F)
Weight_None (UW)
Weight_McCallister-Ianelli (AM)
Weight_McCallister-Ianelli (HM)
Weight_Francis (F0)
Weight_Francis (FA)
Weight_Francis (FB)
X
X
Mean and CV
63,212
77,812
81,534
89,337
na
116,720
112,021
0.29
0.28
0.27
0.27
na
0.25
0.24
151,968
156,987
134,320
168,972
na
240,463
213,398
0.23
0.19
0.15
0.18
na
0.21
0.20
Mean and CV
105,278
134,188
110,451
104,707
105,401
na
na
0.23
0.23
0.23
0.23
0.23
na
na
669,405
734,324
702,207
650,254
666,337
na
na
0.36
0.32
0.35
0.34
0.34
na
na
CV (%)
100 +
Data Weighting Methods
‘Within Assessment’ Variability
MSY
Assmt. CV (MSY)
75
50
25
0
Assessment (species)
CV (%)
Data Weighting Methods
‘Within Assessment’ Variability
75
FMSY
50
25
0
Assessment (species)
Assmt. CV (Fmsy)
CV (%)
Data Weighting Methods
‘Within Assessment’ Variability
100 +
Bcurrent
Assmt. CV (Bcurrent)
75
50
25
0
Assessment (species)
CV (%)
Data Weighting Methods
‘Within Assessment’ Variability
80
Depletion
Assmt. CV (Depletion)
60
40
20
0
Assessment (species)
Data Weighting Methods
‘Between Management Quantity’ Variability
CV (%)
55
50
Mgt. qty. CV
45
40
35
30
25
20
15
10
5
0
MSY
FMSY
Bcurrent
Management quantity
Depletion
‘Within Assessment’ Variability (Relative to Data Weighting Method)
Relative error
MSY
Data weighting method
Species (no. of assessments)
Models (no. of replicates)
Sample size limit implemented (no. of species)
Convergence issues (no. of species)
Unplotted models (pct. extreme positive outliers)
29
126
0
0
4%
27
119
2
2
2%
29
126
0
0
2%
11
43
0
0
2%
15
74
1
0
4%
15
74
1
0
1%
‘Within Assessment’ Variability (Relative to Data Weighting Method)
Relative error
FMSY
Data weighting method
Species (no. of assessments)
Models (no. of replicates)
Sample size limit implemented (no. of species)
Convergence issues (no. of species)
Unplotted models (pct. extreme positive outliers)
29
126
0
0
1%
27
119
2
2
0%
29
126
0
0
2%
11
43
0
0
7%
15
74
1
0
0%
15
74
1
0
0%
‘Within Assessment’ Variability (Relative to Data Weighting Method)
Relative error
Bcurrent
Data weighting method
Species (no. of assessments)
29
Models (no. of replicates)
126
Sample size limit implemented (no. of species)
0
Convergence issues (no. of species)
0
Unplotted models (pct. extreme positive outliers) 6%
27
119
2
2
11%
29
126
0
0
4%
11
43
0
0
12%
15
74
1
0
9%
15
74
1
0
5%
‘Within Assessment’ Variability (Relative to Data Weighting Method)
Relative error
Depletion
Data weighting method
Species (no. of assessments)
Models (no. of replicates)
Sample size limit implemented (no. of species)
Convergence issues (no. of species)
Unplotted models (pct. extreme positive outliers)
29
126
0
0
5%
27
119
2
2
10%
29
126
0
0
2%
11
43
0
0
12%
15
74
1
0
1%
15
74
1
0
3%
Data Weighting Methods
Relative to HM (‘correctly specified’ model)
Relative error
MSY
Data weighting method
Species (no. of assessments)
Models (no. of replicates)
Sample size limit implemented (no. of species)
Convergence issues (no. of species)
Unplotted models (pct. extreme positive outliers)
29
29
0
0
0%
27
27
2
2
0%
11
11
0
0
0%
15
15
1
0
0%
15
15
1
0
13%
Data Weighting Methods
Relative to HM (‘correctly specified’ model)
Relative error
FMSY
Data weighting method
Species (no. of assessments)
Models (no. of replicates)
Sample size limit implemented (no. of species)
Convergence issues (no. of species)
Unplotted models (pct. extreme positive outliers)
29
29
0
0
3%
27
27
2
2
0%
11
11
0
0
0%
15
15
1
0
0%
15
15
1
0
0%
Data Weighting Methods
Relative to HM (‘correctly specified’ model)
Relative error
Bcurrent
Data weighting method
Species (no. of assessments)
Models (no. of replicates)
Sample size limit implemented (no. of species)
Convergence issues (no. of species)
Unplotted models (pct. extreme positive outliers)
29
29
0
0
3%
27
27
2
2
0%
11
11
0
0
0%
15
15
1
0
0%
15
15
1
0
13%
Data Weighting Methods
Relative to HM (‘correctly specified’ model)
Relative error
Depletion
Data weighting method
Species (no. of assessments)
Models (no. of replicates)
Sample size limit implemented (no. of species)
Convergence issues (no. of species)
Unplotted models (pct. extreme positive outliers)
29
29
0
0
0%
27
27
2
2
0%
11
11
0
0
0%
15
15
1
0
0%
15
15
1
0
7%
Conclusions
• Data weighting methods impact on management quantities
o
Terminal biomass estimates most uncertain in most cases (mean CV=35%), depletion
and MSY less so (20%), and FMSY most precise (<10%)
o
Positively-skewed, median-unbiased relative error distributions
o
The harmonic mean-based McCallister-Ianelli method (HM) resulted in precise and
unbiased estimates in most cases, but …
o
Unweighted method (UW) also relatively precise and robust in many comparisons
o
Frances methods (F0, FA, FB) produced generally unbiased estimates, but typically
less precise than HM; more similar for MSY-related quantities
o
FA less bias (equally precise) than FB in many comparisons
o
For correctly-specified assessment based on HM, better off not weighting (UW) than
implementing an alternative data weighting method
Study benefits and further work
• Replicates (assessments) in meta-analysis are realistic
o
o
Replicates associated with typical simulations are unrealistic, i.e., much too
similar to one another … increase number/variety of assessments
However, study (experimental) population based on real assessments provides
limited cause-and-effect information, given the many data/parameter
inconsistencies across replicates
• Meta-analysis provides baseline information for more focused
simulation studies
o
o
Contrast between quality of derived management metrics
Fold into MSEs addressing small pelagic species’ fisheries on the USA Pacific
coast for basing (much needed) new and improved harvest control rules
• Information useful for analysts charged with developing
ongoing assessments for management purposes
o
Data weighting approaches in actual assessments are evolving presently,
research needed to inform good practices
References
Crone, P.R., D.B. Sampson. 1998. Evaluation of assumed error structure in stock assessment models that use
sample estimates of age composition. Pages 355-370 in Fishery Stock Assessment Models. Alaska Sea
Grant College Program Report No. AK-SG-98-01, University of Alaska, Fairbanks, Alaska.
Fournier, D., C.P. Archibald. 1982. A general theory for analyzing catch at age data. Can. J. Fish. Aquat. Sci.
39:1195-1207.
Francis, R.I.C.C. 2011. Data weighting in statistical fisheries stock assessment models. Can. J. Fish. Aquat. Sci.
68:1124-1138.
McAllister, M.K., J.N. Ianelli. 1997. Bayesian stock assessment using catch-age data and the samplingimportance resampling algorithm. Can. J. Fish. Aquat. Sci. 54(2): 284–300.
Methot, R.D., C.R. Wetzel. 2013. Stock Synthesis: a biological and statistical frame-work for fish stock
assessment and fishery management. Fish. Res. 142:86–99.
Pennington, M., L.-M. Burmeister, V. Hjellvik. 2002. Assessing the precision of frequency distributions
estimated from trawl survey samples. Fish Bull. 100:74–80.
Punt, A.E. in press. Some insights into data weighting in integrated stock assessments. Fish. Res.
Stewart, I.J., O.S. Hamel. 2014. Boostrapping of sample sizes for length- or age-composition data used in stock
assessments. Can. J. Fish. Aquat. Sci. 671:581-588.