Modeling biological-composition time series in integrated stock assessments: data weighting considerations and impact on estimates of stock status Fishery 1 Fishery 2 P. R. Crone Southwest Fisheries Science Center (NOAA) Center for the Advancement of Population Assessment Methodology (CAPAM) 8901 La Jolla Shores Dr., La Jolla, CA 92037, USA Presentation outline • Study description • Results • Conclusions • Further work Study description • Motivation and expectations o Better understanding of impact that data weighting considerations in typical assessments have on baseline management statistics … contribute to good practices for stock assessment development is based on a limited pool of assessments … is able to provide quantitative results for particular statistical comparisons, is not a substitute for simulation-based tests o Meta-analysis Study description General •Assessment archive o Pool of recently conducted fish stock (species) assessments used for management o Assessments for small pelagic (3), large pelagic (7), and groundfish (19) species o Assessments based on the Stock Synthesis model o Majority of assessments conducted in 2015, some 2011-14 •Biological-composition time series (‘marginal’, e.g., no./pct. by length bin and time step) o Age (marginal) o Conditional age-at-length (‘random at length’, age-length key format) o Size (marginal, e.g., weight, biological compositions based on different bin structure) o Weight (unfitted empirical weight-at-age data) o Various ways of using/combining biological-composition time series in assessments o Length Study description General (continued) •Data weighting of biological compositions ‘outside’ the model o Initial (input) sample sizes for biological compositions are assessment/analyst-specific o Sometimes o More based on actual number of fish (e.g., sport fishery compositions, CAAL) often based on number of boat trips, hauls, sets, wells, sample adjustment formula, etc. o Can be based generally on variance estimates determined from sample/survey programs o Can be based generally on variance estimates from simulation analysis (e.g., bootstrap methods) o Often caps (thresholds) are used for input sample sizes (e.g., 100-200) o Input sample size determination was not addressed in this evaluation Study description General (continued) •Data weighting of biological compositions ‘inside’ the model of biological-composition time series is based initially on input sample size … subsequently, adjusted internally based on comparing observed and expected values from fits to the time series o Variability data weighting approaches for composition time series in integrated assessment models … McCallister and Ianelli (1997) and Francis (2011) methods often considered in practice o Various o ‘Effective’ sample size in Stock Synthesis model (McCallister and Ianelli methods) reflects number of random samples (drawn from multinomial distribution) needed to produce fit as precise as model’s predicted fit o Actual weighting values (scalars) for composition data reflect various mean estimates calculated from ratios of effective to input sample sizes (multiplicative based) o Francis method basis is variation of mean length/age of the composition time series, accounts for correlation among length or age groups, results in greater variation surrounding composition time series o In practice, ad hoc caps (thresholds) are implemented for estimated scalars >1 o Internally implemented data weighting methods for composition time series were addressed in this evaluation Study description Assessment models Data weighting methods • Baseline (Final) o Assessment model for advising management • Unweighted (UW) o Final model that includes no (internally) weighted composition time series o All scalars (‘weighting values, variance adjustments, lambdas’) = 1 • McCallister-Ianelli (AM) o Scalar estimate reflects arithmetic mean from model fits to composition time series (based on ratios of effective sample size to input sample sizes) • McCallister-Ianelli (HM) o Scalar estimate reflects harmonic mean from model fits to composition time series (based on ratios of effective sample size to input sample sizes) • Francis (F0) o Assessments that included only length- and/or age-composition time series and no CAAL time series (based on FA) • Francis-Method A (FA) o Assessments that included CAAL time series along with length and/or age-composition time series (mean estimates indexed by year) • Francis-Method B (FB) o Assessments that included CAAL time series along with length and/or age-composition time series (mean estimates indexed by year/length bin) Study description General (continued) •Model development/estimation o For each species, final assessment model re-configured according to recommended scalars from respective data weighting method (cap=100 and single iteration) o For a species, from 3-5 alternative models were developed for overall study, depending on the biological compositions, SS version, convergence issues o Data weighting addressed only biological compositions included in the model, i.e., no weighting applicable to other input data (e.g., index of abundance time series) or parameter assumptions (e.g., σR of stock-recruit relationship) o Data weighting methods described in McCallister and Ianelli (1997), Francis (2011), Methot and Wetzel (2013), Punt (in press) •Output o Management quantities of interest: MSY, FMSY, Bcurrent, Depletion (SSBcurrent / SSB0) o Comparisons based on means/CVs and medians/REs Data weighting methods – Example (SS effective sample size) Fishery_Survey Year Input N (sample size) Effective N (sample size) Effective N / Input N 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Means Arithmetic Harmonic 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 106.72 91.64 104.24 120.00 165.16 179.08 143.32 84.56 66.20 79.76 107.52 124.56 108.24 87.56 108.56 90.20 66.64 76.40 84.44 85.80 62.80 101.16 91.96 95.72 64.36 28.92 16.88 19.88 30.84 47.80 71.72 16.00 10.51 16.97 5.64 15.79 15.56 57.87 3.31 17.06 10.33 49.40 38.04 26.31 9.51 110.99 30.41 50.16 49.55 10.37 3.82 2.84 21.12 3.59 14.13 18.89 29.81 21.16 3.43 20.18 8.61 6.68 8.41 22.21 0.10 0.19 0.05 0.13 0.09 0.32 0.02 0.20 0.16 0.62 0.35 0.21 0.09 1.27 0.28 0.56 0.74 0.14 0.05 0.03 0.34 0.04 0.15 0.20 0.46 0.73 0.20 1.01 0.28 0.14 0.12 1.39 85.27 22.27 9.95 0.33 0.12 Data weighting method (scalar calculation) McCallister and Ianelli (AM) Mean (Input N) / Mean (Effective N) = 22.27 / 85.27 = 0.26 Mean (Input N / Effective N) = 0.33 McCallister and Ianelli (HM) Mean (Input N) / Mean (Effective N) = 9.95 / 85.27 = 0.12 Mean (Input N / Effective N) = 0.12 Mean length (cm) Data weighting methods – Example (FA/FB/F0 diagnostic plot) 5 Year Study description Analysis flow chart Species Assessment model (Final) Assessment model (Data weighting method) Unweighted (All scalars=1) UW McCallister/Ianelli (Arithmetic mean) AM P. sardine . . . . . . . N = 29 species Baseline McCallister/Ianelli (Harmonic mean) Final HM Francis (No CAAL) F0 Francis (CAAL, Method A) FA Francis (CAAL, Method B) FB Output (Management quantities) MSY, FMSY, Bcurrent, DEP MSY, FMSY, Bcurrent, DEP MSY, FMSY, Bcurrent, DEP MSY, FMSY, Bcurrent, DEP MSY, FMSY, Bcurrent, DEP MSY, FMSY, Bcurrent, DEP Results Data Weighting Methods Scalar Ranges by Biological Data Type Data weighting method Length Low High McCallister-Ianelli (AM) 0.03 53.91 McCallister-Ianelli (HM) 0.01 11.38 Francis (F0) 0.004 10.13 FA FB Biological data Age CAAL Low High Low High 0.21 8.35 0.12 5.32 0.04 >1000 0.65 >1000 0.03 >1000 0.02 >1000 0.13 >1000 Assessment (species) examples Species Data weighting method Composition time series Management quantity Length Age CAAL MSY (mt) Fmsy Bcurrent (mt) Depletion P. sardine (S3) P. sardine (S3) P. sardine (S3) P. sardine (S3) P. sardine (S3) P. sardine (S3) P. sardine (S3) Final (F) Weight_None (UW) Weight_McCallister-Ianelli (AM) Weight_McCallister-Ianelli (HM) Weight_Francis (F0) Weight_Francis (FA) Weight_Francis (FB) X N.P. albacore (L1) N.P. albacore (L1) N.P. albacore (L1) N.P. albacore (L1) N.P. albacore (L1) N.P. albacore (L1) N.P. albacore (L1) Final (F) Weight_None (UW) Weight_McCallister-Ianelli (AM) Weight_McCallister-Ianelli (HM) Weight_Francis (F0) Weight_Francis (FA) Weight_Francis (FB) X X Mean and CV 63,212 77,812 81,534 89,337 na 116,720 112,021 0.29 0.28 0.27 0.27 na 0.25 0.24 151,968 156,987 134,320 168,972 na 240,463 213,398 0.23 0.19 0.15 0.18 na 0.21 0.20 Mean and CV 105,278 134,188 110,451 104,707 105,401 na na 0.23 0.23 0.23 0.23 0.23 na na 669,405 734,324 702,207 650,254 666,337 na na 0.36 0.32 0.35 0.34 0.34 na na CV (%) 100 + Data Weighting Methods ‘Within Assessment’ Variability MSY Assmt. CV (MSY) 75 50 25 0 Assessment (species) CV (%) Data Weighting Methods ‘Within Assessment’ Variability 75 FMSY 50 25 0 Assessment (species) Assmt. CV (Fmsy) CV (%) Data Weighting Methods ‘Within Assessment’ Variability 100 + Bcurrent Assmt. CV (Bcurrent) 75 50 25 0 Assessment (species) CV (%) Data Weighting Methods ‘Within Assessment’ Variability 80 Depletion Assmt. CV (Depletion) 60 40 20 0 Assessment (species) Data Weighting Methods ‘Between Management Quantity’ Variability CV (%) 55 50 Mgt. qty. CV 45 40 35 30 25 20 15 10 5 0 MSY FMSY Bcurrent Management quantity Depletion ‘Within Assessment’ Variability (Relative to Data Weighting Method) Relative error MSY Data weighting method Species (no. of assessments) Models (no. of replicates) Sample size limit implemented (no. of species) Convergence issues (no. of species) Unplotted models (pct. extreme positive outliers) 29 126 0 0 4% 27 119 2 2 2% 29 126 0 0 2% 11 43 0 0 2% 15 74 1 0 4% 15 74 1 0 1% ‘Within Assessment’ Variability (Relative to Data Weighting Method) Relative error FMSY Data weighting method Species (no. of assessments) Models (no. of replicates) Sample size limit implemented (no. of species) Convergence issues (no. of species) Unplotted models (pct. extreme positive outliers) 29 126 0 0 1% 27 119 2 2 0% 29 126 0 0 2% 11 43 0 0 7% 15 74 1 0 0% 15 74 1 0 0% ‘Within Assessment’ Variability (Relative to Data Weighting Method) Relative error Bcurrent Data weighting method Species (no. of assessments) 29 Models (no. of replicates) 126 Sample size limit implemented (no. of species) 0 Convergence issues (no. of species) 0 Unplotted models (pct. extreme positive outliers) 6% 27 119 2 2 11% 29 126 0 0 4% 11 43 0 0 12% 15 74 1 0 9% 15 74 1 0 5% ‘Within Assessment’ Variability (Relative to Data Weighting Method) Relative error Depletion Data weighting method Species (no. of assessments) Models (no. of replicates) Sample size limit implemented (no. of species) Convergence issues (no. of species) Unplotted models (pct. extreme positive outliers) 29 126 0 0 5% 27 119 2 2 10% 29 126 0 0 2% 11 43 0 0 12% 15 74 1 0 1% 15 74 1 0 3% Data Weighting Methods Relative to HM (‘correctly specified’ model) Relative error MSY Data weighting method Species (no. of assessments) Models (no. of replicates) Sample size limit implemented (no. of species) Convergence issues (no. of species) Unplotted models (pct. extreme positive outliers) 29 29 0 0 0% 27 27 2 2 0% 11 11 0 0 0% 15 15 1 0 0% 15 15 1 0 13% Data Weighting Methods Relative to HM (‘correctly specified’ model) Relative error FMSY Data weighting method Species (no. of assessments) Models (no. of replicates) Sample size limit implemented (no. of species) Convergence issues (no. of species) Unplotted models (pct. extreme positive outliers) 29 29 0 0 3% 27 27 2 2 0% 11 11 0 0 0% 15 15 1 0 0% 15 15 1 0 0% Data Weighting Methods Relative to HM (‘correctly specified’ model) Relative error Bcurrent Data weighting method Species (no. of assessments) Models (no. of replicates) Sample size limit implemented (no. of species) Convergence issues (no. of species) Unplotted models (pct. extreme positive outliers) 29 29 0 0 3% 27 27 2 2 0% 11 11 0 0 0% 15 15 1 0 0% 15 15 1 0 13% Data Weighting Methods Relative to HM (‘correctly specified’ model) Relative error Depletion Data weighting method Species (no. of assessments) Models (no. of replicates) Sample size limit implemented (no. of species) Convergence issues (no. of species) Unplotted models (pct. extreme positive outliers) 29 29 0 0 0% 27 27 2 2 0% 11 11 0 0 0% 15 15 1 0 0% 15 15 1 0 7% Conclusions • Data weighting methods impact on management quantities o Terminal biomass estimates most uncertain in most cases (mean CV=35%), depletion and MSY less so (20%), and FMSY most precise (<10%) o Positively-skewed, median-unbiased relative error distributions o The harmonic mean-based McCallister-Ianelli method (HM) resulted in precise and unbiased estimates in most cases, but … o Unweighted method (UW) also relatively precise and robust in many comparisons o Frances methods (F0, FA, FB) produced generally unbiased estimates, but typically less precise than HM; more similar for MSY-related quantities o FA less bias (equally precise) than FB in many comparisons o For correctly-specified assessment based on HM, better off not weighting (UW) than implementing an alternative data weighting method Study benefits and further work • Replicates (assessments) in meta-analysis are realistic o o Replicates associated with typical simulations are unrealistic, i.e., much too similar to one another … increase number/variety of assessments However, study (experimental) population based on real assessments provides limited cause-and-effect information, given the many data/parameter inconsistencies across replicates • Meta-analysis provides baseline information for more focused simulation studies o o Contrast between quality of derived management metrics Fold into MSEs addressing small pelagic species’ fisheries on the USA Pacific coast for basing (much needed) new and improved harvest control rules • Information useful for analysts charged with developing ongoing assessments for management purposes o Data weighting approaches in actual assessments are evolving presently, research needed to inform good practices References Crone, P.R., D.B. Sampson. 1998. Evaluation of assumed error structure in stock assessment models that use sample estimates of age composition. Pages 355-370 in Fishery Stock Assessment Models. Alaska Sea Grant College Program Report No. AK-SG-98-01, University of Alaska, Fairbanks, Alaska. Fournier, D., C.P. Archibald. 1982. A general theory for analyzing catch at age data. Can. J. Fish. Aquat. Sci. 39:1195-1207. Francis, R.I.C.C. 2011. Data weighting in statistical fisheries stock assessment models. Can. J. Fish. Aquat. Sci. 68:1124-1138. McAllister, M.K., J.N. Ianelli. 1997. Bayesian stock assessment using catch-age data and the samplingimportance resampling algorithm. Can. J. Fish. Aquat. Sci. 54(2): 284–300. Methot, R.D., C.R. Wetzel. 2013. Stock Synthesis: a biological and statistical frame-work for fish stock assessment and fishery management. Fish. Res. 142:86–99. Pennington, M., L.-M. Burmeister, V. Hjellvik. 2002. Assessing the precision of frequency distributions estimated from trawl survey samples. Fish Bull. 100:74–80. Punt, A.E. in press. Some insights into data weighting in integrated stock assessments. Fish. Res. Stewart, I.J., O.S. Hamel. 2014. Boostrapping of sample sizes for length- or age-composition data used in stock assessments. Can. J. Fish. Aquat. Sci. 671:581-588.