Additional Late Reviewer Comments, bp0801842
Dear Editor,
We would like to thank you for giving us additional comments which indeed proved useful for the revision.
Although we were not obliged to take them into account, however, for the sake of improving further the quality
of the revised manuscript, we have taken all of these comments into account during the revision.
We would like to note that most of these comments were raised earlier by the original reviewers (please see the
separate document) and hence were addressed fully (whenever this was the case we have indicated that below).
As for the remaining comments we have provided our response below and consequently revised further the
paper. We have also indicated where we have made the changes in the manuscript for your convenience. Where
relevant, we have also provided our counter arguments to the comments.
We thank you again for the constructive comments of these additional reviewers.
Regards,
Gurkan Sin, PhD
On behalf of the authors.
Reviewer 1)
Abstract
1. Line 2: For clarity, please define the acronym PAT in abstract section since it is its first mention.
Authors: OK! Done. Please see revised abstract. “within
Process Analytical Technology (PAT)”
2. Line 3: Please use full genus name for Streptomyces since it is its first mention.
Authors: OK! Done. Please see revised abstract.
Introduction
1. Modeling can thus be perceived as a vehicle……
Please consider revising sentence for more clarity. Are you
implying that for process understanding the process datum continuously updates the model, how can this be if
you are basing your uncertainty evaluations on a mechanistic model? I would either clarify this sentence or omit
based on the lack of conclusiveness.
Authors: NO! We simply wanted to mean that is a tool or a means to a certain purpose! We have clarified this. Please
see revised paper. Page 4 line 22.
2. Last sentence The paper ends with a set of conclusions….: Please consider combining with the prior sentence or
omit – not necessary.
Authors: OK! Done. The sentence has been removed.
Additional Late Reviewer Comments, bp0801842
Methods
1. Among these parameters, 12 were readily available…… Which parameters were obtained through experimental
measurements, model ID techniques, and literature sources? Perhaps, providing these notations in your
supplemental section would be useful.
Authors: Done! We have addressed this comment in our rebuttal to the original reviewers comments.
Results and Discussion
1. While the model has 19 output variables......... What was your reasoning behind the outputs you chose? Please
describe your justification for focusing on the chosen outputs? Did the output variables not discussed have
similar quality to those discussed? Can you speak about the results in general?
Authors: Done! We have addressed this comment in our rebuttal to the original reviewer’s comments.
2. Using this model, would your uncertainty values decrease if you cultivate a strain with a faster growth rate?
Authors: This question can simply be answered by using a scenario analysis but it is not within the scope of this
paper. But from the results of this fermentation study, we observe that the uncertainty is pronounced in the stationary
and decay phases. Hence the model seems to get the exponential phase right!
3. 3.4 Stochastic….. In a research setting, the predictions…… Sounds like you are implying that non-research or
industrial/manufacturing fermentors are not well-operated. Please consider revising to "successful
fermentations."
Authors: OK! Thanks for highlighting this. We have now rephrased the sentence as follows: “When
models are used in a
research setting, the predictions are usually used to interpret data collected from successful fermentation runs;
hence it remains more important to focus on input uncertainty.
”
4. 3.5 Perspectives: Overall these methods help picturing….. Incorrect usage of picturing, please correct to picture.
Also in same sentence what the quality of the model predictions is change "is" to "are".
Authors: OK! Done! We have replaced picturing with quantifying! And also: we have corrected the grammar error. Please
see the revised paper.
Conclusions
Additional Late Reviewer Comments, bp0801842
1. Does changing any of the three high variance parameters reduce the uncertainty in your model? Can you speak
on the rigidity and standard deviations associated with your model assumptions and how those errors propagate?
Authors: This is a good question and raised in the original reviewers comments as well. This question can be
answered by a dedicated study (by a combination of theoretical and experimental work) and is relevant but it is not
within the scope of this contribution.
2. Could factorial analyses be applied to determine the relationship (if any) to the high variance parameters to
antibiotic production?
Authors: Sure and we have done that in this paper by using the sensitivity analysis. See table 2, 3 and 4.
General Comments and Recommendations
I found this article to be very timely and a well applied application towards the understanding and improvement of
biological processes.
I recommend this manuscript for publication in Biotechnology Progress. I ask that the authors consider my minor
suggestions and corrections for final revision and submission.
Authors: Thank you for your constructive comments.
Reviewer 2)
Comments to the editor:
Based on the review results as outlined in the comments to the authors, the paper is not ready for publication in
Biotechnology Progress. It is also doubtful whether even a major revision would improve the content enough to
justify publication. I suggest that the paper is rejected in its current form. If the editor deems the general topic
interesting enough for the journal, the authors may be advised to resubmit after a substantial reorientation of the
content.
Authors: While we respect the opinion of this reviewer, we disagree with it profoundly. Two reviewers praised its
potential contribution and three other reviewers -although expressing some concerns for revisions - still thought
positive of its novelty and contributions. Second, we wished that the reviewer had supported his/her claim by some
evidence, e.g. has the uncertainty analysis of a fermentation model been presented previously the way it is done in
this manuscript? Has any sensitivity analysis been done, e.g. Morris screening or SRC? Our humble opinion and the
opinion of the other reviewers is: no, it hasn’t! And this is what we did and demonstrated in this study which we
modestly believe deserves merit.
Comments to the authors:
The submitted manuscript summarizes several established methods for uncertainty and sensitivity analysis of
biotechnological processes.
Additional Late Reviewer Comments, bp0801842
These methods are applied to a model for antibiotic production with S.
coelicolor, and they are compared and judged based on the results obtained in this application.
The topic of the paper is in general interesting for readers of Biotechnology process. However, the presented
methods are to well established to justify a research publication on their own. Although there is some merit in
comparison of the different methods, it seems that the outcome is highly dependent on the specific model that
the analysis is applied to. Therefore, a general comparison of the methods based on the one application
discussed in the manuscript would be inappropriate. Moreover, the presentation of the suggested analysis
methods is not done in a way to be suitable for a tutorial introduction.
There are several flaws in the presentation of the analysis methods, that either make the paper unneccessarily
hard to read, or that even shed doubts on the usefulness of the proposed methods for the considered process.
- - In the description of the model (Eq. 1), it is not clear what the state vector x should represent. It seems that
the authors are using an ODE model, which would be written as dx/dt = f(x,t,theta). However, if an output
perspective is taken, the state is not directly relevant, unless it represents the initial condition. The initial
conditions for the model are in fact not mentioned at all, but should also be considered for a batch process.
Authors: We agree with this comment. Indeed we have not described the model in detail –but this choice was made on
purpose for two reasons: to save space and because the model and its development have been presented in a separate
study elsewhere. We have now acknowledged this more clearly in the paper. Please see the revised paper. Page 6 line
31.
Second, we have also extended the equation 1 to indicate that it is an ODE model. Initial conditions are naturally provided
and indeed part of the model structure. These are all addressed in the model development paper mentioned above. That
said, whether a model is ODE type or not will not have any effects on the methods we evaluate, since simply these
statistical methods are applicable to any type of model structure.
For revised Eq.1, please see the revised paper.
dx
 f  x t  , t, θ
dt
y (t )  g (x  t  , t , θ)
- - How the authors obtained the values 0.44 and 0.48 below Eq. 2 is unclear to me. I would have expected 0.23
and 0.69 instead. Also, it usually makes more sense to assume a factor uncertainty instead of a +/- percent
variation for biological systems.
Authors: This was a mistake. We have now corrected the Eq.2. 50% should have been 5%. Please see the revised
manuscript.
- - The presentation of the differential sensitivity analysis is not very clear. Also, Eq. 5 is not appropriate for
continuous-time systems.
Authors: What is not clear about it? It is a standard formalism to denote sensitivity as a first order derivative. This has
been referenced now to ref number 9 which supports our formalism. We have regularly used these formalisms in our
previous publications and on continuous-time systems! We don’t see and understand what is inappropriate about it!
Similarly please check reference number (9) which supports our point here.
References are missing here.
Authors: Which references are missing?
- - It is disturbing that the authors comment so shortly on the scalar output required for the SRCs, as its choice
might influence the results of the analysis significantly. Also, taking the mean over a time series as indicated
may not be very informative for a highly dynamic batch process.
Additional Late Reviewer Comments, bp0801842
Authors: Simply because it is not the prime purpose of this contribution to compare different scalar products of
Monte Carlo simulations and evaluate its effect on the parameter significance ranking. Our aim was to compare
three different sensitivity analysis methods and for that we have used one type of scalar values.
What the reviewer suggests is a technical detail which can be addressed straightforwardly by another and
independent study. Moreover taking mean values of the Monte Carlo simulations is an approach typically used
as standard practice elsewhere; see reference number 8, 13 and 14. Hence there is nothing wrong and flawed in
this choice.
We have now acknowledged this comment in the revised paper as follows:
“The scalar outputs required for the calculation of SRCs were obtained by using the mean of the time series
profiles for each model output. This choice was made on purpose for two particular reasons: (a) to check the
sensitivity of the process characteristics on the average behaviour of the process, (b) to keep the study focused
on comparing three sensitivity analysis methods using one type of scalar product (since otherwise one may
easily get side-tracked by the technical details of each method).”
Reviewer 3)
The paper begins by briefly introducing a parameter-rich ODE model for S. coelicolor cultivation. A Monte
Carlo uncertainty analysis is applied to the model, with parameter variability assigned based on an expert
review process. Additionally, three sensitivity analysis methods are employed. The sensitivity analyses agree
for the most part, with small differences as discussed in the paper.
Overall, the paper provides an interesting application of Monte Carlo uncertainty analysis and a useful
comparison of sensitivity anaysis techniques. I have only a few suggestions and concerns.
Broadly, the paper focuses on constraining model behavior using only direct information on parameter values.
Other, typically Bayesian, methods directly incorporate other experimental information on the model system.
See, for example, Battogtokh et al. PNAS 99:16904 2002 and/or Casey et al. IET Systems Biology 1:190 2007.
It would be helpful for readers of this paper to mention such complementary approaches in either the
introduction or discussion.
Authors: We agree with this comment of the reviewer. Indeed, next to Monte Carlo methods, Bayesian analysis
is relevant for quantifying uncertainty of model outputs. We have now acknowledged this in the introduction
section. But instead of giving specific application references as suggested by the reviewer, we have provided a
general text book “Gelman, A., J.B. Carlin, H.S. Stern and D.B. Rubin (2003). Bayesian data analysis. London,
U.K.: Chapman & Hall/CRC.” on Bayesian analysis which is commonly accepted as the key reference in this
field. Please see the revised introduction:
“Several uncertainty analysis methods are available. For example, Bayesian analysis in combination with
evolutionary optimisation algorithms is emerging as a feasible method for performing uncertainty analysis in
complex numerical models24. In this study, however, the engineering standard Monte Carlo procedure is chosen
to propagate and analyse the input uncertainty, since the method is generally accepted as computationallyeffective and reliable7.”
More technically, the authors mention that only differential sensitivity analysis identifies t_lag as an important
parameter, and they assert that is because this importance is only true locally, at the nominal parameters.
Additional Late Reviewer Comments, bp0801842
Another possible reason is that the diffential analysis is the only one not restricted to analyzing the mean of the
output time series. The authors should test the "locality" of the importance of t_lag by running the differential
analysis about a selection of other parameter sets from within the allowed ranges. Does the differential analysis
identify t_lag as important at other parameter sets? If so, it may be detecting an effect that is not seen in the
mean output traces.
Authors: That is a good question. To start with, we have now explicitly acknowledged the choice of averaging
for the outputs in response to the comments of the second reviewer above. Please see the revised manuscript
page 15 line 33.
Second, the differential analysis is indeed performed locally. However to rank the significance order of
parameters, the dmsqr measure was used. This measure does also perform averaging -albeit not the output but the
output sensitivity functions. Please see equation Eq.5 . Since all of the rankings were somewhat based on
averaging, we don’t think that this is the only factor responsible for that.
Secondly, and actually, Morris Screening does perform differential analysis at randomly selected parameter
points. Hence we have an answer to this question of the reviewer which is that the rank order of the tlag
apparently is high as its was found by differential analysis. Please notice that that doesn’t mean tlag is not
important, it is found important by all the three methods. It is just the ranking which is different! We have now
provided an explanation in the revised manuscript to clarify this issue:
“Two possible reasons can be offered to explain this difference in the ranking of tlag: (a) difference in the
way model outputs were averaged: the SRC and Morris screening were performed on the averaged outputs,
while differential analysis was performed on dynamic (time-series) outputs and later an average was taken as
the mean-squared error,msqr (refer to Eq.4 and Eq.5 to see how msqr measure is calculated), and (b) the
differential analysis is a local technique, while the SRC and Morris screening are global methods. Hence the
resulting rank order of tlag may be relative to the initial conditions at which the analysis is performed.
”
Minor editorial points:
1) The acronym PAT should be defined upon first use in the abstract.
Authors: OK! Done. See above.
2) There appears to be a typo in the example Ysx calculation immediately following equation 2. Should it be
%5 rather than %50?
Authors: That is correct! Done. See above.
3) The arrows in Figure 5 should be explained or removed.
Authors: We have removed the arrows and used instead shading to indicate the axis. Please see revised Figure
5.
4) It is not clear from the paper where one could find the actual model equations simulated. A detailed reference
should be given, or the equations should be included in supplementary material.
Authors: That is true. We now provided a detailed reference. Please see our response above.
5) In the caption for Table 2, it should say "the *larger the
magnitude* of a coefficient".
Authors: OK! Done! Please see revised Table 2.
Additional Late Reviewer Comments, bp0801842
Reviewer 4)
Comments:
This paper evaluates the application of uncertainty and sensitivity analysis techniques in bioprocess model
development. Monte Carlo procedure is applied in the uncertainty study to estimate the model output
uncertainty based on the assumptions made on the uncertainty of input parameters. Three sensitivity analysis
methods (Standardized Regression Coefficients, Morris method and differential analysis) were evaluated and
compared to understand the contribution of the input parameters to the variance in the outputs. The case study is
based on a mechanistic model describing a batch cultivation of S. coelicolor for antibiotic production.
The paper could be benefit if the authors can demonstrate that the uncertainty and sensitivity analysis results are
applied in the process model for improving the model structure, leading to better prediction, or reducing
computing time, etc.
Suggestions on revision:
1. The process information could be briefly introduced to provide background information of the process
and the basis of the process model.
Authors: OK! We have done that. Please see our response above.
2. Compare the model prediction with experimental results if possible to demonstrate that the model
predictions match the trend of experimental results.
Authors: This is done in our earlier publication which we now have referred to explicitly. See above.
3. It would be helpful if the authors can demonstrate the improvement on the model structure and/or
prediction with the application of uncertainty and sensitivity analysis results.
Authors: This is outside the scope of this contribution. On the other had, such an improvement was subject
of another study. See our previous publication.
4. The parameters and symbols used in the model should be stated clearly in the paper, either within the
text or in the appendix. At the moment, it is difficult to follow the context as this information is not
provided with the paper.
Authors: We agree with that. Since the model contains a large number of parameters, they were given as
appendix (please see the Table S1.1). But if the editor requires that, we can provide them in the main text as
well.
5. Information flow needs to be improved. For example, section 3.4 should be placed in the introduction
section rather than the result analysis section.
Authors: We don’t agree with that. Plus the other 5 reviewers did not see any problem in placing section 3.4
where it is now in the manuscript. We have therefore decided not to modify the paper at this point..
6. References should be organised in a standard format.
Additional Late Reviewer Comments, bp0801842
Authors: We don’t understand what is meant by this. We have followed the format of the biotechnology
progress journal for the style of references. We have checked that again and we didn’t find any error.