AN EFFECIENT CONTINUATION APPROACH USING
LEVENBERG MARQUARD ALGORITHM
Hala Omara Elsayed Zakia Gamal Ibrahima Ahmed El-Sawyb
a
b
Department of Basic Engineering Science, Faculty of Engineering , Shebin ElKom, Minoufia University, Egypt.
Department of Mathematics, Faculty of Science ,Qassim University ,KSA.
Abstract
In this paper, a new approach for continuation computation is proposed; that
depends on the Levenberg Marquardt algorithm initiated with a set of random
initial guesses. The Levenberg Marquardt algorithm is used to compute the static
solutions of a bifurcation diagram. The random initiating cuts the search space of
the free parameter which facilitates discovering the bifurcation behavior of the
model without adding a parameterization equation. The performance of the
method is thoroughly investigated for the Lorenz Oscillator, the Co-oxidization
models and the cusp model. The test results are compared to the corresponding
results of the software package AUTO. We find that the proposed approach gives
accurate, fast, and versatile bifurcation results, with one, two bifurcation
parameters.
Keywords Bifurcations, Levenberg Marquardt algorithm, Pseudo Arclength
method
1. Introduction
Dynamic autonomous systems are usually represented by a set of nonlinear
differential equations of the form
𝑢̇ = 𝐺(𝑢, 𝜆)
(1)
Where u is the vector of m-system states and 𝜆is the vector of n-critical
parameters. The qualitative changes in the system caused by the variation of one
or more critical parameters are called bifurcations [1], [2]. Hence, performing
bifurcation analysis is essential to profiling the behavior of dynamic systems and
pinpointing their weak regions and unstable areas. This bifurcation analysis is
typically accomplished by computing all equilibrium and periodic solutions of the
system along the entire range of its stability boundary. These equilibrium
solutions form the (hyper) surfaces in the state parameter space [3]. The
bifurcation analysis is essential for a wide variety of practical applications.
Examples for power system purposes included stability analysis, power system
planning and load dispatch problems as seen in [4],[5] and [Ajjarapu & Lee,
1992]. For chemical applications, the behavior of reaction diffusion systems is a
typical example as described in [6].
Different methods were proposed for continuation studies such as natural
parameter continuation, piecewise linear continuation and Gauss-Newton
continuation methods [3], [7]. The most well known continuation method is the
pseudo arclength method [8]. The method considered the singular point
(bifurcation point) as a solution (uo, λo) to the equation 𝐺(𝑢, 𝜆) = 0, where the
determinant of the Jacobian 𝐺𝑢 (𝑢𝑜 , 𝜆𝑜 ) is singular as described as,
𝜕𝐺
|
𝜕𝑢 𝑢𝑜 ,𝜆𝑜
= 𝐺𝑢 (𝑢𝑜 , 𝜆𝑜 )
(2)
As referenced in [4], the pseudo arclength continuation method eliminates the
drawbacks of the Gauss-Newton method at the singular points by using the
arclength parameterization. The method describes the vector of states u depending
smoothly on the approximated arclength “s”. Differentiating the equation G (u,λ)
=0 with respect to “s” yields
𝑑𝐺(𝑢(𝑠),𝜆(𝑠))
𝑑𝑠
= 𝐺𝑢 𝑢. + 𝐺𝜆 𝜆.
(3)
where 𝑢. , 𝜆. are the derivatives with respect to the approximated arclength “s”.
Then “Eq. (4)” determines the approximate arclength “s” on the curve of the
solution.
‖𝑢. ‖2 + ‖𝜆. ‖2 = 1
(4)
The resulting new solution is calculated through two computational stages. First, a
predicted solution is calculated along the tangent of the current solution. Then the
predicted solution is corrected using Newton’s method to get the new point on the
solution curve [9]. On the other hand, the pseudo arclength method has different
disadvantages. The precise choice of the selected step size ∆𝑠 is highly problem
dependent. As experienced, using a fixed step size does not lead to an ideal
performance for such computation [10]. Accordingly, utilizing a step size control
methodology is recommended. Accordingly, the step size is changed during the
computation process, utilizing suitable reduction and expansion factors to
determine the variation of the step size depending on the success of the corrector.
On the other hand, the selected initial guess remarkably controls the solution
convergence of the computational process. In order to get a suitable initial guess,
the dynamical solution of the constructed model may be required; the computation
will be obviously consuming time [11]. This may complicate performing
bifurcation analysis depending on the Newton method [12]. Also, needing an
additional equation Eq. (4) for parameterization increases the computational
burden.
The Levenberg Marquardt algorithm is an optimization algorithm which possesses
the advantages of both the steepest descent and Newton methods [13], [14]. It
converges to the optimum point whatever the initial guess was close to or away
from this optimum point.It has been used in several applications including the
detection of hopf bifurcation points [15], [16]. Different packages were developed
to perform bifurcation analysis such as AUTO, [17]. In spite of the efficient
performance of AUTO with its efficient computational engine and plotting
facilities, some problems were experienced with selecting the initial step, because
selecting the most suitable initial step with its minimum and maximum values
may be an exhausting issue. Moreover, selecting the initial guess remarkably
affects the ability to smoothly find the initial point on the solution curve.
In this paper, an algorithm is proposed to facilitate discovering the overall
bifurcation diagram of ordinary differential autonomous systems. The proposed
algorithm tries to combine the benefits of random initiating for the search spaces
of states and parameters and the fast response of Levenberg Marquardt algorithm
to overcome the problem of searching for a suitable initial step size, an initial
guess for starting the computations in AUTO or other software packages.
Moreover, the computation is initiated with a group of random initial guesses
within the predetermined search space of the system variables helps to overcome
the problem of slow convergence or divergence from the solution. This algorithm
can be used to explore the overall bifurcation diagram of the studied system,
determines the sensitive area of the continuation curve and introduces the suitable
initial solutions in the suspected areas of search space to be investigated using the
well known software packages without dealing with the computation problems of
these packages. Finally, the proposed algorithm there does not use the
parameterization equation which reduces the computations. This paper is
organized as follows. Section1provides a brief introduction, whereas a description
of the proposed algorithm is given in Section 2.Section 3 is concerned with
verifying the performance of the proposed algorithm. Results are analyzed and
discussed in Section 4.Finally conclusions and remarks are collected and
summarized.
2. Description of the Proposed Algorithm
Fig. 1 demonstrates the proposed algorithm for bifurcation computation. First, a
random set of initial guesses is generated. These random initial guesses are
selected within the predetermined search spaces of the system states and
parameters which try to cut the search spaces of the system states and free
parameter in a random manner. It should be mentioned that as more the size of
this set increases as the search spaces are better explored. This algorithm attempts
to deal with the equations of the system directly there is no need to add an extra
parameterization equation as known in pseudo arclength method. Using the
Levenberg Marquardt algorithm, the set of steady state solutions is generated
along the continuation curve. Then the bifurcation points are detected by
computing those parameters that have a solution branching.
In the steepest descent method, the negative of the gradient vector is used as a
direction for the minimization process. The computation is started from an initial
trial point X1 and moves iteratively along the steepest descent direction until the
optimum point is found [13], [14].
𝑆𝑖 = −𝛻𝐹𝑖 = −𝛻𝐹(𝑋𝑖 )
(6)
𝑋𝑖+1 = 𝑋𝑖 + 𝛼𝑆𝑖
(7)
Newton's method can be extended to minimize the multivariable functions. For
this aim, the quadratic approximation of the function F(X) at X=Xi is expressed
using Taylor's series expansion as
1
𝐹(𝑋) = 𝐹(𝑋𝑖 ) + 𝛻𝐹𝑖𝑇 (𝑋 − 𝑋𝑖 ) + 2 (𝑋 − 𝑋𝑖 )𝑇 𝐽𝑖 (𝑋 − 𝑋𝑖 )
(8)
where Ji is the second partial derivatives (Hessian matrix) of the function F
evaluated at the point Xi. Then minimizing F(X) by setting the partial derivatives
of Eq. (10) to be equal to zero yields
𝛻𝐹 = 𝛻𝐹𝑖 + 𝐽𝑖 (𝑋 − 𝑋𝑖 ) = 0
(9)
Random (N) initial guesses within
the range of variables
For i=1:N
Levenberg Marquardt solver with initial
guess u(i)
No
i=N
yes
Detect bifurcation
points
end
Fig.1 Flowchart of the proposed algorithm
If Ji is nonsingular, the next point can be approximated as
𝑋𝑖+1 = 𝑋𝑖 − 𝐽𝑖−1 𝛻𝐹𝑖
(10)
The steepest descent method reduces the function value when the design vector X1
is away from the optimum point X*. Newton’s method, on the other hand,
converges quickly when the design vector X1 is close to the optimum point X*.
The Marquardt method attempts to take the advantages of both the steepest
descent and Newton methods. This method modifies the diagonal elements of the
Hessian matrix as
𝐽𝑖~ = 𝐽𝑖 + 𝛿𝐴
(11)
where A is the diagonal matrix of [𝐽𝑖𝑇 𝐽𝑖 ]and δ is the damping factor computed for
each iteration. Then the direction of the search can be found as
𝑆𝑖 = −𝐽𝑖~−1 𝛻𝐹𝑖
(12)
If the reduction of Si is rapid, a smaller value of the damping factor δ can be used
[Rao, 1996]. This profiles the algorithm to be closer to the Newton’s method
computation. If iteration gives insufficient reduction in the residual error, δ can be
increased, profiling the algorithm closer to the steepest descent method. On the
other hand, different disadvantages were raised when utilizing the Levenberg
Marquardt algorithm. As remarked, it behaves like an interpolation between both
Newton’s method and the steepest descent methodologies. Accordingly, it
depends strongly on the initial guess. If the employed initial guess is too far from
the aimed solution, convergence may not be attained [12].
3. Performance Evaluation of the Proposed Algorithm
Three different standard systems were selected to investigate the performance of
the proposed algorithm including Lorenz Oscillator, Co-oxidization and cusp
models for both one- and two-parameter bifurcations models, respectively. Test
examples covering these systems were described in the following sub-sections.
3.1Tests with Lorenz Oscillator
The Lorenz Oscillator is a 3-dimensional dynamical system. It can be described
by the following equations [19]:
𝑢1̇ = 𝑝3 (𝑢2 − 𝑢1 )
(13)
𝑢2̇ =𝑝1 𝑢1 − 𝑢2 − 𝑢1 𝑢3
(14)
𝑢3̇ =𝑢1 𝑢2 − 𝑝2 𝑢3
(15)
With p2=8/3, p3=10, it has a super critical pitchfork-bifurcation at p1=1. The range
of 𝑢1 , 𝑢2 , 𝑢3 is -20, 20 and the range of 𝑝1 is 0, 30. Fig. 2 (a) shows the set of the
random initial guesses. Fig. 2 (b) shows the steady state solutions resulting from
the second step of the proposed algorithm. The maximum value of the bifurcation
parameter p1 is reached and the algorithm stops after completing the loop. The
bifurcation point is determined by detecting the change in stability of the
calculated static solutions.
Fig. 3 (a) shows the bifurcation diagram of the state u1 using the well-known
package (ATUO), whereas Fig. 3 (b) shows the bifurcation diagram of the same
state using the proposed algorithm. Similarly, Fig. 4 and 5 demonstrate the
verification of the algorithm, for states u2, u3. The results corroborated the efficacy
of the proposed algorithm verified with the results obtained by the AUTO
package.
20
10
8
6
4
2
0
u1
Initial u1
10
0
-2
-4
-10
-6
-8
-20
0
5
10
15
Initial p1
20
25
-10
0
30
5
10
15
20
25
30
35
p1
(a)
(b)
Fig. 2 Behavior of Levenberg Marquardt algorithm based random initial guessing
(a) Random initial guesses
(b) Resulting state solution
10
u1
5
0
-5
-10
0
5
10
15
20
25
30
p1
(a)
(b)
Fig. 3 u1-p1 Bifurcation diagram for Lorenz Oscillator model
(a) Using AUTO
(b) Using the proposed algorithm
10
u2
5
0
-5
-10
0
5
10
15
20
p1
(a)
(b)
Fig. 4 u2-p1 Bifurcation diagram for Lorenz Oscillator model
(a) Using AUTO
(b) Using the proposed algorithm
25
30
40
30
u3
20
10
0
-10
0
5
10
15
20
25
30
p1
(a)
(b)
Fig. 5 u3-p1 Bifurcation diagram for Lorenz Oscillator model
(a) Using AUTO
(b) Using the proposed algorithm
3.2 Tests with CO-oxidation model
Kinetic oscillations have been observed in many different catalytic reactions.
Research on this phenomenon has its own industrial and practical significance. In
the model derived by [20], the numerical bifurcations of equilibrium were
analyzed depending on two parameters. The following equations represented the
dynamical model of this system. The bifurcation parameters are q2, k.
𝑥̇ = 2𝑞1 𝑧 2 − 2𝑞5 𝑥 2 − 𝑞3 𝑥𝑦
(16)
𝑦̇ = 𝑞2 𝑧 − 𝑞6 𝑦 − 𝑞3 𝑥𝑦
(17)
𝑠̇ = 𝑞4 𝑧 − 𝑘𝑞4 𝑠
(18)
𝑧 =1−𝑥−𝑦−𝑠
(19)
Fig. 6, 7 and 8 illustrate the one bifurcation diagram of the states x, y, s with the
free parameter q2 using AUTO and the proposed algorithm. Fig. 9 demonstrates
the loci of limit point bifurcation diagram considering q2, k as the free parameters,
whereas Fig. 10 shows the corresponding loci of the hopf point bifurcation. As
highlighted from the results, the efficacy of the proposed algorithm was verified
for exploring the bifurcation diagrams for more than one bifurcation parameter
and bifurcation type.
0.2
x
0.15
0.1
0.05
0
0.5
1
1.5
q2
(a)
(b)
Fig. 6 x-q2 bifurcation diagram at k=.4 for CO-oxidation model
(a) Using AUTO
(b) Using proposed algorithm
2
0.8
y
0.6
0.4
0.2
0
0.5
1
1.5
q2
(a)
(b)
Fig. 7 y-q2 bifurcation diagram at k=.4 for CO-oxidation model
(a) Using AUTO
(b) Using proposed algorithm
2
s
0.4
0.2
0
0.5
1
1.5
2
q2
(a)
(b)
Fig. 8 s-q2 bifurcation diagram at k=.4 for CO-oxidation model
(a) Using AUTO
(b) Using proposed algorithm
2
k
1.5
1
0.5
0
0.8
1
1.2
1.4
q2
(a)
(b)
Fig. 9 k-q2 bifurcation diagram for CO-oxidation model
(a) Loci of limit point using AUTO
(b) Using proposed algorit
1.6
1.8
k
0.6
0.4
0.2
0
0.8
1
1.2
1.4
q2
(a)
(b)
Fig. 10 k-q2 two parameter bifurcation diagram for CO-oxidation model
(a) Loci of hopf point using AUTO
(b) Using proposed algorithm
3.3 Tests with the Cusp model
This example illustrates the computation of stationary solutions, locating saddlenode bifurcations of these solutions and the continuation of a saddle-node
bifurcation in two parameters. The cusp normal [17] form equation is given by
𝑥̇ = µ + 𝜆𝑥 − 𝑥 3
(20)
AUTO performs computations of continuation of this model in four separate runs.
In the first run, a branch of stationary solutions is traced out. Along it, one fold
(LP) (limit point, or in this case, a saddle-node bifurcation) is located. The free
parameter is μ. The other parameter λ remains fixed in this run. The second run
does the same thing but now in the negative direction of μ. In the third run, the
fold detected in the first run is followed in the two parameters μ and λ. The fourth
run continues this branch in opposite direction. In our proposed approach, the
ranges of x, µ and λ are -2, 2, -2, 2 and 0, 3 respectively. The algorithm calculates
the bifurcation values of µ at different values of λ which enables to find the µ- λ
bifurcation diagram.Fig.11 (a) shows the distribution of random initial guesses
for µ and x while Fig.11 (b) shows the µ - x bifurcation diagram when λ=1. It is
clear that there are two limit points at µ=.3848 and µ=-.3844. Fig.11 (c) shows the
µ- λ bifurcation diagram after repeating the algorithm at different values of λ
within its range.
4. Results Analysis and Discussion
As experienced from the tests, the bifurcation analysis is a remarkably timeconsuming procedure. The conventional packages have problems of finding a
suitable initial guess and initial step when finding the continuation curve. The
dynamical equations of some models may be required to be solved in order to get
the steady state solution to be used as an initial solution for bifurcation analysis.
As an example, In the Lorenz model, the initial guess was 0, 0, and 0 in AUTO for
tracing the continuation curve. In the case of utilizing an initial guess of 0.01, 0
and 0, the response was generated as seen in Fig. 12 (a) .
2
1.5
1.5
1
1
0.5
0.5
x
0
-0.5
0
-0.5
-1
-1
-1.5
-1.5
-2
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
mu
-2
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
mu
(a)
(b)
2
1.5
lambda
x
2
1
0.5
0
-1.5
-1
-0.5
0
0.5
1
1.5
mu
(c)
Fig. 11 Bifurcation analysis for Cusp model
(a) µ-x random guesses distrubtion of Cusp model
(b)µ-x bifurcation diagram at λ =1
(c) µ- λ bifurcation diagram
This response explains the effects of selection of initial guess in tracing the
solution curve. The proposed algorithm, on the other hand, with its random initial
guesses, eliminates this problem perfectly as shown in Fig. 12 (b) with the spread
points representing the random initial guesses and the solid curve representing the
continuation curve of u1. The coupling between random initiating and Levenberg
Marquardt algorithm can easily explore the overall response of the system’s free
parameters and the shape of its continuation curve directly. This property helps to
explore the system response without wasting time in tracing of the continuation
curve.
The proposed approach can find the solution continuation curve without needing
of a parameterization equation, which leads to dealing with the equation of the
system directly. This reduces the mathematical burden as well. This increases the
efficiency of the algorithm and remarkably reduces the time of computations.
In conventional packages like AUTO, the initial step is needed to trace the
solution curve as well as the minimum and maximum steps. This initial step is
highly problem dependent. When the initial step is not suitable to the problem, it
may fail to trace the continuation curve. Searching for this suitable initial step is a
time-consuming and exhausting process. This problem no longer exists with the
proposed approach. This property reduces the effort and time of computations
which facilitates exploring the bifurcation points (when existing) and discovers
the system response with different cases of free parameters.
u1
10
0
-10
0
(a)
5
10
15
p1
20
25
30
(b)
Fig.12 Effect of initial solution on Lorenz Oscillator
(a) Using AUTO
(b)Using proposed algorithm
5. Conclusion
In this paper, a new approach for continuation of static solutions of autonomous
systems was proposed. This algorithm depends mainly on the Levenberg
Marquardt algorithm, using random initial guesses in the search spaces of the
states and the parameters. The approach explored the overall bifurcation diagram
of the system and defined the areas of bifurcations. It discovers the suitable initial
solutions to the AUTO package to investigate the stability details and overcame its
computation problems. The algorithm was tested with the Lorenz Oscillator, the
Co-oxidization and the Cusp models for one and two parameter bifurcation cases,
respectively. As compared with AUTO, the proposed algorithm provides a
versatile, reliable and fast bifurcation tool. Moreover, the proposed approach
provides a powerful tool for other challenging situations such as three parameter
bifurcation surfaces. This facilitates applying the algorithm to sophisticated
engineering applications, considering their bifurcation diagrams without wasting
time and effort on searching for the initial solutions and suitable initial step.
6. References
[1]
Kubicek, M. &Marek, M., “Computational methods in bifurcation theory
and dissipative structures”, Springer-Verlag, New York Inc., 1983.
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
[17]
[18]
W. Govaerts, 'Numerical bifurcation analysis for ODEs', Journal of
Computational and Applied Mathematics, Volume 125, Issues 1–2, 15
December 2000, Pages 57-68.
Seydel, Rüdiger, “Practical bifurcation and Stability analysis”, SpringerVerlag, Heidelberg, 2000.
Chen, G., Hill, D. & Yu, X. H., “Bifurcation Control: theory and
applications”, Springer-Velrag, Berlin, Heidelberg, 2003.
Ajjarapu, V. & Lee, B., “Bifurcation theory and its application to
nonlinear dynamical phenomena in an electrical power system”, IEEE
Transactions on Power Systems, Vol. 7, No. 1, pp.424-431, 1992.
Leung, W. & Villa, B. R., “Bifurcation of Reaction-Diffusion Systems:
Application to Epidemics of Many Species”, Journal of Mathematical
Analysis and Applications, Vol. 244, No. 2, pp. 542-563(22). 2000.
Jan Broeckhove, Przemysław Kłosiewicz, Wim Vanroose, "Applying
numerical continuation to the parameter dependence of solutions of the
Schrödinger equation", Journal of Computational and Applied
Mathematics, Volume 234, Issue 4, 15 June 2010, Pages 1238-1248.
Keller, H. B., “Numerical solution of bifurcation and nonlinear
eigenvalue problems”, P. H. Rabinowitz, ed., Applications of Bifurcation
Theory, Academic Press, pp. 359–384, 1977.
Keller, H. B., “Lectures on Numerical Methods in Bifurcation Problems”,
INSDOC Regional Centre. Indian Institute of Science, Springer-Verlag,
1986.
Niet, A. de, “Step-size control and corrector methods in numerical
continuation of ocean circulation and fill-reducing orderings in multilevel
ILU methods”, Master’s Thesis, Department of Mathematics, University
of Groningen, Groningen-Germany, 2002.
Iooss, G. & Joseph, D., “Elementary Stability and Bifurcation Theory”,
Springer-Verlag New York Inc., 2002.
Rao, Singiresu S., “Engineering Optimization”, John Wiley & Sons Inc.,
1996.
Levenberg, K., “A Method for the Solution of Certain Non-Linear
Problems in Least Squares”, The Quarterly of Applied Mathematics, vol.
2, pp. 164–168., 1944.
Marquardt, D., "An Algorithm for Least-Squares Estimation of Nonlinear
Parameters". SIAM Journal on Applied Mathematics 11 (2): 431–441,
1963.
A, Trubuil, "Hopf bifurcation point detection. Application of a
continuation method to optimization of multistate systems", Analysis and
optimization of systems, Springer-Verlag London, UK, 1988.
Christian Kanzow, Nobuo Yamashita, Masao Fukushima, "Levenberg–
Marquardt methods with strong local convergence properties for solving
nonlinear equations with convex constraints ', Journal of Computational
and Applied Mathematics, Volume 173, Issue 2, 15 January 2005, Pages
321-343, 2005.
Doedel, E. J., “AUTO-07p continuation and bifurcations software for
ordinary differential equations”, Concordia University Montreal, Canada,
2007.
Chickarmane, V., Paladugu, S. R., Bergmann, F., & Sauro, H. M.,
“Bifurcation discovery tool”, Bioinformatics, Vol. 21, pp. 3688–3690,
2005.
[19]
[20]
Lorenz, J., “Nonlinear boundary value problems with turning points and
properties of difference schemes”, Theory and Applications in Singular
Perturbation (W. Eckhaus & E. M. de Jager, eds) Springer-Verlag,
Berlin, pp. 150-169, 1982.
Bykov, V.I., Yablonski, G.S. & Kim V.F., “On the simple model of
kinetic self oscillations in catalytic reaction of CO oxidation”, Dokl.
Akad. Nauk SSSR 242(3) (1978), pp. 637–639.