scipy.optimize.least_squares
scipy.optimize.least_squares(fun, x0, jac='2-point', bounds=(- inf, inf),
method='trf', ftol=1e-08, xtol=1e-08, gtol=1e-08, x_scale=1.0, loss='linear',
f_scale=1.0, diff_step=None, tr_solver=None, tr_options={}, jac_sparsity=None,
max_nfev=None, verbose=0, args=(), kwargs={})
[source]
Solve a nonlinear least-squares problem with bounds on the variables.
Given the residuals f(x) (an m-D real function of n real variables) and the loss function rho(s) (a
scalar function), least_squares finds a local minimum of the cost function F(x):
minimize F(x) = 0.5 * sum(rho(f_i(x)**2), i = 0, ..., m - 1)
subject to lb <= x <= ub
 Search the docs ...
Clustering package
(
The purpose of the loss function rho(s) is to reduce the influence of outliers on the solution.
)
Parameters: fun : callable
( scipy.cluster )
K-means clustering and vector
Function which computes the vector of residuals, with the signature fun(x,
quantization
*args, **kwargs), i.e., the minimization proceeds with respect to its first
(
( scipy.cluster.vq )
(
Hierarchical clustering
)
argument. The argument x passed to this function is an ndarray of shape (n,)
(never a scalar, even for n=1). It must allocate and return a 1-D array_like of
( scipy.cluster.hierarchy )
)
Constants
complex residuals, it must be wrapped in a real function of real arguments, as
(
shown at the end of the Examples section.
)
( scipy.constants )
Discrete Fourier transforms
( scipy.fft )
)
x0 : array_like with shape (n,) or float
(
Initial guess on independent variables. If float, it will be treated as a 1-D array
with one element.
Legacy discrete Fourier
transforms
(
)
( scipy.fftpack )
jac : {‘2-point’, ‘3-point’, ‘cs’, callable}, optional
Method of computing the Jacobian matrix (an m-by-n matrix, where element (i,
(
Integration and ODEs
Interpolation
j) is the partial derivative of f[i] with respect to x[j]). The keywords select a finite
)
( scipy.integrate )
(
difference scheme for numerical estimation. The scheme ‘3-point’ is more
accurate, but requires twice as many operations as ‘2-point’ (default). The
)
scheme ‘cs’ uses complex steps, and while potentially the most accurate, it is
( scipy.interpolate )
(
Input and output ( scipy.io )
)
Linear algebra
analytically continued to the complex plane. Method ‘lm’ always uses the ‘2return a good approximation (or the exact value) for the Jacobian as an
Low-level BLAS functions
( scipy.linalg.blas )
(
array_like (np.atleast_2d is applied), a sparse matrix (csr_matrix preferred for
performance) or a scipy.sparse.linalg.LinearOperator.
)
Low-level LAPACK functions
( scipy.linalg.lapack )
)
(
bounds : 2-tuple of array_like, optional
Lower and upper bounds on independent variables. Defaults to no bounds.
Each array must match the size of x0 or be a scalar, in the latter case a bound
BLAS Functions for Cython
will be the same for all variables. Use np.inf with an appropriate sign to
LAPACK functions for Cython
disable bounds on all or some variables.
Interpolative matrix
decomposition
applicable only when fun correctly handles complex inputs and can be
point’ scheme. If callable, it is used as jac(x, *args, **kwargs) and should
(
( scipy.linalg )
(
method : {‘trf’, ‘dogbox’, ‘lm’}, optional
( scipy.linalg.interpolative )
)
Miscellaneous routines
( scipy.misc )
)
(
Multidimensional image
processing
shape (m,) or a scalar. If the argument x is complex or the function fun returns
(
( scipy.ndimage )
)
Orthogonal distance regression
(
Algorithm to perform minimization.
( scipy.odr )
)
Optimization and root finding
( scipy.optimize )
)
small problems with bounds. Not recommended for problems with rank-
(
deficient Jacobian.
‘lm’ : Levenberg-Marquardt algorithm as implemented in MINPACK. Doesn’t
( scipy.signal )
Sparse matrices
(
handle bounds and sparse Jacobians. Usually the most efficient method for
small unconstrained problems.
( scipy.sparse )
Sparse linear algebra
(
( scipy.sparse.linalg )
Default is ‘trf’. See Notes for more information.
)
ftol : float or None, optional
Compressed sparse graph
routines
problems with bounds. Generally robust method.
‘dogbox’ : dogleg algorithm with rectangular trust regions, typical use case is
Cython optimize zeros API
Signal processing
(
‘trf’ : Trust Region Reflective algorithm, particularly suitable for large sparse
(
( scipy.sparse.csgraph )
)
Tolerance for termination by the change of the cost function. Default is 1e-8.
The optimization process is stopped when dF < ftol * F, and there was an
adequate agreement between a local quadratic model and the true model in
the last step.
If None and ‘method’ is not ‘lm’, the termination by this condition is disabled. If
‘method’ is ‘lm’, this tolerance must be higher than machine epsilon.
xtol : float or None, optional
Tolerance for termination by the change of the independent variables. Default
is 1e-8. The exact condition depends on the method used:
For ‘trf’ and ‘dogbox’ : norm(dx) < xtol * (xtol + norm(x)).
For ‘lm’ : Delta < xtol * norm(xs), where Delta is a trust-region radius
and xs is the value of x scaled according to x_scale parameter (see below).
If None and ‘method’ is not ‘lm’, the termination by this condition is disabled. If
‘method’ is ‘lm’, this tolerance must be higher than machine epsilon.
gtol : float or None, optional
Tolerance for termination by the norm of the gradient. Default is 1e-8. The
exact condition depends on a method used:
For ‘trf’ : norm(g_scaled, ord=np.inf) < gtol, where g_scaled is the
value of the gradient scaled to account for the presence of the bounds
[STIR].
For ‘dogbox’ : norm(g_free, ord=np.inf) < gtol, where g_free is the
gradient with respect to the variables which are not in the optimal state on
the boundary.
For ‘lm’ : the maximum absolute value of the cosine of angles between
columns of the Jacobian and the residual vector is less than gtol, or the
residual vector is zero.
If None and ‘method’ is not ‘lm’, the termination by this condition is disabled. If
‘method’ is ‘lm’, this tolerance must be higher than machine epsilon.
x_scale : array_like or ‘jac’, optional
Characteristic scale of each variable. Setting x_scale is equivalent to
reformulating the problem in scaled variables xs = x / x_scale. An
alternative view is that the size of a trust region along jth dimension is
proportional to x_scale[j]. Improved convergence may be achieved by
setting x_scale such that a step of a given size along any of the scaled variables
has a similar effect on the cost function. If set to ‘jac’, the scale is iteratively
updated using the inverse norms of the columns of the Jacobian matrix (as
described in [JJMore]).
loss : str or callable, optional
Determines the loss function. The following keyword values are allowed:
‘linear’ (default) : rho(z) = z. Gives a standard least-squares problem.
‘soft_l1’ : rho(z) = 2 * ((1 + z)**0.5 - 1). The smooth approximation
of l1 (absolute value) loss. Usually a good choice for robust least squares.
‘huber’ : rho(z) = z if z <= 1 else 2*z**0.5 - 1. Works similarly to
‘soft_l1’.
‘cauchy’ : rho(z) = ln(1 + z). Severely weakens outliers influence, but
may cause difficulties in optimization process.
‘arctan’ : rho(z) = arctan(z). Limits a maximum loss on a single residual,
has properties similar to ‘cauchy’.
If callable, it must take a 1-D ndarray z=f**2 and return an array_like with
shape (3, m) where row 0 contains function values, row 1 contains first
derivatives and row 2 contains second derivatives. Method ‘lm’ supports only
‘linear’ loss.
f_scale : float, optional
Value of soft margin between inlier and outlier residuals, default is 1.0. The
loss function is evaluated as follows rho_(f**2) = C**2 * rho(f**2 /
C**2), where C is f_scale, and rho is determined by loss parameter. This
parameter has no effect with loss='linear', but for other loss values it is of
crucial importance.
max_nfev : None or int, optional
Maximum number of function evaluations before the termination. If None
(default), the value is chosen automatically:
For ‘trf’ and ‘dogbox’ : 100 * n.
For ‘lm’ : 100 * n if jac is callable and 100 * n * (n + 1) otherwise (because ‘lm’
counts function calls in Jacobian estimation).
diff_step : None or array_like, optional
Determines the relative step size for the finite difference approximation of the
Jacobian. The actual step is computed as x * diff_step. If None (default),
then diff_step is taken to be a conventional “optimal” power of machine
epsilon for the finite difference scheme used [NR].
tr_solver : {None, ‘exact’, ‘lsmr’}, optional
Method for solving trust-region subproblems, relevant only for ‘trf’ and
‘dogbox’ methods.
‘exact’ is suitable for not very large problems with dense Jacobian matrices.
The computational complexity per iteration is comparable to a singular
value decomposition of the Jacobian matrix.
‘lsmr’ is suitable for problems with sparse and large Jacobian matrices. It
uses the iterative procedure scipy.sparse.linalg.lsmr for finding a
solution of a linear least-squares problem and only requires matrix-vector
product evaluations.
If None (default), the solver is chosen based on the type of Jacobian returned
on the first iteration.
tr_options : dict, optional
Keyword options passed to trust-region solver.
tr_solver='exact': tr_options are ignored.
tr_solver='lsmr': options for scipy.sparse.linalg.lsmr. Additionally,
method='trf' supports ‘regularize’ option (bool, default is True), which adds
a regularization term to the normal equation, which improves convergence
if the Jacobian is rank-deficient [Byrd] (eq. 3.4).
jac_sparsity : {None, array_like, sparse matrix}, optional
Defines the sparsity structure of the Jacobian matrix for finite difference
estimation, its shape must be (m, n). If the Jacobian has only few non-zero
elements in each row, providing the sparsity structure will greatly speed up the
computations [Curtis]. A zero entry means that a corresponding element in
the Jacobian is identically zero. If provided, forces the use of ‘lsmr’ trust-region
solver. If None (default), then dense differencing will be used. Has no effect for
‘lm’ method.
verbose : {0, 1, 2}, optional
Level of algorithm’s verbosity:
0 (default) : work silently.
1 : display a termination report.
2 : display progress during iterations (not supported by ‘lm’ method).
args, kwargs : tuple and dict, optional
Additional arguments passed to fun and jac. Both empty by default. The calling
signature is fun(x, *args, **kwargs) and the same for jac.
Returns:
result : OptimizeResult
OptimizeResult with the following fields defined:
x : ndarray, shape (n,)
Solution found.
cost : float
Value of the cost function at the solution.
fun : ndarray, shape (m,)
Vector of residuals at the solution.
jac : ndarray, sparse matrix or LinearOperator, shape (m, n)
Modified Jacobian matrix at the solution, in the sense that J^T J is a GaussNewton approximation of the Hessian of the cost function. The type is the
same as the one used by the algorithm.
grad : ndarray, shape (m,)
Gradient of the cost function at the solution.
optimality : float
First-order optimality measure. In unconstrained problems, it is always
the uniform norm of the gradient. In constrained problems, it is the
quantity which was compared with gtol during iterations.
active_mask : ndarray of int, shape (n,)
Each component shows whether a corresponding constraint is active (that
is, whether a variable is at the bound):
0 : a constraint is not active.
-1 : a lower bound is active.
1 : an upper bound is active.
Might be somewhat arbitrary for ‘trf’ method as it generates a sequence
of strictly feasible iterates and active_mask is determined within a
tolerance threshold.
nfev : int
Number of function evaluations done. Methods ‘trf’ and ‘dogbox’ do not
count function calls for numerical Jacobian approximation, as opposed to
‘lm’ method.
njev : int or None
Number of Jacobian evaluations done. If numerical Jacobian
approximation is used in ‘lm’ method, it is set to None.
status : int
The reason for algorithm termination:
-1 : improper input parameters status returned from MINPACK.
0 : the maximum number of function evaluations is exceeded.
1 : gtol termination condition is satisfied.
2 : ftol termination condition is satisfied.
3 : xtol termination condition is satisfied.
4 : Both ftol and xtol termination conditions are satisfied.
message : str
Verbal description of the termination reason.
success : bool
True if one of the convergence criteria is satisfied (status > 0).
 See also
leastsq
A legacy wrapper for the MINPACK implementation of the Levenberg-Marquadt
algorithm.
curve_fit
Least-squares minimization applied to a curve-fitting problem.
Notes
Method ‘lm’ (Levenberg-Marquardt) calls a wrapper over least-squares algorithms implemented in
MINPACK (lmder, lmdif). It runs the Levenberg-Marquardt algorithm formulated as a trust-region
type algorithm. The implementation is based on paper [JJMore], it is very robust and efficient with
a lot of smart tricks. It should be your first choice for unconstrained problems. Note that it doesn’t
support bounds. Also, it doesn’t work when m < n.
Method ‘trf’ (Trust Region Reflective) is motivated by the process of solving a system of equations,
which constitute the first-order optimality condition for a bound-constrained minimization
problem as formulated in [STIR]. The algorithm iteratively solves trust-region subproblems
augmented by a special diagonal quadratic term and with trust-region shape determined by the
distance from the bounds and the direction of the gradient. This enhancements help to avoid
making steps directly into bounds and efficiently explore the whole space of variables. To further
improve convergence, the algorithm considers search directions reflected from the bounds. To
obey theoretical requirements, the algorithm keeps iterates strictly feasible. With dense Jacobians
trust-region subproblems are solved by an exact method very similar to the one described in
[JJMore] (and implemented in MINPACK). The difference from the MINPACK implementation is that
a singular value decomposition of a Jacobian matrix is done once per iteration, instead of a QR
decomposition and series of Givens rotation eliminations. For large sparse Jacobians a 2-D
subspace approach of solving trust-region subproblems is used [STIR], [Byrd]. The subspace is
spanned by a scaled gradient and an approximate Gauss-Newton solution delivered by
scipy.sparse.linalg.lsmr. When no constraints are imposed the algorithm is very similar to
MINPACK and has generally comparable performance. The algorithm works quite robust in
unbounded and bounded problems, thus it is chosen as a default algorithm.
Method ‘dogbox’ operates in a trust-region framework, but considers rectangular trust regions as
opposed to conventional ellipsoids [Voglis]. The intersection of a current trust region and initial
bounds is again rectangular, so on each iteration a quadratic minimization problem subject to
bound constraints is solved approximately by Powell’s dogleg method [NumOpt]. The required
Gauss-Newton step can be computed exactly for dense Jacobians or approximately by
scipy.sparse.linalg.lsmr for large sparse Jacobians. The algorithm is likely to exhibit slow
convergence when the rank of Jacobian is less than the number of variables. The algorithm often
outperforms ‘trf’ in bounded problems with a small number of variables.
Robust loss functions are implemented as described in [BA]. The idea is to modify a residual
vector and a Jacobian matrix on each iteration such that computed gradient and Gauss-Newton
Hessian approximation match the true gradient and Hessian approximation of the cost function.
Then the algorithm proceeds in a normal way, i.e., robust loss functions are implemented as a
simple wrapper over standard least-squares algorithms.
New in version 0.17.0.
References
STIR(1,2,3) M. A. Branch, T. F. Coleman, and Y. Li, “A Subspace, Interior, and Conjugate Gradient
Method for Large-Scale Bound-Constrained Minimization Problems,” SIAM Journal on
Scientific Computing, Vol. 21, Number 1, pp 1-23, 1999.
[NR] William H. Press et. al., “Numerical Recipes. The Art of Scientific Computing. 3rd edition”, Sec.
5.7.
Byrd(1,2) R. H. Byrd, R. B. Schnabel and G. A. Shultz, “Approximate solution of the trust region
problem by minimization over two-dimensional subspaces”, Math. Programming, 40, pp. 247263, 1988.
[Curtis] A. Curtis, M. J. D. Powell, and J. Reid, “On the estimation of sparse Jacobian matrices”,
Journal of the Institute of Mathematics and its Applications, 13, pp. 117-120, 1974.
JJMore(1,2,3) J. J. More, “The Levenberg-Marquardt Algorithm: Implementation and Theory,”
Numerical Analysis, ed. G. A. Watson, Lecture Notes in Mathematics 630, Springer Verlag, pp.
105-116, 1977.
[Voglis]
C. Voglis and I. E. Lagaris, “A Rectangular Trust Region Dogleg Approach for Unconstrained and
Bound Constrained Nonlinear Optimization”, WSEAS International Conference on Applied
Mathematics, Corfu, Greece, 2004.
[NumOpt] J. Nocedal and S. J. Wright, “Numerical optimization, 2nd edition”, Chapter 4.
[BA] B. Triggs et. al., “Bundle Adjustment - A Modern Synthesis”, Proceedings of the International
Workshop on Vision Algorithms: Theory and Practice, pp. 298-372, 1999.
Examples
In this example we find a minimum of the Rosenbrock function without bounds on independent
variables.
>>> def fun_rosenbrock(x):
...
return np.array([10 * (x[1] - x[0]**2), (1 - x[0])])
Notice that we only provide the vector of the residuals. The algorithm constructs the cost function
as a sum of squares of the residuals, which gives the Rosenbrock function. The exact minimum is
at x = [1.0, 1.0].
>>> from scipy.optimize import least_squares
>>> x0_rosenbrock = np.array([2, 2])
>>> res_1 = least_squares(fun_rosenbrock, x0_rosenbrock)
>>> res_1.x
array([ 1.,
1.])
>>> res_1.cost
9.8669242910846867e-30
>>> res_1.optimality
8.8928864934219529e-14
We now constrain the variables, in such a way that the previous solution becomes infeasible.
Specifically, we require that x[1] >= 1.5, and x[0] left unconstrained. To this end, we specify the
bounds parameter to least_squares in the form bounds=([-np.inf, 1.5], np.inf).
We also provide the analytic Jacobian:
>>> def jac_rosenbrock(x):
...
return np.array([
...
[-20 * x[0], 10],
...
[-1, 0]])
Putting this all together, we see that the new solution lies on the bound:
>>> res_2 = least_squares(fun_rosenbrock, x0_rosenbrock, jac_rosenbrock,
...
bounds=([-np.inf, 1.5], np.inf))
>>> res_2.x
array([ 1.22437075,
1.5
])
>>> res_2.cost
0.025213093946805685
>>> res_2.optimality
1.5885401433157753e-07
Now we solve a system of equations (i.e., the cost function should be zero at a minimum) for a
Broyden tridiagonal vector-valued function of 100000 variables:
>>> def fun_broyden(x):
...
f = (3 - x) * x + 1
...
f[1:] -= x[:-1]
...
f[:-1] -= 2 * x[1:]
...
return f
The corresponding Jacobian matrix is sparse. We tell the algorithm to estimate it by finite
differences and provide the sparsity structure of Jacobian to significantly speed up this process.
>>> from scipy.sparse import lil_matrix
>>> def sparsity_broyden(n):
...
sparsity = lil_matrix((n, n), dtype=int)
...
i = np.arange(n)
...
sparsity[i, i] = 1
...
i = np.arange(1, n)
...
sparsity[i, i - 1] = 1
...
i = np.arange(n - 1)
...
sparsity[i, i + 1] = 1
...
return sparsity
...
>>> n = 100000
>>> x0_broyden = -np.ones(n)
...
>>> res_3 = least_squares(fun_broyden, x0_broyden,
...
jac_sparsity=sparsity_broyden(n))
>>> res_3.cost
4.5687069299604613e-23
>>> res_3.optimality
1.1650454296851518e-11
Let’s also solve a curve fitting problem using robust loss function to take care of outliers in the
data. Define the model function as y = a + b * exp(c * t), where t is a predictor variable, y is
an observation and a, b, c are parameters to estimate.
First, define the function which generates the data with noise and outliers, define the model
parameters, and generate data:
>>> from numpy.random import default_rng
>>> rng = default_rng()
>>> def gen_data(t, a, b, c, noise=0., n_outliers=0, seed=None):
...
rng = default_rng(seed)
...
...
y = a + b * np.exp(t * c)
...
...
error = noise * rng.standard_normal(t.size)
...
outliers = rng.integers(0, t.size, n_outliers)
...
error[outliers] *= 10
...
...
return y + error
...
>>> a = 0.5
>>> b = 2.0
>>> c = -1
>>> t_min = 0
>>> t_max = 10
>>> n_points = 15
...
>>> t_train = np.linspace(t_min, t_max, n_points)
>>> y_train = gen_data(t_train, a, b, c, noise=0.1, n_outliers=3)
Define function for computing residuals and initial estimate of parameters.
>>> def fun(x, t, y):
...
return x[0] + x[1] * np.exp(x[2] * t) - y
...
>>> x0 = np.array([1.0, 1.0, 0.0])
Compute a standard least-squares solution:
>>> res_lsq = least_squares(fun, x0, args=(t_train, y_train))
Now compute two solutions with two different robust loss functions. The parameter f_scale is set
to 0.1, meaning that inlier residuals should not significantly exceed 0.1 (the noise level used).
>>> res_soft_l1 = least_squares(fun, x0, loss='soft_l1', f_scale=0.1,
...
args=(t_train, y_train))
>>> res_log = least_squares(fun, x0, loss='cauchy', f_scale=0.1,
...
args=(t_train, y_train))
And, finally, plot all the curves. We see that by selecting an appropriate loss we can get estimates
close to optimal even in the presence of strong outliers. But keep in mind that generally it is
recommended to try ‘soft_l1’ or ‘huber’ losses first (if at all necessary) as the other two options
may cause difficulties in optimization process.
>>> t_test = np.linspace(t_min, t_max, n_points * 10)
>>> y_true = gen_data(t_test, a, b, c)
>>> y_lsq = gen_data(t_test, *res_lsq.x)
>>> y_soft_l1 = gen_data(t_test, *res_soft_l1.x)
>>> y_log = gen_data(t_test, *res_log.x)
...
>>> import matplotlib.pyplot as plt
>>> plt.plot(t_train, y_train, 'o')
>>> plt.plot(t_test, y_true, 'k', linewidth=2, label='true')
>>> plt.plot(t_test, y_lsq, label='linear loss')
>>> plt.plot(t_test, y_soft_l1, label='soft_l1 loss')
>>> plt.plot(t_test, y_log, label='cauchy loss')
>>> plt.xlabel("t")
>>> plt.ylabel("y")
>>> plt.legend()
>>> plt.show()
In the next example, we show how complex-valued residual functions of complex variables can be
optimized with least_squares(). Consider the following function:
>>> def f(z):
...
return z - (0.5 + 0.5j)
We wrap it into a function of real variables that returns real residuals by simply handling the real
and imaginary parts as independent variables:
>>> def f_wrap(x):
...
fx = f(x[0] + 1j*x[1])
...
return np.array([fx.real, fx.imag])
Thus, instead of the original m-D complex function of n complex variables we optimize a 2m-D real
function of 2n real variables:
>>> from scipy.optimize import least_squares
>>> res_wrapped = least_squares(f_wrap, (0.1, 0.1), bounds=([0, 0], [1, 1]))
>>> z = res_wrapped.x[0] + res_wrapped.x[1]*1j
>>> z
(0.49999999999925893+0.49999999999925893j)
<< scipy.optimize.dual_annealing
© Copyright 2008-2022, The SciPy community.
Created using Sphinx 4.0.2.
scipy.optimize.nnls >>
 Search the docs ...
Clustering package
(
scipy.optimize.minimize
scipy.optimize.minimize(fun, x0, args=(), method=None, jac=None, hess=None,
)
( scipy.cluster )
hessp=None, bounds=None, constraints=(), tol=None, callback=None, options=None) ¶
K-means clustering and vector
quantization
(
( scipy.cluster.vq )
(
Hierarchical clustering
Minimization of scalar function of one or more variables.
)
Parameters: fun : callable
( scipy.cluster.hierarchy )
)
Constants
(
Discrete Fourier transforms
)
(
)
( scipy.fftpack )
(
x0 : ndarray, shape (n,)
Initial guess. Array of real elements of size (n,), where n is the number of
independent variables.
args : tuple, optional
(
Integration and ODEs
Extra arguments passed to the objective function and its derivatives (fun, jac
)
( scipy.integrate )
Interpolation
(
and hess functions).
method : str or callable, optional
)
( scipy.interpolate )
(
Input and output ( scipy.io )
)
Linear algebra
fun(x, *args) -> float
parameters needed to completely specify the function.
Legacy discrete Fourier
transforms
The objective function to be minimized.
where x is an 1-D array with shape (n,) and args is a tuple of the fixed
)
( scipy.constants )
( scipy.fft )
Type of solver. Should be one of
‘Nelder-Mead’ (see here)
‘Powell’ (see here)
(
‘CG’ (see here)
( scipy.linalg )
Low-level BLAS functions
(
( scipy.linalg.blas )
‘BFGS’ (see here)
)
‘Newton-CG’ (see here)
‘L-BFGS-B’ (see here)
(
Low-level LAPACK functions
)
( scipy.linalg.lapack )
‘TNC’ (see here)
‘COBYLA’ (see here)
BLAS Functions for Cython
‘SLSQP’ (see here)
LAPACK functions for Cython
‘trust-constr’(see here)
‘dogleg’ (see here)
Interpolative matrix
decomposition
[source]
(
‘trust-ncg’ (see here)
( scipy.linalg.interpolative )
)
‘trust-exact’ (see here)
‘trust-krylov’ (see here)
(
Miscellaneous routines
( scipy.misc )
)
custom - a callable object (added in version 0.14.0), see below for
description.
Multidimensional image
processing
(
( scipy.ndimage )
If not given, chosen to be one of BFGS, L-BFGS-B, SLSQP, depending on
whether or not the problem has constraints or bounds.
)
Orthogonal distance regression
( scipy.odr )
)
Optimization and root finding
( scipy.optimize )
)
BFGS-B, TNC, SLSQP, dogleg, trust-ncg, trust-krylov, trust-exact and trustconstr. If it is a callable, it should be a function that returns the gradient
where x is an array with shape (n,) and args is a tuple with the fixed
(
parameters. If jac is a Boolean and is True, fun is assumed to return a tuple
(f, g) containing the objective function and the gradient. Methods ‘Newton-
( scipy.sparse )
Sparse linear algebra
(
Method for computing the gradient vector. Only for CG, BFGS, Newton-CG, L-
jac(x, *args) -> array_like, shape (n,)
(
( scipy.signal )
Sparse matrices
jac : {callable, ‘2-point’, ‘3-point’, ‘cs’, bool}, optional
vector:
Cython optimize zeros API
Signal processing
(
(
( scipy.sparse.linalg )
)
CG’, ‘trust-ncg’, ‘dogleg’, ‘trust-exact’, and ‘trust-krylov’ require that either a
callable be supplied, or that fun return the objective and gradient. If None or
Compressed sparse graph
False, the gradient will be estimated using 2-point finite difference estimation
routines
with an absolute step size. Alternatively, the keywords {‘2-point’, ‘3-point’, ‘cs’}
(
( scipy.sparse.csgraph )
)
can be used to select a finite difference scheme for numerical estimation of
the gradient with a relative step size. These finite difference schemes obey any
specified bounds.
hess : {callable, ‘2-point’, ‘3-point’, ‘cs’, HessianUpdateStrategy}, optional
Method for computing the Hessian matrix. Only for Newton-CG, dogleg, trustncg, trust-krylov, trust-exact and trust-constr. If it is callable, it should return
the Hessian matrix:
hess(x, *args) -> {LinearOperator, spmatrix, array}, (n, n)
where x is a (n,) ndarray and args is a tuple with the fixed parameters. The
keywords {‘2-point’, ‘3-point’, ‘cs’} can also be used to select a finite difference
scheme for numerical estimation of the hessian. Alternatively, objects
implementing the HessianUpdateStrategy interface can be used to
approximate the Hessian. Available quasi-Newton methods implementing this
interface are:
BFGS;
SR1.
Not all of the options are available for each of the methods; for availability
refer to the notes.
hessp : callable, optional
Hessian of objective function times an arbitrary vector p. Only for Newton-CG,
trust-ncg, trust-krylov, trust-constr. Only one of hessp or hess needs to be
given. If hess is provided, then hessp will be ignored. hessp must compute the
Hessian times an arbitrary vector:
hessp(x, p, *args) ->
ndarray shape (n,)
where x is a (n,) ndarray, p is an arbitrary vector with dimension (n,) and args
is a tuple with the fixed parameters.
bounds : sequence or Bounds, optional
Bounds on variables for Nelder-Mead, L-BFGS-B, TNC, SLSQP, Powell, and
trust-constr methods. There are two ways to specify the bounds:
1. Instance of Bounds class.
2. Sequence of (min, max) pairs for each element in x. None is used to
specify no bound.
constraints : {Constraint, dict} or List of {Constraint, dict}, optional
Constraints definition. Only for COBYLA, SLSQP and trust-constr.
Constraints for ‘trust-constr’ are defined as a single object or a list of objects
specifying constraints to the optimization problem. Available constraints are:
LinearConstraint
NonlinearConstraint
Constraints for COBYLA, SLSQP are defined as a list of dictionaries. Each
dictionary with fields:
type : str
Constraint type: ‘eq’ for equality, ‘ineq’ for inequality.
fun : callable
The function defining the constraint.
jac : callable, optional
The Jacobian of fun (only for SLSQP).
args : sequence, optional
Extra arguments to be passed to the function and Jacobian.
Equality constraint means that the constraint function result is to be zero
whereas inequality means that it is to be non-negative. Note that COBYLA only
supports inequality constraints.
tol : float, optional
Tolerance for termination. When tol is specified, the selected minimization
algorithm sets some relevant solver-specific tolerance(s) equal to tol. For
detailed control, use solver-specific options.
options : dict, optional
A dictionary of solver options. All methods accept the following generic
options:
maxiter : int
Maximum number of iterations to perform. Depending on the method
each iteration may use several function evaluations.
disp : bool
Set to True to print convergence messages.
For method-specific options, see show_options.
callback : callable, optional
Called after each iteration. For ‘trust-constr’ it is a callable with the signature:
callback(xk, OptimizeResult state) -> bool
where xk is the current parameter vector. and state is an OptimizeResult
object, with the same fields as the ones from the return. If callback returns
True the algorithm execution is terminated. For all the other methods, the
signature is:
callback(xk)
where xk is the current parameter vector.
Returns:
res : OptimizeResult
The optimization result represented as a OptimizeResult object. Important
attributes are: x the solution array, success a Boolean flag indicating if the
optimizer exited successfully and message which describes the cause of the
termination. See OptimizeResult for a description of other attributes.
 See also
minimize_scalar
Interface to minimization algorithms for scalar univariate functions
show_options
Additional options accepted by the solvers
Notes
This section describes the available solvers that can be selected by the ‘method’ parameter. The
default method is BFGS.
Unconstrained minimization
Method CG uses a nonlinear conjugate gradient algorithm by Polak and Ribiere, a variant of the
Fletcher-Reeves method described in [5] pp.120-122. Only the first derivatives are used.
Method BFGS uses the quasi-Newton method of Broyden, Fletcher, Goldfarb, and Shanno (BFGS)
[5] pp. 136. It uses the first derivatives only. BFGS has proven good performance even for nonsmooth optimizations. This method also returns an approximation of the Hessian inverse, stored
as hess_inv in the OptimizeResult object.
Method Newton-CG uses a Newton-CG algorithm [5] pp. 168 (also known as the truncated Newton
method). It uses a CG method to the compute the search direction. See also TNC method for a
box-constrained minimization with a similar algorithm. Suitable for large-scale problems.
Method dogleg uses the dog-leg trust-region algorithm [5] for unconstrained minimization. This
algorithm requires the gradient and Hessian; furthermore the Hessian is required to be positive
definite.
Method trust-ncg uses the Newton conjugate gradient trust-region algorithm [5] for unconstrained
minimization. This algorithm requires the gradient and either the Hessian or a function that
computes the product of the Hessian with a given vector. Suitable for large-scale problems.
Method trust-krylov uses the Newton GLTR trust-region algorithm [14], [15] for unconstrained
minimization. This algorithm requires the gradient and either the Hessian or a function that
computes the product of the Hessian with a given vector. Suitable for large-scale problems. On
indefinite problems it requires usually less iterations than the trust-ncg method and is
recommended for medium and large-scale problems.
Method trust-exact is a trust-region method for unconstrained minimization in which quadratic
subproblems are solved almost exactly [13]. This algorithm requires the gradient and the Hessian
(which is not required to be positive definite). It is, in many situations, the Newton method to
converge in fewer iterations and the most recommended for small and medium-size problems.
Bound-Constrained minimization
Method Nelder-Mead uses the Simplex algorithm [1], [2]. This algorithm is robust in many
applications. However, if numerical computation of derivative can be trusted, other algorithms
using the first and/or second derivatives information might be preferred for their better
performance in general.
Method L-BFGS-B uses the L-BFGS-B algorithm [6], [7] for bound constrained minimization.
Method Powell is a modification of Powell’s method [3], [4] which is a conjugate direction method.
It performs sequential one-dimensional minimizations along each vector of the directions set
(direc field in options and info), which is updated at each iteration of the main minimization loop.
The function need not be differentiable, and no derivatives are taken. If bounds are not provided,
then an unbounded line search will be used. If bounds are provided and the initial guess is within
the bounds, then every function evaluation throughout the minimization procedure will be within
the bounds. If bounds are provided, the initial guess is outside the bounds, and direc is full rank
(default has full rank), then some function evaluations during the first iteration may be outside the
bounds, but every function evaluation after the first iteration will be within the bounds. If direc is
not full rank, then some parameters may not be optimized and the solution is not guaranteed to
be within the bounds.
Method TNC uses a truncated Newton algorithm [5], [8] to minimize a function with variables
subject to bounds. This algorithm uses gradient information; it is also called Newton ConjugateGradient. It differs from the Newton-CG method described above as it wraps a C implementation
and allows each variable to be given upper and lower bounds.
Constrained Minimization
Method COBYLA uses the Constrained Optimization BY Linear Approximation (COBYLA) method
[9], [10], [11]. The algorithm is based on linear approximations to the objective function and each
constraint. The method wraps a FORTRAN implementation of the algorithm. The constraints
functions ‘fun’ may return either a single number or an array or list of numbers.
Method SLSQP uses Sequential Least SQuares Programming to minimize a function of several
variables with any combination of bounds, equality and inequality constraints. The method wraps
the SLSQP Optimization subroutine originally implemented by Dieter Kraft [12]. Note that the
wrapper handles infinite values in bounds by converting them into large floating values.
Method trust-constr is a trust-region algorithm for constrained optimization. It swiches between
two implementations depending on the problem definition. It is the most versatile constrained
minimization algorithm implemented in SciPy and the most appropriate for large-scale problems.
For equality constrained problems it is an implementation of Byrd-Omojokun Trust-Region SQP
method described in [17] and in [5], p. 549. When inequality constraints are imposed as well, it
swiches to the trust-region interior point method described in [16]. This interior point algorithm, in
turn, solves inequality constraints by introducing slack variables and solving a sequence of
equality-constrained barrier problems for progressively smaller values of the barrier parameter.
The previously described equality constrained SQP method is used to solve the subproblems with
increasing levels of accuracy as the iterate gets closer to a solution.
Finite-Difference Options
For Method trust-constr the gradient and the Hessian may be approximated using three finitedifference schemes: {‘2-point’, ‘3-point’, ‘cs’}. The scheme ‘cs’ is, potentially, the most accurate but it
requires the function to correctly handle complex inputs and to be differentiable in the complex
plane. The scheme ‘3-point’ is more accurate than ‘2-point’ but requires twice as many operations.
If the gradient is estimated via finite-differences the Hessian must be estimated using one of the
quasi-Newton strategies.
Method specific options for the hess keyword
method/Hess
None
callable
‘2-point/’3-point’/’cs’
HUS
Newton-CG
x
(n, n) LO
x
x
dogleg
(n, n)
trust-ncg
(n, n)
x
x
trust-krylov
(n, n)
x
x
trust-exact
(n, n)
x
x
trust-constr
x
(n, n) LO sp
where LO=LinearOperator, sp=Sparse matrix, HUS=HessianUpdateStrategy
Custom minimizers
It may be useful to pass a custom minimization method, for example when using a frontend to
this method such as scipy.optimize.basinhopping or a different library. You can simply pass a
callable as the method parameter.
The callable is called as method(fun, x0, args, **kwargs, **options) where kwargs
corresponds to any other parameters passed to minimize (such as callback, hess, etc.), except the
options dict, which has its contents also passed as method parameters pair by pair. Also, if jac has
been passed as a bool type, jac and fun are mangled so that fun returns just the function values
and jac is converted to a function returning the Jacobian. The method shall return an
OptimizeResult object.
The provided method callable must be able to accept (and possibly ignore) arbitrary parameters;
the set of parameters accepted by minimize may expand in future versions and then these
parameters will be passed to the method. You can find an example in the scipy.optimize tutorial.
New in version 0.11.0.
References
[1] Nelder, J A, and R Mead. 1965. A Simplex Method for Function Minimization. The Computer
Journal 7: 308-13.
[2] Wright M H. 1996. Direct search methods: Once scorned, now respectable, in Numerical
Analysis 1995: Proceedings of the 1995 Dundee Biennial Conference in Numerical Analysis
(Eds. D F Griffiths and G A Watson). Addison Wesley Longman, Harlow, UK. 191-208.
[3] Powell, M J D. 1964. An efficient method for finding the minimum of a function of several
variables without calculating derivatives. The Computer Journal 7: 155-162.
[4] Press W, S A Teukolsky, W T Vetterling and B P Flannery. Numerical Recipes (any edition),
Cambridge University Press.
5(1,2,3,4,5,6,7,8) Nocedal, J, and S J Wright. 2006. Numerical Optimization. Springer New York.
[6]
Byrd, R H and P Lu and J. Nocedal. 1995. A Limited Memory Algorithm for Bound Constrained
Optimization. SIAM Journal on Scientific and Statistical Computing 16 (5): 1190-1208.
[7] Zhu, C and R H Byrd and J Nocedal. 1997. L-BFGS-B: Algorithm 778: L-BFGS-B, FORTRAN
routines for large scale bound constrained optimization. ACM Transactions on Mathematical
Software 23 (4): 550-560.
[8] Nash, S G. Newton-Type Minimization Via the Lanczos Method. 1984. SIAM Journal of
Numerical Analysis 21: 770-778.
[9] Powell, M J D. A direct search optimization method that models the objective and constraint
functions by linear interpolation. 1994. Advances in Optimization and Numerical Analysis, eds.
S. Gomez and J-P Hennart, Kluwer Academic (Dordrecht), 51-67.
[10] Powell M J D. Direct search algorithms for optimization calculations. 1998. Acta Numerica 7:
287-336.
[11] Powell M J D. A view of algorithms for optimization without derivatives. 2007.Cambridge
University Technical Report DAMTP 2007/NA03
[12] Kraft, D. A software package for sequential quadratic programming. 1988. Tech. Rep. DFVLRFB 88-28, DLR German Aerospace Center – Institute for Flight Mechanics, Koln, Germany.
[13] Conn, A. R., Gould, N. I., and Toint, P. L. Trust region methods. 2000. Siam. pp. 169-200.
[14] F. Lenders, C. Kirches, A. Potschka: “trlib: A vector-free implementation of the GLTR method
for iterative solution of the trust region problem”, arXiv:1611.04718
[15] N. Gould, S. Lucidi, M. Roma, P. Toint: “Solving the Trust-Region Subproblem using the Lanczos
Method”, SIAM J. Optim., 9(2), 504–525, (1999).
[16] Byrd, Richard H., Mary E. Hribar, and Jorge Nocedal. 1999. An interior point algorithm for
large-scale nonlinear programming. SIAM Journal on Optimization 9.4: 877-900.
[17] Lalee, Marucha, Jorge Nocedal, and Todd Plantega. 1998. On the implementation of an
algorithm for large-scale equality constrained optimization. SIAM Journal on Optimization 8.3:
682-706.
Examples
Let us consider the problem of minimizing the Rosenbrock function. This function (and its
respective derivatives) is implemented in rosen (resp. rosen_der, rosen_hess) in the
scipy.optimize.
>>> from scipy.optimize import minimize, rosen, rosen_der
A simple application of the Nelder-Mead method is:
>>> x0 = [1.3, 0.7, 0.8, 1.9, 1.2]
>>> res = minimize(rosen, x0, method='Nelder-Mead', tol=1e-6)
>>> res.x
array([ 1.,
1.,
1.,
1.,
1.])
Now using the BFGS algorithm, using the first derivative and a few options:
>>> res = minimize(rosen, x0, method='BFGS', jac=rosen_der,
...
options={'gtol': 1e-6, 'disp': True})
Optimization terminated successfully.
Current function value: 0.000000
Iterations: 26
Function evaluations: 31
Gradient evaluations: 31
>>> res.x
array([ 1.,
1.,
1.,
1.,
1.])
>>> print(res.message)
Optimization terminated successfully.
>>> res.hess_inv
array([[ 0.00749589,
0.01255155,
0.02396251,
0.04750988,
0.09495377],
[ 0.01255155,
0.02510441,
0.04794055,
0.09502834,
0.18996269],
[ 0.02396251,
0.04794055,
0.09631614,
0.19092151,
0.38165151],
[ 0.04750988,
0.09502834,
0.19092151,
0.38341252,
0.7664427 ],
[ 0.09495377,
0.18996269,
0.38165151,
0.7664427,
1.53713523]])
# may
vary
Next, consider a minimization problem with several constraints (namely Example 16.4 from [5]).
The objective function is:
>>> fun = lambda x: (x[0] - 1)**2 + (x[1] - 2.5)**2
There are three constraints defined as:
>>> cons = ({'type': 'ineq', 'fun': lambda x:
x[0] - 2 * x[1] + 2},
...
{'type': 'ineq', 'fun': lambda x: -x[0] - 2 * x[1] + 6},
...
{'type': 'ineq', 'fun': lambda x: -x[0] + 2 * x[1] + 2})
And variables must be positive, hence the following bounds:
>>> bnds = ((0, None), (0, None))
The optimization problem is solved using the SLSQP method as:
>>> res = minimize(fun, (2, 0), method='SLSQP', bounds=bnds,
...
constraints=cons)
It should converge to the theoretical solution (1.4 ,1.7).
<< minimize_scalar(method=’golden’)
© Copyright 2008-2022, The SciPy community.
Created using Sphinx 4.0.2.
minimize(method=’Nelder-Mead’) >>
 Search the docs ...
Clustering package
(
scipy.optimize.NonlinearConstraint
class scipy.optimize.NonlinearConstraint(fun, lb, ub, jac='2-point', hess=
)
( scipy.cluster )
<scipy.optimize._hessian_update_strategy.BFGS object>, keep_feasible=False,
K-means clustering and vector
quantization
(
( scipy.cluster.vq )
(
Hierarchical clustering
finite_diff_rel_step=None, finite_diff_jac_sparsity=None)
)
Nonlinear constraint on the variables.
( scipy.cluster.hierarchy )
)
Constants
[source]
The constraint has the general inequality form:
lb <= fun(x) <= ub
(
)
( scipy.constants )
Discrete Fourier transforms
)
( scipy.fft )
Here the vector of independent variables x is passed as ndarray of shape (n,) and fun returns a
(
vector with m components.
Legacy discrete Fourier
It is possible to use equal bounds to represent an equality constraint or infinite bounds to
transforms
represent a one-sided constraint.
(
)
( scipy.fftpack )
(
Integration and ODEs
)
( scipy.integrate )
Interpolation
Parameters: fun : callable
(
The function defining the constraint. The signature is fun(x) -> array_like,
shape (m,).
)
( scipy.interpolate )
(
lb, ub : array_like
Input and output ( scipy.io )
)
Linear algebra
Lower and upper bounds on the constraint. Each array must have the shape
(m,) or be a scalar, in the latter case a bound will be the same for all
(
components of the constraint. Use np.inf with an appropriate sign to specify
a one-sided constraint. Set components of lb and ub equal to represent an
( scipy.linalg )
Low-level BLAS functions
(
( scipy.linalg.blas )
)
equality constraint. Note that you can mix constraints of different types:
interval, one-sided or equality, by setting different components of lb and ub as
(
Low-level LAPACK functions
)
( scipy.linalg.lapack )
necessary.
jac : {callable, ‘2-point’, ‘3-point’, ‘cs’}, optional
BLAS Functions for Cython
Method of computing the Jacobian matrix (an m-by-n matrix, where element (i,
LAPACK functions for Cython
j) is the partial derivative of f[i] with respect to x[j]). The keywords {‘2-point’, ‘3-
Interpolative matrix
point’, ‘cs’} select a finite difference scheme for the numerical estimation. A
decomposition
callable must have the following signature: jac(x) -> {ndarray, sparse
(
( scipy.linalg.interpolative )
)
( scipy.misc )
hess : {callable, ‘2-point’, ‘3-point’, ‘cs’, HessianUpdateStrategy, None}, optional
(
Miscellaneous routines
)
Method for computing the Hessian matrix. The keywords {‘2-point’, ‘3-point’,
‘cs’} select a finite difference scheme for numerical estimation. Alternatively,
Multidimensional image
processing
(
( scipy.ndimage )
objects implementing HessianUpdateStrategy interface can be used to
)
approximate the Hessian. Currently available implementations are:
Orthogonal distance regression
( scipy.odr )
)
Optimization and root finding
( scipy.optimize )
)
(
keep_feasible : array_like of bool, optional
Whether to keep the constraint components feasible throughout iterations. A
( scipy.sparse )
(
( scipy.sparse.linalg )
single value set this property for all components. Default is False. Has no
)
effect for equality constraints.
finite_diff_rel_step: None or array_like, optional
Compressed sparse graph
routines
A callable must return the Hessian matrix of dot(fun, v) and must have the
Lagrange multipliers.
(
Sparse linear algebra
(
SR1
array_like}, shape (n, n). Here v is ndarray with shape (m,) containing
( scipy.signal )
Sparse matrices
BFGS (default option)
(
following signature: hess(x, v) -> {LinearOperator, sparse matrix,
Cython optimize zeros API
Signal processing
matrix}, shape (m, n). Default is ‘2-point’.
(
( scipy.sparse.csgraph )
)
Relative step size for the finite difference approximation. Default is None,
which will select a reasonable value automatically depending on a finite
difference scheme.
finite_diff_jac_sparsity: {None, array_like, sparse matrix}, optional
Defines the sparsity structure of the Jacobian matrix for finite difference
estimation, its shape must be (m, n). If the Jacobian has only few non-zero
elements in each row, providing the sparsity structure will greatly speed up the
computations. A zero entry means that a corresponding element in the
Jacobian is identically zero. If provided, forces the use of ‘lsmr’ trust-region
solver. If None (default) then dense differencing will be used.
Notes
Finite difference schemes {‘2-point’, ‘3-point’, ‘cs’} may be used for approximating either the
Jacobian or the Hessian. We, however, do not allow its use for approximating both simultaneously.
Hence whenever the Jacobian is estimated via finite-differences, we require the Hessian to be
estimated using one of the quasi-Newton strategies.
The scheme ‘cs’ is potentially the most accurate, but requires the function to correctly handles
complex inputs and be analytically continuable to the complex plane. The scheme ‘3-point’ is more
accurate than ‘2-point’ but requires twice as many operations.
Examples
Constrain x[0] < sin(x[1]) + 1.9
>>> from scipy.optimize import NonlinearConstraint
>>> con = lambda x: x[0] - np.sin(x[1])
>>> nlc = NonlinearConstraint(con, -np.inf, 1.9)
<< minimize(method=’trust-exact’)
© Copyright 2008-2022, The SciPy community.
Created using Sphinx 4.0.2.
scipy.optimize.LinearConstraint >>
 Search the docs ...
Clustering package
(
scipy.optimize.LinearConstraint
class scipy.optimize.LinearConstraint(A, lb, ub, keep_feasible=False)
)
( scipy.cluster )
Linear constraint on the variables.
K-means clustering and vector
quantization
(
( scipy.cluster.vq )
(
Hierarchical clustering
[source]
The constraint has the general inequality form:
)
lb <= A.dot(x) <= ub
( scipy.cluster.hierarchy )
)
Constants
Here the vector of independent variables x is passed as ndarray of shape (n,) and the matrix A has
(
shape (m, n).
)
( scipy.constants )
Discrete Fourier transforms
)
( scipy.fft )
(
It is possible to use equal bounds to represent an equality constraint or infinite bounds to
represent a one-sided constraint.
Legacy discrete Fourier
transforms
(
)
Parameters: A : {array_like, sparse matrix}, shape (m, n)
Matrix defining the constraint.
( scipy.fftpack )
(
Integration and ODEs
)
( scipy.integrate )
Interpolation
lb, ub : array_like
(
Lower and upper bounds on the constraint. Each array must have the shape
(m,) or be a scalar, in the latter case a bound will be the same for all
)
components of the constraint. Use np.inf with an appropriate sign to specify
( scipy.interpolate )
(
Input and output ( scipy.io )
)
Linear algebra
a one-sided constraint. Set components of lb and ub equal to represent an
equality constraint. Note that you can mix constraints of different types:
(
interval, one-sided or equality, by setting different components of lb and ub as
necessary.
( scipy.linalg )
Low-level BLAS functions
(
( scipy.linalg.blas )
)
keep_feasible : array_like of bool, optional
Whether to keep the constraint components feasible throughout iterations. A
(
Low-level LAPACK functions
)
( scipy.linalg.lapack )
single value set this property for all components. Default is False. Has no
effect for equality constraints.
BLAS Functions for Cython
<< scipy.optimize.NonlinearConstraint
LAPACK functions for Cython
Interpolative matrix
decomposition
(
( scipy.linalg.interpolative )
)
(
Miscellaneous routines
( scipy.misc )
)
Multidimensional image
processing
(
( scipy.ndimage )
)
Orthogonal distance regression
( scipy.odr )
)
Optimization and root finding
( scipy.optimize )
)
Cython optimize zeros API
Signal processing
(
( scipy.signal )
Sparse matrices
(
( scipy.sparse )
Sparse linear algebra
(
( scipy.sparse.linalg )
)
Compressed sparse graph
routines
(
( scipy.sparse.csgraph )
)
(
(
scipy.optimize.Bounds >>
© Copyright 2008-2022, The SciPy community.
Created using Sphinx 4.0.2.
 Search the docs ...
Clustering package
(
scipy.optimize.Bounds
Bounds constraint on the variables.
K-means clustering and vector
quantization
[source]
class scipy.optimize.Bounds(lb, ub, keep_feasible=False)
)
( scipy.cluster )
(
( scipy.cluster.vq )
(
Hierarchical clustering
The constraint has the general inequality form:
)
lb <= x <= ub
( scipy.cluster.hierarchy )
)
Constants
It is possible to use equal bounds to represent an equality constraint or infinite bounds to
(
represent a one-sided constraint.
)
( scipy.constants )
Discrete Fourier transforms
)
( scipy.fft )
(
Parameters: lb, ub : array_like
Lower and upper bounds on independent variables. Each array must have the
Legacy discrete Fourier
same size as x or be a scalar, in which case a bound will be the same for all the
transforms
variables. Set components of lb and ub equal to fix a variable. Use np.inf with
(
)
( scipy.fftpack )
an appropriate sign to disable bounds on all or some variables. Note that you
(
Integration and ODEs
)
( scipy.integrate )
Interpolation
can mix constraints of different types: interval, one-sided or equality, by
(
setting different components of lb and ub as necessary.
keep_feasible : array_like of bool, optional
)
( scipy.interpolate )
(
Whether to keep the constraint components feasible throughout iterations. A
Input and output ( scipy.io )
)
Linear algebra
effect for equality constraints.
(
( scipy.linalg )
Low-level BLAS functions
(
( scipy.linalg.blas )
<< scipy.optimize.LinearConstraint
)
(
Low-level LAPACK functions
)
( scipy.linalg.lapack )
BLAS Functions for Cython
LAPACK functions for Cython
Interpolative matrix
decomposition
(
( scipy.linalg.interpolative )
)
(
Miscellaneous routines
( scipy.misc )
)
Multidimensional image
processing
(
( scipy.ndimage )
)
Orthogonal distance regression
( scipy.odr )
)
Optimization and root finding
( scipy.optimize )
)
Cython optimize zeros API
Signal processing
(
( scipy.signal )
Sparse matrices
(
( scipy.sparse )
Sparse linear algebra
(
( scipy.sparse.linalg )
)
Compressed sparse graph
routines
single value set this property for all components. Default is False. Has no
(
( scipy.sparse.csgraph )
)
(
(
scipy.optimize.BFGS >>
© Copyright 2008-2022, The SciPy community.
Created using Sphinx 4.0.2.
 Search the docs ...
Clustering package
(
scipy.optimize.curve_fit
scipy.optimize.curve_fit(f, xdata, ydata, p0=None, sigma=None, absolute_sigma=False,
)
( scipy.cluster )
check_finite=True, bounds=(- inf, inf), method=None, jac=None, **kwargs)
K-means clustering and vector
quantization
(
( scipy.cluster.vq )
(
Hierarchical clustering
Use non-linear least squares to fit a function, f, to data.
)
Assumes ydata = f(xdata, *params) + eps.
( scipy.cluster.hierarchy )
)
Constants
Parameters: f : callable
The model function, f(x, …). It must take the independent variable as the first
(
argument and the parameters to fit as separate remaining arguments.
)
( scipy.constants )
Discrete Fourier transforms
)
( scipy.fft )
xdata : array_like or object
(
The independent variable where the data is measured. Should usually be an
M-length sequence or an (k,M)-shaped array for functions with k predictors,
Legacy discrete Fourier
transforms
[source]
(
)
( scipy.fftpack )
but can actually be any object.
ydata : array_like
(
Integration and ODEs
)
( scipy.integrate )
Interpolation
The dependent data, a length M array - nominally f(xdata, ...).
(
p0 : array_like, optional
Initial guess for the parameters (length N). If None, then the initial values will
)
( scipy.interpolate )
(
Input and output ( scipy.io )
)
Linear algebra
(
all be 1 (if the number of parameters for the function can be determined using
introspection, otherwise a ValueError is raised).
sigma : None or M-length sequence or MxM array, optional
Determines the uncertainty in ydata. If we define residuals as r = ydata -
( scipy.linalg )
Low-level BLAS functions
(
( scipy.linalg.blas )
)
f(xdata, *popt), then the interpretation of sigma depends on its number of
dimensions:
(
Low-level LAPACK functions
)
( scipy.linalg.lapack )
A 1-D sigma should contain values of standard deviations of errors in ydata.
In this case, the optimized function is chisq = sum((r / sigma) ** 2).
A 2-D sigma should contain the covariance matrix of errors in ydata. In this
BLAS Functions for Cython
case, the optimized function is chisq = r.T @ inv(sigma) @ r.
LAPACK functions for Cython
New in version 0.19.
Interpolative matrix
decomposition
(
None (default) is equivalent of 1-D sigma filled with ones.
( scipy.linalg.interpolative )
)
( scipy.misc )
If True, sigma is used in an absolute sense and the estimated parameter
(
Miscellaneous routines
)
covariance pcov reflects these absolute values.
If False (default), only the relative magnitudes of the sigma values matter. The
Multidimensional image
processing
(
( scipy.ndimage )
returned parameter covariance matrix pcov is based on scaling sigma by a
constant factor. This constant is set by demanding that the reduced chisq for
)
Orthogonal distance regression
( scipy.odr )
)
Optimization and root finding
( scipy.optimize )
)
pcov(absolute_sigma=True) * chisq(popt)/(M-N)
nonsensical results if the input arrays do contain nans. Default is True.
bounds : 2-tuple of array_like, optional
( scipy.sparse )
(
( scipy.sparse.linalg )
)
Lower and upper bounds on parameters. Defaults to no bounds. Each
element of the tuple must be either an array with the length equal to the
number of parameters, or a scalar (in which case the bound is taken to be the
Compressed sparse graph
routines
after the fit. Default is False. Mathematically, pcov(absolute_sigma=False) =
ValueError if they do. Setting this parameter to False may silently produce
(
Sparse linear algebra
(
other words, sigma is scaled to match the sample variance of the residuals
If True, check that the input arrays do not contain nans of infs, and raise a
(
( scipy.signal )
Sparse matrices
the optimal parameters popt when using the scaled sigma equals unity. In
(
check_finite : bool, optional
Cython optimize zeros API
Signal processing
absolute_sigma : bool, optional
(
( scipy.sparse.csgraph )
)
same for all parameters). Use np.inf with an appropriate sign to disable
bounds on all or some parameters.
New in version 0.17.
method : {‘lm’, ‘trf’, ‘dogbox’}, optional
Method to use for optimization. See least_squares for more details. Default
is ‘lm’ for unconstrained problems and ‘trf’ if bounds are provided. The
method ‘lm’ won’t work when the number of observations is less than the
number of variables, use ‘trf’ or ‘dogbox’ in this case.
New in version 0.17.
jac : callable, string or None, optional
Function with signature jac(x, ...) which computes the Jacobian matrix of
the model function with respect to parameters as a dense array_like structure.
It will be scaled according to provided sigma. If None (default), the Jacobian will
be estimated numerically. String keywords for ‘trf’ and ‘dogbox’ methods can
be used to select a finite difference scheme, see least_squares.
New in version 0.18.
kwargs
Keyword arguments passed to leastsq for method='lm' or least_squares
otherwise.
Returns:
popt : array
Optimal values for the parameters so that the sum of the squared residuals of
f(xdata, *popt) - ydata is minimized.
pcov : 2-D array
The estimated covariance of popt. The diagonals provide the variance of the
parameter estimate. To compute one standard deviation errors on the
parameters use perr = np.sqrt(np.diag(pcov)).
How the sigma parameter affects the estimated covariance depends on
absolute_sigma argument, as described above.
If the Jacobian matrix at the solution doesn’t have a full rank, then ‘lm’ method
returns a matrix filled with np.inf, on the other hand ‘trf’ and ‘dogbox’
methods use Moore-Penrose pseudoinverse to compute the covariance
matrix.
Raises:
ValueError
if either ydata or xdata contain NaNs, or if incompatible options are used.
RuntimeError
if the least-squares minimization fails.
OptimizeWarning
if covariance of the parameters can not be estimated.
 See also
least_squares
Minimize the sum of squares of nonlinear functions.
scipy.stats.linregress
Calculate a linear least squares regression for two sets of measurements.
Notes
With method='lm', the algorithm uses the Levenberg-Marquardt algorithm through leastsq. Note
that this algorithm can only deal with unconstrained problems.
Box constraints can be handled by methods ‘trf’ and ‘dogbox’. Refer to the docstring of
least_squares for more information.
Examples
>>> import matplotlib.pyplot as plt
>>> from scipy.optimize import curve_fit
>>> def func(x, a, b, c):
...
return a * np.exp(-b * x) + c
Define the data to be fit with some noise:
>>> xdata = np.linspace(0, 4, 50)
>>> y = func(xdata, 2.5, 1.3, 0.5)
>>> rng = np.random.default_rng()
>>> y_noise = 0.2 * rng.normal(size=xdata.size)
>>> ydata = y + y_noise
>>> plt.plot(xdata, ydata, 'b-', label='data')
Fit for the parameters a, b, c of the function func:
>>> popt, pcov = curve_fit(func, xdata, ydata)
>>> popt
array([2.56274217, 1.37268521, 0.47427475])
>>> plt.plot(xdata, func(xdata, *popt), 'r-',
...
label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
Constrain the optimization to the region of 0 <= a <= 3, 0 <= b <= 1 and 0 <= c <= 0.5:
>>> popt, pcov = curve_fit(func, xdata, ydata, bounds=(0, [3., 1., 0.5]))
>>> popt
array([2.43736712, 1.
, 0.34463856])
>>> plt.plot(xdata, func(xdata, *popt), 'g--',
...
label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
>>> plt.xlabel('x')
>>> plt.ylabel('y')
>>> plt.legend()
>>> plt.show()
<< scipy.optimize.lsq_linear
© Copyright 2008-2022, The SciPy community.
Created using Sphinx 4.0.2.
scipy.optimize.root_scalar >>