Quadratic forms
Definition of quadratic forms
Let {x1, x2 , ..., xn} be n (non random) variables. A quadratic form Q is, by
definition, an expression such as :
Q=
ij aij xi xj
where the aij (the coefficients of the form) are real numbers.
So a quadratic form is a second degree, homogenous (no constant term) polynom
in the xi.
Studying quadratic forms is made quite a bit easier by using matrix notation.
* The ordered set of variables {x1, x2 , ..., xn} is considered as a vector x with
coordinates (x1, x2 , ..., xn ). So we'll write :
x' = (x1, x2 , ..., xn )
with " ' " denoting transposition, which simply means that the xi are written as a
row (as opposed to "as a column").
* A denotes the matrix whose general term is [aij].
* A quadratic form Q is then defined by the matrix equation :
Q = x'Ax
A is called the matrix of the quadratic form.
It is easily shown that A may be assumed to be symmetric without loss of
generality.
----Quadratic forms are an important chapter of Linear Algebra. We will only retain
the aspects of quadratic forms that are useful to the Statistician.
Expectation of a quadratic form
Let :
* x be a random vector with mean µ and covariance matrix
* Q = x'Ax. be a quadratic form.
.
We'll show that :
E[x'Ax] = tr(A
) + µ'Aµ
This result makes no assumption about the nature of the distribution of x, with the
exception of the existence of the first two orders moments.
1
Quadratic forms in a multivariate normal vector and the Chisquare distribution
Important parts of Statistics like :
* Multiple Linear Regression,
* Variance Analysis,
extensively use the fact that if x is a multivariate normal vector :
x~N(µ,
)
then certain quadratic forms is x are Chi-square distributed.
Chi-square distribution
Let {X1, X2 , ..., Xn} be n standard normal independent variables :
Xi ~ N(0, 1)
for all i
Recall that, by definition, the Chi-square distribution with n degrees of freedom is
the distribution of the variable X² defined as :
X²=
i
Xi ²
and is usually denoted :
X²~
n
So the variable X² is defined as a very special quadratic form in the Xi in which :
* There is no cross-term (aij = 0 for i j),
* and all the aii are equal to 1.
The matrix of the form is then just the identity matrix of order n, denoted In.
----Note that these n variables may conveniently be represented by the single
multivariate normal standard variable :
x~N(0, In)
Quadratic forms in a standard multivariate normal variable and the Chi-square
distribution
It is then natural to ask whether there exist other quadratic forms in a
spherical multinormal variable of unit variance that also have a Chi-square
distribution. In other words, do matrices P exist such that :
x'Px ~
??
for x~N(0, Ip) ?
Recall that a symmetric matrix is said to be "idempotent" if P ² = P.
We'll show that a necessary and sufficient condition for x'Px to have a Chisquare distribution is that P be an idempotent matrix. In addition, we'll show that
the rank of P and the number of degrees of freedom of the
equal.
distribution are then
2
In other words :
* If x~N(0, In)
* Then Q = x'Px ~
r
if and only if :
*P²=P
* rank(P) = r
In fact, we'll show a bit more than that as we'll consider variables
~N(µ, I) that are not necessarily centered on the origin.
A symmetric idempotent matrix P or rank r is interpreted as the matrix of an
orthogonal projection operator on a r-dimensional subspace. The projection of the
vector x is Px.
If P ² = P, then :
x'Px = x'P ²x
= x'PPx
= x'P'Px
For P is symmetric
= (Px)'(Px)
and x'Px is therefore the squared length of the projection of x on the subspace.
The above result is then interpreted as follows :
* A quadratic form in a spherical, unit variance multinormal vector is Chisquare distributed with r degrees of freedom if and only if it is the squared length
of the projection of x on some r-dimensional subspace.
In the above illustration :
* The vector x has a spherical multinormal distribution with unit variance.
* Px is the projection of x on the r-dimensional subspace
.
3
* The squared length of Px is distributed as
r.
Quadratic forms in a general multinormal variable and the Chi-square
distribution
We then address the general issue of quadratic forms in a multivariate normal
variable with an arbitrary covariance matrix
:
x~N(µ,
)
We'll show that the quadratic form Q = x'Ax follows a (non central) Chi-square
distribution with r degress of freedom if and only if both the following conditions
are satisfied :
1) A = A A
2) rank(A) = r
the value of the noncentrality parameter being µ'Aµ.
----So, in the general case, the interpretation in terms of the distribution of the
squared length of the projection of x on a subspace is lost, and A is not a
projection matrix.
----This situation is encountered, for instance when studying the Mahalanobis
distance.
Independence of two quadratic forms in a multivariate
normal vector, Craig's Theorem
Independence of two random variables is a condition that makes life of a
Statistician a lot easier. In particular, many results pertaining to the sum and the
ratio of two random variables explicitely assume that these variables are
independent. For example, the formal definition of Fisher's F distribution involves
the ratio of two independent variables.
It is therefore natural to ask under what conditions two quadratic forms in a
multivariate normal vector x are independent.
Independence of two linear forms
We'll first show that two linear forms A'x et B'x in a multinormal vector x with
covariance matrix
are independent if and only if :
A'
B=0
This result is important by itself but, in addition, it will be needed later when we
address the issue of the independence of quadratic forms.
4
Independence of quadratic forms
We'll then show that :
* Let x~Np(µ, ),
* And let x'Ax and x'Bx be two Chi-square distributed quadratic forms.
* Then these two forms are independent
if and only if
A
B=0
In the special case where x is spherical, this condition becomes the simpler AB =
0.
----It is not assumed that the Chi-square distributions of the quadratic forms are
central or have the same numbers of degrees of freedom.
Craig's Theorem
The above result explicitely assumes that the quadratic forms are Chi-square
distributed. As it happens, this assumption is unnecessary and is introduced just as
to make the demonstration easier.
More generally :
* Let x~Np(µ, ),
* And let x'Ax and x'Bx be two quadratic forms.
* Then these two forms are independent
if and only if
A
B=0
which is the same as before but with the assumption about the
distributions of
the quadratic forms removed. The result is then known as Craig's Theorem,
whose demonstration lies beyond the bounds of this Glossary.
Cochran's Theorem
The results about the nature of the distribution and the independence of quadratic
forms in multinormal vectors as described here are a key element of a most
important theorem in Statistics known as Cochran's Theorem, which is stated and
demonstrated here.
__________________________________________________________________
5
Tutorial 1
In the first section of this Tutorial, we calculate the expectation of a quadratic
form with no assumption about the distribution of the variable. This section is
independent of the remainder of the Tutorial.
----We then establish a necessary and sufficient condition for a quadratic form Q :
Q = x'Px
in a multivariate normal variable to be Chi-square distributed.
* We first address the special case of a spherical multinormal distribution with
unit variance (identity covariance matrix).
- The "sufficient" condition states that if P is a projection matrix with rank r,
then Q is Chi-square distributed with r degrees of freedom.
- The "necessary" part states that if Q is Chi-square distributed with s degrees
of freedom, then P has to be a projection matrix with rank s. The demonstration is
a bit more difficult, and the moment generating function (m.g.f.) of the quadratic
form will be of central importance.
* We then go over the general case (arbitrary covariance matrix) by identifying
a transformation that will turn the problem into the already solved "special case"
problem, a common strategy when studying the multivariate normal distribution.
So the Table of Contents of the general case is quite short, as the brunt of the
work has already been done when solving the special case. Once the
transformation has been identified, only a small additional effort will be needed to
express the result as a function of , the covariance matrix of x.
QUADRATIC FORMS IN MULTINORMAL VECTORS
AND THE CHI-SQUARE DISTRIBUTION
Expectation of a quadratic form
------------------------------Multivariate normal distribution with identity covariance matrix
A sufficient condition for a quadratic form in a multivariate normal vector
with identity covariance matrix to be Chi-square distributed
The condition is also necessary
First form of the m.g.f. of the quadratic form
6
Second form of the m.g.f. of the quadratic form
Final result
Rank of P
Eigenvalues of P
P is idempotent
Noncentrality parameter
General case : multivariate normal distribution with arbitrary covariance matrix
TUTORIAL
______________________________________________________________________
_____
Tutorial 2
We now address the issue of the independence of two quadratic forms is a
multivariate normal vector.
We first solve the simpler problem of the independence of two linear forms. This
result will anyway be needed in the remainder of the Tutorial.
----We then move on to quadratic forms and establish a necessary and sufficient
condition for two such forms to be independent. The reader won't be surprised to
see us first solve the case of quadratic forms in a spherically symmetric, unit
variance multinormal vector. The general case (arbitrary covariance matrix) will
then be solved by identifying a transformation that turns the general case into the
already solved special case.
----Note that we always consider quadratic forms with Chi-square distributions. As it
turns out, this assumption is unnecessary and is introduced only to make the
demonstration easier. The condition remains valid without it and then bears the
name of Craig's Theorem, which is very difficult to demonstrate.
INDEPENDENCE OF QUADRATIC FORMS
Independence of two linear forms in a multivariate normal vector
Independence of two Chi-square distributed quadratic forms
in a standard multinormal vector
The condition is necessary
The condition is sufficient
The final result
7
Independence of two Chi-square distributed quadratic forms
in a multinormal vector with arbitrary covariance matrix
TUTORIAL
___________________________________________________
Related readings :
Chi-square distribution
Multivariate normal distribution
Cochran's Theorem
Projection matrix
8