Ill-posed Computational Algebra
with Approximate Data
Zhonggang Zeng
Northeastern Illinois University, USA
Feb. 21, 2006, Radon Institute for Computational and Applied Mathematics
Example: Matrix rank and nulspace
=4
nullity = 2
Rank
+ eE = 6
nullity = 0
null space
basis
2
A well-posed problem: (Hadamard, 1923)
the solution satisfies
•
•
•
existence
uniqueness
continuity w.r.t data
An ill-posed problem is infinitely sensitive to perturbation
tiny perturbation

huge error
Ill-posed problems are common in applications
- image restoration
- IVP for stiction damped oscillator
- some optimal control problems
- air-sea heat fluxes estimation
……
-
deconvolution
inverse heat conduction
electromagnetic inverse scatering
the Cauchy prob. for Laplace eq.
3
Ill-posed problems are common
in algebraic computing
- Multiple roots of polynomials
- Polynomial GCD
- Factorization of multivariate polynomials
- The Jordan Canonical Form
- Multiplicity structure/zeros of polynomial systems
- Matrix rank
4
Problem: Polynomial factorization
(E. Kaltofen, Challenges of symbolic computation, 2000)
81x 4  16 y 4  648z 4  72 x 2 y 2  648 x 2  288 y 2  1296  0
(9 x 2  4 y 2  18 2 z 2  36)( 9 x 2  4 y 2  18 2 z 2  36)  0
Exact factorization no longer exists under perturbation
Can someone recover the factorization when the polynomial is perturbed?
5
Problem: polynomial root-finding
Exact coefficients
2372413541474339676910695241133745439996376
-21727618192764014977087878553429208549790220
83017972998760481224804578100165918125988254
-175233447692680232287736669617034667590560789
228740383018936986749432151287201460989730173
-194824889329268365617381244488160676107856145
110500081573983216042103084234600451650439725
-41455438401474709440879035174998852213892159
9890516368573661313659709437834514939863439
-1359954781944210276988875203332838814941903
82074319378143992298461706302713313023249
Inexact coefficients
2372413541474339676910695241133745439996376
-21727618192764014977087878553429208549790220
83017972998760481224804578100165918125988254
-175233447692680232287736669617034667590560780 9
228740383018936986749432151287201460989730170 3
-194824889329268365617381244488160676107856140 5
110500081573983216042103084234600451650439720 5
-41455438401474709440879035174998852213892159
9890516368573661313659709437834514939863439
-1359954781944210276988875203332838814941903
82074319378143992298461706302713313023249
Exact roots
1.072753787571903102973345215911852872073…
0.422344648788787166815198898160900915499…
0.422344648788787166815198898160900915499…
2.603418941910394555618569229522806448999…
2.603418941910394555618569229522806448999 …
2.603418941910394555618569229522806448999 …
1.710524183747873288503605282346269140403…
1.710524183747873288503605282346269140403…
1.710524183747873288503605282346269140403…
1.710524183747873288503605282346269140403…
“attainable” roots
1.072753787571903102973345215911852872073…
0.422344648788787166815198898160900915499…
0.422344648788787166815198898160900915499…
2.603418941910394555618569229522806448999…
2.603418941910394555618569229522806448999 …
2.603418941910394555618569229522806448999 …
1.710524183747873288503605282346269140403…
1.710524183747873288503605282346269140403…
1.710524183747873288503605282346269140403…
-61.710524183747873288503605282346269140403…
Example: JCF computation
r



 




1
r
1
r
s
1
s








t
Special case:
r  2,
s  3,
t 5
Maple takes 2 hours
On a similar 8x8 matrix, Maple and Mathematica run out of memory
Question: can we compute the JCF using approximate data?
7
Lancaster matrix derived from stability problem
 0

0




1  2 2  ( 2  2  2 )  

 2  2 2
  2
 1

1

For   0,
(  1   )
1
1
0
0
 2 
 2 2
2
2
 ( 2   2 )  3 2 1   2  2  2
 2 ( 2   2 )
2
2
2



1


1

2
2
2
2
  (1  2  )  (   )




2

spectrum   0,   ,  i,  i,    i 
0 1
 0

i


For   0, JCF  






1
i
i
i
1
i











i
Question:
Can we find the JCF
approximately when
  0?
8
Problem: Jordan Canonical Form (JCF)
0 10
3
0
5
9
3
1
7
-1
1
5
-4 -2 -9
0 -1
1
0
1
9
1
0
1
1 -4 -2
-1
1 -2
5
2
6
-1
4
0
1
0
1
0 -1
4
0
2 12
-4 -1 -5
0 11
8
-2
0
7
3
2
6
5 -12 -10
4 -9
0
2
0
2
0
-1
2
2
-2
-1
-2
0
0
1
2
1
0
-3
-1
-2
1
-2
2
2
1
0
-1 -1 -4 0 0 -5 -5 0 1
3 -2 -1 1 1 -2 -2 1 -1
-2 -11 1 0 6 -4 -3 6 0
3
1 -1 0 0 0 0 0 -1
6 19 -2 0 -8 8 6 -8 1
1
2 1 1 0 0 0 0 0
4 -3 3 3 1 -2 -2 1 0
0 -2 0 3 4 0 0 3 0
1
4 1 0 3 5 4 0 -2
-1
3 -1 -1 -3 3 0 -3 0
-3 -16 1 0 12 -5 -1 12 0
-2 -4 -1 0 0 -5 -4 3 4
0 -2 0 0 2 0 0 2 3
3 -1 1 1 0 0 0 0 -2
2 -7 0 0 2 -4 -3 2 -3
2 12 -1 0 -7 4 3 -7 0
-2 -12 -3 0 6 -9 -8 6 1
5
1 -1 1 -2 0 0 -2 0
-2 -7 1 0 2 -5 -4 2 -2
-3
1 5 -1 0 6 6 0 0
0
1 4 -1 0 4 4 0 -4
0 -3 0 0 3 0 0 3 0
 ( A)   3
0
1
5
0
-7
0
1
2
0
-2
9
0
2
3
2
-6
5
-1
2
0
0
3
0
1
-1
1
1
1
2
0
0
-1
-1
-1
0
3
4
1
0
1
0
-2
0
0
-1
2
0
2
-2
0
1
0
1
0
0
-2
0
1
6
3
-1
1
3
-1
4
0
0
-1
-3
0
4
0
-1
-1
0
1
-5
0
-1
0
-1
4
0
0
-1
0
0
-2
-5
-1
-2
0
4
0
-1
0
3
0
-3
-3
0
0
-2
2
-7
4
-3
6
4
0
-1 0
1 0
-1 0
1 0
2 0
0 0
1 0
0 0
1 0
0 0
-2 0
-1 0
0 0
0 0
0 0
1 0
-2 0
3 -1
1 2
0 3
0 1
0 0
-3
-1
0
-1
0
0
-1
0
1
0
0
-1
0
0
-1
0
-3
-1
0
5
6
0
-1
1
0
1
-1
0
0
0
1
0
0
-2
0
1
3
0
-1
0
2
0
4
3
JCF
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
3

 ( A  10 E )
15
9
Problem: Multiplicity structure of a polynomial system
P( x )  0
A zero x* can only be approximate, even if P is exact
(Abel’s Impossibility Theorem)
An approximate zero x0 is an exact solution to
~
P ( x )  P( x )  P ( x0 )  0
x0 degrades to a simple zero, even if x* is multiple
Can we recover the lost multiplicity structure?
10
Challenge in solving ill-posed problems:
Can we recover the lost solution
when data is inexact?
Solution
P
P
Data
P : Data  Solution
P
11
If the answer is highly sensitive to perturbations,
you have probably asked the wrong question.
Maxims about numerical mathematics, computers,
science and life, L. N. Trefethen. SIAM News
Wilkinson’s Turing Award
(1970):
A numerical algorithm
seeks the exact solution
of a nearby problem
However:
Ill-posed problems
are infinitely sensitive
to data perturbation
Conclusion: Numerical computation is incompatible
with ill-posed problems.
Solution: Ask the right question.
12
Trivia quiz: Is the following matrix nonsingular?
1

10 1





10 



1



10 1
(matrix derived from polynomial division by x+10 )
Answer: It is nonsingular in pure math.
Not so in Numerical Analysis.
1
smallest singular v alue 
n 1
10
(Distance to singular matrices)
13
Rank problem: Ask the right question
B
A
nullity  0
nullity  3
nullity  2
nullity  1
rank ( A)  min rank ( B)
B  A 
N ( A)  N ( B)
B A2 
min
with
rank( C )  rank ( A)
CA2
• nearness
• max-nullity
• mini-distance
i.e. the worst rank of nearby
matrices
I.e. the nulspace of the nearest
matrix with that rank
14
=4
null space
nullity = 2
Rank
basis
+ eE = 6
nullity = 0
After reformulating the rank:
0  e    4.98
=4
nullity = 2
Rank
+ eE = 4
dist ( N ( A)  N ( A  eE )) 
61.26
e
1 e
nullity = 2
Ill-posedness is removed successfully.
15
Computing approximate rank/nullspace:
General purpose:
Singular value decomposition (SVD)
Low-nullity and low-rank cases:
Recursive Inverse iteration/deflation
(T.Y. Li & Z. Zeng, SIMAX 2005)
Software package RankRev available
16
Problem: Polynomial GCD
For given polynomials f and g, find u, v, w such that
u  v  f

u  w  g
u  GCD( f , g )
which is an important application problem in its own right.
Applications: Robotics, computer vision, image restoration,
control theory, system identification, canonical
transformation, mechanical geometry theorem proving,
hybrid rational function approximation … …
17
The question: How to determine the GCD structure
g  uw
If
f  u  v,
then
 w
 f , g     ( u  v ) w  ( u  w) v  0
v


  w
C ( f ), C ( g )     0
v 


nulspace
The Sylvester Resultant matrix
is approximately rank-deficient
At heart of the GCD-finding is calculating
rank and nulspace with an approximate matrix
18
P s  Pt
-- space of polynomial pairs
~
( f , g~ )
( f , g)
deg( ~
u)  0
deg( ~
u)  3
deg( ~
u)  2
deg( ~
u)  1
deg( u~ ) 
max
~
( f , g~ ) ( f , g ) 
~ ~
deg( GCD( f , g ))
• nearness
• max-degree
• mini-distance
i.e. the max degree of
nearby polyn’s
~
AGCD( f , g )  EGCD( f , g~ ) with
~ ~
( f , g )  ( f , g) 
2
min
deg(GCD ( fˆ , gˆ )) deg(u~ )
( fˆ , gˆ )  ( f , g )
I.e. the GCD of the
nearest polyn pair
with that degree
19
P s  Pt
-- space of polynomial pairs
( f , g)
~
( f , g~ )
( fˆ , gˆ ) computed polynomial pair
A two staged approach:
Determine the GCD structure (or manifold)
Reformulate – solve a least squares problem
manifold

  ( f , g )  P s  P r : GCD( f , g )  P k
Minimize
2
~ ~
ˆ
( f , g )  ( f , gˆ )

well-posed, and even well conditioned
20
--- a least squares problem
The strategy:
Given
Reformulate a well-posed problem
( f , g)  P  P
s
t
(u, v, w)  P k  P s k  P t k
Set up a system for
uv

u  w
 (u )

Or, in vector form
and the GCD degree k


f
g

1

  
F ( u , v , w)  b
 
 r Hu 
  


F ( u , v , w )   Ck ( v )u 

Ck ( w)u 


minimize
   
F (u , v , w)  b
and
where
1
  
b f
 
 g 
2
2
21
H 
 r u 
   

F ( u , v , w )   Ck ( v ) u  ,

Ck ( w)u 


Theorem:
Its Jacobian:
H
 r

   

J ( u , v , w )   Ck ( v ) C m  k ( u )

Ck ( w)
Cn k ( w)

The Jacobian is of full rank
if v and w are co-prime
In other words: if the AGCD degree k is correct, then
min
   
F (u , v , w)  b
( u ,v ,w )P k P m  k P n  k
is a regular, well-posed problem!
2
2
22
For univariate polynomials :
Stage I: determine the GCD degree
S1(f,g) =
QR
S2(f,g)
QR
until finding the first rank-deficient Sylvester submatrix
Stage II: determine the GCD factors ( u, v, w )

  
by formulating F (u , v , w)  b
and the Gauss-Newton iteration
23
Start: k = n
Max-degree
k := k-1
k := k-1
no
Is GCD of degree k
possible?
no
probably
Refine with
G-N Iteration
Successful?
yes
Min-distance
nearness
Output GCD
Univariate AGCD algorithm
24
Software packages
Corless-Watt-Zhi
Gao-Kaltofen-May-Yang-Zhi
Kaltofen-Yang-Zhi
etc
Zeng:
uvGCD – univariate GCD computation
mvGCD – multivariate GCD computation
25
Identifying the multiplicity structure
p(x)= (x-1)5(x-2) 3(x-3) = [(x-1)4(x-2) 2] [(x-1)(x-2)(x-3)]
p’(x)=
[(x-1)4(x-2) 2]
[ (x-1)(x-2)+5(x -2)(x -3)+3(x -1)(x -3) ]
GCD(p,p’) = [(x-1)4(x-2) 2]
distinct roots:
u0(x) = [(x-1)4(x-2) 2]
u1(x) = [(x-1)3(x-2) ]
[(x-1)(x-2)(x-3)]
*
*
[(x-1)(x-2)]
*
*
u2(x) = [(x-1)2]
u3(x) = [(x-1)]
u4(x) = [1]
[(x-1)(x-2)]
*
*
[(x-1)]
*
[(x-1)]
*
u0 = p
um =GCD(um-1, um-1’)
*
----------------------------------------multiplicities 5
3
1
26
A squarefree factorization of f:
u0 = f
for j = 0, 1, … while deg(uj) > 0 do
uj+1 = GCD(uj, uj’)
the key
vj+1 = uj/uj+1
end do
Output :
f = v1 v2 … vk
with vj’s being squarefree
-
The number of distinct roots: m = deg(v1)
-
The multiplicity structure
l  [l1 , l2 ,, lk ]
l j  max  t | deg( vt )  m  j  1 ,
-
j  1,2,, k
Roots of vj’s are initial approximation to the roots of f
27
For the ill-posed multiple root problem with inexact data
The key:
Remove the ill-posedness by reformulating the problem
Original problem: calculating the roots
Reformulated problem: finding the nearest
polynomial on a proper pejorative manifold
-- A constraint minimization
28
To calculate roots of
p(x)= xn + a1 xn-1+...+an-1 x + an
( x - z1 ) l1( x - z2 ) l2 ... ( x - zm ) lm =
Let
xn + g1 ( z1, ..., zm ) xn-1+...+gn-1 ( z1, ..., zm ) x + gn ( z1, ..., zm )
Then,
(degree)
p(x) = ( x - z1 ) l1( x - z2 ) l2 ... ( x - zm ) lm
n
g1 ( z1, ..., zm ) =a1
g2( z1, ..., zm ) =a2
... ... ...
gn ( z1, ..., zm ) =an
m
<==>
(m<n)
(number of distinct roots
)
I.e. An over determined polynomial system
G(z) = a
29
a
b
â
computed polynomial
Gl ( zˆ )  aˆ
zˆ  z   l aˆ  a
(
  l aˆ  b  b  a
)
 2 l b  a
It is now a well-posed problem!
30
20
15
10
5
For polynomial ( x  1) ( x  2) ( x  3) ( x  4)
with (inexact ) coefficients in machine precision
Stage I results:
The backward error:
Computed roots
1.000000000000353
2.000000000030904
3.000000000176196
4.000000000109542
6.05 x 10-10
multiplicities
20
15
10
5
Stage II results:
The backward error:
Computed roots
1.000000000000000
1.999999999999997
3.000000000000011
3.999999999999985
Software: MultRoot, Zeng, ACM TOMS 2004
6.16 x 10-16
multiplicities
20
15
10
5
31
JCF computing: For a given matrix A

1

find X, J:
AX = XJ
1


J=
1

Segre characteristics:

[3,2,2,1]
1


find U, S, 
AU = U(I+S)

I+S=



+
+
+
+

+
+
+
+

+
+
+
+
+
+
+
+

+
+
+

Staircase form with Weyr characteristics:
[4,3,1]
3 2 2 1
4
3
1
Ferrer’s diagram
32
If
the Weyr characteristics is known:

find U, S, 
AU = U(I+S)
I+S=



+
+
+
+

+
+
+
+

+
+
+
+
+
+
+
+

+
+
+

leads to
(1)
AU- U(I+S) = 0
UHU – I = 0
BHU = 0
for , U, S
with a staircase structure
constraint
Theorem: Solving system (1) for a least squares
solution is well-posed!
-- Eigenvalues are no longer flying!
-- Gauss-Newton iteration converges locally
33
A two-stage strategy for computing JCF:
Stage I:
determine the Jordan structure and an
initial approximation
Stage II:
Solve the reformulated least squares problem
at each eigenvalue, subject to the structural
constraint, using the Gauss-Newtion iteration.
34
eigenvalue
1
2
3
Segre characteristics
5
3
2
3
3
1
2
2
Minimal polynomials:
p1 ( )  (  1 )5 (  2 )3 (  3 ) 2
p2 ( )  (  1 )3 (  2 )3
p 3 ( )  (  1 ) 2 (  2 )1
p 4 ( )  (  1 ) 2
Let v be a coefficient vector of p1. Then for a random x
x, Ax,, A xv  0
10
A rank/nulspace problem, again
35
By a Hessenberg reduction,
with A1 = A, and x being the first column of Q1,
leads to
p2(), p3(), …
recursively
Rank/nulspace leads to
p1 (  )
(the 1st minimal polynomial)
36
Minimal polynomials
p1 ( )  (  1 )5 (  2 )3 (  3 ) 2
p2 ( )  (  1 )3 (  2 )3
p 3 ( )  (  1 ) 2 (  2 )1
p 4 ( )  (  1 ) 2
Ill-posed root-finding with approximate data
Computing multiple roots of inexact polynomials,
Z. Zeng, ISSAC 03
Rank-revealing
GCD
Root-finding
JCF
37
~
When matrix A is possibly approximate: A  A  eE
C n n
-- space of matrices
Nearness: JCF of Â
Max-defectiveness: on 
~
Mini-distance: from  to A
A
~
A
~
A
We can obtain an approximate JCF of
Â
“nearest” matrix on 
Software AJCF is in testing stage
manifold
  B  C nn with a fixed JCF structure 
38
A two-stage strategy for removing ill-posedness:
For a problem
(data ----> solution)
Stage I:
with data D0
P: D S
determine the structure
S
of the desired solution
this structure determines a pejorative manifold of data
  P-1(S) =  D | P(D) = S fits the structure }
Stage II: formulate and solve a least squares problem
minimize
S S
1
P ( S )  D0
2
by the Gauss-Newton iteration
39
The multiplicity structure of polynomial systems
 x13  0
 2
4
x
x

x
 1 2 2 0
Example:
What’s the multiplicity of (0,0)?
(
x13 ,
Intersection multiplicity = dim C C[[ x1 , x2 ]]
x12 x2  x24
)
(algebraic geometry)
•
•
•
0
1 i j
 ij 
i! j! x1i x2j
Multiplicity = 12
Hilbert function = {1,2,3,2,2,1,1,0,…}
Dual space D(0,0) (I) spanned by
1
2
4
3
5
6
depth = 6
 00 10 ,  01,  20 , 11,  02 , 12 ,  03 , 13 ,  04   02 ,  05   22 ,  06   23
 

 
 
 

 
 
 


1
2
breadth=2
3
2
2
1
1
Multiplicity is not just a number!
For a univariate polynomial p(x) at zero x0:
p( x0 )  p' ( x0 )    p ( m1) ( x0 )  0
Multiplicity = m
Differential functionals
0 ,
1 ,
 2 , ,
 m1
Vanish on the ideal <p>, and span the dual space of <p> at x0
Duality approach (Lasker, Macaulay, Groebner):
Multiplicity is the dimension of the dual space, spanned
by the differential functionals that vanish on the ideal
The dual space


Dx0 f1 ,, f t  c 


 c j j
jN s


c( f )  0, f  f1 ,, f t 


defines the multiplicity structure of the ideal at x0
41
For a polynomial system, or ideal
f1,, f t
at zero z
A differential functional c of order 
c
 c 
j
j
with
j
j   j1 ,, js  N s
z
The functional c is in the dual space if and only if
c( pf i )  0,
Or, equivalently
p  C[ x1,, xs ], i  1,, t
k
(
c

(
x

z
)
fi )( z )  0 for
j j
k  N s , i  1,, s
j 

S c  0
A rank/nulspace problem, again!
42
Multiplicity matrices
S
and nullspaces
N (S )
I  x1  x2  x12 , x1  x2  x22 at (0,0)
S0 
D( 0,0) ( I )  span  00 , 10   01,  10   20  11   02

43
If the polynomial system is inexact, or
if the system is exact, but the zero is approximate
determining the multiplicity structure goes back to solving
Ill-posed rank/nulspace problem with approximate data.
Algorithm:
• for  = 1, 2, … do
- construct matrix S
- compute W  N ( S )  N ( S 1 ) 
- if W = {0}, break
• end do
A reliable algorithm for calculating the multiplicity structure
with approximate data
Dayton & Zeng, ISSAC 2005
Bates, Peterson & Sommese, 2005
44
Conclusion
- Ill-posed problems could be regularized as
well posed least squares problems
- Regularized ill-posed problems permit
approximated data
- Rank/nulspace determination is at the
heart of solving ill-posed problems
Software package PolynMat :
RankRev
uvGCD
mvGCD
MultRoot
AJCF
dMult