Advanced
Algorithmic
Problem Solving
Le 3 – Arithmetic
Fredrik Heintz
Dept of Computer and Information Science
Linköping University
Arithmetic

Range of default integer data types (C++)
 unsigned int = unsigned long: 232 (9‐10 digits)
 unsigned long long: 264 (19‐20 digits)


How to represent 777!
Operations on Big Integer
 Basic: add, subtract, multiply, divide, etc
 Use “high school method”
2
Arithmetic

3
Greatest Common Divisor (Euclidean Algorithm)
 GCD(a, 0) = a
 GCD(a, b) = GCD(b, a mod b)
 int gcd(int a, int b) { return (b == 0 ? a : gcd(b, a % b)); }

Least Common Multiplier
 LCM(a, b) = a*b / GCD(a, b)
 int lcm(int a, int b) { return (a / gcd(a, b) * b); }
▪ // Q: why we write the lcm code this way?

GCD/LCM of more than 2 numbers:
 GCD(a, b, c) = GCD(a, GCD(b, c))

Find d, x, y such that d = ax + by and d = GCD(a,b) (Extended
Euclidean Algorithm)
 EGCD(a,0) = (a,1,0)
 EGCD(a,b)
▪ (d’,x’,y’) = EGCD(b, a mod b)
▪ (d,x,y) = (d’,y’,x’ – a/b*y’)
Arithmetic

4
Representing rational numbers.
 Pairs of integers a,b where GCD(a,b)=1.

Representing rational numbers modulo m.
 The only difficult operation is inverse, ax = 1 (mod m), where an inverse
exists if and only if a and m are co-prime (gcd(a,m)=1).
 Can be found using the Extended Euclidean Algorithm
ax = 1 (mod m) => ax – 1 = qm => ax – qm = 1
(d, x, y) = EGCD(a,m) => x is the solution iff d = 1.
Systems of Linear Equations
5
A system of linear equations can be presented
in different forms
2 x1  4 x2  3 x3  3
4  3  x1  3

2

2.5 x1  x2  3 x3  5   2.5  1 3   x2   5
 1
0  6  x3  7 
x1
 6 x3  7 
Standard form
Matrix form
Solutions of Linear Equations
 x1  1 
 x   2 is a solution to the following equations :
 2  
x1  x2  3
x1  2 x2  5
6
Solutions of Linear Equations

A set of equations is inconsistent if there exists no
solution to the system of equations:
x1  2 x2  3
2 x1  4 x2  5
These equations are inconsistent
7
Solutions of Linear Equations

Some systems of equations may have infinite number of
solutions
x1  2 x2  3
2 x1  4 x2  6
have infinite number of solutions
a
 x1  

 x   0.5(3  a ) is a solution for all a

 2 
8
Graphical Solution of Systems of Linear Equations
x1  x2  3
x1  2 x2  5
Solution
x1=1, x2=2
9
Cramer’s Rule is Not Practical
10
Cramer' s Rule can be used to solve the system
3 1
1 3
5 2
x1 
 1,
1 1
1 5
x2 
2
1 1
1 2
1 2
Cramer' s Rule is not practical for large systems .
To solve N by N system requires (N  1)(N - 1)N! multiplications.
To solve a 30 by 30 system, 2.38  1035 multiplications are needed.
It can be used if the determinants are computed in efficient way
Naive Gaussian Elimination

11
The method consists of two steps:
 Forward Elimination: the system is reduced to upper triangular
form. A sequence of elementary operations is used.
 Backward Substitution: Solve the system starting from the last
variable.
 a11
a
 21
 a31
a12
a22
a32
a13 
a23 
a33 
 x1   b1 
 x   b  
 2  2
 x3  b3 
a11
0

 0
a12 a13   x1   b1 
a22 ' a23 '  x2   b2 '
0 a33 '  x3  b3 '
Elementary Row Operations

Adding a multiple of one row to another

Multiply any row by a non-zero constant
12
Example: Forward Elimination
 6 2
 12  8

 3  13

 6 4
 x1   16 
 x   26 
 2 

 x3    19 
  

 x4   34
Part 1 : Forward Elimination
Step1 : Eliminate x1 from equations 2, 3, 4
6  2
0  4

0  12

0 2
2 4 
6 10 
9 3 

1  18
2 4 
2 2 
8 1 

3  14
 x1   16 
x    6 
 2 

 x3   27
  

 x4    18 
13
Example: Forward Elimination
Step2 : Eliminate x2 from equations 3, 4
6  2 2 4 
0  4 2 2 


0 0 2  5 


0 0 4  13
 x1   16 
x    6 
 2 

 x3    9 
  

x
 4   21
Step3 : Eliminate x3 from equation 4
6  2 2 4 
0  4 2 2 


0 0 2  5


0 0 0  3
 x1   16 
 x    6
 2  
 x3    9
   
 x4    3
14
Example: Forward Elimination
15
Summary of the Forward Elimination :
 6 2
 12  8

 3  13

 6 4
2 4 
6 10 
9 3 

1  18
 x1   16  6  2 2 4 
 x   26  0  4 2 2 
 2 


 x3    19  0 0 2  5
  
 

x

34
0
0
0

3
 

 4 
 x1   16 
 x    6
 2  
 x3   9
   
 x4    3
Example: Backward Substitution
6  2 2 4   x1   16 
0  4 2 2   x    6 

  2  
0 0 2  5  x3   9

   
0 0 0  3  x4    3
Solve for x4 , then solve for x3 ,... solve for x1
3
95
x4 
 1,
x3 
 2
3
2
 6  2(2)  2(1)
16  2(1)  2(2)  4(1)
x2 
 1, x1 
3
4
6
16
Forward Elimination
To eliminate x1
To eliminate x2
 ai1 
aij  aij  
a1 j
 a11 
a 
bi  bi   i1 b1
 a11 
 ai 2 
aij  aij  
a 2 j
 a22 
a 
bi  bi   i 2 b2
 a22 
17

(1  j  n ) 

2  i  n



(2  j  n) 

3  i  n


Forward Elimination
18
 aik 
aij  aij  
akj
 akk 

(k  j  n) 

k  1  i  n


To eliminate xk
 aik 
bi  bi  
bk
 akk 
Continue until xn 1 is eliminated.
Backward Substitution
bn
xn 
a n ,n
xn 1 
xn  2 
bn 1  an 1,n xn
an 1,n 1
bn  2  an  2,n xn  an  2,n 1 xn 1
bi 
xi 
a n  2, n  2
n
 ai , j x j
j i 1
a i ,i
19
Determinant
20
The elementary operations do not affect the determinant
Example :
3
1 2 3 
1 2
A  2 3 2 Elementary
  operations
  A'  0  1  4
3 1 2
0 0 13 
det (A)  det (A')  13
How Many Solutions Does a
System of Equations AX=B Have?
Unique
No solution
det(A)  0
det(A)  0
reduced matrix reduced matrix
has no zero rows has one or more
zero rows
corresponding B
elements  0
21
Infinite
det(A)  0
reduced matrix
has one or more
zero rows
corresponding B
elements  0
Examples
Unique
22
No solution
1 2
1 
3 4 X  2


 

1 2 
 2
 2 4 X   3


 

1 2 
1
0  2 X   1


 
solution :
1 2
2
0 0 X   1


 
No solution
0
X  
0.5
0  1 impossible!
infinte # of solutions
1 2 
 2
 2 4 X   4


 

1 2
 2
0 0  X   0 


 
Infinite # solutions
  
X 

1

.
5



Pseudo-Code: Forward Elimination
Do k = 1 to n-1
Do i = k+1 to n
factor = ai,k / ak,k
Do j = k+1 to n
ai,j = ai,j – factor * ak,j
End Do
bi = bi – factor * bk
End Do
End Do
23
Pseudo-Code: Back Substitution
xn = bn / an,n
Do i = n-1 downto 1
sum = bi
Do j = i+1 to n
sum = sum – ai,j * xj
End Do
xi = sum / ai,i
End Do
24
Problems with Naive Gaussian Elimination
25
o The Naive Gaussian Elimination may fail for very simple cases.
(The pivoting element is zero).
0 1  x1  1
1 1  x   2

  2  
o Very small pivoting element may result in serious computation
errors
10 10 1  x1  1 

   
1  x2  2
 1
How Do We Know If a Solution is Good or Not 26
Given AX=B
X is a solution if AX-B=0
Compute the residual vector R= AX-B
Due to rounding error, R may not be zero
The solution is acceptable if max ri  
i
How Good is the Solution?
1  1 2
3 2 1

5  8 6

4 2 5
 x1   1 
x   1 
 2     solution
 x3   1 
   
 x4   1
0.005
0.002

Residues : R  
0.003


0.001


1
4
3

3
27
 x1    1.8673 
 x   0.3469
 2  

 x3   0.3980 
  

 x4   1.7245 
Karatsuba’s algorithm
28
Polynomials f and g: if each has 2 or more terms
 Split each: if f and g are of degree d, consider

▪ f = f1*x^(d/2)+f0
▪ g= g1*x^(d/2)+g0
 Note that f1, f0, g1, g0 are polys of degree d/2.
 Note there are a bunch of nagging details, but it still




works.
Compute A=f1*g1, C=f0*g0 (recursively!)
Compute D=(f1+f0)*(g1+g0) (recursively!)
Compute B=D-A-C
Return A*x^d+B*x^(d/2)+C (no multiplications)
Karatsuba’s algorithm

Karatsuba’s algorithm
 Cost: in multiplications, the cost of multiplying
polys of size 2r: cost(2r) is 3*cost(r) 
cost(s)=s^log[2](3) = s^1.585… ; looking good.
 Cost: in adds, about 5.6*s^1.585
 Costs (adds+mults) 6.6*s^1.585
 Classical (adds+mults) is 2*s^2
29