MTH371: SAMPLE QUESTIONS FOR MIDTERM: MATRIX-VECTOR
OPERATIONS FOR LARGE MATRICES
PANAYOT VASSILEVSKI
Abstract. In general, you should know the algorithms (input, output), their high level
implementation, be able to estimate the cost in terms memory (dimensions of arrays
needed to store the data), and terms of operations (additions, multiplications, divisions)
as we have done in class and in your homeworks.
1. Topics and Sample Questions
1.1. Evaluation of expressions.
• Consider the following expression
s = a0 b0 + a1 b1 + a2 b2 + ... + an−1 bn−1 .
Write a for loop that implements the evaluation of this sum. Estimate the cost
(depending on n) in terms of number of additions and multiplications.




a0
b0
 a1 
 b1 



• Introduce the vectors a = 
 ...  and b =  ... . What does the above
an−1
bn−1
sum represent ?
• Consider m vectors b0 , b1 , b2 , . . . , bm−1 ∈ Rn and m scalars a0 , a1 , a2 , . . . , am−1
and form the expression
s = a0 b0 + a1 b1 + a2 b2 + ... + am−1 bm−1 .
What does this sum represent ? Write a for loop that computes the vector s.
What is the cost of computing s in terms of arithmetic operations. If we introduce
the matrix B with columns the vectors b0 , b1 , . . . , bm1 , i.e.,


a0
 a1 

B = [b0 , b1 , b2 , . . . , bm−1 ] and the vector a = 
 ... , what does the sum s
am−1
stand for ?
Date: May 7, 2023.
1
2
PANAYOT VASSILEVSKI
1.2. Sparse matrices.
• Given a matrix A as two-dimensional

a00 0
 0
0
A=
 0 a21
a30 0
array (table)

0 a03 0
0 0
0 

0 0 a24 
0 0 a34
– Convert this matrix into Coo format, i.e., create the row index array I,
column index array J and data array data. What are the dimensions of
these arrays ?
– Write down the row pointer array (A.indptr) for the CSR format of A.
• Write a high-level python code for performing w = Av, for A given in CSR
format. The matrix A has n rows, m columns, and nnz number of non-zeros.
This means that w is an array of size n and v is an array of size m. What is the
cost of this matrix-vector operation ?
• Let A be a CSR matrix of size n × n and nnz number of nonzeros, i.e., A is given
as the three arrays A.indptr (row pointer array), A.indices (the column indices
array) and A.data (the data array). Given a vector b ∈ Rn as b[i], i = 0, 1, . . . , n.
What does the following python-like code implement ?
for i in range(0,n):
x[i] = b[i]
for k in range(A.indptr[i], A.indptr[i+1]):
j= A.indices[k]
if (j < i):
x[i]-=data[k]* x[j]
if (j==i):
diag = data[k]
x[i]/=diag
• What portion of the matrix A is used in the above code?
• How would you change the above code if you want to solve a system with the
upper triangular portion of A ?
1.3. Algorithms: Arnoldi, Gram-Schmidt, and solving systems of equations
with least-squares.
• In Arnoldi algorithm we are given a matrix A in CSR format and an input normalized vector q0 . After performing m ≥ 1 steps, Arnoldi algorithm generates m
vectors q0 , q1 , q2 , . . . , qm−1 and a m × m matrix H. What are the properties of
Q = [q0 , q1 , . . . , qm−1 ] and H?
• Arnoldi algorithm involves the following steps:
At step (1), we compute v1 = Aq0 and form q1 = v1 − (v1 , q0 )q0 . Then, we
normalize q1 / = kq1 k.
At step k in range (1, m) :, we compute vk = Aqk−1 and form qk = vk −
(vk , q0 )q0 − · · · − (vk , qk−1 )qk−1 and the normalize it q1 = kqq11 k .
Write a python code that implements the above steps.
MTH371: SAMPLE MIDTERM QUESTIONS: MATRIX-VECTOR OPERATIONS FOR LARGE MATRICES
3
• In Gram-Schmidt algorithm we are given m vectors v0 , v1 , . . . , vm−1 ∈ Rn as
the columns of a matrix B = [v0 , v1 , . . . , vm−1 ]. It produces on output a matrix
Q = [q0 , q1 , . . . qm−1 ] and another matrix R.
What are the properties of Q and R ?
– How can you use Q and R to solve for y the system By = r for a given r,
by least squares?
– In some details, the Gram-Schmidt algorithm performs the following steps:
Step 0:
q[0] =v[0] / ||v[0]||
At general step k in range(1,m):
it computes
q[k] = v[k] - (v[k],q[0]) q[0] - (v[k],q[1]) q[1] - ...
- (v[k],q[k-1]) q[k-1]
and then normalizes
q[k]\=\|q[k]\|.
Write a python-like function that takes as input B and returns Q and R.
– What operations between vectors and matrices would be involved in that
function?
1.4. Stationary iterative methods. For a given n × n matrix A (in CSR format) and
a given r.h.s. b, in a stationary iterative method, we generate a sequence of iterates
x0 , x 1 , x 2 , . . . , xk , . . .
which hopefully approximate the exact solution x∗ of Ax∗ = b with increases accuracy.
In practice, we measure the norm of the residuals rk = b − Axk , krk k, and we want
that krk k 7→ 0 as k 7→ ∞.
A stationary iterative method involves a matrix M (sometimes given as a function)
which is used to compute a correction c by solving the problem M c = r for any current
residual r = rk . Then the iterate reads xk+1 = xk + c.
• Check all of the items that are necessary to be available as input in a stationary
iterative method:
(1) matrix A
(2) r.h.s. b
(3) initial approximation x0
(4) tolerance (5) maximum number of allowed iterations max iter
(6) matrix M , either given explicitly or as function, to compute corrections.
• What would you output from a function that implements a stationary iterative
method ?
• Give examples of M .
4
PANAYOT VASSILEVSKI
• What are the main operations used in the following high-level python code that
implements a stationary iterative method using an M for computing the corrections.
x = 0
r = b - A x
delta = ||r||
delta_0 = delta
iter = 0
while (delta>epsilon*delta_0 and iter< max_iter):
solve for c: M c = r
x+=c
r = b - A x
delta = ||r||
iter +=1
return x, delta, delta_0, iter
MTH371: SAMPLE MIDTERM QUESTIONS: MATRIX-VECTOR OPERATIONS FOR LARGE MATRICES
5
2. Answers
2.1. Evaluation of expressions.
• Consider the following expression
s = a0 b0 + a1 b1 + a2 b2 + ... + an−1 bn−1 .
Write a for loop that implements the evaluation of this sum.
s = 0
for k in range(0,n):
s+= a[k]*b[k]
Estimate the cost (depending on n) in terms of number of additions and multiplications.
cost: 2n additions and multiplications.




a0
b0
 a1 
 b1 



• Introduce the vectors a = 
 ...  and b =  ... . What does the above
an−1
bn−1
T
sum represent ? It represents the inner product b a.
• Consider m vectors b0 , b1 , b2 , . . . , bm−1 ∈ Rn and m scalars a0 , a1 , a2 , . . . , am−1
and form the expression
s = a0 b0 + a1 b1 + a2 b2 + ... + am−1 bm−1 .
What does this sum represent ? It represents a linear combination of the vectors
bk , k = 0, 1, . . . , m − 1.
Write a for loop that computes the vector s.
s =0
for k in range(0,m):
s+= a[k]*b[k]
What is the cost of computing s in terms of arithmetic operations.
2m x the size n of the vectors = 2mn
If we introduce the matrix B with columns the vectorsb0 , b1 , . . . , bm−1 , i.e.,
a0
 a1 

B = [b0 , b1 , b2 , . . . , bm−1 ] and the vector a = 
 ... , what does the sum s
am−1
stand for ?
s = Ba (i.e.,the product matrix B times vector a).
2.2. Sparse matrices.
6
PANAYOT VASSILEVSKI
• Given a matrix A as two-dimensional

a00 0
 0
0
A=
 0 a21
a30 0
array (table)

0 a03 0
0 0
0 

0 0 a24 
0 0 a34
– Convert this matrix into Coo format, i.e., create the row index array I,
column index array J and data array data. What are the dimensions of
these arrays ?
I 0
0
2
2
3
3
J 0
3
1
4
0
4
data a00 a03 a21 a24 a30 a34
The dimension of the three arrays I, J, data is of size nnz = 6.
– Write down the row pointer array (A.indptr) for the CSR format of A.
A.indptr[0] = 0, A.indptr[1] = 2, A.indptr[2] = 2, A.indptr[3] = 4, A.indptr[4] =
6.
• Write a high-level python code for performing w = Av, for A given in CSR
format. The matrix A has n rows, m columns, and nnz number of non-zeros.
This means that w is an array of size n and v is an array of size m. What is the
cost of this matrix-vector operation ?
w =0
for i in range(0,n):
for k in range(A.indptr[i], A.indptr[i+1]):
j = A.indices[k]
w[i]+= A.data[k] * v[j]
The cost is 2nnz.
• Let A be a CSR matrix of size n × n and nnz number of nonzeros, i.e., A is given
as the three arrays A.indptr (row pointer array), A.indices (the column indices
array) and A.data (the data array). Given a vector b ∈ Rn as b[i], i = 0, 1, . . . , n.
What does the following python-like code implement ?
for i in range(0,n):
x[i] = b[i]
for k in range(A.indptr[i], A.indptr[i+1]):
j= A.indices[k]
if (j < i):
x[i]-=data[k]* x[j]
if (j==i):
diag = data[k]
x[i]/=diag
The above code implements forward elimination with the lower triangular portion of A.
• What portion of the matrix A is used in the above code? It uses the lower
triangular portion of A (including the main diagonal of A).
MTH371: SAMPLE MIDTERM QUESTIONS: MATRIX-VECTOR OPERATIONS FOR LARGE MATRICES
7
• How would you change the above code if you want to solve a system with the
upper triangular portion of A ?
Solving with the upper triangular portion of A means we use back substitution.
That is, we have to reverse the for loop:
for i in range(0,n):
which gives
for i in range(n-1, -1, -1):
Also, we change the if statement
if (j < i):
to
if (j > i):
.
2.3. Algorithms: Arnoldi, Gram-Schmidt, and solving systems of equations
with least-squares.
• In Arnoldi algorithm we are given a matrix A in CSR format and an input normalized vector q0 . After performing m ≥ 1 steps, Arnoldi algorithm generates m
vectors q0 , q1 , q2 , . . . , qm−1 and a m × m matrix H. What are the properties of
Q = [q0 , q1 , . . . , qm−1 ] and H?
Q is orthogonal, i.e., QT Q = I, and H are such that QT AQ = H. The matrix
H has zero entries one diagonal below its main diagonal.
• Arnoldi algorithm involves the following steps:
At step (1), we compute v1 = Aq0 and form q1 = v1 − (v1 , q0 )q0 . Then, we
normalize q1 = kq1 k.
At step k in range (1, m) :, we compute vk = Aqk−1 and form qk = vk −
(vk , q0 )q0 − · · · − (vk , qk−1 )qk−1 and the normalize it q1 = kqq11 k .
Write a python-like code that implements the above steps.
for k in range(0,m-1):
v = A * q[k]
q[k+1] = v
for j in range(0,k):
q[k+1] -= (v, q[j]) *q[j]
q[k+1]/=||q[k+1]||
H = QT AQ
• In Gram-Schmidt algorithm we are given m vectors v0 , v1 , . . . , vm−1 ∈ Rn as
the columns of a matrix B = [v0 , v1 , . . . , vm−1 ]. It produces on output a matrix
Q = [q0 , q1 , . . . qm−1 ] and another matrix R.
What are the properties of Q and R ?
Q is orthogonal, i.e., QT Q = I. The columns of Q span the columns of B. R
is upper triangular matrix, and B = QR.
– How can you use Q and R to solve for y the system By = r for a given r,
by least squares?
We solve Ry = QT r which is an upper triangular system.
8
PANAYOT VASSILEVSKI
– In some details, the Gram-Schmidt algorithm performs the following steps:
Step 0:
q[0] =v[0] / ||v[0]||
At general step k in range(1,m):
it computes
q[k] = v[k] - (v[k],q[0]) q[0] - (v[k],q[1]) q[1] - ...
- (v[k],q[k-1]) q[k-1]
and then normalizes
q[k] /=||q[k]||.
Write a python-like function that takes as input B and returns Q and R.
QR(B):
for k in range(0,m):
q[k] = b[k]
for j in range(0,k):
q[k]-=(b[k],q[j])q[j]
q[k]/=||q[k]||
R = QT B
return Q, R
– What operations between vectors and matrices would be involved in that
function?
The QR algorithm involves inner products, and linear combination of vectors,
and at the end the product of matrix transpose (QT ) times a matrix (B).
2.4. Stationary iterative methods. For a given n × n matrix A (in CSR format) and
a given r.h.s. b, in a stationary iterative method, we generate a sequence of iterates
x0 , x 1 , x 2 , . . . , xk , . . .
which hopefully approximate the exact solution x∗ of Ax∗ = b with increases accuracy.
In practice, we measure the norm of the residuals rk = b − Axk , krk k, and we want
that krk k 7→ 0 as k 7→ ∞.
A stationary iterative method involves a matrix M (sometimes given as a function)
which is used to compute a correction c by solving the problem M c = r for any current
residual r = rk . Then the iterate reads xk+1 = xk + c.
• Check all of the items that are necessary to be available as input in a stationary
iterative method:
(1) matrix A
(2) r.h.s. b
(3) initial approximation x0
(4) tolerance (5) maximum number of allowed iterations max iter
MTH371: SAMPLE MIDTERM QUESTIONS: MATRIX-VECTOR OPERATIONS FOR LARGE MATRICES
9
(6) matrix M , either given explicitly or as function, to compute corrections.
All 6 items are necessary.
• What would you output from a function that implements a stationary iterative
method ?
We need to output the obtained approximate solution x, the norm of the last
residual δ = kb − Axk, the norm of the initial residual δ0 = kb − Ax0 k, and the
number of iterations used iter.
• Give examples of M .
(1) the lower triangular portion of A.
(2) the upper triangular portion of A.
(3) the `1 -smoother (the diagonal matrix with entries on its main diagonal being
the rowsums of the absolute values of the entries of A in each row.)
(4) the symmetric Gauss-Seidel matrix.
• What are the main operations used in the following high-level python code that
implements a stationary iterative method using an M for computing the corrections.
x = 0
r = b - A x
delta = ||r||
delta_0 = delta
iter = 0
while (delta>epsilon*delta_0 and iter< max_iter):
solve for c: M c = r
x+=c
r = b - A x
delta = ||r||
iter +=1
return x, delta, delta_0, iter
In each iteration step, the main operations above are: multiplication with matrix A, solving with the matrix M , operations between vectors, and computing
vector norms.