matrix_algebra

Some facts from matrix algebra

1) Vector is defined as a = (a1,a2,,,an). That is a point in n dimensional space



2) Distance between two vectors (x and y) is defined as

n

d p  (  ( x i  y j ) p )1 / p

i 1

When p=2 then subscript is usually dropped. It is a standard Eucilidian distance used

in everyday life.



3) Scalar product of two vectors

n

( x , y )   xi y i

i 1

Norm is defined as:

n

|| x || p  (  xip )1 / p

i 1

If p=2 then subscript is usually dropped and:

|| x || ( x, x )1 / 2



3) Matrix. nxk matrix is defined:

 a11 a12 ... ... a1k 

 

 a 21 a 22 ... ... a 2 k 

A   ... ... ... ... ... 

 

 ... ... ... ... ... 

a 

 n1 a n 2 ... ... a nk 



If n=k then it is called a square matrix.

Transpose of a matrix is defined:

( A T ) ij  a ji

If aij=aji for all i and j pairs then square matrix is called symmetric. Obviously in this

case - A T  A.

Identity matrix is defined as a nxn matrix that has only diagonal elements:

 1 0 ... 0 

 

 0 1 ... 0 

In  

... ... ... ...



 0 0 ... 1  

 

Usually subscript is dropped.

Multiplication matrix by vector is defined as:

k

(Ab) i   aij b j

j 1

Obviously b is a k dimensional vector and Ab is an n dimensional vector.

If a matrix is square then bi-linear form is defined as:

n n

( Ax, y )  x Ay   aij xi y j

T



i 1 j 1

When x=y it is called a quadratic form. Matrix is called positive semi-definite if for

all x the condition x T Ax  0 holds. If for all non-zero vectors above inequality is

strict then the matrix is positive definite. Negative semi-definite and definite matrices

are defined similarly.

Multiplication of nxk matrix by kxm matrix is defined as (the result is nxm) matrix):

k

(AB) ij   ail blj

l 1

The following relations are true:

1) ( AB ) T  B T A T

2) ( ABC)  CT B T A T

3) det( AB )  det ( A )det ( B)



Scalar product of two vectors can also be written

( x, y )  x T y

This notation means that multiplication of a column vector with a row vector. It is a

scalar. However multiplication of a row vector by a colum vector is nxn matrix.

 x1 y1 x1 y 2 ... x1 y n 

 

 x2 y1 x2 y 2 ... x2 y n 

xy  

T



... ... ... ... 



 x y x y ... x y  

 n 1 n 2 n n



7) Orthogonal matrix is defined as a matrix for which row vectors are orthogonal to

each other. It can be defined as:

A 1  A T  AT A  I

In general matrix is called orthogonal if

AAT  I

where I is a diagonal matrix with min(n,k) elements equal to 1. All other elements are

equal to 0. In effect it means that vector composed of rows of this matrix are

orthogonal for different rows. Similarly column orthogonality can be defined.



If multiplication of a matrix by itself is equal to matrix itself then it is call idempotent:

AA  A



Psuedo-inversion ( A  ) of a matrix is defined as:

AA  A  A



Rank of matrix: Rank of the matrix is maximum number linearly independent row

vectors or column vectors. It is also defined maximum square submatrix with non-

zero determinant.



9) Trace of a quadratic matrix is defined as:

n

tr (A )   aii

i 1

sum of the diagonal terms of the matrix. Trace has the following properties (they can

be easily verified):

1) tr ( I n )  n

2) tr ( A T )  tr ( A )

3) tr ( sA )  s tr ( A )

4) tr ( AB )  tr ( BA )

5) tr ( ABC)  tr ( BCA )

n

6) tr ( A )   i

i 1



where s is a scalar s are eigenvalues of the matrix.



10) Eigenvalue decompositions:

Eigenvalues of a matrix called a scalar value  for which there is a non-zero solution

to the equation:

Ax  x

Corresponding solution vecto is called eigenvector. Symmetric nxn matrix has n

eigenvectors. Some of the eigenvalues can be equal to each other. In this case they are

called degenerate. If a matrix has n eigenvalues and we build a matrix from this

eigenvectors so that each row is one of the eigenvectors of the matrix:

 e1 

 

e 

E 2

...

 

e

 

For symmetric matrix eigenvectors are orthonormal i.e.:

e i e j   ij

where  ij are Kronecker symbols that satisfy :

1 if i  j

 ij  

0 if i  j

It implies that the matrix E is orthogonal.

We define:

 1 0 ... 0 

 

 0 2 ... 0 

  diag(1 , 2 , , , n )  

... ... ... 0 



 0 0 ...   

 n



A symmetric matrix can be decomposed by its eigenvalues and eigenvectors:

A  ET E

Then matrix inversion can be written:

A1  ET 1E



11) Singular value decomposition. There are several version of this. One of them is.

Any matrix of nxk can be decomposed as:

A  UDV

Where U is nxn orthogonal matrix, V is kxk orthogonal matrix. D is a diagonal matrix

(with min(n,k) diagonal elements and remaining element 0).

Another version uses the fact about a matrix rank. If rank of a matrix is r then matrix

can be decomposed using:

A  UDV T

. Now U is nxr orthogonal matrix, V is kxr orthogonal matrix and D is rxr diagonal

matrix of positive elements. In the first version diagonal terms of D may have 0

elements. In the second version all elements are non-0.





Some useful matrices for data analysis



Some facts about derivative of function

w.r.t. vectors

If a function depends on a vector x of length n then its derivative (if exists) w.r.t.

vector is defined as:

f f f

( ,,,, )

x x1 xn

Note that derivative of a function w.r.t vector elements is a vector. The second

derivative is defined similarly:

 2 f 2 f 2 f 

 .... 

 x1x1 x1x2 x1xn 

 2 f 2 f 2 f 

2 f

 ... 

xx T  x2 x1 x2 x2 x2 xn 

 ... ... ... ... 

  f  f 2 f 

2 2



 x x x x ...

 n 1 n 2 xn xn 



Derivative of the scalar product:

(x T y )

y

x

And derivative of the square of the vector norm:

(x T x)

 2x

x

 2 (x T x)

 In

xx T



3) Derivative of a quadratic form

 ( x T Ax )

 ( A  A T )x

x

 ( x T Ax )

2

 A  AT

xx T

Derivatives of bi-linear forms:

(x T Ay)

 Ay

x

(y T Ax)

 ATy

x

Some facts about derivative of function w.r.t.

matrix

Definition: If a function depends on a matrix then its derivative w.r.t. matrix is

defined as a matrix composed of derivative of the function w.r.t. elements o the

matrix:

 f f f 

 ... 

 a11 a12 a1k 

 f f f 

f ( A)  ...

 a 21 a 22 a 2 k 

A  

 ... ... ... ... 

 f f

...

f 

 a a nk 

 n1 a n 2 

Similraly the second derivatives are defined. This time it is an array of 4th order.



2) Derivative of a trace is the identity matrix

(tr ( A)

I

A

Derivativ of determinant:

 (det( A)  det( A) A 1 if matrix is not symmetric



A 1

det( A)(2A  diag ( A ))

1

if matrix is symmetric

Derivative of a bi-linear form

f (x T Ay)  xTy if matrix is not symmetric

 T

A x y  y x  diag(x y)

T T

if matrix is symmetric

diag is defined as a matrix consisting of diagonal elements only.

If we put y=x we can get derivative of a quadratic form w.r.t matrix.