Matrix Algebra
Matrix algebra is a means of expressing large
numbers of calculations made upon ordered
sets of numbers.
Often referred to as Linear Algebra
Many equations would be completely
intractable if scalar mathematics had to be
used. It is also important to note that the
scalar algebra is under there somewhere.
Definitions - scalar
scalar - a number
denoted with regular type as is scalar
algebra
[1] or [a]
Definitions - vector
vector - a single row or column of numbers
denoted with bold small letters
row vector a = 1 2 3 4 5
column vector x =
x1
x
2
x3
x4
x5
Definitions - Matrix
A matrix is a set of rows and columns of
numbers 1 2 3
4 5 6
Denoted with a bold Capital letter
All matrices (and vectors) have an order
- that is the number of rows x the
number of columns.
Thus A = 1 2 3
4 5 6
2 x 3
Matrix Equality
Two matrices are equal iff (if and only
if) all of their elements are identical
Note: your data set is a matrix.
Matrix Operations
Addition and Subtraction
Multiplication
Transposition
Inversion
Addition and Subtraction
Two matrices may be added iff they are
the same order.
Simply add the corresponding elements
a11 a12 b11 b12 c11 c12
a a22 b21 b22 c 21 c 22
21
a31 a32 b31 b32 c31 c32
Addition and Subtraction
(cont.)
a11 b11 c11
Where a12 b12 c12
a21 b21 c 21
a22 b22 c 22
a31 b31 c31
a32 b32 c32
Hence 1 2 4 6 5 8
3 4 4 6 7 10
5 6 4 6 9 12
Scalar Multiplication
To multiply a scalar times a matrix,
simply multiply each element of the
matrix by the scalar quantity
a11 a12 2a11 2a12
2 2a
a21 a22 21 2a22
Matrix Multiplication (cont.)
To multiply a matrix times a matrix, we
write
A times B as AB
This is pre-multiplying B by A, or post-
multiplying A by B.
Matrix Multiplication (cont.)
In order to multiply matrices, they must
be conformable (the number of columns
in A must equal the number of rows in
B.)
an (mxn) x (nxp) = (mxp)
an (mxn) x (pxn) = cannot be done
a (1xn) x (nx1) = a scalar (1x1)
Matrix Multiplication (cont.)
The general rule for
matrix multiplication
is:
N
cij aik bkj where i 1,2,..., M , and j 1,2,..., P
k 1
Matrix multiplication is not
Commutative
AB does not necessarily equal BA
(BA may even be an impossible
operation)
Yet matrix multiplication is
Associative
A(BC) = (AB)C
Special matrices
There are a number of special matrices
Square
Diagonal
Symmetric
Null
Identity
Square matrix
A square matrix is just what it sounds like, an nxn
matrix
a11 a12 a13 a14
a
21 a22 a23 a24
a31 a32 a33 a34
a41 a42 a43 a44
Square matrices are quite useful for describing the
properties or interrelationships among a set of
things – like a data set.
Diagonal Matrices
A diagonal matrix is a square matrix that
has values on the diagonal with all off-
diagonal entities being zero.
a11 0 0 0
0 a 0 0
22
0 0 a33 0
0 0 0 a44
Symmetric Matrix
All of the elements in the upper right portion
of the matrix are identical to those in the
lower left.
For example, the correlation matrix
Identity Matrix
The identity matrix I is a diagonal
matrix where the diagonal elements all
equal one. It is used in a fashion
analogous to multiplying through by "1"
in scalar math.
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
Null Matrix
A square matrix where all elements equal
zero.
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
Not usually „used‟ so much as sometimes the
result of a calculation.
Analogous to “a+b=0”
The Transpose of a Matrix A'
Taking the transpose is an operation
that creates a new matrix based on an
existing one.
The rows of A = the columns of A'
Hold upper left and lower right corners
and rotate 180 degrees.
Example of a transpose
1 4
2 5, A' 1 2 3
A 4 5 6
3 6
The Transpose of a Matrix A'
If A = A', then A is symmetric (i.e.
correlation matrix)
If AA‟ = A then A' is idempotent
(and A' = A)
( A B C )' sum B C'
The transposeof a A'= 'sum of transposes
The transpose of a product = the product of
the transposes in reverse order
(ABC)‟ = C‟B‟A‟
An example:
Suppose that you wish to obtain the
sum of squared errors from the vector
e. Simply pre-multiply e by its
transpose e'.
which, in matrices looks like
e' e e1 e2 ..en
2 2 2
An example - cont
Since the matrix product is a scalar
found by summing the elements of the
vector squared.
The Determinant of a Matrix
The determinant of a matrix A is
denoted by |A|.
Determinants exist only for square
matrices.
They are a matrix characteristic, and
they are also difficult to compute
The Determinant for a 2x2 matrix
If A = a11 a12
a
21 a22
Then
A a11a22 a12a21
That one is easy
The Determinant for a 3x3 matrix
If A =
a11 a12 a13
a
21 a22 a23
a31 a32
a33
Then
A a11a22a33 a11a23a32 a12a23a31 a12a21a33 a13a21a32 a13a22a31
Determinants
For 4 x 4 and up don't try. For those
interested, expansion by minors and
cofactors is the preferred method.
(However the spaghetti method works
well! Simply duplicate all but the last
column of the matrix next to the
original and sum the products of the
diagonals along the following pattern.)
Spaghetti Method of |A|
a11 a12 a13 a11 a12
a a22 a
a23 21 a22
21
a31 a32
a33 a31 a32
Properties of Determinates
Determinants have several mathematical
properties which are useful in matrix
manipulations.
1 |A|=|A'|.
2. If a row of A = 0, then |A|= 0.
3. If every value in a row is multiplied by k,
then |A| = k|A|.
4. If two rows (or columns) are interchanged
the sign, but not value, of |A| changes.
5. If two rows are identical, |A| = 0.
Properties of Determinates
6. |A| remains unchanged if each
element of a row or each element
multiplied by a constant, is added to any
other row.
7. Det of product = product of Det's
|AB| = |A| |B|
8. Det of a diagonal matrix = product
of the diagonal elements
The Inverse of a Matrix (A-1)
For an nxn matrix A, there may be a B such
that AB = I = BA.
The inverse is analogous to a reciprocal)
A matrix which has an inverse is nonsingular.
A matrix which does not have an inverse is
singular.
An inverse exists only if A 0
Inverse by Row or column
operations
Set up a tableau matrix
A tableau for inversions consists of the
matrix to be inverted post multiplied by
a conformable identity matrix.
Matrix Inversion by Tableau
Method
Rules:
You may interchange rows.
You may multiply a row by a scalar.
You may replace a row with the sum of that row
and another row multiplied by a scalar.
Every operation performed on A must be
performed on I
When you are done; A = I & I = A-1
The Tableau Method of Matrix
Inversion: An Example
Step 1: Set up Tableau 1 4 3 1 0 0
2 5 4 0 1 0
1 3 2 0 0 1
Step 2: Add –2(Row 1) 1 4 3 1 0 0
0 3 2 2 1 0
to Row 2
1 3 2 0 0 1
Matrix Inversion – cont.
1 4 3 1 0 0
0 3 2 2 1 0
1 3 2 0 0 1
Step 3: Add –1(Row 1 4 3 1 0 0
0 3 2 2 1 0
1) to Row 3
0 7 5 1 0 1
Step 4: Multiply Row 1 4 3 1 0 0
0 1 2 / 3 2 / 3 1 / 3 0
2 by –1/3
0 7 5 1
0 1
Matrix Inversion – cont.
1 4 3 1 0 0
0 1 2 / 3 2 / 3 1 / 3 0
0 7 5 1
0 1
Step 5: Add –4 (Row 2) 1 0 1 / 3 5 / 3 4 / 3 0
to Row 1 0 1 2 / 3 2 / 3 1 / 3 0
0 7 5 1
0 1
1 0 1 / 3 5 / 3 4 / 3 0
Step 6: Add 7(Row 2) 0 1 2 / 3 2 / 3 1 / 3 0
to Row 3 0 0 1 / 3 11 / 3 7 / 3 1
Matrix Inversion – cont.
1 0 1 / 3 5 / 3 4 / 3 0
0 1 2 / 3 2 / 3 1 / 3 0
0 0 1 / 3 11 / 3 7 / 3 1
Step 7: Add Row 3 1 0 0 2 1 1
0 1 2 / 3 2 / 3 1 / 3 0
to Row 1
0 0 1 / 3 11 / 3 7 / 3 1
1 0 0 2 1 1
Step 9: Add 2(Row 0 1 0 8 5 2
3) to Row 2 0 0 1 / 3 11 / 3 7 / 3 1
Matrix Inversion – cont.
1 0 0 2 1 1
0 1 0 8 5 2
0 0 1 / 3 11 / 3 7 / 3 1
Step 9: Multiply Row 3
by -3 1 0 0 2 1 1
0 1 0 8 5 2
The original matrix is 0 0 1 11 7 3
now an identity, and
the original identity has
been transformed to the
Inverse
Checking the calculation
Remember AA-1=I
1 4 3 2 1 1 1 0 0
2 5 4 8 5 2 0 1 0
1 3 2 11 7 3 0 0 1
Thus
1 * 2 4 * 8 3 * 11 1
1 * 1 4 * 5 3 * 7 0
etc
The Matrix Model
The multiple regression model may be
easily represented in matrix terms.
Y XB e
Where the Y, X, B and e are all
matrices of data, coefficients, or
residuals
The Matrix Model (cont.)
The matrices in Y XB e are
represented by
Y1 X 11 X 12 ... X ik B1 e1
Y X X ... X 2 k B e
Y
2
X 21 22 B
2
e
2
... ... ... ...
Yn X n1 X n 2 ... X nk Bk en
Note that we postmultiply X by B since this
order makes them conformable.
The Assumptions of the Model
Scalar Version
1. The ei's are normally distributed.
2. E(ei) = 0
3. E(ei2) = 2
4. E(eiej) = 0 (ij)
5. X's are nonstochastic with values fixed in repeated
samples and (Xik-Xbark)2/n is a finite nonzero
number.
6. The number of observations is greater than the
number of coefficients estimated.
7. No exact linear relationship exists between any of
the explanatory variables.
The Assumptions of the
Model: The Matrix Version
These same assumptions expressed in
matrix format are:
1. e N(0,)
2. = 2I
3. The elements of X are fixed in repeated
samples and (1/ n)X'X is nonsingular and
its elements are finite
Derivation of B's in matrix
notation
Given the matrix algebra model we can
replicate the least squares normal
equations in matrix format.
We need to minimize ee’ which is the sum
of squared errors
Setting the derivative equal to 0 we get
Note that X‟X is called the sums-of-squares
and cross-products matrix.