Multiple Regression Analysis:
Multiple Regression Model
y = ß0 + ß1x1 + ß2x2 + …+ ßkxk + u
-ß is still the intercept
-ß to ß all called slope parameters
-u is still the error term (or disturbance term)
-Zero mean assumption
E(u) = 0
-Still minimize the sum of squared residuals
Multiple Regression Model: Example
-Dependent variable: Q, tile cases (in 1000 of cases)
-Right-hand side variables: tile price per case (p), income per
capita I (in 1000 of $), and advertising expenditure A (in
Regression: Q = ß0 + ß1P + ß2I + ß3A + u
-ß1 measures the effects of the tile price on the tile consumption,
holding all other factors fixed
-ß2 represents the effects of income, holding all other factors fixed
-ß3 represents the effects of advertising, holding all other factors fixed
Q = 17.513 – 0.296P + 0.066I + 0.036A
1. What is the impact of a price change on tile scales?
2. What is the impact of a change in income on tile scales?
3. What is the impact of a change in advertising
expenditures on tile scales?
Calculation of own-price elasticity?
Calculation of income elasticity?
Calculation of advertising elasticity?
-Collecting sales data of 23 tile stores in 2002 in the
-For each observation, Q = ß + ß P + ß I + ß A + u
i 0 1 i 2 i 3 i i
-Goal: Estimate ß , ß , ß , ß
0 1 2 3
Dependent Price Income Advertising
Q1 P1 I1 A1 Using OLS to estimate the
coefficients to minimize the
Q2 P2 I2 A2 sum of squared errors.
Q3 P3 I3 A3
… … … …
Q23 P23 I23 A23
ˆ 1Pi 2 I i 3 Ai
0, 1, 2, 3 i 1
0, 1, 2, 3 i 1
The Generic Multiple Regression Model
Yi 0 1X1i 2 X 2i ... k X ki i i 1,...,n
Y1 1 X11 X12 ... X1k
Y 1 X X 22 ... X 2 k
Yn nx1 1 X n1 X n 2 ... X nk nx ( k 1)
k ( k 1) x1 n nx1
Estimation of regression parameters:
-Least Squares (no knowledge of the distribution of the error or disturbance terms is required).
-The use of the matrix notation allows a view of how the data are housed in software programs.
Components of the Model
-Endogenous Variables—dependent variables, values of
which are determined within the system.
-Exogenous Variables—determined outside the system
but influence the system by affecting the values of the
-Structural Parameters—estimated using statistical
techniques and relevant data.
-Lagged Endogenous Variables
-Lagged Exogenous Variables
The Disturbance (or Error) Term
Stochastic, a random variable.
Statistical distribution often normal.
1. Omission of the influence of other variables.
2. Measurement error.
Recognition that any regression model is a parsimonious
stochastic representation of reality. Also recognition that any
regression model is stochastic and not deterministic.
OLS Estimates Associated with the
Multiple Regression Model
( x x) x y
1 x11 x21 xk1 y1
1 x x22 xk 2 y
x 12 (nx(k 1)) y 2 (nx1)
1 x1n x2 n xkn yn
xT is the transposeof x ((k 1) xn) ˆ 1 ((k 1) x1)
The Gauss-Markov Theorem
Given the assumptions below, it can be shown that the OLS
estimator is “BLUE.”
- Linear in parameters
- Corr (εi, εj) = 0
- Zero mean
- No perfect collinearity
Communication and Aims for
- A technician can run a program and get output.
- An analyst must interpret the findings from examination of this output.
- There are no bonus points to be given to terrific hackers but poor
1. Improve your ability in developing models to conduct structural analysis and to
forecast with some accuracy.
2. Enhance your ability in interpreting and communicating the results, so as to
improve your decision-making.
1. The analyst transforms the economic model/idea to a mathematical/statistical
2. The technician estimates the model and obtains a mathematical/statistical
3. The analyst transforms the mathematical/statistical answer to an economic
yi yi u i
Definition s :
yi y 2 is the total sum of squares (SST)
ˆ yi y 2 is the regressionsum of squares (SSR)
ˆi2 is the residual (or error)sum of squares (SSE)
Then SST SSR SSE
Goodness-of-Fit (continued . . .)
How well does our sample regression line fit our sample
R-squared of regression is the fraction of the total sum
of squares (SST) that is explained by the model.
R² = SSR/SST = 1 – SSE/SST
More about R-Squared
R² can never decrease when another explanatory
or predetermined variable is added to a
regression; usually R² will increase.
Because R² will usually increase (or at least not
decrease) with increases in the number of right-
hand side or explanatory variables, it is not
necessarily a good way to compare alternative
models with the same dependent variable.
R² and Adjusted R²
yi y 2
Explained sample variabili ty SSR SSE
R2 i 1
Total sample variabili ty 2 SST SST
2 SSE /(n k 1)
Adjusted R² R 1
SST /(n 1)
(a)Why do we care about the adjusted R² ?
(b)Is adjusted R² always better than R² ?
(c)What’s the relationship between R² and adjusted R² ?
Model Selection Criteria
MSE t 1
T e 2
s2 t 1
T p T T p MSE
Is the " penalty factor"
Akaike Information Criterion (AIC) AIC e T
(SIC) or (BIC)
SIC( BIC) T T
p is the number of parameters to be estimated
Model Selection Criteria Example
Model 1 Model 2 Model 3
AIC 19.35 15.83 17.15
SIC 19.37 15.86 17.17
Which model to choose?
Estimate of Error Variance
i n k 1 SSE df
-df = n – (k + 1), or df = n – k – 1
-df (i.e. degrees of freedom) is the (number of
observations) – (number of estimated parameters)
Variance of OLS Parameter Estimates
Var s 2 (x T x)-1
Variance Covariance Matrix of OLS Parameter Estimates
this matrix is a function of the residual variance s 2 .
Example: SAS Output of the
Demand Function for Shrimp
■ Price of shrimp
■ Price of finfish
■ Price of other shellfish
■ Advertising for shrimp
■ Advertising for finfish
■ Advertising for other
Model Selection Criteria for the
MSE SSE/T 1580.90 / 97 16.29
s2 MSE (16.29) 17.56
AIC e T
MSE e 97 (16.29) 18.82
SIC T T MSE 97 97 (16.29) 22.67