# Significance and Meaningfulness

Document Sample

```					Significance and
Meaningfulness
Effect Sizes
KNR 445
Statistics
Effect sizes
Slide 2
Significance vs. meaningfulness

 Is your significant difference a real
difference?
KNR 445
Statistics
Effect sizes
Slide 3
Significance vs. meaningfulness

 Is your significant difference a real
difference?
KNR 445
Statistics
Effect sizes
Slide 4
Significance vs. meaningfulness

 Statistical Power
KNR 445
Statistics
Effect sizes
Slide 5
Significance vs. meaningfulness

 Statistical Power
 Smaller difference between means reduces power
 Larger SEM reduces power
KNR 445
Statistics
Effect sizes
Slide 6
Significance vs. meaningfulness

 Statistical Power
 Smaller  reduces power
KNR 445
Statistics
Effect sizes
Slide 7
Significance vs. meaningfulness

 As sample size increases, likelihood of
significant difference increases

X1  X 2
The fact that this sample   t
size is buried down here
in the denominator of
SE X 1  X 2
the test statistic means
that as n  , p  0. So    SEX1  X 2  SEX1  SEX 2
enough, it will generate
significant results              SD sample
SE X 
n
KNR 445
Statistics
Effect sizes
Slide 8
Significance vs. meaningfulness

 As sample size increases, likelihood of
significant difference increases
 So statistical difference does not always mean
important difference
 Calculate a measure of the difference that is
standardized to be expressed in terms of the
variability in the 2 samples
 = EFFECT SIZE
KNR 445
Statistics
Effect sizes
Slide 9
Significance vs. meaningfulness

 EFFECT SIZE - FORMULA

X1  X 2               X1  X 2
d          
SDpooled               SS1  SS2
n1  n2  2
KNR 445
Statistics
Effect sizes
Slide 10
Significance vs. meaningfulness

 EFFECT SIZE – from SPSS
 Using appendix B data set 2, and submitting DV
salary to test of difference across gender, gives
the following output (squashed here to fit):
T-Test                 Group Statistics

Std. Error
SEX         N            Mean       Std. Deviation   Mean
SALARY      male              6    36833.33      19913.9817 8129.8490
female            6    32500.00      14110.2799 5760.4977

Independent Samples Test

Levene's Test for
Equality of Variances                                  t-test for Equality of Means
95% Confidence
Interval of the
Mean       Std. Error        Difference
F          Sig.      t             df        Sig. (2-tailed)   Difference   Difference    Lower        Upper
SALARY   Equal variances
.011        .918        .435           10            .673     4333.3333    9963.8235    -17867.4   26534.12
assumed
Equal variances
.435      9.010              .674     4333.3333    9963.8235    -18202.6   26869.31
not assumed
KNR 445
Statistics
Effect sizes
Slide 11
Significance vs. meaningfulness

 EFFECT SIZE – from SPSS
T-Test                                                                                                Mean
Group Statistics                                                 difference
Std. Error                      to use
SEX            N           Mean        Std. Deviation   Mean
SALARY     male                 6   36833.33       19913.9817 8129.8490
female               6   32500.00       14110.2799 5760.4977

SD’s to
pool
Independent Samples Test

Levene's Test for
Equality of Variances                                     t-test for Equality of Means
95% Confidence
Interval of the
Mean       Std. Error        Difference
F          Sig.          t            df        Sig. (2-tailed)   Difference   Difference    Lower        Upper
SALARY   Equal variances
.011        .918            .435          10            .673     4333.3333    9963.8235    -17867.4   26534.12
assumed
Equal variances
.435     9.010              .674     4333.3333    9963.8235    -18202.6   26869.31
not assumed
KNR 445
Statistics
Effect sizes
Slide 12
Significance vs. meaningfulness

 EFFECT SIZE – from SPSS

SSsample
SDest    
n 1
So…                 Mean diff
d
(n1  1)SD  (n2  1)SD
2
1
2
2
n1  n2  2
KNR 445
Statistics
Effect sizes
Slide 13
Significance vs. meaningfulness

 EFFECT SIZE – from SPSS
Mean diff
d
(n1  1)SD12  (n2  1)SD2
2

n1  n2  2

Substituting…
4333 .33
d
(5)19913 .98 2  (5)14110 .28 2
10
KNR 445
Statistics
Effect sizes
Slide 14
Significance vs. meaningfulness

 EFFECT SIZE – from SPSS
4333 .33
d
(5)19913 .98  (5)14110 .28
2                 2

10

Calculating…
4333.33
d           0.25
17257.85
KNR 445
Statistics
Effect sizes
Slide 15
Significance vs. meaningfulness

 From Cohen, 1988:
 d = .20 is small
 d = .50 is moderate
 d = .80 is large
 So our effect size of .25 is small, and concurs
on this occasion with the insignificant result
 The finding is both insignificant and small
 (a pathetic, measly, piddling little difference of no
consequence whatsoever – trivial and beneath us)
Statistical Power

Maximizing the likelihood
of significance
KNR 445
Statistics
Effect sizes
Slide 17
Statistical Power

 The likelihood of getting a significant
relationship when you should (i.e. when there
is a relationship in reality)
 Recall from truth table, power = 1 - 
( = type II error)
KNR 445
Statistics
Effect sizes
Slide 18
Factors Affecting Statistical Power

The big ones:
 Effect size (bit obvious)
 Select samples such that difference between
them is maximized
 Sample size
 Most important: as n increases, SEM decreases,
and test statistic then increases
KNR 445
Statistics
Effect sizes
Slide 19
Factors Affecting Statistical Power

The others:
 Level of significance
 Smaller , less power
 Larger , more power
 1-tailed vs. 2-tailed tests
 With good a priori info (i.e. research literature),
selecting 1-tailed test increases power
 Dependent samples
 Correlation between samples reduces standard
error, and thus increases test statistic
KNR 445
Statistics
Effect sizes
Slide 20
Calculating sample size a priori

1. Specify effect size
2. Set desired level of power
3. Enter values for effect size and power in
appropriate table, and generate desired
sample size:
 Applet for calculating sample size based on above:
http://www.stat.uiowa.edu/~rlenth/Power/
 Applets for seeing power acting (and interacting) with
sample size, effect size, etc…
http://statman.stat.sc.edu/~west/applets/power.html