Significance and Meaningfulness

Document Sample
Significance and Meaningfulness Powered By Docstoc
					Significance and
 Meaningfulness
       Effect Sizes
  KNR 445
 Statistics
Effect sizes
   Slide 2
                  Significance vs. meaningfulness

                Is your significant difference a real
                 difference?
  KNR 445
 Statistics
Effect sizes
   Slide 3
                  Significance vs. meaningfulness

                Is your significant difference a real
                 difference?
  KNR 445
 Statistics
Effect sizes
   Slide 4
                  Significance vs. meaningfulness

                Statistical Power
  KNR 445
 Statistics
Effect sizes
   Slide 5
                  Significance vs. meaningfulness

                Statistical Power
                   Smaller difference between means reduces power
                   Larger SEM reduces power
  KNR 445
 Statistics
Effect sizes
   Slide 6
                  Significance vs. meaningfulness

                Statistical Power
                   Smaller  reduces power
  KNR 445
 Statistics
Effect sizes
   Slide 7
                  Significance vs. meaningfulness

                As sample size increases, likelihood of
                 significant difference increases

                                               X1  X 2
                The fact that this sample   t
                size is buried down here
                  in the denominator of
                                               SE X 1  X 2
                 the test statistic means
                that as n  , p  0. So    SEX1  X 2  SEX1  SEX 2
                   if your sample is big
                enough, it will generate
                     significant results              SD sample
                                            SE X 
                                                              n
  KNR 445
 Statistics
Effect sizes
   Slide 8
                  Significance vs. meaningfulness

                As sample size increases, likelihood of
                 significant difference increases
                  So statistical difference does not always mean
                   important difference
                  What to do about this?
                Calculate a measure of the difference that is
                 standardized to be expressed in terms of the
                 variability in the 2 samples
                  = EFFECT SIZE
  KNR 445
 Statistics
Effect sizes
   Slide 9
                 Significance vs. meaningfulness

                EFFECT SIZE - FORMULA


                   X1  X 2               X1  X 2
                d          
                   SDpooled               SS1  SS2
                                         n1  n2  2
  KNR 445
 Statistics
Effect sizes
  Slide 10
                        Significance vs. meaningfulness

                EFFECT SIZE – from SPSS
                    Using appendix B data set 2, and submitting DV
                     salary to test of difference across gender, gives
                     the following output (squashed here to fit):
               T-Test                 Group Statistics

                                                                                                   Std. Error
                                                SEX         N            Mean       Std. Deviation   Mean
                                    SALARY      male              6    36833.33      19913.9817 8129.8490
                                                female            6    32500.00      14110.2799 5760.4977


                                                            Independent Samples Test

                                      Levene's Test for
                                    Equality of Variances                                  t-test for Equality of Means
                                                                                                                                  95% Confidence
                                                                                                                                   Interval of the
                                                                                                        Mean       Std. Error        Difference
                                       F          Sig.      t             df        Sig. (2-tailed)   Difference   Difference    Lower        Upper
         SALARY   Equal variances
                                        .011        .918        .435           10            .673     4333.3333    9963.8235    -17867.4   26534.12
                  assumed
                  Equal variances
                                                                .435      9.010              .674     4333.3333    9963.8235    -18202.6   26869.31
                  not assumed
  KNR 445
 Statistics
Effect sizes
  Slide 11
                        Significance vs. meaningfulness

                EFFECT SIZE – from SPSS
                      T-Test                                                                                                Mean
                                                        Group Statistics                                                 difference
                                                                                           Std. Error                      to use
                                     SEX            N           Mean        Std. Deviation   Mean
                          SALARY     male                 6   36833.33       19913.9817 8129.8490
                                     female               6   32500.00       14110.2799 5760.4977



                          SD’s to
                           pool
                                                                Independent Samples Test

                                       Levene's Test for
                                     Equality of Variances                                     t-test for Equality of Means
                                                                                                                                      95% Confidence
                                                                                                                                       Interval of the
                                                                                                            Mean       Std. Error        Difference
                                        F          Sig.          t            df        Sig. (2-tailed)   Difference   Difference    Lower        Upper
          SALARY   Equal variances
                                         .011        .918            .435          10            .673     4333.3333    9963.8235    -17867.4   26534.12
                   assumed
                   Equal variances
                                                                     .435     9.010              .674     4333.3333    9963.8235    -18202.6   26869.31
                   not assumed
  KNR 445
 Statistics
Effect sizes
  Slide 12
                 Significance vs. meaningfulness

                EFFECT SIZE – from SPSS

                              SSsample
                   SDest    
                               n 1
                 So…                 Mean diff
                        d
                               (n1  1)SD  (n2  1)SD
                                           2
                                           1
                                                     2
                                                     2
                                       n1  n2  2
  KNR 445
 Statistics
Effect sizes
  Slide 13
                 Significance vs. meaningfulness

                EFFECT SIZE – from SPSS
                                 Mean diff
                    d
                           (n1  1)SD12  (n2  1)SD2
                                                    2


                                   n1  n2  2

                  Substituting…
                                       4333 .33
                      d
                             (5)19913 .98 2  (5)14110 .28 2
                                           10
  KNR 445
 Statistics
Effect sizes
  Slide 14
                 Significance vs. meaningfulness

                EFFECT SIZE – from SPSS
                                   4333 .33
                     d
                          (5)19913 .98  (5)14110 .28
                                      2                 2


                                       10

                  Calculating…
                        4333.33
                    d           0.25
                       17257.85
  KNR 445
 Statistics
Effect sizes
  Slide 15
                  Significance vs. meaningfulness

                From Cohen, 1988:
                   d = .20 is small
                   d = .50 is moderate
                   d = .80 is large
                So our effect size of .25 is small, and concurs
                 on this occasion with the insignificant result
                   The finding is both insignificant and small
                      (a pathetic, measly, piddling little difference of no
                       consequence whatsoever – trivial and beneath us)
Statistical Power

  Maximizing the likelihood
  of significance
  KNR 445
 Statistics
Effect sizes
  Slide 17
                           Statistical Power

                The likelihood of getting a significant
                 relationship when you should (i.e. when there
                 is a relationship in reality)
                Recall from truth table, power = 1 - 
                               ( = type II error)
  KNR 445
 Statistics
Effect sizes
  Slide 18
                Factors Affecting Statistical Power

               The big ones:
                Effect size (bit obvious)
                   Select samples such that difference between
                    them is maximized
                Sample size
                   Most important: as n increases, SEM decreases,
                    and test statistic then increases
  KNR 445
 Statistics
Effect sizes
  Slide 19
                Factors Affecting Statistical Power

               The others:
                Level of significance
                   Smaller , less power
                   Larger , more power
                1-tailed vs. 2-tailed tests
                   With good a priori info (i.e. research literature),
                    selecting 1-tailed test increases power
                Dependent samples
                   Correlation between samples reduces standard
                    error, and thus increases test statistic
  KNR 445
 Statistics
Effect sizes
  Slide 20
                  Calculating sample size a priori

               1. Specify effect size
               2. Set desired level of power
               3. Enter values for effect size and power in
                  appropriate table, and generate desired
                  sample size:
                   Applet for calculating sample size based on above:
                            http://www.stat.uiowa.edu/~rlenth/Power/
                   Applets for seeing power acting (and interacting) with
                    sample size, effect size, etc…
                        http://statman.stat.sc.edu/~west/applets/power.html

                        http://acad.cgu.edu/wise/power/powerapplet1.html

                    http://www.stat.sc.edu/%7Eogden/javahtml/power/power.html

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:43
posted:3/27/2012
language:
pages:20