Learning Center
Plans & pricing Sign in
Sign Out



									Using Simulations to Teach
Statistical Inference

Beth Chance, Allan Rossman (Cal Poly)

                  ICTCM 2011            1
Joint Work with

n   Soma Roy, Karen McGaughey (Cal Poly),
    q   Alex Herrington (Cal Poly undergrad) 
n   John Holcomb (Cleveland State), 
n   George Cobb (Mt. Holyoke), 
n   Nathan Tintle, Jill VanderStoep, Todd 
    Swanson (Hope College)
n   This project has been supported by the 
    National Science Foundation, DUE/CCLI  
                           ICTCM 2011           2

n   Motivation/Goals
n   Examples
    q   Binomial process, randomized experiment- binary, 
        randomized experiment - quantitative response
    q   Series of lab assignments
    q   Discussion points
n   Student feedback, Evaluation results
n   Design principles & implementation
n   Observations, Open questions
                           ICTCM 2011                   3

n   Cobb (2007) – 12 reasons to teach 
    permutation tests…
    q   Model is “simple and easily grasped”
    q   Matches production process, links data production 
        and inference
    q   Role for tactile and computer simulations
    q   Easily extendible to other designs (e.g., blocking)
    q   Fisherian logic
                           --”The Introductory Statistics Course: 
                                A Ptolemaic Curriculum” (TISE)
                            ICTCM 2011                          4

n   Develop an introductory curriculum that 
    focuses on randomization-based approach to 
    q   vs. using simulation to teach traditional inference 
    q   From beginning of course, permeate all topics
n   Improve understanding of inference and 
    statistical process in general
    q   More modern (computer intensive) and flexible 
        approach to inferential analysis

                             ICTCM 2011                        5
Brief overview of labs

n   Case-study focus
n   Pre-lab
    q   Background, Review questions submitted in advance
n   50-minute (computer) lab period
n   Online instructions
    q   Directed questions following statistical process
    q   Embedded applets or statistical software
n   Application/Extension
n   Lab report with partner

                              ICTCM 2011                    6
Example 1: Friend or Foe
n   Videos
n   Research question
n   Pre-lab
n   Descriptive analysis
n   Introduction of null hypothesis, 
        p-value terminology
n   Plausible values
n   Conclusions

                        ICTCM 2011      7
Discussion Points

n   Can this be done on day one?
    q   Yes if can motivate the simulation
        n   Loaded dice
        n   Before reveal the data?

                                ICTCM 2011   8
<<After tactile simulation>> How many infants
would need to choose the helper toy for you to be
convinced the choice was not made “at random,”
but they actually prefer the helper toy?
n   Many students can reason inferentially
    q   “If a choice is made at complete random, then 
        having 13 infants would be highly unlikely”
    q   “Based on the coin flipping experiment, the results 
        stated that at/over 12 was extremely rare. 
        Therefore, at least 12 infants …
    q   “Would be around 12-16 because it seems highly 
        unlikely that given a 50-50 option 12-16 would 
        choose the helper toy”

                            ICTCM 2011                     9
<<After tactile simulation>> How many infants
would need to choose the helper toy for you to be
convinced the choice was not made “at random,”
but they actually prefer the helper toy?
n   But maybe not as well “distributionally”
    q   Is it unusual? = “barely over half”
        n   vs. unusual compared to distribution
n   Examine language carefully
    q   “Unlikely that choice is random”
    q   “Prove”
    q   “Simulate”, “Repeated this study”
    q   “At random” = 50/50, “model” 
n   “Random” = anything is possible

                                   ICTCM 2011       10
Discussion Points

n   Can this be done on day one?
    q   Yes if can motivate the simulation
        n   Loaded dice
        n   Before reveal the data?
        n   Enough understanding of “chance model”?
        n   Use of class data instead? (“observed” vs. research 
    q   Yes, if return to and build on the ideas throughout 
        the course
        n   So what comes next?

                                ICTCM 2011                         11
Discussion Points

n   Tactile simulation
    q   One coin 16 times vs. 16 coins
n   Population vs process
    q   Defining the parameter
n   3Ss: statistic, simulate, strength of evidence
    q   “could have been” distribution of data
    q   “what if the null was true” distribution of statistic
n   Fill in the blank wording
n   Timing of final report
    q   Follow-up in-class discussion

                                ICTCM 2011                      12
Example 2: Two Proportions

n   Is Yawning Contagious?
    q   Modelling entire process: data collection, 
        descriptive statistics, inferential analysis, 
    q   Parallelisms to first example
    q   Could random assignment alone produce a 
        difference in the group proportions at least this 
    q   Card shuffling, recreate two-way table
    q   Extend to own data
                             ICTCM 2011                      13
Lab Instructions

                   ICTCM 2011   14
Exam Questions

n   Horizontal axis
n   Shade p-value
n   Make up a research question

                      ICTCM 2011   15
Discussion Points

n   Starting with a significant result but when 
    ready to discuss insignificant?
n   How critical is authentic data?
n   Choice of statistic (count vs. difference in 
n   Role of traditional symbols and notation?
n   Visualization of bar graphs from trial to trial
n   Implementation of predict and test

                         ICTCM 2011                   16
Example 3: Two means

n   Are there lingering effects to sleep 
    q   Randomized experiment
    q   Quantitative data
    q   Parallel inferential reasoning process
        n   Index cards

n   Possible follow-up/extensions: what if -4.33?, 
    medians, plausible values

                            ICTCM 2011            17
Discussion Points

n   Role of tactile simulation
n   Scaffolding of lab report
    q   Introductory sentences, labeling of graphs
    q   Write conclusion to journal
n   When should “normal-based” methods be 
    q   Alternative approximation to simulation
    q   Position, method for confidence intervals
n   Choice of technology
    q   Advantages/Disadvantages
        n   Applets, Minitab, R, Fathom

                                  ICTCM 2011         18
Post-Lab Assessment (Fall 2010)

n   Following the lab comparing two groups on a 
    quantitative variable (65 responses)
    q   Discuss the purpose of the simulation process
    q   What information does the simulation process reveal 
        to help you answer the research question?
n   Essentially correct: 35.4% demonstrated 
    understanding of the big picture (looking at 
    repeated shuffles to assess whether the 
    observed results happened by chance)
n   Partially: 38.5% (one of null or comparison)
n   Incorrect: 26.1% (“better understand the data”)

                             ICTCM 2011                        19
Post-Lab Assessment (Fall 2010)

n   Did students address the null hypothesis?
    q   33.9% E/ 38.5% P/ 27.7% I
n   Did students reference the random assignment?
    q   36.9% E/ 36.9% P/ 26.2% I
n   Did students focus on comparing the observed 
    q   64.6% E/ 13.8% P/ 21.5% I
n   Did students explain how they would link the 
    pieces together and draw their conclusion?
    q   24.6% E/ 60% P/ 15% I

                            ICTCM 2011              20
Student Surveys

                  ICTCM 2011   21
Student Surveys

                  ICTCM 2011   22
Student Surveys

n   Example 3 simulation

                      ICTCM 2011   23
Student Surveys

                  ICTCM 2011   24
Student Surveys

                  ICTCM 2011   25
Student Surveys

n   Helper/Hinderer (Winter 2011) – Did the lab 
    help you understand the overall process of a 
    statistical investigation?

                       ICTCM 2011               26
Student Surveys

n   Did subsequent labs increase understanding?

                      ICTCM 2011              27
Remainder of labs
n   Lab 4: Random babies
n   Lab 5: Reese’s Pieces (demo)
    q   Normal approximation, CLT for binary
    q   Transition to formal test of significance (6 steps)
n   Lab 6: Sleepless nights (finite population)
    q   t approximation, CLT for quantitative, conf interval
n   Lab 7: Simulation of matched-pairs
n   Lab 8: Simulation of regression sampling
n   Chi-square, ANOVA
                             ICTCM 2011                       28
Lab Report

             ICTCM 2011   29
Student Feedback (Winter 2011)

n   Google docs survey during last week of 
n   Two instructors

                      ICTCM 2011              30
Student end-of-course surveys (W 11)

                ICTCM 2011             31
Student end-of-course surveys

                ICTCM 2011      32
Student end-of-course surveys

                ICTCM 2011      33
Student end-of-course surveys

                ICTCM 2011      34
Student end-of-course surveys

                ICTCM 2011      35
Student end-of-course surveys

                ICTCM 2011      36
Student end-of-course surveys

                ICTCM 2011      37
Student end-of-course surveys

                ICTCM 2011      38
Student end-of-course surveys

                ICTCM 2011      39
Student end-of-course surveys

                ICTCM 2011      40
Student end-of-course surveys

                ICTCM 2011      41
Student end-of-course surveys

                ICTCM 2011      42
Student end-of-course surveys

                ICTCM 2011      43
Student end-of-course surveys

                ICTCM 2011      44
Student end-of-course surveys

                ICTCM 2011      45
Top 2 most interesting labs

n   Instructor A
    q   Is Yawning Contagious?
    q   Heart Rates (matched pairs)

n   Instructor B
    q   Friend or Foe
    q   Is Yawning Contagious?
    q   Reese’s Pieces

                           ICTCM 2011   46
Top 2 most/least helpful labs

n   Most helpful:
    q   Friend or Foe

n   Least Helpful (Instructor B):
    q   Random babies
    q   Melting away (intro two-sample t, paired)

                            ICTCM 2011              47
Exam 1

n   In a recent Gallup survey of 500 randomly 
    selected US adult Republicans, 390 said they 
    believe their congressional representative 
    should vote to repeal the Healthcare Law. 
    Suppose we wish to determine if significantly 
    more than three-quarters (75%) of US adult 
    Republicans favor repeal.
n   The coin tossing simulation applet was used to 
    generate the following two dotplots (A) and (B). 
    Which, if either, of the two plots (A) and (B) was 
    created using the correct procedure? Explain 
    how you know.

                          ICTCM 2011                  48
Exam 1

n   35% picked B (usually citing null .75´500)
    q   But some look at shape, or later p-value
n   29% picked A (observed result)
n   23% neither (wanted .5´500 = 250)
n   13% other responses: 0, .75, 50, can’t tell, 
    anything possible, label is wrong
                            ICTCM 2011              49
Exam 2

n   Heights of females are known to follow a normal 
    distribution with a mean of 64 inches and a 
    standard deviation of 3 inches. Consider the 
    behavior of sample means. Each of the graphs 
    below depicts the behavior of the sample mean 
    heights of females. 
     a. One graph shows the distribution of sample 
    means for many, many samples of size 10. The 
    other graph shows the distribution of sample 
    means for many, many samples of size 50. 
    Which graph goes with which sample size?

                        ICTCM 2011                 50
Exam 2

n   85% matched n=10 and n = 50

                     ICTCM 2011   51
Exam 2

n Suppose we wish to test the following 
  hypotheses about the population of Cal Poly 
  undergraduate women:
n For which graph (A or B) would you expect 
  the p-value to be smaller? Explain using the 
  appropriate statistical reasoning.

                     ICTCM 2011                   52
Exam 2

n   77% picked B
    q   Mixture of appealing to smaller SD/outliers, larger 
        sample size means smaller p-value, and thinking 
        in terms of test statistic
    q   A few choices not internally consistent
                             ICTCM 2011                    53
Student understanding of p-value

n   CAOS questions (final exam)
    q   Statistically significant results correspond to small 
        n   Traditional (National/Hope/CP): 69/86/41%
        n   Randomization (Hope/CP): 95%/95%
    q   Recognize valid p-value interpretation
        n   Traditional (National/Hope/CP): 57/41/74%
        n   Randomization (Hope/CP): 60/72%
    q   p-value as probability of Ho - Invalid
        n   Traditional (National/Hope/CP): 59/69/68%
        n   Randomization (Hope/CP): 80%/89%

                                ICTCM 2011                   54
Student understanding of p-value

n   CAOS questions (final exam)
    q   p-value as probability of Ha – Invalid
        n   Traditional (National/Hope/CP): 54/48/72%
        n   Randomization (Hope/CP): 45/67%
    q   Recognize a simulation approach to evaluate 
        significance (simulate with no preference vs. 
        repeating the experiment)
        n   Traditional (National/Hope/CP): 20/20/30%
        n   Randomization (Hope/CP): 32%/40%

                                ICTCM 2011               55
Student understanding of p-value

n   p-value interpretation in regression (final 

                        ICTCM 2011                 56
Student understanding of process

n   Video game question (Final exam: NCSU, Hope, 
    Cal Poly, UCLA, Rhodes College)
    q   What is the explanation for the process the 
        student followed? 
    q   Which of the following was used as a basis for 
        simulating the data 1000 times?
    q   What does the histogram tell you about whether 
        $5 incentives are effective in improving 
        performance on the video game?
    q   Which of the following could be the approximate p
        -value in this situation? 

                           ICTCM 2011                   57
Student understanding of process

n   Simulation process
    q   Fall: over 40% chose “This process allows her to 
        determine how many times she needs to replicate 
        the experiment for valid results.”
    q   About 70% pick “The $5 incentive and verbal 
        encouragement are equally effective at improving 
        performance.” as underlying assumption
    q   Still evidence some look at center at zero or 
        shape as evidence of no treatment effect
    q   1/3 to ½ could estimate p-value from graph

                           ICTCM 2011                   58
Example – 2009 AP Statistics Exam

n   A consumer organization would like a method 
    for measuring the skewness of the data. One 
    possible statistic for measuring skewness is 
    the ratio mean/median….  
    q   Calculate statistic for sample data…
    q   Draw conclusion from simulated data …

                          ICTCM 2011            59
Design Principles

n   Tactile simulation
n   Visual, contextual animation of tactile simulation
n   Intermediate animation capability
n   Level of student construction
    q   Ease of changing inputs
    q   Connect elements between graphs
n   Carefully designed, spiraling activities
    q   “Stop!”
    q   Thought questions
n   Allow for student exploration

                            ICTCM 2011                   60
n   Early in course
n   Repetition through course, connections
n   Normal approximations
n   Lab assignments 
    q   Focus on entire statistical process
    q   Motivating research question
    q   Follow-up application
    q   Thought questions
    q   Screen captures 
    q   Pre-lab questions
    q   Minitab demos (Adobe Captivate)
n   Exam questions

                                ICTCM 2011    61

n   Students quickly get sense of trying to 
    determine whether a result could be “just due 
    to chance”
n   Still struggle with more technical 
    q   Under the null hypothesis
    q   Observed vs. hypothesized value
n   Students may fail to see connections 
    between scenarios
                          ICTCM 2011             62
Suggestions/Open Questions

n   Begin with class discussion/brain-storming on 
    how to evaluate data before show class 
    q   Loaded dice, biased coin tossing
    q   Thought questions
n   Student data vs. genuine research article
    q   “the result” vs. “your result”
n   Choice of first exposure
    q   Significant?
    q   Random sampling or random assignment
                              ICTCM 2011        63
Suggestions/Open Questions

n   Scaffolding
    q   Observational units, variable
        n   How would you add one more dot to graph?
    q   At some point, require students to enter the 
        correct “observed result” (e.g., Captivate)
    q   At some point, ask students to design the 
    q   Start with fill in the blank interpretation?

                               ICTCM 2011               64
Suggestions/Open Questions

n   One crank or more?
n   When connect to normal approximations?
    q   How make sure traditional methods don’t overtake 
        once they are introduced?
    q   How much discuss exact methods?
    n   Confidence intervals

                           ICTCM 2011                  65

n   Very promising but also need to be very 
    careful, and need a strong cycle of repetition 
    closely tied to rest of course…

                        ICTCM 2011                66

To top