Fishing Expeditions
Purpose
This model allows the user to generate Type I error through the process of multiple testing
(fishing.) The user will define twenty binominal factors (i.e can only have two states: present
and absent) and then the computer will random assign observations form these parent
populations into two samples. Next, the user will have the opportunity to discover if there any
real or apparent differences between the two samples. Keep in mind that since the members
of both samples came from the same parent distribution, any such differences were due to
Directions
Define the names of the twenty variables in the shaded column labeled "Factor" You may
also choose to use the default values.
In the shaded cells besides each of these variable names, indicate the proportion of the
population that has that Factor present. This should be a number from 0.00 to 1.00. Note
that the computer will automatically calculate and display the proportion of the population who
Press the F9 key. This will automatically randomly draw members from the defined
population and assign them to Sample A and Sample B. This process will actually take place
twice. At that top of the screen, the computer will draw samples with 10 observations each
and at the bottom of the spreadsheet it will draw samples of size 50. The computer will place
the expected number of observations with the factor in Sample A in the column labeled "#
If you scroll to the right, you can see the actual observations in each sample. A "1" indicates
the factor is present and a "0" indicates it is absent.
The computer will then report on summary statistics which include: a) the actual number of
observation in sample A that has the factor (column labeled "# in Sample A"), b) the actual
number of observations in sample B that has the factor (column labeled "# in Sample B"), c)
the difference in these two number of observations (column labeled "diff"), and d) the ratio of
the # with the factor in sample A versus the # in sample B (column labeled "ratio").
The second set of two samples (n=50 each) also reports on whether the difference found
between the empirical proportions observed in the random samples is statistically significant.
These are reported as either significant as a one tailed or two tailed test.
You may change the values of any shaded cell and/or press F9 to draw new samples.
Questions
Define the population factor proportions as described in the directions and press F9. Look at
the top table (in which sample sizes are 10 each.) Were there any cases in wiich there is a
ratio of 2 to 1 or higher between the two samples? If yes, why? If no, press F9 until that
Scroll down to the results of the samples of size 50 can be seen. Look at the column labeled
"Statisical significance." Press F9 until there is significance. How can there be a statistically
significant difference between the groups if the observations in both samples were drawn
Continue to press F9 for about a dozen times. What can you say about the pattern of
statistically significant results?
Developed by Dr. Scott Wetstone, (860) 679-4440
Factor present absent Expected sample A sample B diff
Male 0.50 0.5 5 4 7 3
Married 0.50 0.5 5 6 5 1
Wears glasses 0.50 0.5 5 3 4 1
Electric blanket 0.20 0.8 2 3 2 1
Lead paint 0.60 0.4 6 5 4 1
>1 hour TV daily 0.70 0.3 7 9 8 1
>2 meals out/wk 0.80 0.2 8 7 7 0
Has a cat 0.40 0.6 4 3 1 2
Has a dog 0.33 0.67 3 3 6 3
Speeding 0.20 0.8 2 2 2 0
Childbearing years 0.90 0.1 9 9 8 1
>2 cups coffee/day 0.60 0.4 6 6 7 1
Plays video games 0.10 0.9 1 2 0 2
Cancer in family 0.30 0.7 3 2 5 3
Visited Mexico 0.10 0.9 1 1 0 1
Exercises regularly 0.30 0.7 3 2 3 1
Full-time employment 0.60 0.4 6 5 6 1
Daily vitamin 0.40 0.6 4 6 4 2
Leo astrological sign 0.08 0.92 1 1 0 1
8+ hours sleep/night 0.50 0.5 5 5 5 0
ratio
1.8
1.2
1.3
1.5
1.3
1.1
1.0
3.0
2.0
1.0
1.1
1.2
***
2.5
***
1.5
1.2
1.5
***
1.0