Prediction of Pre-college Level English Performance by Writing Lab Hours at the Student Level Salvador Castillo, Director Allan Hancock College Institutional Research & Planning This paper is third in a series on the effectiveness and optimal use of Basic Skill level English and writing training, For Fall 2007 data, I look at the individual effect of Writing Lab hours on English course pass rates, controlling for academic ability. Additionally, we should control for instructor effects, however, such analysis are more complex, requiring special software, and not necessarily more illuminating. An approximation to that is to run analysis for each instructor and look to see if effects differ across instructors. WRITING LAB HOURS A number of the pre-college level English courses, including credit ESL courses, require a set amount of weekly writing lab hours as part of the general course requirements. The English 300 and 500 series require about 26 hours from each student per semester. ESL courses require 16 hours. Of course, a passing grade is affected by other factors beyond writing lab hours. General ability, facility with language, personal work ethic, motivation are all student level factors. Faculty perceptions, integration of writing lab into curriculum, etc, also affect student performance at the course level. Some, but not most of these factors, are measurable and available. METHOD The outcome variable for most of the courses associated with the writing lab is binary, since the grade will be “CR” (passed with a “C” or better) or “NC” (not passed, no credit). The withdraw grade may also be considered a “not pass”, although some students withdraw even if they are not in academic trouble. A problem with binary outcome variable is that traditional regression analysis is not appropriate, as it is geared toward continuous variables, not those with only two outcomes. Logistic regression is a technique that can be used for such variables, but a certain compromise must be made. Namely, the outcome itself cannot be modeled (pass, no-pass or 1, 0), but rather we model the probability of a given outcome. This is perfectly fine for a statistical point test of a variable (either it predicts or it doesn’t), however, it makes interpretation of the magnitude of the effect more abstract and problematic. The probability distribution of an outcome is not linear, so we don’t have a constant amount that a predictor variable changes an outcome. One common measure of effect is the “odds ratio”. Essentially, this is a measure of how much a 1 unit change a predictor variable changes the “odds”, where the statistician’s definition of odds is the ratio of the probability of the positive outcome over the probability of the negative outcome. When
-1-
things have no effect, we have even odds and the ratio is 1.00. If we have an odds ratio of 1.1, then a one unit increase in the predictor increases the probability of the outcome by 10%. Similarly, an odds ratio of 0.9 reduces the probability by 10%. In addition to writing lab hours, I computed the cumulative grade point average (GPA) up to and including the Fall 2007 semester. Since many students were first time students to AHC, the only way for me to determine an ability level was to use the current semester GPA. SAMPLE The cumulative writing lab hours for Fall 2007 were taken from the EWL 150 report and converted into an SPSS data file. The data was then merged with Fall 2007 final grades. Students with data under multiple versions of names in the EWL 150 were combined into one record within a given course. Thus, all Fall 2007 students enrolled in the 300 or 500 series English courses, including credit ESL, whose courses had any students with writing lab house, were included in the analysis. Students with missing lab hours were counted as having zero lab hours. RESULTS The first analysis looks at the entire data set, using cumulative GPA and writing lab hours as predictor variables. Table 1 demonstrates that both variables are statistically significant predictors of achieving a passing grade (CR) grade in English 300 and 500 series classes. Note that effect sizes (B) and odds ratios are cast in the scale of the original variable and thus could be misleading. For example, the slope for Writing Lab Hours is 0.042, while that for GPA is 0.585. This is seemingly a large difference. However, the Writing Lab Hours slope indicates how much the probability distribution increases for a 1 hour increase in writing lab hours, while the GPA slope indicates the change for a 1 grade increase in GPA. Clearly, 1 hour increase is a much smaller change than a 1 grade GPA change (say from a 2.00 to a 3.00). A ten hour increase in Writing Lab hours increases the slope by .42, almost as strong as the GPA increase. Table 1. Logistic Regression Predicting A Passing Grade in English 300 and 500 Series Courses in Fall 2007
Overall Model B Intercept Writing Lab Hours Overall GPA N % passing grade (CR) Model χ2 Pseudo r2 * * The Cox & Snell pseudo r2 -1.690 0.042 0.585 447 69% 63.1 .132 S.E. 0.373 0.120 0.373 Wald χ2 20.57 22.80 23.60 sig .000 .000 .000 Odds ratio 1.043 1.795
Note: statistically significant values are in bold.
-2-
Eighteen additional hours of Writing Lab per semester increase the probability of passing an English 300 or 500 level course by the same amount as a 1 grade GPA difference.
A 2.00 GPA English student (300 or 500 series) who takes 18 writing lab hours more than the average 2.00 GPA person has the same probability of a passing grade as someone with a 3.00 GPA
Also of interest is modeling by English course type. The following tables provide the outcomes by the English 300 course, English 501/506 and the ESL courses (all considered together). Table 2. Logistic Regression Predicting a Passing Grade in English 300 in Fall 2007
Overall Model B Intercept Writing Lab Hours Overall GPA N % passing grade (CR) Model χ2 Pseudo r2 * * The Cox & Snell pseudo r2 -2.686 0.055 0.479 174 55% 33.9 .177 S.E. 0.675 0.014 0.211 Wald χ2 15.85 16.17 5.16 sig .000 .000 .023 Odds ratio 1.057 1.615
Note: statistically significant values are in bold.
For English 300, the results are similar to the overall model except that writing lab hours are a stronger effect in relation to a cumulative GPA differential. The hours “equivalency here is about 11. In other words, all else being equal, 11 more hours of writing lab in a semester is about as good as a 1 grade GPA increase. For English 501/506, both predictor variables are statistically significant. The effect is about the same as the general model. About 17 hours of “above and beyond” writing lab hours work out to the equivalent probability increase of passing to a 1 GPA increase.
-3-
Table 3. Logistic Regression Predicting a Passing Grade in English 501/506 in Fall 2007
Overall Model B Intercept Writing Lab Hours Overall GPA N % passing grade (CR) Model χ2 Pseudo r2 * * The Cox & Snell pseudo r2 -1.721 0.059 0.707 104 75% 19.8 .173 S.E. 0.762 0.021 0.263 Wald χ2 5.10 7.62 7.24 sig .024 .006 .007 Odds ratio 1.061 2.027
Note: statistically significant values are in bold.
Table 4. Logistic Regression Predicting a Passing Grade in Credit ESL in Fall 2007
Overall Model B Intercept Writing Lab Hours Overall GPA N % passing grade (CR) Model χ2 Pseudo r2 * * The Cox & Snell pseudo r2 -1.290 0.056 0.781 54 80% 11.9 .198 S.E. 0.902 0.042 0.312 Wald χ2 2.05 1.80 6.28 sig .153 .180 .012 Odds ratio 1.057 2.184
Note: statistically significant values are in bold.
For Credit ESL students, Writing Lab Hours was not a statistically significant predictor of the probability of obtaining a passing grade (at the 95% confidence level). Cumulative GPA was statistically significant and as strong a predictor as in the previous models if not stronger. However, the Credit ESL analysis is hampered by the fact that many students in this population do not take any courses for a letter grade, and thus about half of such students (53 of 107) were not included in the analysis because they were missing a GPA. Indeed, for this population, cumulative GPA is of little value as an indicator of academic ability. A simple comparison of mean hours demonstrates that those who withdrew or did not pass their ESL courses averages 16 writing lab hours while those that passed average 27 hours or about 11 hours more.
-4-
Figure 1. Fall 2007 Credit ESL Pass Rates by Writing Lab Hours
Pass Rate 120% 100% Passing Percent 80% 60% 40% 20% 0% 0 to 4 5 to 9 10 to 14 15 to 19 20 to 24 25 to 29 30 to 34 35 to 39 40 to 44 45 to 49 W riting Lab Hours
Writing lab hours are most efficient for ENGL 300 students, as an increase of 11 hours over the average is about equal to an increase of 1 grade in GPA
ENGL 500 series students need similar hours to the overall model, about 17, to see large increases in probability of passing.
Credit ESL students with less than 10 writing lab hours have about a 50% chance of passing whereas ESL students with more than 25 hours have about 100% chance of passing.
-5-
Ability differences are not the sole factor in determining pass rates; the difference for the general population is about a 10% increase in probability of passing for a 1 GPA increase in ability. This is a moderately significant increase (indeed, for education, it is on the high side), but by itself does not assure passing. In addition, integration of lab hours into the curriculum is a complex issue that is not easily measured by simple counts of lab hours. CONCLUSION Writing Lab hours are clearly associated with higher pass rates for all pre-collegiate level English courses. With the non-linearity of the selected model, it is not possible to provide a single estimate increase in pass rates. But relative comparisons demonstrate that a difference of 10 to 20 hours is equates to a difference of 1 grade level in GPA. GPA is a proxy for ability.
-6-