The SCALE Efficiency Projects by malj


									The SCALE Efficiency Projects

           By Lanny Arvan
             John C. Ory
          Cheryl D. Bullock
         Kristine K. Burnaska
          Matthew Hanson
               May 1998
                     The SCALE Efficiency Projects

        This study is essentially the only one of which we are aware that actually
documents increases in instructional productivity attributable to using computer
technology in instruction in any form. (The notable exception is some work done at
Michigan State University by Ed Kashy, Michael Thoennessen, et. al., [1], [2].) That
such work is indeed rare is confirmed by Polley Ann McClure [3]. “While there are
some cases in which we can document improved educational output as the result of
technology intervention, in a brief survey of the literature, I could find no studies
documenting improved educational output per unit cost. The educational gains have been
at huge cost, in terms of investment in both equipment and software, but more
significantly, in faculty and support staff time.” Similarly, David Noble [4], a notable
opponent of online education, cites the work of Kenneth Green [5] when arguing,
“Recent surveys of the instructional use of information technology in higher education
clearly indicate that there have been no significant gains in either productivity
improvement or pedagogical enhancement.”

        That such documentation is so rare suggests two possible hypotheses: (1) it is not
possible to generate productivity increases with computer technology, and (2) it is
possible, but the incentives are not right for us to witness them. Robert Koob [6] argues
for the second hypothesis. If Koob is correct and the „right‟ incentives could be
identified, it is nevertheless not trivial to tinker with faculty reward schemes. There is a
natural reluctance in doing so, owing to institutional inertia and for fear of making
matters worse. Consequently, it seems important to first refute hypothesis (1) or at the
least provide substantial evidence that it is unlikely to hold before making wholesale
changes in the way faculty are compensated for their teaching. The SCALE Efficiency
projects are meant as a first step in this direction.

        While there has been scant evidence on the enhancement of productivity
attributed to instructional technology, there is an emerging cottage industry of thought
pieces on how instructional technology should be used for advancing productivity ends.
Much of this work has come out of Educom‟s National Learning Infrastructure Initiative
(NLII). Examples include the papers by Carolyn Twigg [7], [8], D. Bruce Johnstone [9],
and William F. Massy and Robert Zemsky [10]. The ideas behind the SCALE Efficiency
projects have been influenced by this work. Yet it should be understood that making
these ideas operational requires compromise, in both the implementation and in the
measurement. It is our hope that this paper gives the reader some insight into the type of
compromises that are needed to get actual productivity projects underway and the variety
of measurement problems that arise as a consequence.

       Furthermore, there is a fundamental conceptual point that should be considered
where the NLII philosophy departs from the ALN philosophy. The basis of the NLII
thinking is that educational technology is capital and that any productivity gains must
come as capital input substitutes for labor input. While this capital for labor approach is
not entirely absent in the ALN approach, it is not the whole story. With ALN, much of
the productivity increase comes from labor for labor substitution – inexpensive student
labor for expensive faculty labor. Viewing the students‟ time as a productive input, as
suggested by Lanny Arvan [11], some of this productivity gain arises from peer-to-peer
communication. (Note that we don‟t cost-out this student time in the measurement
component of this paper, though we do provide some demographic evidence that suggests
how such a costing-out should be done.) And some of the productivity gain emerges
from student interaction with „peer tutors‟ who receive remuneration for the help they
provide. In the ALN approach, it is critical to view networked computers as, in part,
communication tools. This allows the ALN approach to make the instruction more
personal while simultaneously increasing productivity. At least, that is the ideal.

        The reader will likely benefit from a very brief history of SCALE and where the
Efficiency Projects fit into the bigger picture. We hope this will also serve to dispel some
criticisms of this work, by allowing the reader to better understand what it is that we are

        SCALE was formed in spring 1995 with a three-year grant from the Alfred P.
Sloan foundation and a generous match from the University of Illinois at Urbana-
Champaign. SCALE‟s primary mission was to support ALN course development in an
on-campus setting. Initially, Sloan had set four targets for this on-campus ALN to
achieve. These were to improve retention, to decrease time to degree, to demonstrate
verifiable increases in student learning, and to lower the cost of instruction. Over time,
these targets have been modified, based on the experience with ALN and its
implementation on the UIUC campus.

        There has been an independent evaluation team from the start of the SCALE
project. That team is headed by John C. Ory and includes Cheryl Bullock and Kristine
Burnaska. Earlier evaluation work of the SCALE project can be found at [12], [13], [14],
and [15]. These are semester-by-semester evaluations starting in fall 1995 and
culminating in spring 1997. Matthew Hanson joined the evaluation team in summer
1997, to work exclusively on the Efficiency Projects. While the evaluation team has been
in frequent contact with SCALE administration and, in particular, the evaluation strategy
of the Efficiency Projects was discussed extensively, the actual data collection effort has
been the sole province of the evaluation team. This independence helped to minimize the
chance of misrepresentation of the findings and to reduce the awkwardness involved in
the data collection, particularly in those cases when students or faculty reported that
things weren‟t going so well.

        There has been a change of thinking within SCALE administration about how to
deliver on the Sloan objectives. During the first year of the SCALE project, there was an
expectation that the desired efficiency outcomes would come as a byproduct of ALN
implementation. It was also expected that efficiency gains could be had in all ALN
courses. In fact, most SCALE-affiliated faculty reported increased time involved in
instruction as a long-term proposition, because of the increased contact with the students
online. It became apparent that the byproduct approach would not achieve the desired

results. Moreover, it also became clear that efficiency outcomes would be difficult or
impossible to attain in small ALN classes. There were two reasons for this that perhaps
should have been obvious at the outset of the project but were not. First, if there was
substantial up-front development in a small class such development could not be
amortized over a large number of students. Second, in a small class there is very limited
opportunity to exploit labor for labor substitution. When SCALE administration
ultimately contracted for the efficiency projects [16], SCALE targeted large classes only.

        Another consequence of abandoning the byproduct approach was the growing
realization that efficiencies could not be had right off the bat (unless the project was a
derivative of an already successful project). Instructors need to learn how to run a class
with ALN. This means more than learning the mechanics of the software. It means
getting comfortable with this style of teaching. When this comfort zone has been
reached, opportunities for efficiencies may become apparent to the instructor that were
not so when the instructor first started out with ALN. This is not a readily testable
proposition. Yet it seems a sensible warning for others who might try to replicate the
SCALE approach.

        The SCALE Efficiency projects, then, represent mature ALN development in
large classes that has now been focused on efficiency ends. There are many other ALN
courses that SCALE currently supports where no attempt is being made to produce
efficiency outcomes. Among these are some large classes. Thus, we are not arguing that
large size per se makes a class a good candidate for an efficiency project. For example,
SCALE supports an introductory comparative literature course that enrolls about 250
students a semester. The course is taught with a lecture once a week. There are also
small sections run by graduate assistants under the supervision of the faculty member
who delivers the lecture. The course is writing-intensive and satisfies the campus
„Composition II‟ requirement. In spite of the course size, the possibility for capital
substitution is limited here. Competent evaluators must assess the students‟ written work.
Computer assessment of the writing is not possible, because the assessment is so
contextually based. It can‟t be done via a search for key words. This requirement of
competent assessment also limits the possibility of labor for labor substitution in this
course. We think that ALN is improving learning in this course. But we have no way to
quantify that learning, so the course is not one of our Efficiency Projects. There are also
SCALE-supported courses currently taught in such an inexpensive manner – large lecture
with few if any graduate assistants to support the course – that it seems foolhardy to try to
further reduce the cost of instruction.

        We are also not arguing that the SCALE approach can work everywhere. The
reliance on peer tutors, in particular, requires highly able students who can serve in this
capacity and feel they are doing something socially beneficial in the process. The
SCALE approach likely can work well at other institutions in the Big Ten and at other
similarly regarded public campuses. To what other institutions the approach can be
profitably extended is an open question.

         One further point bears mention here. There has been a negative reaction to using
educational technology for efficiency ends, emerging from various pockets of concerned
faculty [17], [18]. Much of this reaction relates to the effect on faculty employment. The
capital substitution argument would seem to suggest a need for fewer faculty. Certainly
there is a fear that this will be the case. Reducing faculty employment is viewed as „bad‟
in many quarters. It is our view that on the UIUC campus the SCALE Efficiency
Projects will have little or no impact on faculty employment, though we do anticipate a
big impact from these projects overall. It is graduate student employment that will be
effected the most dramatically, if the SCALE Efficiency Projects become more
widespread on campus. The reason for this is simple. In the vast majority of the courses
that SCALE has been targeting, graduate students do the bulk of the teaching. The
course coordination function remains, even with ALN, in the hands of a faculty member.
The upside of this is to reduce the pressure on graduate student enrollment to staff large
introductory undergraduate courses. This should allow graduate student enrollment to
better track the new Ph.D. job market in the individual discipline and to better match the
quality of the particular degree program. Furthermore, to the extent that the changes in
graduate student enrollment can be made without disenfranchising students who are
currently enrolled, simply by adjusting the size of entering cohorts, it is not obvious that
there is a downside to this approach.

        The remainder of this paper is divided into two parts. Part 1 is the cost
accounting component. In addition to brief descriptions of each project, costs are broken
up into operating and development costs and comparisons are made to the pre-ALN mode
of teaching the course regarding operating costs, to get some sense of how long it takes to
recover the development costs. Some discussion is provided regarding issues of cost
measurement. Part 2 is the outcomes-assessment component. There is also a section on
measurement issues in this part. Some demographic results are presented. There is a
discussion of common findings across projects. Where possible, comparative objective
performance data and comparative attitudinal results are presented. In some cases, only
absolute attitudinal results are available.


1. Kashy, E., Thoennessen, M., Tsai, Y., Davis, N. E., and Wolfe, S. L., “Using
    Networked Tools to Enhance Student Success Rates in Large Classes,” Proceedings
    of the Frontiers in Education,, 1997.
2. Kashy, E., Thoennessen, M., Tsai, Y., Davis, N. E., and Wolfe, S. L., “Application of
    Technology and Asynchronous Learning Networks in Large Lecture Classes,” 31st
    Hawaii International Conference on System Sciences, Volume I, Collaboration
    Systems and Technology Track, page 321, edited by Jay F. Nunamaker, Jr., 1997.
3. McClure, P. A., “„Growing‟ Our Academic Productivity,” in Reengineering Teaching
    and Learning in Higher Education: Sheltered Groves, Camelot, Windmills, and Malls,
    edited by Robert C. Heterick, Jr.,
    resources/ir-library/abstracts/pub3010.html, 1993.
4. Noble, D. F., “Digital Diploma Mills: The Automation of Higher Education,” First
    Monday,, 1998.
5. Green, K., “The Campus Computing Project,”, 1997.
6. Koob, R., “New Funding Paradigms: The Need for New Incentives,” NLII Viewpoint,
    Vol. 1, Issue 1,, 1996.
7. Twigg, C., “Academic Productivity: The Case for Instructional Software,”, 1996.
8. Twigg, C., “The Need for a National Learning Infrastructure,”, 1994.
9. Johnstone, D.B., “Learning Productivity: A New Imperative for American Higher
    Education,” NLII Viewpoint, Vol. 1, Issue 1,, 1996.
10. Massy, W. F., and Zemsky, R., “Using Information Technology to Enhance
    Academic Productivity,”,
11. Arvan, L. “The Economics of ALN: Some Issues,” Journal of Asynchronous
    Learning Networks, Volume 1, Number 1,, 1997.
12. Ory, J. C., Bullock, C.D., and Burnaska, K.K., “SCALE Fall 95 Evaluation Results,”, 1996.
13. Ory, J. C., Bullock, C.D., and Burnaska, K.K., “SCALE Spring 96 Evaluation
    Results,”, 1996.
14. Ory, J. C., Bullock, C.D., and Burnaska, K.K., “SCALE Fall 96 Evaluation Results,”, 1997.
15. Ory, J. C., Bullock, C.D., and Burnaska, K.K., “SCALE Spring 97 Evaluation
    Results,”, 1997.
16. Arvan, L., “Bottom Up or Top Down, Using ALN to Attain Efficiencies in
    Instruction,” Proceedings of the Third International Conference on Asynchronous
    Learning Networks, , 1997.
17. Berube, M. “Why Inefficiency is Good for Universitites,” Chronicle of Higher
    Education, Vol. 44, Number 29, 1998.

18. Young, J. R., “Technology May Not Be All That Great, Say Professors at 'Second
    Look' Meeting,” Chronicle of Higher Education, Vol. 44, Number 34, 1998.


To top