Information Retrieval Cost by jle31578


More Info
									A Cost Model for Evaluating Information Retrieval Systems

This paper develops a mathematical model for use in             performance level of the system and the combination
studying how to minimize the cost of operoting a me-            of user and system time that is expended in working
chanized retrieval system. Through the use of cost anal-        with the system. With this approoch it is possible to
ysis, the model provides a method for comporative               determine the allocation of user and system. This
evaluation between information retrieval systems. The           minimizes the total cost of operating the system. This
cost model divides the costs of a retrieval system into         allocation is done for a given performance level and
two components: system costs and user costs. In addi-           for a given cost per unit of user and system time.
tion, it suggests that a trade ofF exists between the

                                                                MICHAEL D. COOPER

                                                                School of Librarianship
                                                                University of California
                                                                Berkeley, Califonna 947S0

* Introduction                                                  amount that must be paid to use the system.
                                                                  3. The benefit that the patron can gain by using the
   One method whereby information retrieval systems can         system.
he evaluated is through the use of costs. In order for            4. The most economic division of effort between the
accurate comparative analysis to be conducted, the cost         user and the system in accomplishing the user's search
methods used must be consistent and comprehensive. The          objective.
model developed in this paper has two facets. It develops
equations for a total cost of operating the system and            In this model, the user formulates a query and the sys-
thus allows comparative evaluation between other sys-           tem decides, on the basis of control parameters aud pre-
tems. Further, once the cost equations for the system           vious experience, the retrieval rule most likely to yield
have been presented, it shows that an optimal division          documents relevant to the user's request.
can be made between those functions that the user should
perform and those that the system should perform in
optimizing the search process.                                  • Retrieval Activities
   Most information retrieval systems that have been de-
signed to date use a technique of comparing a query rep-           Acquisition of information from an automated retrieval
resentation with stored document representations in             system involves an interaction between the user and the
order to retrieve documents that satisfy the request.           computer. As with any man-machine interaction, the
Those documents whose representations are 'closest' to          more demanding and more sophisticated the user is in his
the query are retrieved. It is hypothesized that this type      requests, the greater will have to be the system effort to
of comparison process is not sufficient to insure the 'best'    achieve the desired goal. In this section the trade off
operation of the retrieval system. Not only should the          between user and system effort is explored and a schema
system perform matching but it should take into account:        is developed for analyzing it.
                                                                   It is possible to distinguish three stages in the interac-
   1. The cost of the search to the system. This implies        tion of a user with the retrieval system. The process be-
that the system provides a service to the user at a spe-        gins with pre-search activities. For the user this involves
cifiable price.                                                 determining what is to be asked of the retrieval .system
   2. The cost of the search to the user. This suggest.s that   and mapping the request into the system's formal qucr\'
the user places a value on the time and effort he spends        language. Since it is unlikely that the user will enter the
using the system. This quantity is in addition to the           correct query the first time he tries (perhaps due to syn-

306     Journal of the American Society for Information Science—September-October 1972
tax or spelling errore) there will be some user-system dia-                             TABLE 1. User and System Activities
log involved in putting the request into a form acceptable                               User Activity             System Activity
by The system.
   Query negotiation may be a simple process of correct-                                 Determine information Syntax check of
ing syntax, as noted above, or can be more elaborate. For                                 need                   f|iier>'

example, the system may be in a position to aid the user                                 Enter quer>'              Thrsaunis lookup
in query formulation by the use of a thesaurus and/or a                   Pre-Search     Query negotiation         Quer>' term frequency
word frequency list. The thesaurus is employed to tell                                                               analysis
the user the generality or specificity of the words that are
                                                                                                                   Map qucr>' into
present in the query. This allows the user to broaden or                                                             fornial language
narrow the scope of the query depending on the search
objective.                                                                               Wait                       Select comparison
    Through the use of word frequency distributions, the                                                              method (retrieval
                                                                          Search                                      rule)
system can tell the user the extent to which a particular
term has been used as, for example, an index term in the                                                            Search file
indexing of the document collection. When this informa-                                  Read output                Display output
tion is supplied, the user can judge the quantity of mate-
rial that will be retrieved for a f!;iven request.                                       Mark relevant              Calculate perform-
                                                                                          material                    ance measure
    The second stage in the retrieval process is the search               Post-Search
 activity. It is in this st^ge that the comparison of the                                Use relevant               Kevise strategy
 formal representation of the user's request is made to                                   material                    and/or query with
stored document representations. Consequently the sys-                                                                feedback
 tem's effort in this stage of the process is greatest, and
 the user is resigned to waiting for the results to be
 displayed.                                                               background, degrees, current research interests, employer,
    The final stage is concerned with post-search activities.             etc.), the journal in which the paper was published, the
 The retrieval system has predicted which documents                       editor, the references, etc.
 satisfy the request and now must display the output for                    Thus one can see that there are a number of alternate
 the user. With the documents displayed in front of him,                  document surrogates that can be stored in the file and
 the user then evaluates the retrieved documents in terms                 used for retrieval. In this paper these alternate forms
 of their relevance to his information need. The system                   are called representations of a document. Examples of
 uses this information to calculate a performance measure                 document representations include the title, author(s),
 for the search. In addition, a feedback mechanism op-                    abstract(s), full text, index terms, citation, cluster cen-
 erates to revise the search procedure in light of the user's             ter descriptions, etc. of a document.
 satisfaction. Table 1 summarizes the user and system
                                                                          • User-System Interaction
* Document Representation                                                    The user has a number of alternative strategies that
                                                                          he can employ in his information socking behavior. In-
   A number of developments suggest that useful informa-                  stead of employing an information retrieval system, the
tion is being ignored when the only keys that can be used                 user may browse through his personal library, consult a
to retrieve documents from a file are author, title and                   co-worker, phone a friend, or consult a reference librarian.
index terms. Consider, for example, the MARC format
                                                                             The user's time is an economic quantity. Given the
for the interchange of bibliographic information on mag-
                                                                          cost of this time and the fact that there are a number of
netic tape iiS). It is suggested that a large number of
                                                                          information seeking alternatives, Simon's concept of sat-
the fields in this format are useful means of accessing
                                                                          isficing appears particularly applicable (9, 14)** The
bibliographic information. Kessler's work on bibliographic
                                                                          user pursues a selected information seeking strategy until
cou]}ling suggests the iisefulness of storing document cita-
                                                                          the cost incurred exceeds the level of satisfaction received.
tions in the file to allow citation indexing and searching
                                                                          At this point another strategy could be adopted or the
 (13). Another possibility is that of including non-content
                                                                          process could stop with the user satisficed.
 information about the document (i.e. context informa-
tion) in the file (10).* Context information, as distin-                     It is suggested that a number of variables are tested
guished from content information is that material that                    by the user in deciding whether his cost of a particular
 describes characteristics of the author (his academic                    strateg>' has exceeded the benefit. These variables include
                                                                          the time the user spends at the console of the informa-
    * Preliminary evidence using a amall corpus has not indicated the
 usefulness of this data for searching, but these results are liaaed on      ** Baker and Nancp ( I ) have also suggested the applicability of
 too small H Gsniple to be considered indicative or final (^2).           this idea to information seeking behavior.

                           Journal of the American Society for Information Science—September-October 1972                                 307
tion retrieval system, the time required to map the re-                      be employed. Traditionally one or perhaps two of these
quest into the retrieval system's query language, and the                    have been implemented in a given operational system. In
waiting time until the results of the search are displayed.                  the proposed system, however, all the comparison meth-
   Another group of variables that determine user cost,                      ods will be available to the user. This is done on the
more specifically relate to man-machine interaction (3,                      theorj- that a specific strategy will have certain properties
p. 67). The design of the console, the flexibility of the                    that make its use advantageous in specific circumstances.
programs in allowing the user to go as slow or as fast as                    For example, comparison method X may be found to be
he wants in a dialog witii the machine, all contribute to                    extremely exliaustivc in its search for documents meeting
his willingness to use the machine and the value that he                     the quer>' objective. Method Y, on the other hand, may
places in the retrieval sj'stem.                                             be particularly useful when searching for one specific sub-
   Finally, the user Is influenced by the results that he                    ject. Then the user will have the ability to decide which
obtains from the system. It is in this area that measures                    strategy to employ. Alternatively he may rely on the
of retrieval effectiveness are valuable. They provide the                    system to pick the strategy, or may be forced to pick one
user with a measure of the degree to which he is satisfied                   because of a requirement for a specific performance level
with the system.*                                                            or becau.sp of a given budget constraint.
   The cost that the user assigns to the employment of                          In addition to the need to pick a particular search
the .system is a combination of the factor described iu                      comparison method or retrieval rule for a specific pur-
this section. If the system does not satisfy the user re-                    ITOse. the user ha.s the ability to select a document rep-
quirements, it is not used. Thus the user cost-bene6t                        resentation that will be used to compare the query
function is a constraint that is considered by tho system.                   against.
This is accomplished in a number of ways. For a given                           A document has associated with it a number of repre-
quer>' the retrieval system predicts for the user the cost                   sentations. These surrogates include index terms, ab-
of the search and the time required to perform it. The                       stracts, subject headings, etc. Wlien a search i;- made,
user can then broaden or narrow his request Riven his                        the retrieval system picks a particular representation to
budget constraint.                                                           compare the query against. For example, if it is desired
                                                                             to find an article written by author W. a search of the
                                                                             author rejiresentation is conducted.
• Seorcli Methodology                                                           The search for a particular author is the easiest case
                                                                             for the system to handle. This is so because the system
   A complex proces,s is undertaken when a retrieval S>"F-
                                                                             knows which representation to compare against. Bvit
tem attempts to find material relevant to a user need.
                                                                             take the case of a request which is in the form of a set of
The model of user interaction in the previous section sug-
                                                                             words characterizing a cho.sen subject. Here tho problem
gests that information seeking patterns var\" according
                                                                             is more complex because there arc a number of repre-
to user cojit and benefit. This section elaborates the prob-
                                                                             sentations that could be used for the comparison, ,such as
lem further by suggesting the need to pick an ojitimal
                                                                             the document index terms or the document abstract.
combination of search comparison method and document
representation for the search.                                                  The user then has the flexibility to decide which repre-
   There are a number of search comparison methods                           sentation will be used in his search or alternatively to
that can be used in an information retrieval system.                         let the system decide. If a broad survey of literature is
These included the simple matcliing technique, in which                      desired, there may be more benefit in using subject head-
the quer\' is compared with the document representations                     ing--; than author assigned index terms for the .search
and the degree of similarity between the two is calculated.                  com]iarlson. On the other hand, if a ver>' specific ques-
FAiensions of the methodology include the elaboration                        tion is yiosed, the query may only be able to be answered
of the terms in the query with related terms to efFect as-                   through searching the abstracts or full text of a docu-
sociative searching. Alternatively it is possible that in-                   ment. Here again there is a trade off between the degree
stead of looking at cver>' document representation in the                    of generality or specificity required in the search and
file, chistering could be employed, and only cluster center                  the cost and benefit of conducting it. By allowing this
representations be compared to the query representa-                         flexibility to exist, the system stands a better chance of
tion.**                                                                      satisfying a variety of users.
    Thus there are a number of different strategies that can
    • An cvaiuatiTP mpasiire with properlies Bimilnr to the n'wari)-
coiit concept (tiscuMed OBrlier in this nectinn hns been propoacd by         • Retrieval Model
I. J. Good (t).      He BujrcPstR that a linear relation between num-
ber o( documentg retrieved and the value of those dorument« to tho
URer hHR not been eHtabiished. Tho author hyimtheBizes thnt a more              Optimal performance of a retrieval system requires
complex niapplnK function between vnlue and number of documents
is involved.                                                                 that both system and user resources be considered in de-
    " T h e desrription of the rPtrievnl system nivPii in tliis spctitm i*   termining an operating level. This section considers the
Htructured to convey the roncept of resource allocation in retrieval
syFit«m design. The Aiternntive approach of a retrieval system de-           issues involved in selecting such a level. It also sketches
Bifcn foT a partieuliir file utrurture is not considered. The allocation
model in independent of the die structure used.                              the manner in which the system's strat^y can be modi-

308       Joumal of the American Society for Information Science—September-October 1972
fied in the light of changes in U8er assessment of system     that the curves are L shaped or even straight lines. How-
benefits.                                                     ever, assume for the following discussion that the per-
   The total cost of a retrieval system's operation for a     formance can be characterized by an equation such as
querj', Cy, is the sum of the sj'stem cost and the user       (3). Then it is possible to solve equations (1) and (3) to
cost. That is                                                 find the optimal level of („ and fg that minimizes total
                                                              cost for a given performance level.
                     C =UCu-^-t,c,                    (1)        First rewrite equation (3) as
where c,, is the cost per unit of user time and („ is the
amount of user time required for a given search. Similarly       Then using the Lagrange multiplier A, form the equa-
€„ is the cost per unit of system time and tg is the amount   tion
of system time required for a given search.
   It is presumed that retrieval system performance can                     Fx = (-Cu -h t,C. -\- \(tut. — P).         ^
be characterized by a measure called P. It is believed that   The application of partial differentiation yields
P is a complex function that may include variables such
as those used in calculating measures of retrieval effec-
tiveness. For this model the measure is considerably sim-
plified so that it has the form             ,
                      P = f{t^,t,).                   (2)
That is, the performance is a function of the amount of                            3\          "'
user time and system time expended on the search. A           Then
number of more specific formulations are possible. For                                   t, = — c./\             "   0>:-
example, the relation could be                                and
                                                                                    t^ = — c./\.                   (10)
This is the form of an isoquant curve from economic           Substituting equations (9) and (10) into (8) yields
theory. (See Figure 1). Each point on an isoquant curve                                             0             (11)
represents the maximum output that can be produced                                      X =
with a given combination of inputs. Each of the curves
                                                              Then the optimal („ is
of Figure 1 show combinations of user and system time
that yield the same level of performance, P^.
  Very little information is available about the precise      and
shape of the performance curves. It could very well be
                                                                                 t* = V{c./c.)P.                   (14)
                                                              Thus the optimal allocation of resources depends on the
                                                              performance and the cost coefficients of each of the two

                                                              SYSTEM RESOURCES
                                                                System activity is divided into three areas; pre-search,
                                                              search, and post-search activities. The system cost, Cgtg,
                                                              of equation (1) can be written
                                                                          C,t,=:          p p t f ' f ~i

USER                                                          The variables represent the costs per unit of tinae mul-
TIME                                                          tiplied by the time used for each of the three activities.
                                                                 The pre-search system cost per unit of time is given by
                                                                           Cp,,-. = a.C/i + piCPU -\- yiCore.        (16)
                                                              Here Ch is the cost of the computer channels, CPU is the
                                                              cost of the central processing unit and Core is the cost
                                                              of the core storage per unit of time. The vahie of a. ^.
                                                              and y represent the utihzation rates of each of the com-
                                                              ponents for the pre-search activity.
                                                                System search cost per unit of time is a function of the
                                                              search comparison method employed and the document
                           SYSTEM TIKE
                                                              representation used. Thus
                 FIG. 1. Isoquant Curves                                                                   -(-       (17)

                      Journal of the American Society for Information Science—September-October 1972                 309
  Finally, the post-search cost is given in a form anal-       the central processing unit cost, the channel cost, and
ogous to equation (16):                                        the core storage cost. The basic form of the processing
              Cp«, -. = aXh -I- ^iCPV + '^tCore.        (18)   cost equation is the same as in equation (16):
   Each of the search comparison methods or retrieval                      Cfr,c,.. = fJL£h -1- j3,CP[/ + 'iiCore.    (19)
rules employed in a retrieval system is presumed to have          While the costs of each of the computer components
a coat associated with it. No general statements can be        remain constant in the processing cost equation, the
made about the exact formulae for the cost of a compa-         values of a, /? and y vary depending on whether updat-
rison method because the cost is highly dependent on the       ing or retrieval is being performed.
way in which a strategy is implemented in a computerized          To summarize, the cost of a document representation
system. For example, the internal representation of the        is
query and the documents will influence the cost. Never-
theless, it is passible to suggest the form that such an
equation might take.                                           Preliminary analysis suggests that the cost differences be-
   The comparison cost will be a function of the number        tween document representations in the same class (e.g.
of terms in the quer>', the number of logical operators in     the index terms assigned to document number one and
the query {e.g. 'and,' 'or.' 'not'), the number of document    those assigned to document number two) may be so
representations in the file, and the number of words in        small as to minimize the need for cost computation for
each representation. Additionally the cost will depend on      each representation of each document. Instead costs
the computer resources used: the central processing unit,      could be computed for each representation class. This
core storage and the channels. Finally, the amortized          follows the approach of standard costing suggested
cost of programming a particular comparison method             earlier (5).
will have to be included. For associative searching, the
association files need to be constructed, and for compari-
                                                               SYSTEM RESOURCE ALLOCATION
son on cluster centers, the clustering will have to be
performed.                                                        When the user begins a dialog with a system employ-
   Tlie retrieval system must calculate costs for documet^t    ing this resource allocation scheme, he specifies a de-
 representations stored in the system. Total cost for a        sired performance level and a budget constraint. Then
 representation is made up of three components: creation       using the solutions in equations (13) and (14) the
 cost, storage cost, and processing cost.                      system is able to divide the user's fixed budget between
   When documents are received at an information center        system activities and user activities. This section ex-
a certain amount of pre-processing is performed before         plores possible approaches whereby the system can di-
 the doniment can be stored in the systeai's data bank.        vide its time between pre-search, search, and post-search
 For example, the document may have to be indexed, as-         activities.
signed subject classifications, abstracted, etc. In addi-         It was postulated earlier in this paper that a relation
tion, if it is not already in machine readable form, the       exists between user time, system time, and system per-
conversion will have to be performed. All these functions      formance. It is also possible to establish a relation be-
are considered part of the information center's cost of        tween performance, search comparison method, and docu-
creation of a document surrogate, c^gate-                      ment representation.
   No uniform method exists for accurately determining         P = /(comparison method, document representation) (21)
retrieval system costs or document .surrogate costs. Sur-
veys by Landau (f?), Olsen {11) and Penner (13) have           Using the equations reflecting representation and com-
summarized the cost work that has been performed to            parison method costs per unit time, it is possible to ar-
date. Leimkuhler and Cooper have proposed the use of           rive at an optimal choice of document representation and
standard cost accounting techniques (S) and mathemati-         comparison method that minimizes system search cost,
cal models (7) as a solution to the problem. Keith U)          ^gparch-a* ^0^ ^ given performance level. This is done in a
has also outlined a cost model.                                manner similar to that used previously in this section.
                                                                  Once the comparison method and document representa-
   The second surrogate cost component is storage cost,
                                                               tion have been selected, the search cost is determined us-
c^ton- There are a number of variables that determine
                                                               ing equation (17). The search, t^^gi-^^.g, is calculated us-
this cost: the rental rost of the computer storage de-
                                                               ing the average number of documents to be searched or
vice, the proportional cost of the control unit for the
                                                               the average number of index entries to be searched. The
device, the capacity and utilization of the device, and the
                                                               user is charged based on the average cost figure, and
number of characters in the representation.
                                                               variances are accumulated and at periodic intervals are
   The final component of the surrogate cost function is       used to readjust the cost coefficients. Tn this manner the
the processing cost, Cp. Two types of processing are per-      total search cost,
formed in the retrieval system: retrieving records from
the file and creating and maintaining the file. In addi-
tion, there are three components of the processing cost:       is determined.

310     Journal of the American Society for Information Science—September-October 1972
  The remaining problem facing the system is to allocate            The values of c,, and t,, in e<iuation (24) are a function
the remaining funds between the pre-search and post-              of the qualifiactions of the user, U. That is
search activities. No precise rules can be given for this.
However, it is possible to delimit the values of tp^c-s ^^^       Many different people presumably use an information
t ^^1^^ that are feasible. For instance the structure of the      system. Each i)erson most likely values his time at a
system may be such that pre-?earch activity requires a            certain price. The user cost per unit of time is related
minimum of 'a' time units and cannot use more than 'b'            to this as.sessment of value by the user. For example, a
time units no matter what performanee is required. Then           senior member of an organization will command a higher
                        a < f.r.-. < b                  (22)      salar>' than a clerk. If both of them use the retrieval
Similar bounds could be developed for post-search time            system, the allocation of resources between user and sys-
requirements. As an additional future stage, it would be          tem will vary depending on the c^ and c^ values.
desirable to determine if a relation could be established            Similarly the time that a user spends at the console
between the system performance and pre- and post-search           will depend on his experience with the system as well as
time allocations.                                                  his qualifications. Thus
                                                                                           t.==}(V).                      (27)
                                                                  It is possible to conceive of a situation in which users
• User Resources                                                  with similar c,, vjilues have different („ values simi)ly due
                                                                  to extensive practice with the system or a moro agile
  The second cati^ory of resources that are used in the           mind. Equation (27) is intended to reflect this disparity.
retrieval process is user resources. The total cost of these
factors is
                                                                   • Cost Model Evaluation
where c,, is the cost per unit of user time and t,, is the
amount of user time expended for a given query. .\s be-              The cost model that was developed divides the activi-
fore, a distinction is made between the three activities:        ties involved in the retrieval system operation in several
pre-search, search and iiost-search. Then the total user         ways. The first division involves an allocation of effort
cost is given by                                                 for a given search between the user of the retrieval sys-
                                                                 tem and the system itself. That is, cither the user can
                                                                 spend time and effort in correctly specifying his query,
                                                                 understanding what kind of material is in the document
 Here Cprc-u. ^»earch-in ^^^ <^poat-u ^""^ the per unit costs of file, how terms are related in the document file, etc., or a
 the user's time for each activity, and iprc-w *«nrt;(-«' ^"^    dialog between the user and the system ran take place
 ^po»i-u ^re the number of time units of each activity used       in which this information is established by negotiation.
 for a given search.                                             The negotiation process shifts some of the effort from the
    It should be noted that equation (23) is again a sim-         user to the system. Thus there is a trade off between
 plification of the actual situation. Post-search activity        the cost to the user and the cost to the system for the
 time for the user is in actuality a function of the amount       search. In addition to the division bptworn user and sys-
 of time spent in pre-search activity.                            tem effort, the model divides the total time during which
    Ill this model it is assumed that when the user is not        an interaction is taking place into three parts: pre-search
 availing himself of system services, the system can service      activity, search activity and post-search activity. During
 other users or other jobs. Thus it is as.sumed that the          the pre-search phase the user negotiates the query with
 system is never idle, or if it is the user does not pay for      the system; during the search phase the user waits while
 the idle system time. On the other hand, if the system is         the system searches the file; and during the post-search
  heavily loaded with other tasks, the user may have to            phase the system displays the output for the user.
  wait for a response to his dialog with the system. This            As with all models, the cost model nf a literature search-
  suggests that equation (23) should be modified as follows:       ing system is a simplified description of the real situation.
                                                                   There are a number of deficiencies in the model. The
                  '^''i,.,7(l,..i"+0p".')'.''' ' "         (24)    performance measure that is used in determining the opti-
                                                                   mal allocation of effort between the user and the system
    The variable S represents the additional time that the         is simplified. The measure only considers performance
  user must wait for the system for each of the activities.        as a function of user and system time. In all prob-
  For example, search activity will require a small amount         ability, a performance measure is much more complex
  of user time perhaps to initiate the searching once the          than this cost model assumes. Another deficiency is that
  query has been accepted. Then the user will have to wait         the model has not yet been verified with operating data.
  S^rarch "^its of time until the system completes the search,     Aside from these problems it is believed that, the frame-
  where                                                            work that the model presents is a useful way of evaluat-

                         Journal of the American Society for Information Science—September-October 1972                     311
 ing retrieval systems as well as a meaningful method for             of the American Society for Information      Seience,
 arriving at an optimal allocation between user and sys-              22:6 (November-December 1971) 390-398.
 tem resources.                                                  8. LEIMKUHLER, FERDINAND F . and MICHAEL D . COOPER.
                                                                      "Cost Accounting and Analysis for University Li-
                                                                      braries," College and Research Libraries, 32:6 (No-
                                                                      vember 1971) 449-464.
 References                                                      9. MARCH, JAMES G . and HERBERT A. SIMON. Organiza-
                                                                      tiom, John Wiley, New York, 1958.
  1. BAKEH, NORMAN R . and RICHARD E . NANCE. "The Use          10. MARON. M . E . and R. M. SHOFFNER.      The Study of
        of Simulation in Studying Information Storage and             Context: An Overview, NSF Grant No. GN643. In-
         Retrieval Systems," American Documentation, 19:4             stitute of Library Research. University of California,
         (October 1968) 363-370.                                      Berkeley, Januar>-1969.
 2.   GOOD, I. J. "The Decision-Theory Approach to the          11. OLSEN, HAROLD A. The Economics of          Information:
         Evaluation of Information-Retrieval Systems," In-           Bibliography and Commentary on the Literature,
        formation Storage and Retrieval, 3:2 (April 1967)            ERIC Clearinghouse on Library and Infonnation
        31-34.                                                       Science. January 1971. ED 044 545.
 3.   KATTER, ROBERT V. "Design and Evaluation of Infor-        12. PATRICK, RUTH J. and MICHAEL D . COOPER. "A User
        mation Systems," In: Carlos A. Ciiadra (Ed.) An-              Study" I n : Laura Gould, Deborah D. Barrett, and
        nual Review of Information Science and Technology,             Ralph M. Shoffner A?i Exjjerimental Inquiry into
        4, Brilannica, Chicago, 1969, pp. 31-70.                       Context Information Processing, NSF Grant No. GN
 4.   KEITH, NATHAN R., JR. "A General Evaluation Model               643. Institute of Library Research, University of
        for un Information Storage and Retrieval System,"             California, Berkeley, January 1969, pp. 79-91.
        Journal of the America/i Society for Information Sci-   13. PENNER, RUDOLF J. "The Practice of Charging Users
        ence, 21:4 (July-August 1970) 237-239.                        for Information Ser\qces: A State of the Art Re-
 5.   KESSLER, M . M . "Bibliographic Coupling between Sci-           port." Joumal of the American Society for Informa-
        entific Papers," American Documentation, 14:1                 tion Science, 21:1 (January--February 1970) 67-74.
        (January 1963) 10-25.                                   14. SIMON, HERBET A. Administrative Behavior: A Study
 6.   LANDAU, HERBERT B . "The Cost Analysis of Document              of Decision-Making Processes in Administrative Or-
        Surrogation: A Literature 'Review." American Docu-            ganization, MacmiUan. New York, 1957.
        mentation, 20:4 (October 1969) 302-310.                 15. USA Standard for a Format for Bibliographic Informa-
 7. LEIMKUHLER, FERDINAND F . and MICHAEL D . COOPER.                 tion Interchange on Magnetic Tape," Journal of Li-
       "Analytical Models for Library Planning." Journal              brary Automation, 2:2 (June 1969) 53-65.

312      Journal of the American Society for Information Science—September-October 1972

To top