Benchmarking Academic Programs Methods and Examples

Document Sample
Benchmarking Academic Programs Methods and Examples Powered By Docstoc
					             Benchmarking Academic Programs:
             Methods and Examples

                                                    Robert L. Armacost
                                                     Alicia L. Wilson
                                                University Analysis and Planning Support
                                                      University of Central Florida

                                                2004 AIR Annual Forum
                                                             June 2, 2004
Presentation available at http://uaps.ucf.edu
Overview of Presentation
      what is prominence?
      what is benchmarking?
      why benchmark academic programs?
      what to measure?
      where to get data?
      how to make the comparisons? (methods)
      what are the results?
      how to use them?



June 2, 2004       Benchmarking Academic Program: Methods and Examples   2
The University of Central Florida
               From Promise to Prominence: Celebrating 40 Years
   established in 1963 in Orlando Florida (first
    classes in 1968), Metropolitan Research
    University
   grown from 1,948 to 41,700 students in 35 years
        34,400 undergraduates and 7,300 graduates
        12 instructional sites in regional campus system
   doctoral intensive
        84 Bachelors, 64 Masters, 3 Specialist, and 23 PhD programs
   second largest undergraduate enrollment in
    state
        projected largest undergraduate enrollment in 2005
   approximately 1,100+ faculty and 3,100 staff
   eight colleges
        Arts and Sciences, Business Administration, Education,
         Engineering and Computer Science, Health and Public Affairs,
         Honors, Optics and Photonics, and Hospitality Management


June 2, 2004                   Benchmarking Academic Program: Methods and Examples   3
UCF Strategic Initiative
      increase prominence in graduate studies
              UCF will increase its emphasis on high-quality graduate
               education, providing professional education to meet the needs
               of the metropolitan area while achieving international
               prominence in engineering, optics, education, and the
               physical, biological, social, environmental, and space
               sciences, as well as other selected programs.




June 2, 2004               Benchmarking Academic Program: Methods and Examples   4
What is Prominence?
          prominent adj. 1. Projecting outward or upward from
           a line or surface. 2. Immediately noticeable;
           conspicuous. 3.Widely known; eminent. (American
           Heritage Dictionary, 1996)

          who determines who is prominent?
          what are the key measures used for this judgment?
          what actions are anticipated when ―rankings‖ are
           known?
                  new strategies
                       improve marketing?
                       improve accomplishments?

June 2, 2004                  Benchmarking Academic Program: Methods and Examples   5
What is Benchmarking?
      benchmark n. 1. A standard by which something can
       be measured or judged. 2. Often bench mark. A
       surveyors mark made on a stationary object of
       previously determined position and elevation and
       used as a reference point in tidal observations and
       surveys. --benchmark tr.v. To measure (a rival’s
       product) according to specified standards in order to
       compare it with and improve one’s own product.
       (American Heritage Dictionary, 1996)



June 2, 2004              Benchmarking Academic Program: Methods and Examples   6
What is Benchmarking?
       the continuous process of measuring our products,
        services and business practices against the toughest
        competitors or those companies recognized as
        industry leaders    (Xerox Corp.)

       a basis for establishing rational performance goals
        through the search for industry best practices that will
        lead to superior performance (Camp, 1989)




June 2, 2004         Benchmarking Academic Program: Methods and Examples   7
What is Benchmarking?
    ―benchmarking involves
          first examining and understanding your own internal work
           procedures,
          then searching for "best practices" in other organizations that
           match those you identified, and finally,
          adapting those practices within your organization to improve
           performance. It is, at bottom, a systematic way of learning
           from others and changing what you do.‖ (Epper, 1999)
    process for identifying gaps so that you can improve
    not about performance measurement or rankings
          although measures are used


June 2, 2004             Benchmarking Academic Program: Methods and Examples   8
Benchmarking Menu (Spendolini, 1992)




June 2, 2004   Benchmarking Academic Program: Methods and Examples   10
Approaches to Benchmarking
        problem-based or process-based
        types
              competitive
              functional
              performance
              strategic

        internal or external




June 2, 2004                 Benchmarking Academic Program: Methods and Examples   15
Related, But Not Benchmarking
        comparative analysis
              requires identification of comparables for whatever objective
               one has in mind, but not generally for improvement purposes
        key performance indicators (KPI)
        accountability measures
        institutional characteristics




June 2, 2004              Benchmarking Academic Program: Methods and Examples   16
Benchmarking ―Experience‖ at UCF
      common use implies ―comparison for _____,‖ not
       necessarily for process improvement
      initial efforts to identify ―prominent‖ graduate programs
      some comparative analysis as part of academic
       program reviews
      limited process studies (e.g., transfer credit evaluation,
       1996)




June 2, 2004         Benchmarking Academic Program: Methods and Examples   17
What Do We Want Benchmarking to
Mean for Academic Programs?
      process improvement
              process benchmarking—comparison against ―best-in-class‖ for
               a specified process
                   external—admissions process
                   internal—departmental advising practices
      comparative analysis
              curiosity, potentially leading toward process improvement
              competitive benchmarking—how are we doing relative to our
               ―competitors‖? (e.g., Florida schools admitting National Merit
               Scholars)
              best-in-class benchmarking—how are we doing relative to a
               specified class of comparable institutions? (e.g., Metropolitan
               Research Universities)
              world-class benchmarking—how do we rank among the best
               universities?
June 2, 2004                 Benchmarking Academic Program: Methods and Examples   18
Benchmarking Processes
        plan
            functions or processes to benchmark
            benchmark measures (key performance variables)
            who to benchmark (best-in-class, partner)
        collect data                                                             Do
              acquire data, observe                                      Plan          Check
        analyze data
                                                                                  Act
              identify actions to close gap
        adapt for improvement
            specify improvement programs and actions                                    benchmarking

            implement plans
        focus
                                                                                          assessment
            assessment—continuous improvement
            benchmarking—discontinuous improvement


June 2, 2004                Benchmarking Academic Program: Methods and Examples                 19
Approaches
       lone ranger
              third party data
              inference
       partner
              win-win
              mutual exchange on best-in-class processes
              data exchanges and visits
       consortium
              participant
              observer
              requires significant effort


June 2, 2004                Benchmarking Academic Program: Methods and Examples   20
Practical Questions
      what do I measure (benchmark)?
      who do I compare to?
      what process should I use?
      where do I get data?


      answer: it depends on what you want to accomplish




June 2, 2004        Benchmarking Academic Program: Methods and Examples   21
Measures
      primary determinant of measures is the purpose of
       the comparison
              process improvement
              comparison
                   rankings
                   prominence
      number and type of measures will depend on
       program or process
      will typically have multiple measures
              best-in-class will generally not be dominant on all measures
              identification of best-in-class is difficult

June 2, 2004                   Benchmarking Academic Program: Methods and Examples   22
Who Do I Compare To?
       identification of ―class‖
       peers—similar institutions
          •    differ by program
          •    differ by process
       requires insight and knowledge—no reference lists
        generally available
       comparables
          •    similar-sized operations in similar-sized institutions
       best-in class
          •    strong reputation

June 2, 2004               Benchmarking Academic Program: Methods and Examples   23
What Process to Use?
       partner approach is good for non-competitors
       consortium is preferred approach for process
        improvement benchmarking




June 2, 2004        Benchmarking Academic Program: Methods and Examples   24
Getting Benchmark Data
       published data
              reports
              websites
       information sharing
              establish relationship with benchmarks




June 2, 2004              Benchmarking Academic Program: Methods and Examples   25
UCF Graduate
―Programs of Prominence‖
              strategic initiative—programs of prominence
              who determines who is prominent?
              what are the key measures used for this judgment?
              what actions are anticipated when ―rankings‖ are
               known?
                  new strategies
                       improve marketing?
                       improve accomplishments?

              how to identify candidate programs?

June 2, 2004                 Benchmarking Academic Program: Methods and Examples   26
Identifying Programs
       Dean’s recommendations
       selected programs
              Computer Science (MS, PhD)
              Counselor Education (MS, PhD)
              Criminal Justice (MS)
              Environmental Engineering (MS, PhD)
              Applied Experimental and Human Factors Psychology (MS, PhD)
              K-8 Math/Science Education (MS)
              Nursing (MS)
              Optics (MS, PhD)



June 2, 2004              Benchmarking Academic Program: Methods and Examples   27
Process
      led by Vice Provost and Dean of Graduate Studies
              UAPS provided guidance and technical support
      initial meeting to define terms, set goals, evoke
       commitment ($$$)
      agreement on common measures
      bi-weekly progress meetings
      templates
              report
              data formats
      Dean ―sign-offs‖ on approach, measures, results
      final report
      follow-up meetings to develop action plans
June 2, 2004                  Benchmarking Academic Program: Methods and Examples   28
Candidate Benchmark Institutions
          types of Institutions
                  best in class
                  top institutions
                  unique institutions
                  peer institutions
                  direct competitors
          where to find them
                  general knowledge within the discipline
                  rankings in discipline-specific association journals
                  US News discipline rankings



June 2, 2004                  Benchmarking Academic Program: Methods and Examples   29
    Benchmark Institutions
   Counselor Education                 K-8 Math/Science Ed                  Environmental Engineering
      U Minnesota                         UC Berkeley                          Stanford U
      Indiana U                           U Wisconsin Madison                  UC Berkeley
      UNC Greensboro                      Ohio State                           Georgia Tech
      U of MD College Park                Clemson                              UT Austin
      U South Carolina                    Oregon State                         U Michigan
      Kent State                          Hofstra                              U Ill. Urbana Champaign
      Portland State                      George Mason                         Cal Tech
   Criminal Justice                       San Diego State                      U Florida
      U Louisville                        5 others didn’t respond              Virginia Tech
      Michigan State                   Human Factors                           NC State
      SUNY Albany                         U Ill. Urbana Champaign           Nursing
      Cal State LB                        George Mason U                       U Washington
      Rutgers U                           Georgia Tech                         UNC Chapel Hill
      Georgia State                       U Cincinnati                         Ohio State
      U Cincinnati                        New Mexico State                     U Kansas
      UNC Charlotte                       NC State                             U Kentucky
   Optics                                 Wright State                         Arizona State
      Stanford U                       Computer Science                        U Florida
      MIT                                 UC Berkeley                          Georgia State
      UC Berkeley                         UCLA                                 U North Dakota
      5 others didn’t respond             NYU
      U Arizona                           Columbia U                    Red-Top Tier; Pink-Second Tier;
      U Rochester                         Duke U                        Green-Peers; Black –No delineation
    June 2, 2004                 Benchmarking Academic Program: Methods and Examples                      30
What Measures?
          both program characteristics and performance
           measures
          national studies
                  National Research Council
                  US News and World Report
                  TheCenter Report
                  discipline-specific studies (e.g., American Association of
                   Colleges of Nursing)
          varies by program
                  set of core measures for all programs
                  discipline specific measures
          looked at both raw data and ratios

June 2, 2004                 Benchmarking Academic Program: Methods and Examples   31
Sample Benchmark Measures
       faculty characteristics                      student characteristics
              # of faculty                               # of students
              # of society fellows                       # of minority students
              # of national awards                       # of international students
              # of publications                          GRE scores
                                                          # of students supported
              # of faculty publishing                     (GTAs, GRAs)
              # of faculty with research                 # of national fellowships
               support                                     (other fellowships)
              amount of external and                program characteristics
               federal funding                            # of degrees awarded
                                                          amount of lab space


June 2, 2004               Benchmarking Academic Program: Methods and Examples       32
Where’s the Data?
         if program is general enough
                  national studies
                  discipline specific studies
                  number of graduates from IPEDS
         if program is more narrowly defined (e.g.,
          environmental engineering)
                  more difficult to find data
                  program data is grouped with other programs (e.g., civil
                   engineering) or departments (e.g., optics)
                  create consortiums or partnerships
                  request data directly from colleagues


June 2, 2004                  Benchmarking Academic Program: Methods and Examples   33
How To Get the Data?                                        Benchmark Data Request Form
                                                             University of Central Florida

                       UCF Program:

                       Benchmark Program:

                       Benchmark Institution:


        templates:     Contact Name:

                       Contact Email:


                       The University of Central Florida is conducting a benchmarking study in order to compare some of its programs to
                       other programs that are considered to be among the best in the nation. We are requesting your assistance in
                       completing our study. Please provide the information requested below from the most recent 12-month
                       reporting period at your institution. The student data should come from fall 2001 only. Define your
                       reporting period by specifying your inclusive dates in the space provided below (e.g., 1 July 2001-30June
                       2002). After completing this form, please return it by email to XXXX. Your assistance in this regard is
                       greatly appreciated.

                       Reporting Period:

                       Benchmark Characteristics                                                                            Data
                          Criterion Measures                                                                               Report

                       Faculty Characteristics

                           Number of tenured and tenure earning faculty in your program

                           No. of society Fellows among your program faculty

                           No. of national awards received by your program faculty

                           No. of peer reviewed publications by your program faculty appearing in print in the period

                           No. of program faculty publishing peer reviewed publications in the period

                           No. of program faculty receiving external funding (as PI or Co-PI) in the period

                           Total amount of external research funding received by your program faculty in the period

                           Total amount of federal research funding received by your program faculty in the period


June 2, 2004         Benchmarking Academic Program: Methods and Examples                                                                  34
How To Get the Data?
       department representative contacted colleagues
              sent template (with program specific questions) via email
              followed up with multiple phone calls
              took 2-3 months to gather data
              not all benchmark programs cooperated
              attempt to fill in missing pieces
                   web of science for publications (labor intensive)
                   search discipline-specific journals
                   search university web pages
                   follow-up with colleagues



June 2, 2004                  Benchmarking Academic Program: Methods and Examples   35
     How To Organize The Data?
                UCF Program:

                                                                      Benchmarks:                                                   Benchmarks:
                                                                 Selected Top Institutions                                    Selected Peer Institutions
Criterion Measures                       University    Georgia      George       Ohio        University   Average    Wright     N. Carolina New Mexico Average
                                UCF      of Illinois    Tech         Mason      State        Cincinnati    Top       State      State Univ   State Univ  Peer

Faculty Characteristics

Number of Faculty                                                                                         #DIV/0!                                     #DIV/0!
No. Society Fellows                                                                                       #DIV/0!                                     #DIV/0!
No. National Awards                                                                                       #DIV/0!                                     #DIV/0!
No. of Publications                                                                                       #DIV/0!                                     #DIV/0!
Publications per Faculty       #DIV/0!   #DIV/0!       #DIV/0!      #DIV/0!    #DIV/0!       #DIV/0!      #DIV/0!   #DIV/0!      #DIV/0!   #DIV/0!    #DIV/0!
No. of Faculty Publishing                                                                                 #DIV/0!                                     #DIV/0!
% of Faculty Publishing        #DIV/0!   #DIV/0!       #DIV/0!      #DIV/0!    #DIV/0!       #DIV/0!      #DIV/0!   #DIV/0!      #DIV/0!   #DIV/0!    #DIV/0!
No. of Faculty Funded                                                                                     #DIV/0!                                     #DIV/0!
% of Faculty Funded            #DIV/0!   #DIV/0!       #DIV/0!      #DIV/0!    #DIV/0!       #DIV/0!      #DIV/0!   #DIV/0!      #DIV/0!   #DIV/0!    #DIV/0!
Amount of External Funding                                                                                #DIV/0!                                     #DIV/0!
Funding per Faculty            #DIV/0!   #DIV/0!       #DIV/0!      #DIV/0!    #DIV/0!       #DIV/0!      #DIV/0!   #DIV/0!      #DIV/0!   #DIV/0!    #DIV/0!
Amount of Federal Funding                                                                                 #DIV/0!                                     #DIV/0!




     June 2, 2004                            Benchmarking Academic Program: Methods and Examples                                                      36
Comparison Methods
       look at summary data to develop impressions of where
        programs ―ranked‖
       analytic approaches
              dominance ranking
                   primary method used
              data envelopment analysis (DEA)
              hierarchical ―weight and rate‖ approach




June 2, 2004                Benchmarking Academic Program: Methods and Examples   37
Dominance Ranking
   based on an approach used in TheCenter's annual
    report, The Top American Research Universities
         classification of universities into groups based upon quality
          indicators
         method relies on counts of the number of times that a university
          is included in the top 25 on a given measure or in the second
          group (26-50)
         number of counts of those occurrences in the first tier or second
          tier are used to group or rank the institutions
         see http://thecenter.ufl.edu/


June 2, 2004             Benchmarking Academic Program: Methods and Examples   38
Dominance Ranking
        UCF graduate programs are compared to 6-10 other
         graduate programs
        comparable approach is to rank order the programs
         for each measure and use that ranking to determine
         in which tier each program falls
              Tier 1   ranked number one or two
              Tier 2   ranked number three or four
              Tier 3   ranked below number four

        uses counts of those occurrences, ranking first by the
         number of counts in Tier 1, then by the counts in Tier
         2, and then by the counts in Tier 3
June 2, 2004             Benchmarking Academic Program: Methods and Examples   39
Dominance Ranking
      data acquisition and limitations
              methodology depends on having consistent data
              serious limitation associated with missing data
                   missing data results in a university not being ranked on those
                    given measures (equivalent to being in Tier 3)
                   for an otherwise highly ranked university, missing data will
                    necessarily lower its rank
              assumes that each measure is equally important
              could conduct the analysis using only those measures for
               which complete data are available
              the overall ranking needs to be used with care

June 2, 2004                  Benchmarking Academic Program: Methods and Examples    40
Example Faculty Summary                                                                                                                                                                                                                                                                                      Overall Ranking




                                                                                                                                                                                                                                                                       Amount of External Funding
                                                                                                                                                                                                                                  Amount of Federal Funding
                                                                                                                                                   # of Faculty Publishing
                                                                               # of National Awards
                                                 # of Society Fellows




                                                                                                                                                                                     # of Faculty Funded
                                                                                                                  # of Publications
                   # of Faculty

                                          Rank




                                                                        Rank



                                                                                                           Rank




                                                                                                                                            Rank



                                                                                                                                                                             Rank




                                                                                                                                                                                                                         Rank




                                                                                                                                                                                                                                                              Rank




                                                                                                                                                                                                                                                                                                    Rank



                                                                                                                                                                                                                                                                                                           Tier1


                                                                                                                                                                                                                                                                                                                               Tier2


                                                                                                                                                                                                                                                                                                                                                   Tier3
    Counts:
    U of Arizona             79              1              48             1 177  1                                171  4                           57   1                                                          26      2   30,732,000  1                        43,472,000                        1           7                   1               0
    Stanford                 14              5              10             4 25   2                               3000  1                           14   4                                                          14      5   10,000,000  2                        12,000,000                        2           4                   2               2
    UC Berkeley              50              2              25             2 NA ###                                NA ###                           50   2                                                          50      1          NA ###                         9,000,000                        4           4                   1               0
    MIT                       9              6               6             6 20   3                               1200  2                            9   5                                                           9      6    4,500,000  4                         5,000,000                        5           1                   2               5
    UCF                      23              3              16             3 12   4                                184  3                           23   3                                                          22      3    6,413,000  3                        10,426,000                        3           0                   8               0
    Rochester                16              4              10             4 NA ###                                NA ###                           NA ###                                                          15      4    3,105,040  5                         4,196,000                        6           0                   3               2


                                                                                                                                                                                                                                                                                                             Overall Ranking




                                                                                                                                                                                     External Funding per Faculty
                                                                               Publications per Faculty




                                                                                                                  % of Faculty Publishing
                   Students per Faculty




                                                                                                                                                   % of Faculty Funded
                                                 % Society Fellows
                                          Rank




                                                                        Rank



                                                                                                           Rank




                                                                                                                                            Rank



                                                                                                                                                                             Rank




                                                                                                                                                                                                                         Rank




                                                                                                                                                                                                                                                                                                                       Tier1


                                                                                                                                                                                                                                                                                                                                           Tier2

                                                                                                                                                                                                                                                                                                                                                           Tier3
    Ratios:
    Stanford       8.57                      6   0.71                      1 214  1                                      100  1 100                                             1   857,143                                 1                                                                                      5                   0               1
    MIT            6.11                      4   0.67                      3 133  2                                      100  1 100                                             1   555,556                                 3                                                                                      3                   3               0
    UC Berkeley    4.40                      2   0.50                      6 NA ###                                      100  1 100                                             1   180,000                                 6                                                                                      3                   0               2
    UCF            5.17                      3   0.70                      2   8  3                                      100  1 96                                              4   453,304                                 4                                                                                      2                   4               0
    U of Arizona   2.41                      1   0.61                      5 2.2  4                                       72  5 49                                              6   820,226                                 2                                                                                      2                   1               3
    Rochester      6.13                      5   0.63                      4 NA ###                                      NA ###  94                                             5   262,250                                 5                                                                                      0                   1               3

June 2, 2004                                                                                              Benchmarking Academic Program: Methods and Examples                                                                                                                                                                                      41
Example Student Summary                                                                                                                                                                                                  Overall Ranking




                                                                    # of International Students




                                                                                                         # of Students Supported




                                                                                                                                                                                               # of Fellowships
                                 # of Students




                                                                                                                                                                 # of GRAs
                                                                                                                                          # of GTAs
                                                            Rank




                                                                                                  Rank




                                                                                                                                   Rank




                                                                                                                                                          Rank




                                                                                                                                                                                        Rank




                                                                                                                                                                                                                  Rank




                                                                                                                                                                                                                           Tier1

                                                                                                                                                                                                                                   Tier2

                                                                                                                                                                                                                                           Tier3
               Counts:
               UC Berkeley      220                          1      53                             4 220   1                              70    1 150   1                                      88                  1      5        1       0
               U of Arizona     190                          2      55                             3 146   2                              19    2  95   3                                      13                  4      3        3       0
               UCF              119                          4      82                             2 NA ####                               2    4 103   2                                      10                  6      2        2       1
               Stanford         120                          3      85                             1 120   3                              NA #### NA ####                                      40                  2      2        2       0
               U of Rochester    98                          5      30                             5  86   4                              19    2  46   4                                      18                  3      1        3       2
               MIT               55                          6      15                             6  55   5                               2    4  41   5                                      12                  5      0        1       5


                                                                                                                                                                                                                         Overall Ranking
                                 % International Students




                                                                                                                                                                 % Students Supported
                                                                                                                                          % Fellowships
                                                                                                         % GRAs
                                                                    % GTAs
                                                            Rank




                                                                                                  Rank




                                                                                                                                   Rank




                                                                                                                                                          Rank




                                                                                                                                                                                        Rank




                                                                                                                                                                                                                           Tier1

                                                                                                                                                                                                                                   Tier2

                                                                                                                                                                                                                                           Tier3
               Ratios:
               UC Berkeley      24                           6     0.32   1 0.68  3 0.40                                                                   1 100   1                                                      3        1       1
               Stanford         71                           1      NA #### NA #### 0.33                                                                   2 100   1                                                      3        0       0
               MIT              27                           5     0.04   4 0.75  2 0.22                                                                   3 100   1                                                      2        2       1
               UCF              69                           2     0.02   5 0.87  1 0.08                                                                   5 NA ####                                                      2        0       2
               U of Rochester   31                           3     0.19   2 0.47  5 0.18                                                                   4  88   4                                                      1        3       1
               U of Arizona     29                           4     0.10   3 0.50  4 0.07                                                                   6  77   5                                                      0        3       2
June 2, 2004                                                  Benchmarking Academic Program: Methods and Examples                                                                                                                                  42
Example Program Summary
                                                                                               Overall Ranking




                                                              Lab Space (Square Feet)
                                # of Degrees Awarded

                                                       Rank




                                                                                        Rank




                                                                                                   Tier1


                                                                                                               Tier2

                                                                                                                       Tier3
               Counts:
               UC Berkeley          30    1                   45000   1                        2           0           0
               Stanford             20    2                   30000   4                        1           1           0
               U of Arizona         NA ####                   42000   2                        1           0           0
               UCF                  15    3                   40000   3                        0           2           0
               MIT                  10    4                    9000   5                        0           1           1
               U of Rochester       NA ####                     NA ####                        0           0           0


                                                                                               Overall Ranking

                                                              Degrees per Student
                                Degrees per Faculty

                                                       Rank




                                                                                        Rank




                                                                                                   Tier1


                                                                                                               Tier2

                                                                                                                       Tier3
               Ratios:
               Stanford         1.43   1                              0.17   2                 2           0           0
               MIT              1.11   2                              0.18   1                 2           0           0
               UCF              0.65   3                              0.13   4                 0           2           0
               UC Berkeley      0.60   4                              0.14   3                 0           2           0
               U of Arizona      NA ####                               NA ####                 0           0           0
               U of Rochester    NA ####                               NA ####                 0           0           0
June 2, 2004             Benchmarking Academic Program: Methods and Examples                                                   43
Data Envelopment Analysis
      multi-dimensional approach
              analyzes inputs and outputs
              assesses overall effectiveness
      advantages
              assigns mathematically optimal weights
              simultaneous comparisons of performance measures
              calculates ―distance‖ from ―best-practice‖ organizations
      efficiency
              weighted sum of outputs (more is better) divided by weighted
               sum of inputs (less is better)
              find best set of weights to maximize efficiency—best possible
               case—use Solver in Excel (linear program)
June 2, 2004                Benchmarking Academic Program: Methods and Examples   44
  Sample Excel DEA Output
                                                  Computer Science

                       -------- Outputs --------          ----- Inputs -----                                      DEA
                          PhD         Grants               MS             PhD  Weighted Weighted
 Unit     University   Awarded          ($M)     TTF       Students Students Output       Input    Difference   Efficiency
  1          UCF             8       $     3.70     21          88          88  0.5832   1.0000     -0.4168       0.5832
  2        Berkeley         37       $ 20.00        43           0         300  2.8946   2.9983     -0.1037       1.0000
  3         UCLA            25       $ 10.00        29         151         151  1.7159   1.7159      0.0000       1.0000
  4       Columbia          10       $     8.00     27          73          86  0.9595   0.9595      0.0000       1.0000
  5         Duke             8       $     6.80     17          13          80  0.7949   0.8174     -0.0225       1.0000
  6         NYU              4       $     7.00     31         353          16  0.6432   0.6432      0.0000       1.0000
  7                                                                             0.0000   0.0000      0.0000
  8                                                                             0.0000   0.0000      0.0000
  9                                                                             0.0000   0.0000      0.0000
  10                                                                            0.0000   0.0000      0.0000
  11                                                                            0.0000   0.0000      0.0000
  12                                                                            0.0000   0.0000      0.0000

Weights                 0.0413       0.0683      0.0000    0.0014    0.0100

Unit                       1
Output                  0.5832
Input                   1.0000




  June 2, 2004                           Benchmarking Academic Program: Methods and Examples                          45
Weight and Rate Approach
         identify key benchmark measures
         create hierarchy to group similar dimensions
         develop weights for each measure to determine
          relative importance
                  use any decision analysis method
                  pairwise comparisons--Analytic Hierarchy Process (AHP)
         use value of measure or separate evaluation to rate
          institution on the measure
         compute overall ―score‖ to rank institutions


June 2, 2004                 Benchmarking Academic Program: Methods and Examples   46
Rating Hierarchy
                  Graduate Program Benchmarking
                                                                         determine weight
                                                                         for relative
                         Overall Performance                             importance of
                                                                         measures

       Faculty                Student               Program
    Characteristics        Characteristics        Characteristics


           Capability           Capability              Productivity

         Publications               Size                 Capability

               Funding           Support                              determine rating of
                                                                      individual measures

               compute overall score for institution
June 2, 2004                Benchmarking Academic Program: Methods and Examples             47
Results
        programs used the results of the dominance rankings
         to identify what areas were competitive with other top
         institutions (i.e., faculty productivity) and what areas
         needed improvement (i.e., number of faculty)
        programs submitted a report which included:
              programs benchmarked
              measures used
              results of the dominance ranking
              plan of action
        review held with Vice Provost of Graduate Studies to
         further refine action plans
June 2, 2004              Benchmarking Academic Program: Methods and Examples   48
Action Plans
      identified what was needed to elevate the program
       to prominence
      examples:
              # of additional faculty needed
              necessary increases in faculty productivity
              revise recruitment strategies
              market the program
              increase in lab space
              increase financial support for students
      identified areas where additional data was needed
              breakdown of masters vs. doctoral
              benchmark clinical costs

June 2, 2004                Benchmarking Academic Program: Methods and Examples   49
How Will the Results Be Used?
      assist programs to get the help they need
              marketing
              recruiting
      provides documentation and support for requesting
       additional funds and university specials
      strategic plan provides justification for additional
       funding and support for those programs identified as
       ―programs at or near national or international
       prominence‖



June 2, 2004                Benchmarking Academic Program: Methods and Examples   50
Resources
   Internal sources:
        IR: http://pegasus.cc.ucf.edu/%7Eirps/character/current.html
        OEAS: http://www.oeas.ucf.edu/benchmarking.htm
        UAPS: http://www.uaps.ucf.edu/benchmarking.html
   Websites to Universities:
        IR Offices: http://airweb.org/links/offices.cfm
        Florida Colleges and Universities:
         http://iea.fau.edu/fair/flacol.htm
        Florida Colleges and Universities IR Offices:
         http://iea.fau.edu/fair/flair.htm
        Southern Association IR Offices:
         http://sair.org/Resources/Links.htm#SAIR%20IR%20Office%20
         Web%20Sites
        Coalition of Urban & Metropolitan Universities:
         http://cumu.uc.iupui.edu
June 2, 2004            Benchmarking Academic Program: Methods and Examples   51
Resources
   Websites to Industry Resources:
         Higher Education Associations: http://iea.fau.edu/fair/edasoc.htm
         Higher Education Research Centers:
          http://iea.fau.edu/fair/edres.htm
         Institutional Research Internet Resources:
          http://www.airweb3.org/air-new/page.asp?page=21
         http://airweb.org/links/linkmap.html
         The Learning Alliance for Higher Education:
          http://www.thelearningalliance.info/index.php
         National Center for Educational Statistics:
          http://www.nces.ed.gov/
         National Organizations: http://oeas.ucf.edu/related_links.htm


June 2, 2004             Benchmarking Academic Program: Methods and Examples   52
Resources
      Websites to Data:
              Common Data Sets: http://airweb.org/links/cds.cfm
              Census Data: http://airweb.org/links/census.cfm
              Data Warehousing: http://airweb.org/links/datawarehouse.cfm
              Enrollment Statistics: http://airweb.org/links/enroll.cfm
              Environmental Scanning: http://airweb.org/links/scanning.cfm
              Peer Comparison Data: http://airweb.org/links/peers.cfm
              Performance Indicators: http://airweb.org/links/indicators.cfm
              Statistics/Research Methods: http://airweb.org/links/stats.cfm
              Southern Universities-Common Data Sets:
               http://sair.org/Resources/Links.htm#SAIR%20School%20Comm
               on%20Datasets
              Resource Identification for Programs:
               http://oeas.ucf.edu/SourceID.html
June 2, 2004               Benchmarking Academic Program: Methods and Examples   53
Resources
   Websites to Data:
         Resource Sheet Identifying available data elements:
          http://oeas.ucf.edu/InstitutionResourceSheet.html
         Variables Available About Specific Programs:
          http://oeas.ucf.edu/VariablesAvailableAboutSpecificPrograms.html
         Population Characteristics: http://site.conway.com/ez/
         Student Characteristics: http://nces.ed.gov/ipeds/
         http://www.usnews.com/usnews/edu/college/coworks.htm
         Faculty Characteristics: http://nces.ed.gov/ipeds/
         Financial Characteristics: http://nces.ed.gov/ipeds/
         Research Characteristics: http://www.nsf.gov
          http://caspar.nsf.gov/cgi-bin/WebIC.exe?template=nsf/srs/webcasp/start.wi
         List of Published Rankings:
          http://oeas.ucf.edu/PublishedRankings.html
         List of Program Rankings:
          http://oeas.ucf.edu/ProgramRankings.html
June 2, 2004              Benchmarking Academic Program: Methods and Examples   54
Resources
       Websites to Benchmarking Literature:
              American Productivity and Quality Center:
               http://www.apqc.org/best/
              Ebenchmarking: http://ebenchmarking.com/ (note: scroll down
               page for list of additional resources including industry specific)
              Consortium for Higher Education Benchmarking Analysis:
               http://www.cheba.com/
              National AAU Peer Benchmarking for Quality:
               http://www.ir.ufl.edu/compare/intro.htm




June 2, 2004               Benchmarking Academic Program: Methods and Examples   55
Questions




Contacts:
                              ???
Dr. Robert L. Armacost                     Ms. Alicia L. Wilson
Director, University Analysis and          Assistant Director, University Analysis and
    Planning Support                           Planning Support
University of Central Florida              University of Central Florida
12424 Research Parkway, Suite 215          12424 Research Parkway, Suite 215
Orlando, FL 32826-3207                     Orlando, FL 32826-3207
407-882-0286                               407-882-0287
armacost@mail.ucf.edu                      awilson@mail.ucf.edu
http://uaps.ucf.edu                        http://uaps.ucf.edu
June 2, 2004         Benchmarking Academic Program: Methods and Examples       56