Graduate Assessment Report AY06

Document Sample
Graduate Assessment Report AY06 Powered By Docstoc
					                       Graduate Assessment Report AY091
This report offers information concerning graduate program assessment at Eastern
Ilinois University. In AY09 twenty-nine graduate programs submitted annual
assessment plans to the Director of the Center for Academic Support and
Achievement. 2 The following chart indicates how many graduate programs in the
four colleges are using the various measures for assessment purposes.


                     Graduate Measures by College AY09
    120%
           100%100%       100%                                          100%              100%
    100%                                                                                     90%
                    80%     80%             80% 80%                            80%
    80%                       75%
                                 70%                                 67%             67%
                 63%                          63%                                      60%
    60%                                                50%                 50%
                                                         40% 40%
    40%                                 33%                38%


    20%

     0%
              Exams         Written         Oral         Practica      Surveys          Other

                                      CAH     LCBAS      CEPS       COS


The most prevalent measures used for assessment purposes are masters’ exams
and theses with the majority of programs using both of these measures. Theses and
seminar papers are used by 70%-100% of programs, which shows a growth from the
previous year. Fewer programs are assessing oral competency with a range from
33% (CAH) to 80% (LCBAS and COS). Laboratory experiments/exercises and
internships/practica make up 38%-50% of the measures. The widest range in usage
of measures is the indirect measures with 50% (CEPS) to 100% (LCBAS) employing
surveys and/or interviews. Measures by college and program are included as
Appendix A or can be found on the submitted plans themselves, which are available
on-line at www.eiu.edu/~assess.


1
  All information provided in this chart was taken from the annual assessment summaries submitted
to the Director of CASA by July 14, 2009.
2
  Twenty-six programs submitted plans. An additional three graduate programs were placed on a
two-year reporting cycle in AY08: M.A., Family and Consumer Sciences—Gerontology; M.S., Family
and Consumer Sciences—Dietetics; and M.A., Clinical Psychology. Data from these programs’ 2008
reports are included here for consistency.

                                                    Prepared by Karla Sanders, CASA, Summer 2009, p.1
The following chart follows the changes in measures from AY05 to AY09 with all
graduate programs submitted for each year included. As shown on the char below,
there has been a slight drop in the usage of exams by programs and a slight
increase in the adoption of theses and papers by graduate programs. Oral
presentations has grown by 10% in the past year while the use of laboratory
work/practica has declined since 2006.


                          Graduate Measures by Year
 120%

          96%
 100%       90%
              85%84%83%                                                                              83%
                              79%
   80%                      76%                                                                    76%
                          73%
                     70%72%                                                    69% 69%
                                                     66%
                                                                                 60%
                                            52% 56%                                           54%
   60%                                                       55%
                                               50%              50%     48%48%           48% 45%
                                                           43%    44%
                                       39%                           41%
   40%

   20%

    0%
            Exams         Thesis &           Oral            Labs &      Surveys &           Other
                           Papers                            Practica    Interviews

                                     AY05     AY06     AY07     AY08    AY09


The “other” category in the above chart refers to a variety of measures that are
either not measures assessing student learning outcomes directly (such as numbers
of students presenting at conferences or receiving awards/scholarships, employment
rates, admission requirements, numbers of theses completed, and number of
students applying to and being accepted by Ph.D. programs) or are very
field/program specific (such as participation with peers in coursework).

Use of indirect measures such as surveys of students or employers or exit interviews
has increased 9% in the past year. Twenty of the twenty-nine submitted programs
are using at least one indirect measure. Every plan submitted from the Lumpkin
College of Business and Applied Sciences and all but two submitted from the
College of Sciences have included both direct and indirect measures in their plans.
This incorporation of both kinds of assessment measures indicates maturation in
assessment work. While all program plans submitted have included direct
measures, some programs still do not have multiple measures but are using one
summative measure only.


                                                      Prepared by Karla Sanders, CASA, Summer 2009, p.2
The following chart indicates the level of progress for the graduate programs by the
five criteria on the primary trait analysis. These levels have been given to
department chairs and coordinators on their 2009 Response to Summary Report.
These responses are also on the assessment web site.


                       Graduate Program Levels AY09

  80%
                                                                         69%
  70%                                               62% 62%
  60%                                                                                       55%

  50%                                                        45%               45%
                                              41%
  40%                                   31%
                                                                                    28% 28%
  30%
  20%           14%
                      10% 10%
  10%
           0%                    0%
   0%
                  Level 1                      Level 2                          Level 3

           Learning Objectives    Assessment Measures     Expectations    Results    Feedback Loop


While our goal is to move more programs into level three in all categories, each year
there are fewer and fewer programs still at level one, which does show progress. A
chart listing progress by college is included as Appendix B. The best gauge of each
program’s progress is the analysis provided on the summary reports in Parts Two
and Three.

Several programs are making great progress at the graduate level. The percentage
of programs at level three for all categories has grown since last year. The highest
number of programs at level one is in the measures section of the plan followed by
expectations and results. These categories are so integral to each other that
programs that struggle in one section tend to struggle with other parts of the plan.

But, the percentage of programs at level one has decreased in the last year with no
plans at this level in the learning objectives and feedback loop categories.
Objectives and the feedback loop have the most programs at level three. These
categories tend to be the most stable; once a program has a solid set of objectives
and a plan for sharing and using data, it tends to leave these parts of the plan alone.
The Council for Graduate Studies’ mandate to include the graduate goals in all
graduate plans acted as an impetus for departments to discuss their objectives and
move into more specific and measurable objectives for this program.


                                                    Prepared by Karla Sanders, CASA, Summer 2009, p.3
Plans cannot reach a level three designation in measures until they use both direct
and indirect measures and employ multiple measures of assessment, so some plans
remain at level two while nearly half have reached level three. Reaching level three
for results takes time; programs must display several years of collection and analysis
of data and use of data for program improvement in order to reach level 3 for the
results section of the plan.

The following chart shows the program levels in AY08 for the sake of comparison.


                        Graduate Program Levels AY08

    80%                                                    76%
                                               68%
    70%
    60%
                                                     52%         52%
                                         48%                               48%
    50%
                                                                                                   40%
    40%
                                                                                 28%
    30%              24%                                                               24%

    20%                                                                                      16%

                           8%
    10%      4% 4%               4%

    0%
                   Level 1                       Level 2                             Level 3

           Learning Objectives    Assessment Measures       Expectations   Results     Feedback Loop



In AY09 all graduate programs submitted plans, so this year’s data provides the
most accurate information on graduate assessment we have had to date. This
100% compliance also accounts for the increase in level two and three programs in
all categories.

In addition to measures and progress levels, the number of programs that had
incorporated the graduate learning goals was also tracked. These goals are:
     A depth of content knowledge (including technology skills and ethical
       behaviors)
     Critical thinking and problem-solving skills
     Effective oral and written communication skills
     Evidence of advanced scholarship through research and/or creative activity
The percentage of programs currently incorporating these goals into their program
objectives is given in the chart below:3


3
 These data are based on the assessment summaries and the Director’s understanding of those
summaries and the graduate learning goals themselves.
                                                     Prepared by Karla Sanders, CASA, Summer 2009, p.4
                      Incorporating Graduate Goals AY06-AY08
  120%
                       100%
                96%
  100%
                   100%                                                    85%    83%
             90%                         79%                   76%
    80%                                                                        88%
                                   69%84%             69%

    60%                                                                69%
                              62%                         64%
                                                  55%
    40%

    20%

     0%
              Knowledge            CT & PS       Communication          Scholarship

                            AY06     AY07    AY08       AY09


The depth of knowledge was the goal that the majority of programs had already
incorporated as is shown by the high percentages for the last four years. Critical
thinking and problem solving have been incorporated into the 23 graduate programs.
Oral and written communication skills are part of the program objectives for 22 of the
graduate programs. Twenty-four graduate programs currently articulate advanced
scholarship through research or creative activity as a program objective (more
programs are using scholarship activities such as the thesis as a measure to show
depth of knowledge or more program-specific objectives).

Some programs may be examining student learning of these goals, but because
these goals have not been specifically articulated in their program objectives, they
have not been included in the numbers given here. I suggest that in the next couple
of years, programs should be encouraged to respond to these goals in their annual
summaries as part of the information given in Part Two.

The following chart shows adoption of graduate learning goals by college. All
colleges have adopted content knowledge objectives for all of their programs.
Three of the colleges (CAH, CEPS, and LCBAS) have attained 100% adoption of
two of the graduate goals. CAH and CEPS reached this level with content
knowledge and research and creative activity while LCBAS attained 100% adoption
with content knowledge and critical thinking and problem solving.




                                             Prepared by Karla Sanders, CASA, Summer 2009, p.5
           Incorporation of Graduate Learning Goals by College AY09
  120%
          100%100%100%100%                     100%                           100%100%
  100%                                                            90%
                             83%
                                         80%                            80%
   80%                                                                                   70%
                                                      67%
                                   63%                      63%
                                                                                               60%
   60%

   40%

   20%

    0%
            Knowledge              CT & PS            Communication             Scholarship

                              CAH         CEPS        COS         LCBAS


As this report shows, the graduate programs have been making steady progress
over the last few years with their assessment plans.




                                                 Prepared by Karla Sanders, CASA, Summer 2009, p.6