Docstoc

Ranking and Grading

Document Sample
Ranking and Grading Powered By Docstoc
					                                              1


                         Grades and Purposes of Grades

The individual teacher must think about grades in the context of learning reality, must
formulate a philosophy and approach to grades and report cards to act accordingly. The
grades are with us and it is our job to make them as accurate and effective as possible.
After accepting the need for grades, let us review some of their historical purposes as
given by Karmel and Karmel (1978):

1. They provide data for parents on their children's progress.

2. They certify promotional status and graduation.

3. They serve as an incentive to do school lessons.

4. They help in educational and vocational guidance by presenting a realistic basis for
future choices.

5. They serve as a reference point for personal development.

6. They provide a basis for awarding honors.

7. They enable the school to ascertain the amount of extracurricular activities, if any, in
which the student should participate.

8. They may be used as a source for communication to prospective employers.

9. They provide information for curriculum research.

10. The provide data to a school that the student may letter attend through transfer or
graduation.

ASSIGNING LETTER GRADE

Because most schools use A, B, C, D, F. system, most teachers will be faced with the
Problem of assigning letter grades. This involves the question such as the following:

      What should be included in a letter?

      How should achievement data be combined in assigning letter grades?
                                             2


      What frame of reference should be used in grading?

      How should the distribution of letter grades be determined?

       Each of these issues will be discussed in turn.

Determining What to Include in a Grade

As noted earlier, letter grades are likely to be most meaningful and useful when they
represent achievement only. If they are contaminated by such extraneous factor, amount
of work completed (rather than the quality of work), personal conduct, and so on, their
interpretation will become hopelessly confused. When letter grades combined various
aspects of pupil development, not only the lose their meaningfulness as a measure of
achievement, but they also suppress information concerning other important aspects of
development, A letter grade of B, for example, may represent average achievement with
outstanding effort and excellent conduct or high achievement with little effort and some
disciplinary infractions. Only by making the letter grade as pure as measure of
achievement is possible and reporting on these other aspects separately, can we hope to
improve our descriptions of pupil learning and development.

If letter grades are to serve as valid indicators of achievement, they must be based on
valid measures of achievement. This involves the process described earlier define course
objective as intended learning outcomes and developing or selecting tests and other
evaluation devices that measure these outcomes most directly. How much emphasis
should be given to tests, rating, written reports and other measures of achievement in a
letter grades is determined by the nature of course and the objective being stressed. Thus
a grade in English might be determined largely by tests and writing project. A grade in
science by tests and evaluations of laboratory performance. And a grade in music by tests
and rating on performance skill. The types of evaluation data to include in a course grade
and the relative emphasis to be given to each type of evidence are determined primarily
by examining the instructional objectives. Other things being equal, the more important
the objective is, the greater the weight it should receive in the course grade. In the final
analysis, letter grade should reflect the extent to which pupils have achieved the learning
                                            3


outcomes specified in the course objective, and these should be weighed according to
their relative importance.

Combining Data in Assigning Grades

When the aspects of achievement (e.g., tests, written reports, performance ratings) to be
included in a letter grade and the emphasis to be given to each aspect have been decided,
out next step is to combine the various element so that each element receives its intended
weight. If we decide, that the final examination should count 40%, the mid term 30%,
laboratory performance 20%, and written reports 10%, we will want our course grades to
reflect this emphasis; a typical procedure is to combine the element into a composite
score by assigning appropriate weights to each element and then uses these composite
scores as basis for grading.

Combining data into a composite score in order to produce the desired weightage is not
sample as it may appear at first glance. This can be illustrated by a simple example. Let
us assume that want to combine scores on a final examination and a term report and that
we want them to be given equal weight. Our ranges of scores on the two measures are as
follows:

                                    Range of Scores

              Final Examination                                   80 to 100

              Term Report                                          10 to 15

Because the two sets of scores are to be given equal weight, we may be inclined to simply
add together the final examination score and the term report score for each pupil. We can
check on the effectiveness of this procedure by comparing the composite score of pupil
who is highest on the final examination and lowest on the term report (10 + 10 = 110)
with a pupil who is lowest on the final examination and highest on the term report (80 +
50 = 130). It is obvious from this comparison that simply adding together the two scores
will not give them equal representation. Likewise there may be other erroneous methods
to equate scores without considering a larger difference between two composite scores.
This is due to the fact that the influence each component has on the composite score
                                             4


depends son the variability, or spread, of scores and not on the total number of points.
Thus, to weigh properly the components in a composite score, the variability of the scores
must be kept into account.

The range of scores in the example provides a measure of score variability, or spread, and
this can be used to equate the two sets of scores. We can give the final examination and
the term report equal weight in the composite score, by sing a multiplier that makes the
two ranges equal. Because the final examination scores a ranges of 40 (50-10), we would
need to multiply each final examination scores by 2 obtain the desire equal weight. If
wanted out final examination to count twice as much as the term report, it would be
necessary to multiply each final examination score by 4 rather them by 2.

A more refined weight age system can be obtained by using the standard deviation as the
measure of variability, but the ranges is satisfactory for must classroom purposes.

The components in a composite score can also be weighted properly by converting all
sets of scores to STANINES (Standard Scores, 1 through 9) .When all scores have been
converted to the Stanine system; the scores in each set have the same variability. They
then are weighed by simply multiplying each stanine score by the desired weight. Thus, a
pupil’s composite score would be determined as follows:




                      Desired Weight         Pupil’s Stanines       Weighed Scores


Examination                   2                      9                      18
Laboratory Work               1                      7                      7
Written Reports               1                      8                      8


                                             Composite Scores       =       33




These composite scores can be used to rank pupils according to an overall weighed
measure of achievement in order to assign letter grades.
                                             5


Selecting the proper Frame of Reference for Grading
Letter grades are typically assigned on the basis of one of the following frames of
reference:
I. Performance in relation to other group members (norm referenced).
II. Performance in relation to prespecified standards (criterion referenced).
III. Performance in relation to learning potential or amount of improvement
Assigning grades on a norm-referenced basis involves comparing a pupil’s performance
with that of a reference grouped, typically one’s classmates. With this system, the grade
is determined by the pupil’s relative ranking in the total group, rather than by some
absolute standard of achievement. Because the grading is based on relative performance,
the grade is influenced by both the pupil’s performance and the performance of the
group. Thus, one will fare much batter, grade wise, in a low achieving group then in a
high achieving group.
Although norm-referenced has the disadvantage of a shifting frame of reference (i.e.,
grades depends on group ability),it is widely used in the school, because much of
classroom tasting is norm referenced. That is, the tests are designed to rank the pupils in
order of achievement, rather than to describe achievement in absolute terms. Although
relative position is group is the key element in the norm-referenced system of grading,
the actual grades assigned are also likely to be influenced to some extent by the
achievement expectations that the teacher has acquired from teaching other groups. Thus,
a high-achieving group of pupils is likely to receive a larger proportion of good grades
then a low achieving group.
Assigning grades on a criterion referenced basis involves comparing a pupil’s
performance to prespecified standers set by the teacher. These standers are usually
concerned with the degree of mastery to be achieved by pupils and may be specified as

1. Tasks to be formed (e.g., type 40 words per minute without error) or

2. The percentage of correct answers to be obtained on test designed to measure a clearly
defined set of learning tasks.
                                             6


Thus, with this system, letter grades are assigned on the basis of an absolute standard of
performance rather than a relative standard. If all pupils demonstrate a high level of
mastery, all will receive high grades.

The criterion referenced system of grading is much more complex than it first appears.
To use absolute level of achievement as a basis for grading requires requires that

      The domain of learning tasks be clearly defined,

      The standards of performance be clearly specified and justified .and

      The measures of pupil achievement be criterion referenced. These conditions are
       difficult to meat except in a mastery learning situations. When complete mastery
       is the goal, the leaning tasks tend to be more limited and easily defined. In
       addition. Percentage correct scores, which are widely used in setting absolute
       standards, are must meaning gull in mastery learning because they include how
       far a pupil is from complete. All too frequently, schools are absolute grading
       based on percentage correct scores (e.g., A = 5 100. B = 85 - 94, C = 75 - 84, D =
       65 - 74 F below 65) but the domain of learning tasks has not been clearly defined
       and the standard have bee set in a complete arbitrary manner. To fit the grading
       system teacher attempt to build test (norm referenced) with scores in the 60 to 100
       range. If the test turns out to be too difficult or too easy, they represent adjusted
       level of performance on some ill defined conglomerate tasks. 4. Although
       reporting pupil performance in relation potential or amount of improvement
       shown has been fairly widely used at the elementary schools level, this type of
       grading is fraught with difficulties. Making reliable estimates of learning
       potential, with or without test, is a formidable task, because judgments or
       measurements of potential are likely to be contaminated by achievement to some
       unknown degree. Similarly, improvement (i.e. growth in achievement) over shorts
       spans of time is extremely difficult to estimate reliably with classroom measure of
       achievement in relation to potential, and in judging degree of improvement will
       result in grades of low dependability. If used at all (e.g., to motivate low ability
       pupil’s). Such grades should be used as supplementary. In dual marking, for
                                             7


       example, one letter grade might be used to represent achievement in relation to
       potential, or the degree of improvement shown since the marking period.

                     Determining the Distribution of grades

As noted in the previous section, there are two ways of assigning letter grades to measure
the level of pupil achievement the norm referenced system based on relative low level of
achievement and criterion referenced based on absolute level of achievement.

Norm-Referenced Grading
The assignment of norm-referenced grades is essentially a mater of ranking the pupils in
order of overall achievement and assigning latter grade on the basis of each pupil’s rank
in the group. This ranking might be limited to a single classroom group or might be
based on combined distribution of several class room groups taking the same course. In
any event, before letter grades can be assigned, the proportion of As, Bs, Cs, Ds and Fs to
be used must be deter-mind.
One method of grading that has been widely used, in the past is to grade on the basis of
the norm curve. The procedure results in a distribution of grades like that shown on the
next page Grading on the normal curve results in a equal percentage of As and Fs, Bs and
Ds. Thus, regardless of a group’s level of ability, the proportion of high grades is
balances by and equal proportional of low grades. Such grading is seldom defensible for
classroom groups because (1) the groups are usually too small to yield a normal
distribution: (2) classroom evaluation instruments are usually not designed to yield
normally distributes scores, and (3) the pupil population becomes more select as it
improve through the grades and the less-able pupil’s fail or drop out of school. It is not
only when a course or combined courses have a relatively large and unselected group of
pupils that grading on the normal curve might be defined. Even than, however, one might
ask whether the decision concerning the distribution of grades should be left to a
statistical model (i.e., normal curve) or should be made on a more relational basis.
The most sensible approach in determining the distributions of letter grades to be used in
a school is to have the school staff set general guidelines for the approximate
distributions of mark. This might involve separate distributions for introductory and
advanced courses, for gifted and slow learning classes, and the like. In any event, the
                                             8


distributions should be flexible enough to allow for variation in the caliber of pupils from
one course to another and from one time to another in the same course. Indicating ranges
rather than fixed percentages of pupils who should receive each letter grades offer this
flexibility. Thus, suggested distributions for an introductory course might be as follows’
A= 10 to 20 percent of the pupils
B= 20 to 30 percent of the pupils
C= 30 to 50 percent of the pupils
D= 10 to 20 percent of the pupils
F= 0 to 10 percent of the pupils
These percentage ranges are presented for illustrative purposes only; there is no simple or
scientific means of determining what these ranges should be for a given situation. The
local school staff, taking into account the school’s philosophy, the pupil population and
the purposes of the grades must make the decision. All staff members must understand
the basis for assigning grades and this basis must be clearly communicated to the users of
the grades.
In setting approximate distributions of grades for teacher to follow, the distributions
should provide for the possibility of no falling grades. Whether pupils pass or fail a
course should be based on their absolute level of learning rather than their relative
position in some group. If all low-ranking pupils have mastered enough of the material to
succeed at the next highest level of instruction, they all probably should pass. On the
other hand, if some have not mastered the minimum essentials needed at the next highest
level, these pupils probably should tail weather minimum performance has been attained
can be determined by reviewing the low-ranking pupils, performance on tests and other
evaluation instruments or by administering a special mastery test on the course’s
minimum essentials. Thus, even when grading is done on a relative basis, the pass-fail
decision must be based on an absolute standard of achievement if it is to be educationally
sound.
Criterion-Referenced Grading
Criterion referenced grading is most useful when a mastery learning approach is used,
because mastery learning provides the necessary conditions for grading on an absolute
basis. This includes delimiting the domain of learning tasks to be achieved, defining the
                                            9


instruct-ional objective in performance to be attained, and measuring the intended
outcomes with criterion referenced instruments.
If the course’s objectives have been clearly specified and the standards for mastery
appropriately set, the letter grades in a criterion referenced system may be defined as the
degree to which the objectives have been attained, as follows:
A = Out standing. Pupil has mastered all of the course’s major and minor instructional
objectives.
B = Very Good. Pupil has mastered all of the course’s major instructional objectives and
most of the minor objectives.
C = Satisfactory. Pupil has mastered all of the course’s major objectives but just a few
of the minor objectives.
D =     Very weak. Pupil has mastered just a few of the course’s major and minor
instructional objectives and barely has the essentials needed for the next highest level of
instruction. Remedial work would be desirable.
F =    Unsatisfactory. Pupil has not mastered any of the course’s major instructional
objectives and lakes the essentials needed for the next highest level of instruction.
Remedial work is needed.
In the tests and other evaluation instruments have been designed to yield scores in terms
of the percentage of correct answers, criterion referenced grading then might be defined
as follows:
A = 95 to 100 percent correct
B = 85 to 94 percent correct

C = 75 to 84 percent correct

D = 65 to 74 percent correct

F= below 65 percent correct

As noted earlier, defined letter grades in this manner is defensible only if the necessary
condition of a criterion referenced system have been met. Using percentage correct scores
when the measuring instruments are bases on some undefined hodgepodge of learning
tasks produce uninterruptible grades. With criterion referenced grading system such as
                                            10


these, the distribution of grades is not predetermined. If all pupils demonstrate a high
level of mastery, all will receive high grades. If some pupil’s demonstrate a low level of
performance, they will receive low grades. Thus, the distribution of grades is determined
by each pupil’s absolute level of performance, and not by the pupil’s relative position in
the group.

There are basically two procedures for determining what letter grade a pupil should
receive in a criterion referenced system. The first called’ the one shot system’ provides a
single opportunity to achieve the prespecified standard. The pupils assigned whatever
grade is earned on the first attempt. The second procedure, which is widely used in
mastery learning, permits the pupils to make repeated attempts to achieve the
prespecified standard. In this approach may result in some failing grades, where as the
second approach eliminates failure. Typically, only the letter grades A, B, C, D, and f are
used, and pupils are permitted to repeat examination unit all satisfactory level of
performance is achieved.

The criterion referenced system for reporting on pupil’s progress seldom uses letter
grades alone. A comprehensive report generally includes a check list for objective to
inform both pupil and parents which objective have been mastered and which have not
been mastery by the end of each marking period. In the some mastery learning
programme letter grades are assigned to each objective to indicate the level of mastery
achieved.

Individual differences among students are inevitable, regardless of the type of instruction
or evaluation measure employed. Even with individualized, mastery learning approaches,
students will differ in rate and degree of mastery. As Eble (1979) has pointed out:

“The notion that criterion referenced testing will avoid problems of marking seems to be
based on quite unrealistic expectation of uniform achievement in learning by all
students.….Programmes of mastery learning can not abolish individual differences of in
ability, interest, and determination.”
                                             11


Students will continue to have strengths and weaknesses irrespective of the instructional
strategies employed, and these should be diagnosed and communicated to the students
and their parents.

                                         Ranking

A ranking is a relationship between a set of items such that, for any two items, the first is
either 'ranked higher than', 'ranked lower than' or 'ranked equal to' the second.
In mathematics, this is known as a weak order or total preorder of objects. It is not
necessarily a total order of objects because two different objects can have the same
ranking. The rankings themselves are totally ordered. For example, materials are totally
preordered by hardness, while degrees of hardness are totally ordered.

By reducing detailed measures to a sequence of ordinal numbers, rankings make it
possible to evaluate complex information according to certain criteria. Thus, for example,
an Internet search engine may rank the pages it finds according to an estimation of
their relevance, making it possible for the user quickly to select the pages they are likely
to want to see.

Ranking in Statistics

In statistics ranking refers to the data transformation in which numerical or ordinal values
are replaced by their rank when the data are sorted. For example, the numerical data 3.4,
5.1, 2.6, 7.3 are observed, the ranks of these data items would be 2, 3, 1 and 4
respectively. For example, the ordinal data hot, cold, warm would be replaced by 3, 1, 2.
In these examples, the ranks are assigned to values in ascending order. (In some other
cases, descending ranks are used.) Ranks are related to the indexed list of order statistics,
which consists of the original dataset rearranged into ascending order.

                        Strategies for Assigning Rankings

It is not always possible to assign rankings uniquely. For example, in a race or
competition two (or more) entrants might tie for a place in the ranking. When computing
an ordinal measurement, two (or more) of the quantities being ranked might measure
                                             12


equal. In these cases, one of the strategies shown below for assigning the rankings may
be adopted.

A common short-hand way to distinguish these ranking strategies is by the ranking
numbers that would be produced for four items, with the first item ranked

ahead of the second and third (which compare equal) which are both ranked ahead of the
fourth. These names are also shown below.

Standard competition ranking ("1224" ranking)

In competition ranking, items that compare equal receive the same ranking number, and
then a gap is left in the ranking numbers. The number of ranking numbers that are left out
in this gap is one less than the number of items that compared equal. Equivalently, each
item's ranking number is 1 plus the number of items ranked above it. This ranking
strategy is frequently adopted for competitions, as it means that if two (or more)
competitors tie for a position in the ranking, the position of all those ranked below them
is unaffected (i.e., a competitor only comes second if exactly one person scores better
than them, third if exactly two people score better than them, fourth if exactly three
people score better than them, etc.).

Thus if A ranks ahead of B and C (which compare equal) which are both ranked ahead of
D, then A gets ranking number 1 ("first"), B gets ranking number 2 ("joint second"), C
also gets ranking number 2 ("joint second") and D gets ranking number 4 ("fourth"). In
this case, nobody would get ranking number 3 ("third") and that would be left as a gap.

Modified competition ranking ("1334" ranking)

Sometimes, competition ranking is done by leaving the gaps in the ranking
numbers before the sets of equal-ranking items (rather than after them as in standard
competition ranking). The number of ranking numbers that are left out in this gap
remains one less than the number of items that compared equal. Equivalently, each item's
ranking number is equal to the number of items ranked equal to it or above it. This
ranking ensures that a competitor only comes second if they score higher than all but one
of their opponents, third if they score higher than all but two of their opponents, etc.
                                             13


Thus if A ranks ahead of B and C (which compare equal) which are both ranked ahead of
D, then A gets ranking number 1 ("first"), B gets ranking number 3 ("joint third"), C also
gets ranking number 3 ("joint third") and D gets ranking number 4 ("fourth"). In this case,
nobody would get ranking number 2 ("second") and that would be left as a gap.

Dense ranking ("1223" ranking)

In dense ranking, items that compare equal receive the same ranking number, and the
next item(s) receive the immediately following ranking number. Equivalently, each item's
ranking number is 1 plus the number of items ranked above it that is distinct with respect
to the ranking order.

Thus if A ranks ahead of B and C (which compare equal) which are both ranked ahead of
D, then A gets ranking number 1 ("first"), B gets ranking number 2 ("joint second"), C
also gets ranking number 2 ("joint second") and D gets ranking number 3 ("third").

Ordinal ranking ("1234" ranking)

In ordinal ranking, all items receive distinct ordinal numbers, including items that
compare equal. The assignment of distinct ordinal numbers to items that compare equal
can be done at random, or arbitrarily, but it is generally preferable to use a system that is
arbitrary but consistent, as this gives stable results if the ranking is done multiple times.
An example of an arbitrary but consistent system would be to incorporate other attributes
into the ranking order (such as alphabetical ordering of the competitor's name) to ensure
that no two items exactly match.

With this strategy, if A ranks ahead of B and C (which compare equal) which are both
ranked ahead of D, then A gets ranking number 1 ("first") and D gets ranking number 4
("fourth"), and either B gets ranking number 2 ("second") and C gets ranking number 3
("third") or C gets ranking number 2 ("second") and B gets ranking number 3 ("third").

In computer data processing, ordinal ranking is also referred to as "row numbering"....

Fractional ranking ("1 2.5 2.5 4" ranking)

Items that compare equal receive the same ranking number, which is the mean of what
they would have under ordinal rankings. Equivalently, the ranking number of 1 plus the
number of items ranked above it plus half the number of items equal to it. This strategy
                                             14


has the property that the sum of the ranking numbers is the same as under ordinal
ranking. For this reason, it is used in computing Borda counts and in statistical tests (see
below).

Thus if A ranks ahead of B and C (which compare equal) which are both ranked ahead of
D, then A gets ranking number 1 ("first"), B and C each get ranking number 2.5 (average
of "joint second/third") and D gets ranking number 4 ("fourth").

What is Assessment?

Assessment is the ongoing process of:


   Establishing clear, measurable, expected outcomes that demonstrate
    institutional effectiveness;
   Implementing programs and practices designed to achieve those outcomes;
   Systematically gathering, analyzing, and interpreting evidence to determine
    how well programs and practices are working at meeting their expected
    outcomes; and,
   Using the resulting information to understand and improve institutional
    effectiveness.
Analyzing and Reporting Assessment Results

In presenting qualitative and/or quantitative assessment results, there are a number of best
practices to consider in order to most effectively high lighting the important findings in
your data. These pages provide guidance on how to best present data visually (in graphs,
charts, and tables), and also detail general rules governing the presentation of data in text.


Reporting Quantitative Results


Reporting Quantitative Results provides general rules governing how to report numerical
and statistical findings, including the consideration of the visual presentation of data in
tables as well as charts and graphs.
                                                  15




Coding and Reporting Qualitative Data
Coding and Reporting Qualitative Data considers the most efficient ways to code and
analyze qualitative data, as well as how to present these data in the most expressive and
effective manner.


Considerations for Writing an Assessment Report


Considerations for Writing an Assessment Report provides useful guidelines for
assessment report content and organization.


Reporting Quantitative Results

          General Practices in Reporting Quantitative Data
          Presenting Data in Charts and Graphs
          Pie Charts
          Bar Graphs
          Line Graphs
          Presenting Data in Tables
General Practices in Reporting Quantitative Data
            Data can be presented in text, table, or chart form. When presenting data in all
             three forms, care should be taken to include only information and/or images that
             help to clarify points being made.
            For reference purposes, tables are usually the sensible option. Extensive tables
             should usually appear as appendices at the end of a report.
            In general, tables are better than graphs for giving structured numeric
             information. Graphs are better for demonstrating trends, making comparisons or
             showing relationships.
                                               16


        Text alone should not be used to convey more than three or four numbers. Sets
         of numerical results should usually be presented as tables or pictures, rather than
         included in the text.
        When whole numbers are given in text, numbers less than, or equal to, nine
         should be written as words, while numbers from 10 upwards should be written
         in digits.
        When decimal numbers are quoted, the number of significant digits should be
         consistent. Generally, one point past the decimal point is sufficient.
        Tables and graphs should be self-explanatory. The reader should be able to
         understand them without detailed reference to the text. The title should be
         informative and rows and columns of tables or axes of graphs should be clearly
         labeled. On the other hand, the text should always include mention of the key
         points in a table or figure. If a table does not warrant discussion, it should not be
         there.
        Statistical information beyond means and frequencies (e.g., standard deviations,
         p-values, t-values), is usually required in formal scientific papers, but may not
         be necessary for a more general readership. When presented, care should be
         taken to do so in a way that does not obscure the main message of the table or
         graph.
Presenting Data in Charts and Graphs
Charts and graphs are often the best way to demonstrate trends in data and make
comparisons between different groups. Different types of graphs are required to most
efficiently and effectively present different types of findings. The following sections
detail the most common types of charts and graphs and the important rules governing
their use.

Pie Charts
        Pie charts have limited utility. They can only be used to show parts of a whole (if all
         parts total 100%).
        Pie charts emphasize general findings, but do not make small differences apparent.
        Pie charts should only be used to represent categorical data with a relatively small
         number of values and should not consist of more than five or six slices.
                                                 17

        When presenting a pie chart, it is better not to use 3-D features, or break out the pieces,
         as this often makes it more difficult to compare the relative size of the slices.




        It is always necessary to include category labels or a legend that describes which
         slice corresponds with which category. If labels are brief enough, it is better to
         place category labels directly next to the pie slices to which they correspond.
        It is good practice to include value labels (indicating the percentage of the pie
         represented by a given slice).
        It is also good to pre-sort data so that, clockwise or counter-clockwise, the
         relative size of pie slices is most apparent.




The purpose of color in pie charts is to differentiate between pie slices to further facilitate
comparison. When using color, it should be used thematically.
                                             18


The best use of color in a size-ordered pie chart is a progression of dark to light hues
from the largest slice to smallest slice (see 2-D pie chart above).
A bar graph, rather than multiple pie graphs, is the better option if data need to be
compared by more than one value. Pie graphs should not be used to represent more than
one categorization of data.

                                Multiple Pie Graphs




                                A Single Bar Graph
                                              19


Bar Graphs
      Bar graphs are used for direct comparison of data (e.g., student GPA’s by class
       year).
      Bar graphs can also be used to show time series data when the number of time
       intervals is small.
      If all values are positive integers, the scale should generally use 0 as a baseline.
       In the event that values include both positive and negative integers (e.g., in
       graphing differences in means), 0 should be the midpoint of the scale.
      Scale ranges should be standardized and not vary between graphs, when
       possible.
      Always try to avoid using 3-D features in a bar graph. The complexity of 3-D
       graphs makes them ineffective in conveying results to most audiences and there
       is usually a greater amount of data distortion that occurs.
      Bar graphs may be vertical or horizontal. The only difference between
       horizontal and vertical bars is that horizontal bar charts are seldom used to
       portray time series.
      To facilitate comparison and analysis, it is desirable that columns be sorted in
       some systematic order. The most common and visually effective schema is
       according to size of value.




      It may also be desirable to order findings by a particular category such as class year (see
       Clustered Bar Graph below), where it is best to order sequentially from freshman to
       senior year or visa versa, or by grade achieved, where it is best to order by the
       standardized grade scale.
      Stacked bar graphs, which consist of one or more segmented bars where each segment
       represents the relative share of a total category, are generally not preferred because it is
                                                  20

           difficult to make comparisons among the second, third, or subsequent segments without
           a standard baseline.
          When graphing data from two or more different series, or different classes within the
           same series, it is preferable to create a bar graph that groups these values together, side
           by side (see below).




Line Graphs
           Line graphs are most often used to display time series data (e.g., the
            average GPA of students in a starting cohort over their first eight
            semesters, or program enrollment over the past 10 years). See graph
            below.
                                                21


          Compared with bar graphs, line graphs are more effective in presenting
           five or more data points, but less effective in providing emphasis on
           differences over relatively few periods of time.
          When plotting time series data in a line graph, it is convention that the
           x-axis (horizontal) contains the categories of time (e.g., days of the
           week, months, years – depending on the data), and the y-axis (vertical)
           has frequencies of what is being measured (see graphs below).
          Graphs with more than four or five lines tend to become confusing
           unless the lines are well separated.




      In a line graph with more than one line, different line styles (e.g., solid line, dashed line,
       etc.), colors and/or plotting symbols (e.g., asterisks, circles, etc.) should be used to
       distinguish the lines.
Presenting Data in Tables
      Tables are the most effective way to present data for reference purposes.
      A table should always be given a meaningful, self-explanatory title.
      Each part of a table should be labeled clearly and abbreviations should be
       avoided.
                                             22


   The number of digits and decimal places presented should be consistent and
    should be the minimum number that is compatible with the purpose of the table.
   It is usually better to convert counts into percentages, unless providing a simple
    frequencies table. More readers will care that 78% of students agreed with a
    statement rather than 325 students agreed.
   It is always important to include information in a table about the size of the
    sample from which a percentage is derived.
   A table should be constructed so that it is easy for readers to see differences and
    trends. If a table is presenting results from two or more different groups, years or
    survey cycles, it is good to include a column that indicates either the percentage
    change or the significance of differences observed.

                                          Table 1

           Comparision of Course Satisfaction Measures for Course X
                                          2008 vs. 2009

                   Satisfaction Measure                   2009         2008




              Applicability of Material                   88% 86%
              Covered
              Access to Professor                         79%    68%
              Course Content                              72%    67%
              Course Organization                         56%    53%
              Class Size                                  43%    44%
                                          23




                        Improved Version of Table 1
    Comparision of Course Satisfaction Measures for Course X


                                   2008 vs 2009


                         (% "Satisfied" or "Very Satisfied")




     Satisfaction Mea                  2009(N = 134 )   2008(N = 123 )        Difference




     Applicability of Material         88%              86%              2
     Covered
     Access to Professor               79%              68%              11
     Course Content                    72%              67%              5
     Course Organization               56%              53%              3
     Class Size                        43%              44%              -1


   It is best to present information in an order that makes sense to the
    reader by sorting from most frequently chosen response or highest score
    to lowest (see tables above).
                                                            24


       A table should draw attention to the most salient points. Use boldface,
        italics, borders, and/or colors to draw attention to the most important
        figures, and put totals in boldface (see tables above).
       Always note the source of data presented in a table (see tables above).
       More complex tables that organize information by more than one level
        should be constructed to best reflect how data are grouped. It is best to
        merge cells that apply to more than one column in a table, rather than
        repeating the grouping information in more than one column. Shading
        can also provide greater organization and distinction between groups of
        data (see tables below).

                                                    Table 2
                                        2008 vs 2009 Divisional Enrollments by Gender


                       Academic Division           2009          2009      2008          2008
                                                   Males       Females     Males       Females
                   Physical Sciences             368         182         355         173
                   Natural Sciences              658         495         642         505
                   Humanities                    352         435         375         415
                   Social Sciences               786         962         801         1002
                   Total                         2164        2074        2173        2095



                           Improved Version of Table 2
                                     2008 vs 2009 Divisional Enrollments by Gender



                                                    2009                     2008

                                        Males     Females        Males                   Females

                 Physical Sciences      368       182            355                     173
                 Natural Sciences       658       495            642                     505
                 Humanities             352       435            375                     415
                 Social Sciences        786       962            801                     1002
                 Total                  2164      2074           2173                    2095




                 Coding and Reporting Qualitative Results

Coding Qualitative Data
       It is always best to read through all comments or responses before attempting to
        group or code data.
                                           25


       It is good to start simply with one broad layer of coding organized around three
        to seven major themes (and no more than 10). Try to use categories that, from
        your initial reading of all comments, you are certain a good number of responses
        will fit into.
       For comments that appear to be outliers, create a miscellaneous category.
        Results from this category can later be revisited to see if they fit within the
        developed framework, but coding them individually will only muddy attempts at
        analysis.
       Some comments will address more than one category. These comments should
        be coded in each of the categories they address.
       Once comments are grouped in overarching categories, review each created
        category for an additional layer(s) of organization.
Reporting Qualitative Findings
       It is best to start any report of qualitative findings with a brief explanation of
        how data were processed and coded, as well as how data exemplars were chosen
        for presentation. Remaining open about the process of how qualitative data were
        made empirical builds trust in the findings presented.
       It is best to allow readers to visualize the structure of data coding in the
        presentation of results. Data should be presented so that the reader is able to
        view the different levels of coding and how data were categorized and
        subcategorized.
       Quantifying top layer coding results can be a way of demonstrating the relative
        importance of different ideas or concepts in the data, and helps to legitimize
        findings for quantitatively-oriented audiences.
       It is best, however, not to quantify results beyond the first layer of coding. This
        can give a false sense of the nature of the data, by misplacing the emphasis on
        numerical representation, as opposed to the rich description they provide.
       When presenting findings, it is most effective to include actual examples of
        comments that reflect the concept being coded. This also helps to build trust in
        findings presented, as it elucidates the connection between the open text
        comments analyzed and the coding structure imposed in analysis.
                                            26


An Assessment Report Should

     Include information about the author, the originating office and the date it was
      produced.
     Consider who its audience(s) is and how much information they need.
     Describe methodologies and procedures used to conduct assessments (include
      sample sizes, response rates, when and how assessments were conducted).
     Organize presentation of data around stated goals.
     Synthesize data from various sources to tell a story with a meaningful point.
     Consider prior research and findings.
     Use a mixture of visuals and text to emphasize key findings.
     Make sure all graphs and tables used within the body of the report are addressed in
      the text.
     Use appendices for lengthy, detailed tables.
     Not include analysis that reaches beyond the data.
     Include any hypothesizing about findings beyond the data collected in a separate,
      properly labeled section (e.g., suggestions for future research).
     Conclude with recommendations based on assessment findings presented.
                                          27




References:

             Karmel, L.J. and Karmel, M.O. (1978); Measurement and Evaluation in
              the Schools, 2nd Edition , Macmillan Publishing Co., Inc., New York.
             http://en.wikipedia.org/wiki/Ranking
             http://virg.vanderbilt.edu/assessmentplans/About.aspx

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:24
posted:4/27/2012
language:English
pages:27