GUIDE TO PERFORM

Document Sample
GUIDE TO PERFORM Powered By Docstoc
					                            GUIDE TO PERFORM
            SELF-ASSESSMENT OF PERFORMANCE MEASUREMENT DATA
                      SUBMITTED IN THE ANNUAL REPORT

This guide has been provided to help you prepare the self-assessment documents for
each performance measure in your State Action Plan (Strategic Plan and Annual
Report).


STEPS

   1.  Choose a measure to review.
   2.  Review measure definition and other data relevant to the measure.
   3.  Identify departmental staff involved in gathering data for the measure.
   4.  Determine if you can re-create the numbers reported in the strategic plan and
       annual report.
   5. Collect information on the process used to generate performance data.
   6. Develop a high-level process flowchart. An example is found at the end of this
           guide.
   7. Determine whether adequate controls over performance measure data exist to
       ensure consistent reporting of accurate information.
   8. Determine if source documents support the measure results reported in strategic
       plan and annual report.
   9. Determine the certification category for the measure.
   10. If applicable, describe plans for improvement.
   11. Repeat for each performance measure in your report. If more than one division’s
           numbers help develop the composite measure, assessment should be
           performed for each part.

1. Choose measure(s) to review.

Select one of your performance measures to review.

2. Review measure definition and other data relevant to the measure.

   A) Obtain a copy of the measure definition developed by your agency. If one has
      not been written, determine an accurate definition and reduce it to writing. The
      definition should include, if applicable, the following information:

           Program
           Measure Name
           Description
           Data Sources
           Calculation Methodologies
           Persons Responsible
           Data Limitations



D:\Docstoc\Working\pdf\28f94ddb-1a8c-47f9-9c49-769f79c57d74.doc                          1
   B) Determine if the measure definition is clear, specific, and not open to
      interpretation. If the measure definition is not clear, not specific, and open to
      interpretation indicate in the self-assessment document what needs to be
      changed. Also, indicate when and by whom it will be changed.

   C) Determine if the calculation method is consistent with the measure’s definition.

   D) Determine if the measure name accurately describes what it is measuring.

      Tip:   Having an accurate measure name is also important because
             stakeholders will often not take the time to delve into the details of
             the measure definition.

   E) Has this measure been reviewed in the last three years by sources external to
      the agency (federal, state, Public Funds, etc.)? Yes or No

     If yes, discuss the results in the appropriate section of the self-assessment
document.

Measure Definition Example:

Measure Name: Percent of Indigent Consumers Whose Dependent Living Risk was
diminished

Description: This measure addresses the extent to which services provided by the
program enable indigent individuals to minimize their dependency on others due to
blindness or severe vision loss.

Frequency: Quarterly

Data Sources: Visually impaired database (VI6)

How it is collected: Service personnel in field offices enter all data for the consumers
they serve into the department’s database. A record of each consumer is begun at the
point an application for services is taken or a referral is received. After assessing the
consumer’s situation, service personnel note in the consumer’s database record
whether the individual is at risk of increased dependency on others. At the time the
consumer’s case is closed, the staff enters a code noting whether or not the consumer’s
risk for dependent living is diminished because of services provided. Consumer coding
is presented in a quarterly custom report that extracts this database information.

Calculation (methodology): A percentage is obtained by dividing the number of
consumers coded as having a diminished dependent living risk at closure by the
number of consumers coded as being at risk during the eligibility phase of their
rehabilitation process.



D:\Docstoc\Working\pdf\28f94ddb-1a8c-47f9-9c49-769f79c57d74.doc                           2
Data limitations: The determination of risk of dependence at application and the
degree of dependence at closure is based on the judgment of professional staff.

3. Identify departmental staff involved in gathering data for the measure.

   A) Obtain the name(s) and title(s) of all personnel who collect data for the measure
      and list their names and titles in the self-assessment document.

   B) Find out how long have they had responsibility for this information and document.

   C) Identify and document who prepares and submits the information for preparation
      of the strategic plan and Annual Report.


   Tip:   Ensure that the names and title of all individuals involved with the
          measure is kept current.

4. Determine if the numbers reported can be recreated.

   A) Gather summary documentation for the measure.

   *   Attach to the self-assessment document a copy of the first page of the summary
       documentation showing field headings. Also, attach any totals pages.

Definition of summary documentation:             Summary documentation shows the
final calculations that support the performance data reported. Documentation should
also include any reports that support the measure result reported.

Examples of summary documentation:

    Current computer printouts that reproduce summary calculations of the reported
     performance
    Archived computer printouts produced at the reporting date that document the
     summary calculations
    Quarterly summary calculation documentation
    Spreadsheets
    Manual calculation sheets

   Important Note: Data to be reviewed should be from the time period reported in
your annual report.

   B) Recreate the number reported and determine if the summary documentation is
      within a tolerable range of plus or minus three percent of the number that was
      reported. If you find that the number is not within three percent, investigate the
      reasons for the discrepancy, record your results, and provide recommendation(s)



D:\Docstoc\Working\pdf\28f94ddb-1a8c-47f9-9c49-769f79c57d74.doc                           3
      for improvement. As you continue the certification review for this measure, try to
      determine what the actual number reported should have been. Record the
      results of this step.

      Note: If the summary documentation were not within +/-3 percent, you
            would place the measure in the “Inaccurate” certification category.
            Continue with the review.

Example

A jobs training program reported 1,000 degrees were awarded during the previous
year. The documented measure definition for Number of Degrees Awarded states “the
number of degrees awarded after completing all of the training courses.”

To determine the number of degrees awarded for the entire fiscal year, we must
consider the number of degrees awarded each semester. The following is an example
of a document with supporting summary calculations.

                                            Total number of
                                                degrees
                          Semester             awarded
                          Fall                          350
                          Spring                        500
                          Summer I and
                          II                             150
                          Year Total                   1,000

If the summary documentation for the number of degrees awarded is between 970 and
1,030, then the measure is within +/-3 percent of the number reported.

   Tip:   Review summary documents on a regular basis to ensure that these
          numbers are the same as the numbers reported.

   C) One of the reasons that a number may not be within +/- 3 percent is that the
      department deviated from the measure definition.

   If the measure calculation deviated from the measure definition and caused more
   than a three percent difference between the performance reported and the correctly
   calculated number based on performance measures data, explain in self-
   assessment document.

5. Collect information on the process used to generate performance data.

   A) Identify and document the event(s) that begin the process of collecting data for
      this measure.



D:\Docstoc\Working\pdf\28f94ddb-1a8c-47f9-9c49-769f79c57d74.doc                          4
   B) Identify and document the events that occur from the beginning of the process
      until the data is reported.

   C) Identify and document how and where the data is stored (automated or manual)
      during the period of collection.

   D) Identify and document how the performance measure is calculated.

   E) Identify and document if anyone reviews the calculation.

   Important Note: If data comes from a third party, describe the procedures that
   are followed to ensure that the data is accurate.

6. Develop a high-level process flowchart.

Using the information gathered in the previous step develop a flowchart that represents
the key points in the flow of the measurement data. Attach to the self-assessment
document. An example is included at the end of this guide.

7. Determine whether adequate controls over performance measure data exist to
   ensure consistent reporting of accurate information.

Using the high level flowchart that you developed, determine whether the necessary
controls exist at each key point in the data flow. Controls should be reviewed from the
initial point that performance information is recorded until the accumulated measure
information is reported. You should examine input, process, and review controls to
ensure that an effective control structure exists. The system can be manual, automated
or a combination of both.

Each performance measurement system is different and may need greater, fewer, or
different controls than the one in the examples below to be effective.

   A) Identify and then determine whether input, process, and review controls should
      and do exist at each step in the data flow.


Example of Controls for an Automated System:

An automated system is one in which the computer is the major source of information
and is the major source of calculations.

Input controls:

The data entry supervisor should review information entered into the computer system
for accuracy.




D:\Docstoc\Working\pdf\28f94ddb-1a8c-47f9-9c49-769f79c57d74.doc                           5
   A) Controls should exist over third-party sources of information.

           Departments should obtain written documentation of third-party controls
            when possible.

           If the third party has no controls, the department should conduct
            necessary inquires for assurance that the information received is accurate.

Process Controls:

    The person responsible for calculating the performance data should understand
     the origin of the information and stay current with any changes in the form of the
     information.
    Procedures should be in place to ensure that the computer program used to
     calculate any portion of the performance data is capturing the correct information
     and is performing the correct mathematical calculations.
    Controls should be in place to ensure only authorized individuals have the ability
     to enter or change data in the system.

Review Controls:

    The department should periodically review information submitted by field offices
     and third parties.
    A supervisor should review calculations of the performance data to ensure that
     they are consistent with the measure definition and to check for mathematical
     errors.
    The person responsible for the accuracy of the data should review the
     information reported before the Annual Report or strategic plan is submitted as
     complete.


8. Determine if source documents support the measure results reported.

The purpose of this step is to test the source documentation that supports the measure
results reported. Source documents are documents associated with the events that
prove the activity occurred.

   A) Obtain/gather a list of all items counted for the measure for the period in which
      you are doing the review.

   B) Document the criteria that you are going to test for each item.

   C) Choose a sample of items to test to see if the correct information has been
      transferred from the “tested” item to the database or report.

   *   In the self-assessment document, list the record numbers you have sampled,



D:\Docstoc\Working\pdf\28f94ddb-1a8c-47f9-9c49-769f79c57d74.doc                           6
       and which, if any, had errors.

9. Determine the certification category for the measure if you have not already
done so.

Four categories have been developed to evaluate each performance measure. The
categories are as follows:

    Certified (C) – indicates the reported number can be replicated and procedures
     used to test the reported number indicate the result is within an acceptable range
     of accuracy.

    Certified with Qualifications (CQ) – indicates the reported number can be
     replicated and tested procedures indicate the reported number is within an
     acceptable range of the reported number; however, controls over the data
     collection and reporting are not adequate to ensure continued accuracy.

    Factors Prevented Certification (FPC) – indicates data is unavailable or
     incomplete, data cannot be recreated, and controls are not adequate to ensure
     accuracy of the data reported.

    Inaccurate (I) – indicates the actual performance measure is not within an
     acceptable range of the reported performance results, or is not within an
     acceptable error rate of the documents tested.

   *   Place the reviewed measure in one of four certification categories.

10. If applicable, describe plans for improvement.

An action plan should be developed if the measure was categorized as “factors
prevented certification, “Inaccurate”, or if you found control weaknesses during the
review.

In the self-assessment document, describe your action plan for improvement. The
action plan should include the individual(s) responsible for the action step(s) and the
estimated time frame to implement.




D:\Docstoc\Working\pdf\28f94ddb-1a8c-47f9-9c49-769f79c57d74.doc                           7
         Flowchart of Measure " Cost Per Acre Mowed with Contract for Industry Standard of Mode II"
                                                                                                   Step No. 6 in the Self-Assessment Guidelines

          Parks and Recreation            Mowing                     Parks and Recreation
          Inspector calls mowing          completed**                Inspector inspects
          contractor when                                            zone*** to ensure that                             Approved?
          mowing is needed*                                          cut meets standards
                                                                                                                                                        Yes

                                                                                                         No

                                                                    Mowing redone

          Parks and                      Parks and Recreation's                                          Financial                  Inspector
          Recreation's                   Financial Manager             Approved?                         Manager                    prepares
          Accounts                       submits paperwork to                                            reviews
          Payable
                                        Yes and Recreation's
                                         Parks
                                                                                                                                    paperwork****
                                                                                                         paperwork                  to pay
          Department                     Accounts Payable Office                                         for                        contractor and
          submits                                                                                        approval                   forwards to
          paperwork to                                                                                                              Parks and
          the City's                                                                          No
                                                                                                                                    Recreation's
          Accounts                                                                                                                  Financial
          Payable Office                                                                                                            Manager




          City's Accounts                Monthly the               Monthly the                                                      Quarterly the
          Payable Office                 Financial                 Financial                       Monthly the Performance          Performance
          cuts check and                 Manager                   Manager                         Measures Administrator           Measures
          sends to the                   calculates cost           forwards                        enters results in                Administrator
          mowing                         per acre                  performance                     performance                      enters
          contractor                     mowed for                 measure                         measurement database             measurement
                                         contractor                results to                                                       data results into
                                                                   Department's                                                     a different
                                                                   Performance                                                      tracking
                                                                   Measures                                                         database
                                                                   Administrator



                                      *Contractor has one week in which to begin mowing
                                     **Minimum of 25 acres per day must be cut
                                    ***For the purpose of bidding out mowing services, the City's parkland is divided into ten zones
                                   ****Paperwork includes payment documents, inspection checklist, Parks and Recreations purchase
                                       agreement, submitted for payment document, grounds maintenance work order




D:\Docstoc\Working\pdf\28f94ddb-1a8c-47f9-9c49-769f79c57d74.doc                                                                                               8