Performance Accountable Measures Measures by pou19125

VIEWS: 20 PAGES: 33

More Info
									   ORLANDO POLICE DEPARTMENT

 PERFORMANCE MEASURES REVIEW

Exit Conference Date:      November 11, 2003

Release Date:                January 16, 2004

           Report No. 04 – 01M




           CITY OF ORLANDO
      OFFICE OF AUDIT AND EVALUATION



        Beryl H Davis, CPA, CGFM
       Audit and Evaluation Director

         George J. McGowan, CPA
       Audit and Evaluation Manager
                                                    TABLE OF CONTENTS
Memorandum..................................................................................................................................1
Executive Summary ....................................................................................................................... 2
Background .................................................................................................................................... 9
Summary of Recommendations and Responses ..........................................................................10
Performance Measure Certification Results.................................................................................12
Issues and Recommendations ......................................................................................................14
   Develop Measures for Managing Operations and Hold Employees Accountable ................... 15
   Establish Smaller Number of Measures that Demonstrate Department Outcomes................ 17
   Include the Public in the Determination of Performance Measures and Reported Results....19
   Document and Review Performance Measures Prior to Reporting ........................................ 20
   Define Key Information Used in Calculating Performance Measures .................................... 22
   Consider New Measures And Benchmark Measures Whenever Possible............................... 23
   Identify Acceptable Methodology for Tracking Cases Investigated and Cases Cleared...........27
Appendix A - OPD Performance Measures – FY 2003-04 ......................................................... 29
Appendix B – Auditee Response Memorandum ..........................................................................31




   Orlando Police Department Performance Measures Review                                                                               i
MEMORANDUM

To:           Michael J. McCoy, Police Chief

From:         Beryl H. Davis, CPA, CGFM, Audit and Evaluation Director

Date:         Exit Conference: November 11, 2003; Report Issued: January 16, 2004

Subject:      Orlando Police Department Performance Measures Review
              (Audit Report No. 04-01M)

In accordance with our annual operations plan, the Office of Audit and Evaluation has
performed a review of the Performance Measures reported by the Orlando Police Department
(OPD). The performance measures are compiled and calculated by staff members throughout
the department and reported each quarter to the Performance Optimization Program (POP).
These measures are also reported annually in the City’s Budget Book. Our review was
designed to determine whether the performance measures of OPD are linked to the City’s and
department’s mission statements; how the measures are used by OPD to manage the police
operation and ensure individual accountability; and whether controls exist over the data
collection methods to ensure consistent and accurate reporting of the performance measures.

Our examination was conducted in accordance with generally accepted government auditing
standards, except for that we did not perform tests of data gathered from computer-based
resources. Our procedures included a review of documents, discussions with OPD personnel,
testing of performance measures data, and such other auditing steps as we considered
necessary. Our review revealed that some of the source information that is compiled and
reported by OPD resides on the computer applications and databases used by the department.
We did not perform tests of the integrity of the software that accumulates this information.

Our review revealed that OPD has devised acceptable methodologies for calculating and
compiling the current performance information it has been asked to report. We have
summarized these conclusions into “Performance Measurement Certification Results” for the
measures selected for review. We have also identified several recommendations that
challenge OPD to lead an effort to move the performance information it has been guided to
report by the POP Program toward meaningful measures that can be used in managing
department operations and ensuring employee accountability.

We would like to express our appreciation to the employees of the Orlando Police Department
for their courtesy and cooperation during this review.

BHD/am
c:      Honorable Buddy Dyer, Mayor
        Wayne Rich, City Attorney
        Joe Robinson, Deputy Chief of Staff
        Robert R. Garner, Management, Budget and Accounting Director
        Linda M. Jennings, General Administration Department Deputy Director
        Paul Rooney, Police Captain
                               EXECUTIVE SUMMARY

       Objective,      Our review of the performance measures used by the Orlando
      Scope, and       Police Department (OPD) was designed to determine whether
     Methodology
            Page 14
                       such measures are complete, tied to the mission statements of
                       the department, and used to manage the operations and hold
                       employees accountable. We reviewed the methodology used by
                       the department to collect, calculate and report performance
                       measurement, interviewed key staff and tested the accuracy of
                       reported performance results. Our detailed review was limited
                       to data reported for the quarter-ended June 30, 2003.

                       We performed a certification process on the performance
                       measures to conclude whether the information reported is
                       accurate when traced to supporting data. As a result, we report
                       that of the 9 performance measures selected for review, 5 are
                       Certified, 3 are Certified with Qualifications, and in the case of
                       one measure, Factors Prevented Certification. These results are
                       more fully explained in the following sections.


       Conclusion      OPD has effective systems for reporting the performance
                       measure information it is required to report to the Performance
                       Optimization Program (POP). The performance information
                       gathered by the department primarily consists of measures
                       developed under POP’s guidelines and are not generally
                       accepted as valuable information by the upper level managers of
                       OPD, as further explained below.




Permitting Field Inspections                                                            2
        Develop      We learned that the current performance measures reported by
   Measures for      OPD are not used by police managers to either manage the
       Managing
 Operations and      operation or hold employees accountable for performance. The
 Hold Employees      performance measures are separate from the everyday
    Accountable
                     operations of the department and carry no meaning for those
           Page 15
                     charged with managing the department. We believe that the
                     ideal performance management system is interconnected with
                     the management of the operation being measured. That is, the
                     performance measures are meaningful, linked to the
                     department’s mission, of interest to stakeholders, and are
                     utilized by departmental managers to improve operations and
                     establish accountability for department performance.

                     At present, the efforts made to gather, calculate and report
                     performance results are not fully benefiting either the
                     department or City government. This is due to a “disconnect”
                     between the philosophy of the City’s performance management
                     system and the management culture of OPD. We found that the
                     policy makers at OPD had limited input into the development of
                     the performance measures currently being reported. We believe
                     that this disconnect can be corrected if the POP Program can
                     allow increased involvement by OPD management in the
                     development of performance measures. In order to get “buy in”
                     from OPD managers, a collaborative effort is needed to agree to
                     measures that provide information useful to managing and
                     controlling police operations, while also ensuring individual
                     accountability.

                     Our research into the use of performance measures by peer
                     governments shows that these governments report performance
                     information that highlights the outcomes of quality police
                     services and demonstrates whether service efforts are



Orlando Police Department Performance Measures Review                                3
                     accomplishing the expected results. We recommend that OPD
                     review its current performance measures with POP to ensure
                     the measures are relevant to the department’s mission, goals
                     and expected results. If necessary, we recommend that OPD
                     develop new performance measures that are meaningful to
                     management and useful to the evaluation of police services. In
                     order to fully integrate these measures, it is important to
                     establish accountability so we recommend that the department
                     incorporate specific performance measures into the goals of its
                     upper managers (Lieutenant, Captain, Deputy Chief). This can
                     be accomplished through using the “optional factors” in the
                     City’s employee evaluation system or through other internal
                     methods used to ensure employee accountability.


         Establish   OPD currently reports 57 performance measures, as listed in
 Smaller Number      Appendix A. The department assigns four “coordinators” to
  of Performance
   Measures that     gather, calculate and report these measures each quarter and
     Demonstrate     also update the Business Plans prepared for the department.
      Department
       Outcomes      We obtained information on police performance measures from
           Page 17
                     several local governments and noted a wide variation in the
                     number of measures used by these governments. We found that
                     the number of performance measures utilized by these
                     governments varies from five to 145 police performance
                     measures, with a median number of 20 measures. Our research
                     also showed that one Florida peer city, St. Petersburg, reports
                     on just eleven critical or significant performance factors. In light
                     of the amount of effort required to accumulate data, calculate
                     and report performance information, and given evidence that
                     many peer governments report fewer measures, we recommend
                     that OPD seek the approval of the POP Program to review its



Orlando Police Department Performance Measures Review                                  4
                     measures and determine if they can be reduced to a more
                     manageable and meaningful number.


     Include the     While researching how other governments use their
    Public in the    performance measures, we learned that Kansas City, Missouri,
Determination of
    Performance      organized “focus groups” of citizens and small business owners
   Measures and      to get a better idea of what these groups expect from their police
Reported Results
                     services. We believe that the inclusion of stakeholders in the
           Page 19
                     performance management system is a worthwhile aspect of
                     learning what is meaningful to them. We believe that since OPD
                     already has a firm track record of including the public in its
                     policy-making, it can be a leader by including the public in the
                     determination of performance measures and reporting its
                     performance to interested citizens’ groups. We recommend that
                     OPD continue to seek public input to determine its performance
                     measures and what performance results are reported to
                     interested citizens’ groups.


  Document and       During our review of selected performance measure
         Review      information reported to POP, we inquired whether the
    Performance
Measures Prior to    methodology and procedures used by the coordinators in
      Reporting      gathering, calculating and reporting periodic performance data
           Page 20
                     are documented in a reference guide or other similar document.
                     We were informed that no such guidelines have been prepared
                     to assist the coordinators to consistently gather the information
                     necessary to report performance data each quarter.

                     We also inquired whether the coordinators have the
                     performance data reviewed by a supervisor or peer prior to
                     transmitting the information to the POP Program. We were




Orlando Police Department Performance Measures Review                                 5
                     informed that this review process is not performed.

                     While reviewing the nine selected performance measures we
                     found two instances where the information reported was not
                     completely supported by the backup documentation retained by
                     the coordinators. While the impact of these cases was minor, it
                     demonstrates the importance of having a documented
                     methodology for gathering, calculating and reporting
                     performance data and an associated review process to ensure
                     the accuracy of reported performance information. For this
                     reason we concluded that these two measures are “Certified
                     with Qualifications.” We recommend that OPD establish a
                     committee of performance measure coordinators and that this
                     committee be tasked with preparing a short guideline document
                     that lists the steps necessary to accurately calculate each
                     performance measure. We also recommend that the
                     coordinators have a supervisor review performance measure
                     information prior to it being released to management or others.
                     We recognize, however, from an efficiency standpoint, that
                     these recommendations may make more sense if the number of
                     performance measures used by the department is decreased to a
                     more manageable and meaningful number.


       Define Key    While reviewing detailed information on performance measures
Information Used     reported by OPD, we found an inconsistency in the
    in Calculating
     Performance     determination of the cost of the sections under review. This was
        Measures     due to the fact that one section used the midpoint of the salary
           Page 22
                     range to estimate labor costs while another section used actual
                     salaries to calculate labor costs.

                     While this may not materially affect the performance results of
                     the department, it would improve the accuracy of the data


Orlando Police Department Performance Measures Review                               6
                     reported if consistent methods were used in the calculations.
                     We recommend that OPD ask the POP Program to recommend
                     a consistent method for determining section costs.


   Consider New      We performed research into the performance measures used by
   Measures And      other local governments and found some interesting alternative
     Benchmark
      Measures       performance measures that could be meaningful and useful to
      Whenever       OPD. These measures are listed in this report and we believe
        Possible
                     that they are worth considering as they can be benchmarked to
           Page 23
                     these governments. We feel that benchmarking OPD
                     performance to a peer group will provide information to OPD
                     management that can be beneficial to institute performance
                     improvements. We acknowledge that benchmarking of this sort
                     must be done carefully as police agencies have different
                     methods, practices and services; different service areas; and
                     different customers (residents and visitors). Nevertheless, we
                     believe that benchmarking whenever possible, both internally
                     and to an appropriate peer group, can be an informative
                     addition to the performance management system.

                     We reviewed the measures currently used by OPD and noted
                     that OPD does not have many measures that track “time
                     effiencies.” We believe that the timely delivery of services may
                     be as important as the economical cost measures currently used
                     by the department. We found some interesting examples of
                     these performance measures and have listed them in this report.
                     Similarly, we found that the department could add “customer
                     satisfaction” measures to have a regular feedback mechanism
                     for the services it provides to the public.

                     We encourage OPD to devise measures to track the areas of time
                     efficiency, customer satisfaction and other important


Orlando Police Department Performance Measures Review                                   7
                     performance aspects utilized by other local governments, such
                     as those listed in the body of this report. We also encourage
                     OPD to strive to identify performance measures that can be
                     benchmarked to those of peer governments.


         Identify    One of the performance measures selected for detailed review
      Acceptable     concerns the percentage of criminal cases investigated and
 Methodology for
  Tracking Cases     cleared. We traced the reported performance information to the
Investigated and     Police Records Management System (RMS) computer database
         Cleared
                     to determine the accuracy of the information. We learned that
           Page 27
                     the system utilized to gather information for this measure
                     included having investigators manually report case information.
                     Our tests of this information showed that the information
                     gathered manually differed from the case information in the
                     RMS system. For this reason, we concluded that “Factors
                     Prevented Certification” of this measure.

                     The coordinator over this section informed us that he plans to
                     initiate a discussion among police managers regarding the
                     proper methodology to report the clearance rate for cases
                     investigated. We suggest that such an effort include a
                     representative of the POP Program to offer guidance and
                     assistance. We also believe that any discussion in this area
                     should address how best to utilize the RMS system to create
                     reports of performance data rather than have investigators
                     manually report this information.




Orlando Police Department Performance Measures Review                                 8
                                       BACKGROUND

The City of Orlando has a history of gathering and publishing performance information on
each of its departments. In the past, City departments and offices were required to identify
several performance measures to demonstrate their service efforts and accomplishments for
each year. The actual results for each of these performance measures were reported on a
quarterly basis to the former Office of Management and Budget and were summarized for
executive management’s use. Each year, the performance measures were published in the
City’s Annual Budget document.

In the fall of 2002, the City created a Performance Optimization Program (POP) to guide and
manage the process of performance management in City government. This program requires
the reporting of performance data through the creation of automated annual Business Plans.
Each department and office is now required to develop a Business Plan that outlines the
area’s mission, strategy and approach for accomplishing this mission. These Business Plans
are prepared at the Bureau or Division level and include detailed goals, objectives and
performance measures for each subsection of City government. During the first phase of this
effort, departments and offices are required to report performance results for each quarter to
POP by posting these results on spreadsheets available in a common directory on the City’s
computer network. This year, these quarterly reports will be accomplished using Lotus Notes.
It should be noted that the above steps and other policies governing the performance
management system have not been formalized in City Policies and Procedures.

For FY 03-04, the Orlando Police Department (OPD), in collaboration with POP, established
Business Plans and 57 associated performance measures for four areas: Professional
Standards Division (17 measures), Police Administrative Services Bureau (16), Police Patrol
Services Bureau (6), and Police Special Services Bureau (18). See Appendix A. This
collaborative process included an effort by POP staff to refine the department’s performance
measures from over 100 measures to the present 57.

OPD has four “coordinators” who are assigned to be the primary individuals responsible for
gathering, calculating and reporting quarterly and annual performance measure results.
These coordinators gather information from the various sections of the department and
report this data on behalf of these sections. The coordinators for the department include the
Planning Supervisor, Police Communications Manager, a Patrol Lieutenant and an
Investigations Lieutenant. As staff is promoted or transferred, new coordinators may be
named. During this review, one coordinator transferred out of the Patrol Bureau and moved
responsibility for performance measures to his replacement. Another coordinator planned to
transfer coordination to a fellow Lieutenant in the Special Services Bureau.

Orlando Police Department Performance Measures Review                                 9
             SUMMARY OF RECOMMENDATIONS AND RESPONSES

                             RECOMMENDATIONS                                RESPONSES
1.   The Orlando Police Department should review its current                  Concur
     performance measures with the Performance Optimization Program
     to ensure that these measures are meaningful to management and
     most useful in managing department operations or develop new
     performance measures that better serve the department’s mission.

2.   The Orlando Police Department should incorporate specific                Do Not
     department performance measures into the goals of its upper              Concur
     managers, through using the “optional factors” in the City’s
     employee evaluation system or through other internal methods used
     to ensure employee accountability.

3.   The Orlando Police Department should seek the approval of the            Concur
     Performance Optimization Program to review its current
     performance measures and determine if they can be reduced to a
     more manageable and meaningful number.

4.   The Orlando Police Department should continue to seek public input     Concur with
     and use this input to determine its performance measures and what      Reservations
     performance results are reported to interested citizens’ groups.

5.   The Orlando Police Department should establish a committee to          Concur with
     include the performance measurement coordinators and this              reservations
     committee should be instructed to prepare a short guideline
     document for each section that lists the steps needed to compile and
     calculate the performance measures.

6.   The Orlando Police Department performance measurement                  Concur with
     coordinators should have a supervisor review the compilation and       Reservations
     calculation of performance measure data prior to this information
     being transmitted to Orlando Police Department management, the
     Performance Optimization Program staff, and others outside the
     department.

     See also Appendix B - Auditee Response Memorandum



Orlando Police Department Performance Measures Review                               10
                             RECOMMENDATIONS                               RESPONSES
7.   The Orlando Police Department performance measurement                   Do Not
     coordinators should ask the Performance Optimization Program to         Concur
     recommend a consistent method for determining section costs for
     use in calculating performance measures.

8.   The Orlando Police Department should consider adding “customer        Concur with
     satisfaction” measures and “time efficiency” measures, as noted in    Reservations
     the report examples.

9.   The Orlando Police Department should continue its efforts to          Concur with
     benchmark its performance to peer governments and extend these        Reservations
     efforts by adopting performance measures used by peer
     governments, such as those listed in the report examples.

10. The Orlando Police Department’s performance measures                     Concur
    coordinator for the Police Special Service Bureau should meet with
    colleagues to discuss: 1) alternatives to the present method of
    compiling data to report a “clearance rate” for investigative cases,
    and 2) resolving the differences between the case information
    entered in the computerized Records Management System and the
    data manually reported by the investigators.

     See also Appendix B - Auditee Response Memorandum




Orlando Police Department Performance Measures Review                               11
                 PERFORMANCE MEASURE CERTIFICATION RESULTS

                                                                   Factors
Measure Selected for          Certified      Certified with       Prevented        Inaccurate
Review                                       Qualifications      Certification


Budgeted Positions per          XXX
1,000 Service Population
(Day Population)


Priority 1 Police Calls per     XXX
1,000 Residents


Monthly Attrition of Sworn                        XXX
Officers


Percent of 911 Calls                              XXX
Answered Within 10
Seconds


Average Queue Time of           XXX
Code 2 Calls (in Minutes)


Aggregate Case Clearance                                             XXX
Rate


Average Response Time for       XXX
Code 2 Calls


Average Response Time for       XXX
Code 3 Calls


   See next page for definitions of terms and method used to select measures for review.




   Orlando Police Department Performance Measures Review                              12
Performance Measure Certification Notes

Measures were selected for review as follows:

All OPD performance measures were reviewed and the Office of Audit and Evaluation
made a judgment selection of the more important measures. We focused on those
measures we perceived were of interest to the public and considered as “outcome”
measures. The information reported by OPD to POP for the third quarter of FY 2002-03
was selected for our detailed review for two reasons. First, it was the most recent period
of information reported and second, the procedures followed by OPD performance
measure coordinators were the procedures currently being used to gather performance
measure information.

Certification categories are defined as follows:

A measure is considered Certified if reported performance information is accurate and
it appears that controls are in place to ensure accuracy in collecting and reporting
performance data.

A measure is considered Certified with Qualifications when reported performance
information is materially accurate but testing showed immaterial differences between
reported information and information verified through audit testing. In these cases,
controls are in place to ensure future accuracy for collecting and reporting performance
data.

Factors Prevented Certification occurs when reported performance information is
unable to be efficiently tested due to inaccuracies in the system used to calculate the
information. For example, inaccuracies between the backup information retained by
those reporting the information and information obtained from independent sources.

A measure is Inaccurate when the reported performance information is materially
different from the actual performance as calculated during the review.




Orlando Police Department Performance Measures Review                                  13
                     ISSUES AND RECOMMENDATIONS

       Objectives    Our objectives were to determine whether the performance
                     measures used by the Orlando Police Department (OPD) are
                     complete, are tied to the mission statements of the department,
                     are used to manage the operations, and hold employees
                     accountable. We also reviewed the methodology used by the
                     department to collect, calculate and report performance
                     measurement data to ensure that it is consistent and provides
                     accurate information.

            Scope    We reviewed Business Plans, performance measures and
                     associated documents for FYs 2002-03 and 2003-04. We
                     limited our detailed reviews of performance measures to
                     selected results reported to the Performance Optimization
                     Program (POP) for the quarter-ended June 30, 2003.

    Methodology      We interviewed the OPD performance management
                     coordinators for each section that reports performance
                     information. We reviewed detailed information compiled by
                     OPD staff and available from OPD computer applications. We
                     tested the accuracy of this information by tracing selected data
                     to source documents. We interviewed OPD management to gain
                     insights on the use of performance data in the management
                     process.




Orlando Police Department Performance Measures Review                             14
        Develop      During our review, we discussed the current performance
   Measures for      measures and the performance management reporting system
       Managing      with the Operations Committee of OPD. The Operations
 Operations and
                     Committee is made up of the department’s Police Captains. We
 Hold Employees
    Accountable      learned from this group that police management does not find
                     the information reported as performance measures as
                     meaningful, useful or necessary to their effective management
                     of the department. Similarly, these managers did not believe
                     that individuals should be held accountable for performance
                     using the current performance measures.

                     We believe that the ideal performance management system
                     accomplishes the following: gathers meaningful performance
                     information that is linked to the mission of the department;
                     reports information that is of interest to the stakeholders in the
                     department (i.e., the public, the elected officials, etc.); reports
                     information that is utilized by the department’s managers to
                     manage and improve operations; and establishes accountability
                     over performance to the appropriate individuals, managers or
                     operating sections of the department.

                     We discussed the importance of these issues with various
                     members of OPD management and observed that the
                     performance measures currently reported to POP and published
                     in the City’s Annual Budget are not utilized by the managers of
                     the Orlando Police Department. That is, the managers have not
                     embraced the performance management information system as
                     it is currently designed, nor the performance measures
                     determined through police staff’s interaction with POP. Rather,
                     OPD managers utilize other management information and
                     performance data to guide the police services operation. Our
                     observation is that the reporting of performance information to
                     POP stands outside the regular operations of the department
                     and is not integrated into the operations of the department.

                     The effect of this situation is that the efforts of the individuals
                     tasked with performing the duties of performance measures
                     “coordinators” may not be beneficial and do not add to the


Orlando Police Department Performance Measures Review                                  15
                     success of the department. One coordinator commented that he
                     wanted to get back to “police work” rather than continue to
                     perform the duties of the coordinator. This indicates that the
                     work performed by these coordinators may not be a “value-
                     added” activity.

                     Cities throughout the United States have embraced the
                     reporting of performance information. Our research shows that
                     many peer governments report performance information that
                     highlights the outcomes of the police services and demonstrates
                     that service efforts are accomplishing the expected results. We
                     encourage OPD to review its current measures with POP for
                     their relevance to the department’s mission, goals and expected
                     results. We acknowledge the development of performance
                     measures is not entirely under the control of OPD. Therefore,
                     we suggest that OPD management work in collaboration with
                     POP to ensure that performance measures are meaningful to
                     management and utilized to improve the department as a whole
                     and encourage improvements in individual performance. If
                     necessary, we recommend that OPD develop new performance
                     measures that better serve the department’s mission.

                     We understand that the Police Chief is beginning an effort to
                     update the employee evaluation policies and procedures used by
                     the department. As more meaningful and useful performance
                     measures are utilized by the department, we believe that OPD
                     should incorporate the accomplishment of specific performance
                     measures into the goals of the upper managers (Lieutenant,
                     Captain, Deputy Chief). This can be done through the use of
                     “optional factors” in the City’s employee evaluation system, or
                     through other internal methods.

 Recommendation 1.   We recommend that the Orlando Police Department review its
                     current performance measures with the Performance
                     Optimization Program to ensure that these measures are
                     meaningful to management and most useful in managing
                     department operations or develop new performance measures
                     that better serve the department’s mission.



Orlando Police Department Performance Measures Review                            16
          Response   We concur. Re-evaluating our performance measures will result
                     in accomplishing an increased involvement by upper
                     management.

 Recommendation 2.   We recommend that the Orlando Police Department
                     incorporate specific department performance measures into the
                     goals of its upper managers, through using the “optional
                     factors” in the City’s employee evaluation system or through
                     other internal methods used to ensure employee accountability.

          Response   We do not concur due to most of the current performance
                     targets being beyond our control, such as average cost to
                     complete formal investigations, percent reduction in vehicle
                     accidents from previous year totals, targeted numbers of violent
                     felons arrested for the year, and average cost to answer a call for
                     service by a uniform patrol unit.


Establish Smaller    We reviewed the number of performance measures currently
       Number of     reported by OPD. We found that performance information will
   Measures that     be reported for 57 measures in FY 2003-04 (see Appendix A).
    Demonstrate
                     This includes 17 measures for the OPD Professional Standards
     Department
       Outcomes      Division (includes Planning and Evaluation Section, Internal
                     Affairs Section and Fiscal Management Section), 16 measures
                     for the Police Administrative Services Bureau (includes Support
                     Services Division, Communications Division, Recruiting Unit,
                     and Special Operations Division), 6 measures for the Police
                     Patrol Services Bureau (includes East, North and West Patrol
                     Divisions and Training Section) and 18 measures for the Police
                     Special Services Bureau (includes Criminal Investigations
                     Division, Drug Enforcement Division, Community Oriented
                     Policing Division, International Airport Division and Homeland
                     Security Section). See Appendix A for a full listing of current
                     performance measures.

                     The department assigns four individuals to be the designated
                     “coordinators” responsible for gathering, calculating and
                     reporting performance measure information reported to the
                     POP program each quarter. These coordinators may also be
                     consulted for updates to the four Business Plans established for

Orlando Police Department Performance Measures Review                               17
                     the department for each of the areas identified above. The
                     coordinators perform their performance measure reporting
                     duties above and beyond their normal duties. While we did not
                     determine exactly how many hours each of these coordinators
                     dedicate to the gathering, calculation and reporting of
                     performance measures each quarter, one might assume that it
                     could be significant to gather information each quarter on
                     performance results for 57 different performance indicators.

                     We obtained information on police performance measures from
                     several local governments and found that the number of
                     performance measures utilized by these governments varies
                     from five to 145 police performance measures. The median
                     number of measures reported was 20. Our research also showed
                     that one Florida peer city, St. Petersburg, reports on just eleven
                     critical or significant performance factors. This indicates that
                     the number of measures reported by OPD may be high. The
                     high number may contribute to the fact that OPD performance
                     measures are not currently utilized in the management of the
                     police operations.

                     In addition to evaluating the measures for meaningfulness and
                     usefulness, we believe that OPD should review the current
                     measures with an idea to decrease the total number. This would
                     make the information gathered more manageable and free the
                     coordinators to perform their regular assigned duties during the
                     time currently taken to gather information on the current 57
                     measures.

 Recommendation 3.   We recommend that the Orlando Police Department seek the
                     approval of the Performance Optimization Program to review its
                     current performance measures and determine if they can be
                     reduced to a more manageable and meaningful number.

          Response   We concur. As stated in our auditee response memorandum
                     (Appendix B), we have 57 performance measures that could be
                     drastically reduced or redefined to approximately ten.




Orlando Police Department Performance Measures Review                              18
     Include the     During our review, we researched what other governments are
    Public in the    doing with their performance management system, as they
Determination of     related to police services. We found an interesting practice of
    Performance
                     the City of Kansas City, Missouri. Kansas City representatives
   Measures and
Reported Results     organized “focus groups” of citizens and small business owners
                     to get a better idea of what services citizens expect from their
                     police department and what they want to know about their
                     department. A semi-structured format directed discussion to
                     several pre-selected issues and after this discussion a
                     questionnaire was given to participants to gather their opinions
                     on specific types of performance measures that might be used to
                     monitor police performance.

                     We believe that the inclusion of stakeholders, including citizens’
                     groups, is a very important element of a performance
                     management system. As of now, the City does not currently
                     emphasize this aspect of the performance management system.
                     However, we feel that OPD is one of the functions of City
                     government that directly interacts with the public every day and
                     yet, that aspect of the department’s performance is not included
                     in the current performance measures of the department. OPD
                     has a good rapport with the citizens and currently solicits input
                     from the public on important matters such as the Use of Force
                     Policy and the Pursuit Policy and regularly reports the
                     investigation of formal citizen complaints to the Citizen’s Police
                     Review Board. In addition, OPD has a successful Citizen Police
                     Academy program.

                     While we address the specific measures that OPD may want to
                     consider to track its effectiveness in serving the public in a
                     future section of this report, we want to encourage OPD to be a
                     leader by including the public in the determination of
                     performance measures to be tracked and reported by the
                     department. Additionally, we feel that OPD can lead the way in
                     reporting performance results directly to citizens. While the
                     information currently reported by OPD coordinators eventually
                     is published in the Annual Budget of the City, which is available
                     on the Internet and distributed to interested parties, we feel


Orlando Police Department Performance Measures Review                              19
                     that more timely performance information could be published
                     by OPD and the other City departments either electronically or
                     in published updates to the citizens.

 Recommendation 4.   We recommend that the Orlando Police Department continue to
                     seek public input and use this input to determine its
                     performance measures and what performance results are
                     reported to interested citizens’ groups.

          Response   We concur, with reservations, if the performance measures were
                     reduced to a more manageable and meaningful number that
                     measured efficiency and effectiveness.


  Document and       During our review, we examined the detailed information
         Review      gathered by the coordinators to calculate and report data for
    Performance      several selected performance measures. During this review
Measures Prior to
                     process, we inquired of the coordinators whether the
      Reporting
                     methodology or procedures they follow to gather this
                     information is memorialized in a document that outlines the
                     steps necessary to compile and calculate the performance
                     measures. We were informed that no such guidelines have been
                     prepared to document the steps taken each quarter to gather
                     performance information for reporting.

                     We also inquired whether the information gathered and
                     calculations made each quarter are reviewed by a supervisor or
                     peer prior to the information being reported to the POP
                     program. We were informed that the OPD performance
                     measures “coordinators” either calculate the measures
                     themselves or simply gather information calculated by others.
                     We found that performance information is not reviewed by a
                     second individual for accuracy or completeness.

                     When we were reviewing the details of performance measure
                     calculations, we found two examples of the impact of not having
                     someone review performance measure calculations: “Monthly
                     Attrition of Sworn Officers” and “Percent of 911 Calls Answered
                     Within 10 Seconds.” While reviewing the measure regarding
                     attrition rate, we found that the backup documentation

Orlando Police Department Performance Measures Review                           20
                     prepared by the coordinator for this area resulted in a different
                     number than the number reported to POP on the quarterly
                     report. While reviewing the measure regarding answering 911
                     calls, we found that OPD Communications staff did not keep the
                     system reports and notes used to calculate the measure. While
                     staff recreated these reports at our request, they did not include
                     the detailed information necessary to properly recalculate the
                     measure reported to POP.

                     Although one could conclude that the difference in the reported
                     number and the number noted on the backup documentation
                     was immaterial, these instances indicate that a review process
                     would increase the accuracy of reported performance
                     information.

                     The use of an internal guide for preparing the information
                     would also alleviate this occurrence. The apparent reason for
                     not preparing an internal guide for preparing performance data
                     is the time necessary to prepare such a guide. The individuals
                     preparing performance measure information are performing
                     this task as a secondary duty of their position. Some
                     coordinators also opined that having a supervisor review the
                     performance measures gathered or calculated by the
                     coordinators is an inefficient use of this individual’s time. We
                     assert that as performance data becomes more meaningful to
                     citizens and other stakeholders, it will become increasingly
                     important for coordinators to both document the steps taken to
                     calculate performance data and seek assistance to validate
                     selected data to ensure accuracy and continuity. This is
                     especially true when coordinators and others providing
                     information to coordinators change duty assignments within
                     OPD as they are reassigned and/or promoted.

                     As a result of the above, OPD is reporting information that has
                     not been double-checked or reviewed by supervisory personnel
                     prior to its being reported for future publishing, benchmarking
                     and/or analysis. We suggest that a committee be established to
                     include the performance measurement coordinators. This
                     committee could meet to prepare internal guidelines for


Orlando Police Department Performance Measures Review                              21
                     preparing performance reports and thus help institute a policy
                     of having a supervisor review key performance data calculations
                     prior to reporting the information to members of the OPD
                     management team, POP and others outside the department. We
                     recognize, however, from an efficiency standpoint, that these
                     recommendations may make more sense if the number of
                     performance measures used by the department is decreased to a
                     more manageable and meaningful number.

 Recommendation 5.   We recommend that the Orlando Police Department establish a
                     committee to include the performance measurement
                     coordinators and that this committee be instructed to prepare a
                     short guideline document for each section that lists the steps
                     needed to compile and calculate the performance measures.

          Response   We concur, with reservations, if the performance measures were
                     reduced to a more manageable and meaningful number that
                     measured efficiency and effectiveness.

 Recommendation 6.   We recommend that the Orlando Police Department
                     performance measurement coordinators have a supervisor
                     review the compilation and calculation of performance measure
                     data prior to this information being transmitted to Orlando
                     Police Department management, the Performance Optimization
                     Program staff, and others outside the department.

          Response   We concur, with reservations, if the measures are reduced to a
                     more manageable and meaningful number. It will be the
                     responsibility of the Professional Standards Division
                     Commander to review the data prior to dissemination.


       Define Key    During our review of the detailed calculations made by the
Information Used     performance measures coordinators, we found an inconsistency
    in Calculating   in the determination of the costs of the sections under review.
     Performance
                     We found that the costs are calculated based on the midpoint of
        Measures
                     the salary range for the positions assigned to the sections under
                     review (Police Patrol Bureau) or are calculated using the actual
                     salaries of the individuals assigned to these sections (Police
                     Special Services Bureau). This cost information is used by the

Orlando Police Department Performance Measures Review                             22
                     Police Patrol Bureau to report the average cost to answer calls
                     for service and by the Police Special Services Bureau to report
                     the average cost to clear investigation cases.

                     This instance indicates a need for coordination within the
                     department regarding the methodology used to calculate key
                     information. In fact, it may be necessary for this coordination to
                     occur across City departments so that reported performance
                     measures are comparable both internally and against any
                     external benchmarks. The OPD coordinators should seek
                     guidance from POP regarding the proper way to calculate this
                     information so that they can be consistently applied, if such
                     measures continue to be used by the department.

 Recommendation 7.   We recommend that the Orlando Police Department
                     coordinators ask the Performance Optimization Program to
                     recommend a consistent method for determining section costs
                     (using actual salaries or midpoint salaries) for use in calculating
                     performance measures.

          Response   We do not concur, based on the fact that our current
                     performance measures are of little value.


   Consider New      We performed research to determine what other local
   Measures And      government police agencies are reporting for their performance
     Benchmark       measures. This was done to determine if any of these measures
      Measures
                     could be meaningful or useful to OPD and to find out whether
      Whenever
        Possible     any of these measures can be used as benchmarks. Benchmarks
                     provide information for OPD management to compare its
                     performance to peers and identify areas of concern or areas
                     where OPD is excelling.

                     We were informed by POP staff that the City is participating in
                     an International City/County Management Association (ICMA)
                     effort to publish comparative performance data for local
                     governments. We support and encourage this effort to identify
                     performance measures that are both meaningful and can be
                     benchmarked against our peers to help us gain a better



Orlando Police Department Performance Measures Review                               23
                     understanding of City performance.

                     We gathered police agency performance measures information
                     from colleagues in the National Association of Local
                     Government Auditors (N.A.L.G.A.) who responded to our
                     inquiry on this subject. We reviewed the materials gathered and
                     noticed several possible performance measures that we thought
                     deserved to be considered by OPD as possible additions or
                     replacements to the current performance measures reported.

                     We observed that the performance measures currently used by
                     OPD include several measures that track “cost efficiencies” (e.g.,
                     Cost per Service Call, etc.). During our discussions with OPD
                     management we learned that the managers do not believe that
                     the cost per service measures are measures that accurately
                     document the performance of their sections. They stated that
                     many factors could lead to a high cost per service. For example,
                     the calls for service responded to by the Patrol are primarily
                     generated by 911 calls, which can vary from period to period
                     without any predictable cause and then may cause the cost per
                     service call measure to vary from period to period without an
                     apparent reason. Similarly, the cost to perform investigations
                     can vary from one investigative discipline to another, or could
                     vary because a period could contain a small number of major
                     investigations or a large number of minor investigations.

                     We also found that OPD does not have many measures that
                     track “time efficiencies.” That is, there are only a limited
                     number of measures that demonstrate the department’s
                     timeliness in completing some important administrative duties.
                     We believe timely delivery measures may be as important as
                     economical cost measures. During our discussion of
                     performance measures of the OPD Records section, the
                     manager of the section stated that he believed that tracking his
                     section’s efforts to process subpoenas or traffic citations in a
                     timely manner was a more useful performance measure than
                     tracking the cost of these services. In addition, we found that
                     the current measures do not include those that track customer
                     satisfaction with police services. While this area may be


Orlando Police Department Performance Measures Review                              24
                     included in a future Mayor’s Report Card Survey, we feel it
                     would be beneficial to the department to have a regular
                     feedback mechanism and reporting of customer satisfaction
                     with the services provided by OPD.

                     For these reasons, we offer the following performance measures
                     as possible additions/substitutions for consideration by OPD:

                     “Customer Satisfaction” Measures (would be measured by
                     performing customer surveys or holding focus groups):

                     Overall Customer Satisfaction

                     Adequacy of Perceived Patrol Presence

                     Percentage of Crime Victims Reporting Satisfaction with
                     Services

                     “Time Efficiency” Measures:

                     Percentage of Media Calls Responded to Within an Established
                     Time Frame

                     Percentage of Subpoenas Processed Within 72 Hours

                     Percentage of Traffic Citations Processed Within 72 Hours

                     Other Measures Used by Peer Governments:

                     Number of Traffic Fatalities Per 100,000 Population

                     Value of Property Stolen / Recovered

                     Percentage Below National Crime Rate

                     Percentage Above National Case Clearance Rate

                     Crime Ranking With Cities Over 100,000 Population

                     Percentage Compliance With Accreditation Standards

                     Percentage of Calls for Service Handled Without Requiring a
                     Police Response


Orlando Police Department Performance Measures Review                              25
                     While other local governments have reported information using
                     these specific measures, we encourage OPD to devise its own
                     meaningful measures to track the areas of customer
                     satisfaction, time efficiency and other important performance
                     aspects. We were informed that that the identification of
                     customer satisfaction measures may become more important
                     because police accreditation agencies are discussing a
                     requirement for regular customer surveys as a standard for
                     receiving future accreditation.

                     We also believe that some of the current performance
                     information reported by OPD can be benchmarked against a
                     peer group. We understand that OPD has participated in the
                     effort by the POP program to report performance information to
                     the ICMA and applaud this initial effort to benchmark its
                     performance. We also encourage OPD to strive to identify other
                     performance measures that can be used to benchmark its
                     performance in the context of peer government performance.

 Recommendation 8.   We recommend that the Orlando Police Department consider
                     adding “customer satisfaction” measures and “time efficiency”
                     measures, as noted in the report examples.

          Response   We concur with reservations. The current performance
                     measures have little or no value to the community that we serve.

 Recommendation 9.   We recommend that the Orlando Police Department continue
                     its efforts to benchmark its performance to peer governments
                     and extend these efforts by adopting performance measures
                     used by peer governments, such as those listed in the report
                     examples.

          Response   We concur with reservations. As long as the benchmark
                     performance measures are directly related to the effective and
                     efficient operation of the police department and the community
                     served, it would be beneficial to share our performance
                     information to peer governments.




Orlando Police Department Performance Measures Review                            26
         Identify    We reviewed the methodology used to compile the data
      Acceptable     reported by the Police Special Services Bureau for criminal
 Methodology for
  Tracking Cases     cases investigated and cleared. Due to the complexity of the
Investigated and     information, we selected one criminal investigation section and
   Cases Cleared
                     one month of information to trace to the Police Records
                     Management System (RMS) computer database that is used to
                     keep all of the data generated for criminal investigation cases.

                     At present, the method utilized to gather information on case
                     clearance rates by the coordinator for this section requires each
                     investigator to compile their statistics onto hand written logs
                     that are presented to the sergeant assigned to one of several
                     investigative subdivisions (sex crimes, crimes against children,
                     homicide, economic crimes, robbery, assault and battery,
                     property, and auto theft). These sergeants summarize the data
                     for the lieutenant tasked with reporting the compiled
                     performance information. In order to test the accuracy of these
                     reports, we attempted to trace this compiled information to the
                     RMS computer records kept for investigative cases and found
                     some discrepancies.

                     As a result, the information reported for cases cleared may be
                     accurate but is not supported by the data contained in the RMS
                     for the period. It may be possible that the information reported
                     by each investigator on the hand-written logs is the
                     investigator’s best recollection of current caseload and
                     disposition of these cases. If this is the case, this information
                     should be reflected in the RMS system, as it is the management
                     system used by sergeants and managers to review current case
                     status online.

                     While reviewing this area, we learned from the coordinator for
                     the Police Special Services Bureau that he plans on initiating a
                     discussion among the division’s managers to update the
                     methodology used to determine the “clearance rate” for criminal
                     investigation cases. This is due to several factors, including that
                     the section has decided to report a clearance rate for the entire
                     section rather than individual clearance rates for the many
                     investigative subdivisions in this section and also because the

Orlando Police Department Performance Measures Review                               27
                     current method of gathering information is so labor-intensive.

                     The performance measures selected for the investigations
                     section should be designed to provide management with
                     meaningful and useful information to assist in the determining
                     the effectiveness and efficiency of the unit being measured. We
                     believe that an effort to review measures to ensure they are
                     accurate and meaningful is appropriate. We suggest that any
                     discussion of this area should include how to best utilize the
                     RMS system to create reports of performance data than can be
                     used to replace the cumbersome system of having each
                     investigator manually report his or her caseload and cases
                     cleared each month.

Recommendation 10.   We recommend that the Orlando Police Department’s
                     performance measures coordinator for the Police Special
                     Service Bureau meet with colleagues to discuss: 1) alternatives
                     to the present method of compiling data to report a “clearance
                     rate” for investigative cases, and 2) resolving the differences
                     between the case information entered in the computerized
                     Records Management System and the data manually reported
                     by the investigators.

          Response   We concur. The Criminal Investigations Division Commander
                     did review the recommendation and will address the
                     recommendation with his section commanders, and attempt to
                     clarify the discrepancies by March 2004.




Orlando Police Department Performance Measures Review                             28
     APPENDIX A - OPD PERFORMANCE MEASURES – FY 2003-04

Police Administration (Professional Standards Division) – 17 measures

Sworn Officers Per 1,000 Service Population (Day Population)
Budgeted Positions per 1,000 Service Population (Day Population)
Sworn Officers Per 1,000 Residents
Sworn and Civilian FTEs per 1,000 Residents
Priority 1 Police Calls per 1,000 Residents (includes Airport calls)
Percentage of Formal Investigations completed in 45 calendar days or less
New grant dollars received
Average Cost to complete Formal Investigations
Average Cost to complete Informal Investigations
Average Cost to prepare, update and publish a Crime Bulletin
Average Cost to publish a Criminal Information/ Intelligence Bulletin
Average Cost to process policy review requests
Average Cost to review a TRC/MPB case
Average Cost for Receiving/Returning Media related telephone calls
Average Cost per Media Interview
Average Cost per Case Submitted to the State Attorney Office
Total Operating and Maintenance Expenditures Per 1,000 Residents

Police Administrative Services Bureau – 16 measures

Percent of 911 calls answered within 10 seconds
Average Code 2 Call Queue Time in Minutes
Attrition Rate of Sworn Officers
Percent Increase in K-9 Activity over previous year total
Average Decreased Cost per Unit of K-9 Activity from previous year
Percent Increase in Mounted Patrol activity over previous year total
Percent Increase in Traffic Enforcement activity over previous year total
Percent reduction in vehicle accidents from previous year total
Percent hit and run cases cleared each year
Percent increase of taxi meters checked, vehicles inspected, citations and Notices of
   Violation issued over previous year
Average Cost to Fingerprint a Citizen
Average Cost to Process a Subpoena
Average Cost per Sworn Position Recruited
Average Cost per unit of Mounted Patrol activity
Cost per unit of Mounted Patrol Activity from previous year
Percent of revenue increase over previous year (Vehicles for Hire)




Orlando Police Department Performance Measures Review                                   29
Police Special Services Bureau – 18 measures

Percent of Cases Cleared
Targeted Number of Violent Felons Arrested for the Year
Percent of plausible complaints received through the crack line investigated
Percent Change in the number of prostitution arrests
Percentage of Code 2 Calls responded to within 10 minutes (Airport)
Percent of all cases assigned for investigation solved (Airport)
Percent increase in the number of students taught the GREAT curriculum
Percent Parramore Heritage activity unit level maintained over previous year
Percent Neighborhood Team activity unit level maintained over previous year
Percent increase in the number of Citizen Police Academies conducted
Percent increase in the number of volunteers
Percent increase in the number of Neighborhood Watch Programs
Percent increase in National Night Out participants
Percent variance in the number of Super Kid classes taught at the desired level
Percent decrease in the number of activity units by School Resource Officers
Bureau Expenditures per Resident Population
Bureau Expenditures per Day Population
Bureau Expenditures per FTE

Police Patrol Services Bureau – 6 measures

Average Response Time for Code 2 Calls for service
Average Response Time for Code 3 Calls for service
Average Cost to Answer Call for Service by a Uniform Patrol Unit
Bureau Expenditures per Resident Population
Bureau Expenditures per Day Population
Bureau Expenditures per FTE




Orlando Police Department Performance Measures Review                             30
           APPENDIX B – AUDITEE RESPONSE MEMORANDUM




Orlando Police Department Performance Measures Review   31

								
To top