Flowchart of Hiring Through Job Portals - PDF by blw19492

VIEWS: 284 PAGES: 49

More Info
									Career Service Authority
     Recruiting Process
          Performance Audit




               August 2009




                        Office of the Auditor
                       Audit Services Division
                     City and County of Denver




                      Dennis J. Gallagher
                           Auditor
The Auditor of the City and County of Denver is independently elected by the citizens of Denver. He is
responsible for examining and evaluating the operations of City agencies for the purpose of ensuring the
proper and efficient use of City resources and providing other audit services and information to City
Council, the Mayor and the public to improve all aspects of Denver’s government. He also chairs the
City’s Audit Committee and oversees the City’s Comprehensive Annual Financial Report (CAFR)


The Audit Committee is chaired by the Auditor and consists of seven members. The Audit Committee
assists the Auditor in his oversight responsibilities of the integrity of the City’s finances and operations,
including the integrity of the City’s financial statements. The Audit Committee is structured in a manner
that ensures the independent oversight of City operations, thereby enhancing citizen confidence and
avoiding any appearance of a conflict of interest.



Audit Committee
    Dennis Gallagher                                      Maurice Goodgaine
    Robert Haddock                                        Jeffrey Hart
    Charles Husted                                        Bonney Lopez
    Timothy O’Brien


Audit Staff
    John Carlson, JD, CIA, CICA, Deputy Director
    Nancy Howe, Internal Audit Supervisor
    Jacob Claeys, CICA, Lead Internal Auditor
    Marcus Garrett, CIA, CGAP, Lead Internal Auditor
    Mary Mutchler, CICA, Senior Internal Auditor
    Travis Henline, Senior Internal Auditor
    Emily Gibson, Staff Internal Auditor



    You can obtain free copies of this report by contacting us at:



                         Office of the Auditor
                         201 W. Colfax Avenue, Dept. 705  Denver CO, 80202
                         (720) 913-5000  Fax (720) 913-5026


    Or view an electronic copy by visiting our website at:
    www.denvergov.org/auditor
                        City and County of Denver
                        201 West Colfax Ave., Dept. 705      Denver, Colorado 80202      720-913-5000     FAX 720-913-5247
                        www.denvergov.org/auditor
Dennis J. Gallagher
        Auditor
                                                                 August 20, 2009


Jeff Dolan, Executive Director
Career Service Authority
City and County of Denver

Dear Mr. Dolan:

Attached is the Auditor‘s Office Audit Services Division‘s report of their audit of the Career
Service Authority Recruiting Process for the year ended December 31, 2008. The purpose of the
audit was to evaluate the operational efficiency and effectiveness of the City‘s recruiting
process and to determine whether internal controls in place were adequate under the
circumstances.

Audit work identified opportunities for improvement in all four aspects of the recruiting function
examined, including strategic planning, the timeliness of the recruiting function, consistency in
recruiting practices, and oversight. The weaknesses identified in CSA‘s recruiting process inhibit
the efficiency and effectiveness of the process and limit accountability.

If you have any questions, please call Kip Memmott, Director of Audit Services, at
720-913-5029.

                                                                 Sincerely,



                                                                 Dennis J. Gallagher
                                                                 Auditor


DJG/mkm


cc:     Honorable John Hickenlooper, Mayor
        Honorable Members of City Council
        Members of Audit Committee
        Ms. Kelly Brough, Chief of Staff
        Mr. Claude Pumilla, Chief Financial Officer
        Mr. David T. Roberts, Chief Services Officer
        Mr. David Fine, City Attorney
        Mr. L.Michael Henry, Staff Director, Board of Ethics
        Ms. Lauri Dannemiller, City Council Executive Staff Director
        Ms. Beth Machann, Controller




To promote open, accountable, efficient and effective government by performing impartial reviews and other audit
services that provide objective and useful information to improve decision making by management and the people.
           We will monitor and report on recommendations and progress towards their implementation.
                        City and County of Denver
                        201 West Colfax Ave., Dept. 705      Denver, Colorado 80202      720-913-5000, FAX 720-913-5247
                        www.denvergov.org/auditor
Dennis J. Gallagher
      Auditor

                                                 AUDITOR’S REPORT

We have completed an audit of the Career Service Authority‘s Recruiting Process for the period
January 1, 2008 through December 31, 2008. The purpose of the audit was to examine and
assess the City‘s recruiting processes to identify possible inefficiencies and opportunities for
improvement. Audit work focused primarily on strategic planning, the timeliness of the recruiting
function, consistency in recruiting practices, and adequate oversight as key performance
indicators.

This performance audit is authorized pursuant to the City and County of Denver Charter, Article
V, Part 2, Section 1, General Powers and Duties of Auditor, and was conducted in accordance
with generally accepted government auditing standards. Those standards require that we plan
and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis
for our findings and conclusions based on our audit objectives. We believe that the evidence
obtained provides a reasonable basis for our findings and conclusions based on our audit
objectives.

Audit work identified opportunities to improve all four aspects of the recruiting function
examined. Specifically:

                           CSA currently lacks both a clear definition of what the merit system seeks
                           to accomplish and a strategic plan that establishes goals and helps the
                           agency adjust to future developments;
                           CSA‘s recruiting and hiring process appears to take longer than other
                           jurisdictions;
                           CSA personnel are not consistent in how they document or perform their
                           recruiting activities and weak controls such as poor security over tests
                           create situations that potentially threaten fair and open competition for
                           City positions; and
                           Accountability is hampered by the lack of sound performance measures,
                           limited data, and ineffective tools for both evaluating employees‘
                           performance and reviewing the timeliness and quality of recruiting efforts.

We extend our appreciation to the CSA personnel who assisted and cooperated with us during
the audit.

                                                            Audit Services Division




                                                            Kip Memmott, MA, CGAP, CICA
                                                            Director of Audit Services

To promote open, accountable, efficient and effective government by performing impartial reviews and other audit
services that provide objective and useful information to improve decision making by management and the people.
           We will monitor and report on recommendations and progress towards their implementation.
        TABLE OF CONTENTS

EXECUTIVE SUMMARY                                                             1
 Opportunities Exist to Improve the Recruiting Function                       1

INTRODUCTION & BACKGROUND                                                     4
 Overview of Denver’s Career Service Authority                                4
 Overview of CSA’s Recruiting Process                                         5
 CSA Recruiting Uses a Variety of Assessment Methods                          8
 Equal Employment Opportunity Laws Applicable to the City’s Recruiting
 Process                                                                      9
 CSA Has Recently Experienced Organizational Change                          10

SCOPE                                                                        10

OBJECTIVE                                                                    10

METHODOLOGY                                                                  11

FINDING                                                                      12
 Opportunities Exist to Improve the Recruiting Function                      12
 Lack of a Clearly Defined Merit System and a Weak Strategic Plan Hinder
 CSA’s Effectiveness                                                         12
 The City’s Recruiting and Hiring Process Does Not Produce Timely Results    15
 Inconsistencies and Problems Executing CSA’s Processes Hinder Objectivity
 in the City’s Recruiting Efforts                                            17
 Accountability Is Hampered by Limited Management Information and Poor
 Oversight Mechanisms                                                  22

RECOMMENDATIONS                                                              28

APPENDIX A                                                                   32
 City Positions Not in the Career Service (Charter § 9.1.1 E)                32

AGENCY RESPONSE                                                              33
EXECUTIVE SUMMARY
Opportunities Exist to Improve the Recruiting Function
      Audit work identified opportunities to improve all four aspects of the recruiting function
      examined — strategic planning, the timeliness of the recruiting function, consistency in
      recruiting practices, and oversight. The weaknesses identified in CSA‘s recruiting process
      inhibit the efficiency and effectiveness of the process and limit accountability. In
      summary:

             CSA currently lacks both a clear definition of what the merit system seeks to
             accomplish and a strategic plan that establishes goals and helps the agency
             adjust to future developments;

             CSA‘s recruiting and hiring process appears to take longer than other jurisdictions;

             CSA personnel are not consistent in how they document or perform their
             recruiting activities, and weak controls such as poor security over tests create
             situations that potentially threaten fair and open competition for City positions;
             and

             Accountability is hampered by the lack of sound performance measures, limited
             data, and ineffective tools for both evaluating employees‘ performance and
             reviewing the timeliness and quality of recruiting efforts.

      Specifically, audit work identified the following areas for improvement.

Lack of a Clearly Defined Merit System and a Weak Strategic Plan Hinder CSA‘s
Effectiveness

      Audit work determined that the components of the City‘s merit system are not
      specifically defined. As a result, key stakeholders do not have a consistent understanding
      of what the merit system should encompass. This lack of clarity has implications for how
      the recruiting process is carried out.

      Without a clear understanding and definition of what a merit system means, CSA is
      limited in its ability to determine whether current recruiting processes reflect merit
      principles, whether some processes such as testing are even necessary, and whether a
      merit system is effective or necessary.

      In addition, audit work found CSA does not have a comprehensive strategic plan.
      Although CSA does have information typically contained in a strategic plan, it is located
      in three separate documents. This limits the organization‘s ability to use the information to
      plan effectively for the future.

The City‘s Recruiting and Hiring Process Does Not Produce Timely Results

      A key issue for a recruiting function is whether it succeeds in placing new employees in
      positions as expeditiously as possible. The most qualified candidates are generally desired
      by many employers, so as the length of time to hire increases, the opportunity to hire the
                                                                                 O f f ice of t he A udit or
                                               Page 1
            most qualified decreases. Audit work found that, relative to other jurisdictions, CSA‘s
            recruiting process takes longer. A lengthy recruiting and hiring process can negatively
            affect overall customer satisfaction. There are steps CSA can take to improve the overall
            efficiency of the process.

            For example, an audit sample found the average time to fill a position in 2008 was 79.9
            days. Compared against two industry standards, the City‘s process is slower. CSA is
            primarily a customer service agency with a variety of customers, including hiring
            agencies, potential job candidates, as well as current City employees. While most
            agencies reported they are fairly satisfied with the relationship and service they receive
            from CSA, several agencies also felt the recruiting process is too complicated or overly
            stringent. Audit work identified several steps that could be taken to shorten the review
            process to include:

                      Eliminating or compressing steps;

                      Identifying ways to increase efficiency by the hiring agency; and

                      Shortening the process for certain types of positions.

 Inconsistencies and Problems Executing CSA‘s Processes Hinder Objectivity in
 the City‘s Recruiting Efforts

            Audit work found CSA lacks policies and procedures to help prevent discrepancies and
            potential problems in the recruiting process. Detailed examination of the recruiting
            process showed numerous inconsistencies in how CSA personnel conducted and carried
            out various steps of the process. In addition, gaps and lapses in some procedures create
            situations in which fair and open competition for positions is potentially affected.

            Rule 3 serves as the primary policies and procedures for Recruiting. Audit work showed
            that while Rule 3 does provide some guidance, it is inadequate on its own because it is
            broad and vague. Recruiting has no additional written policies and procedures, and the
            result is incomplete or inconsistent recruiting practices. Specific examples include:

                      Key recruiting steps are not documented by Recruiters;

                      Recruiting does not have procedures for deciding when to eliminate recruiting
                      steps; and

                      Recruiters are inconsistent in utilizing special and preferred qualifications.

            Additionally, audit work revealed significant weaknesses in the physical security of test
            books. Auditors conducted an inventory of the test books stored in CSA‘s test vault and
            identified three discrepancies.

 Accountability Is Hampered by Limited Management Information and Poor
 Oversight Mechanisms

            Audit work determined CSA has little ability to monitor and report on its performance. It is
            generally not collecting the kinds of measures needed to accomplish this task effectively.
            Further, in several instances when it did report information about its performance in order
            for the Board to make key decisions about the recruiting process, the information was
Cit y and Count y of Denver
                                                        Page 2
inaccurate. Likewise, key oversight mechanisms, including performance evaluations of
employees and quality control reviews of the agency‘s work are weak.

Specifically, audit work determined there are two ways in which the current
performance measurement system limits the ability to assess the efficiency and
effectiveness of the recruiting function. First, monitoring and collection of performance
information is inconsistent. Second, the performance indicators monitored yield little
information about efficiency and effectiveness, and the indicators do not align with best
practices.

The limited focus on such indicators reduces CSA‘s and the Board‘s ability to analyze the
efficiency and effectiveness of recruiting activities. For example, the cost of recruiting is
unknown. CSA does not know how much time Recruiters spend on each recruitment or
the total cost to hire an employee. Weaknesses in current performance measures
impede CSA‘s ability to determine whether the City‘s resources are being wisely and
effectively spent. The limited information also reduces CSA‘s ability to plan strategically
for the future.

Furthermore, audit work identified several instances of inaccurate reporting. These
involved performance information in various CSA reports and the content of
presentations to the CSA Board. Reporting must be accurate in order for key
stakeholders to hold Recruiting accountable for its performance. Inaccurate information
diminishes the ability of stakeholders to evaluate the overall effectiveness of Recruiting.
Additionally, accurate information is necessary to determine whether CSA is adhering to
the merit system.

Audit work found existing quality controls over the recruiting process are vague and
largely undocumented. Recruiting has developed a general reporting structure for
quality review of Recruiters‘ work, but no way to monitor whether reviews occur. The lack
of a well defined and documented quality control process limits CSA‘s knowledge of
performance, their ability to hold individual personnel accountable, and increases the
risk of inaccuracies.

CSA was specifically established as an independent agency to limit inappropriate
political influence, but because of this independence, the CSA Board should actively
provide oversight and hold CSA accountable for carrying out the agency‘s mission and
accomplishing its goals. Action is needed on multiple fronts to strengthen the City‘s
recruiting function. In many respects, these issues are interrelated.




                                                                           O f f ice of t he A udit or
                                         Page 3
 INTRODUCTION & BACKGROUND
 & BACKGROUND
 Overview of Denver’s Career Service Authority
            The City and County of Denver is one of the largest employers in the Denver
            metropolitan area. About 68% of the City and County‘s more than 13,000 employees are
            part of what is called the Career Service. These employees include general positions
            such as administrative personnel, laborers, and professionals. 1 Filling these positions with
            qualified applicants is a sizeable task. In fiscal year 2008, the City and County had 1,769
            new hires for positions within the Career Service.

            The Career Service Authority (CSA) is the primary agency that provides human resource
            services for Career Service employees. CSA has over 80 authorized positions, including
            Recruiters, Training Specialists, Classification/Compensation Specialists, and Benefit
            Specialists. The agency‘s duties include:

                      Recruiting, examining, and certifying applicants for employment and promotion;

                      Administering the personnel rules that govern appointments, promotions,
                      demotions, transfers, and layoffs;

                      Providing Citywide training;

                      Setting compensation and classification details for City employees;

                      Administering employee benefits; and

                      Managing employee grievances and other employee relations.

            Established under the City Charter in 1954, CSA's overall mission is to uphold and
            administer a merit system. Among other things, a merit system allows job applicants to
            compete for and obtain employment based on their knowledge, skills, and abilities.2
            Merit systems have been utilized by government for decades and were intended to
            combat the "spoils" system in which government jobs were openly bought and sold.

            The City Charter establishes CSA as an independent agency—that is, an agency
            removed from the Mayor‘s direct control. CSA‘s operations are overseen by a governing
            board, called the Career Service Authority Board (the Board). The Board is comprised of


 1
   Examples of positions that are not part of the Career Service include elected officials and uniformed staff of the police and fire
 departments. Appendix A lists the positions not included in the Career Service.
 2
   The federal government has a detailed framework for achieving fair and open human resource policies and practices called
 the Merit System Principles and Prohibited Personnel Practices. The City and County of Denver is not required to abide by these
 practices.



Cit y and Count y of Denver
                                                                 Page 4
         five members who are appointed by the Mayor and confirmed by City Council for five-
         year staggered terms. The Board‘s duties include:

                   Overseeing the Career Service Board Personnel Rules (Rules) that govern
                   employment practices within the Career Service. These Rules address hiring, pay,
                   classification, transfers, dispute resolution, promotions, training, and other
                   personnel matters. The Board may revise or rescind proposed Rule changes at
                   any time provided the process is performed in accordance with the rule-making
                   process specified in the Rules;

                   Overseeing CSA, including appointing CSA‘s Executive Director; and

                   Overseeing the Career Service Hearing Office, and serving as a quasi-judicial
                   body to decide appeals of decisions of the Career Service Hearings Officers. 3

Overview of CSA’s Recruiting Process
         The Recruiting Section (Recruiting) within CSA‘s Workforce Management Division is
         responsible for providing recruitment services for Career Service positions. Rule 3 governs
         the process for recruiting potential employees. Candidate selection is a key component
         of the recruiting and hiring process and is instrumental in helping to ensure the City hires
         the most qualified candidates and upholds principles of a merit system. City agencies
         play an important role in the hiring process; however, CSA will perform some agency
         functions upon request.

         The general process to fill open positions is depicted in Figure 1.4 As the figure shows,
         CSA is only one of the City agencies involved in the process. In general, CSA‘s focus is on
         advertising the position and establishing a list of the most highly qualified candidates. The
         hiring agency then decides which of these candidates to hire. The process can be
         separated into three main phases as follows:

                   Planning and Position Announcement. When a hiring agency has a vacancy, it
                   typically contacts the City‘s Budget and Management Office to verify available
                   funding. If the Budget and Management Office determines funding is available, it
                   completes an approval to fill form and returns it to the hiring agency, which then
                   contacts CSA Recruiting. Then recruiting preparation begins. This preparation
                   includes a CSA Recruiter working with the hiring agency to determine the
                   recruitment strategy, the methodology for testing applicants, and the posting
                   (job announcement) plan. When complete, Recruiting posts the open position to
                   the City website and any other agreed-upon external websites.

                   Applicant Screening. As individuals apply for an open position, Recruiting reviews
                   every application received and creates what is called an eligible list. This list


3
  City and County of Denver Career Service Board Personnel Rules, Rule 2-11A: Duties and Organization of the Board, Effective
December 23, 2005, http://denvergov.org/HumanResources/RulesandPolicies/tabid/432517/Default.aspx (accessed March 20,
2009).
4
  The process described here is used for most recruitments, but not all. Other approaches taken by CSA are discussed elsewhere
in this report.
                                                                                                        O f f ice of t he A udit or
                                                             Page 5
                      includes all applicants who meet the position‘s minimum qualifications. For these
                      applicants, Recruiting administers an assessment—that is, a test designed to
                      further determine applicants‘ relative abilities—and ranks applicants in order of
                      their scores. Using these scores, Recruiting creates what is called a certified list
                      (also known as a referred list or referral list), which comprises the top 20 ranked
                      candidates. It then transmits this certified list to the agency.

                      Interview and Selection. The hiring agency determines which candidates from the
                      certified list it wishes to interview. CSA Rules require agencies to interview a
                      minimum of three candidates from the certified list. The hiring agency negotiates
                      salary, makes a verbal offer, conducts background checks, and sends an offer
                      letter. CSA may partner with the hiring agency or perform these tasks as well. The
                      CSA Recruiter verifies that the hiring agency complied with City policies, such as
                      interviewing at least three candidates from the certified list. After the selected
                      candidate begins work, the City Controller‘s Office completes I-9 paperwork.




Cit y and Count y of Denver
                                                       Page 6
Figure 1: City Recruiting Process




                                    O f f ice of t he A udit or
               Page 7
 CSA Recruiting Uses a Variety of Assessment Methods
            CSA Recruiting uses a variety of assessment methods, both in screening applicants‘
            qualifications and in developing and administering examinations. When an open position
            is posted, the position description lists the required minimum qualifications, such as the
            minimum level of education or relevant experience required. These minimum
            qualifications are set for each job class by CSA‘s Classifications Division. Neither
            Recruiting nor hiring agencies may amend these qualifications under any circumstances.
            At the same time, agencies may include two additional types of qualifications, as
            follows:

                      Special qualifications. Rule 3 allows agencies to require additional qualifications
                      beyond the minimum if the skills are necessary to perform the job. For example,
                      an agency seeking to recruit an Administrative Support Assistant may require an
                      additional qualification to be bilingual if, as a regular part of that job‘s duties, the
                      hired person will be interacting with non-English speakers. CSA refers to these as
                      special qualifications and Recruiters may screen applicants for special
                      qualifications in the same manner they screen for minimum qualifications.

                      Preferred qualifications. These qualifications are not covered in Rule 3 or in any
                      other written CSA policy. They are qualifications that a hiring agency would prefer
                      the ideal candidate to have, but are not necessary to perform the position. For
                      example, the minimum qualifications required for a Senior Accountant are a
                      Bachelor‘s Degree and two years of accounting experience, but a particular
                      agency may prefer their Senior Accountant have government accounting
                      experience. This could be stated on the job announcement as a preferred
                      qualification. There are no written restrictions regarding when an agency may
                      request preferred qualifications in a recruitment. Additionally, there are no written
                      instructions on how these are to be used in recruiting.

            CSA administers an examination for every position in the Career Service. These
            examinations are developed by CSA Recruiters and Test Specialists in consultation with
            the hiring agency. According to CSA‘s Test Operations Manual, the following are the
            most common tests given to Career Service job applicants:

                      Written test questions. These are test booklets typically administered to high
                      volume recruitments or at agency request. Questions originate from the County
                      of San Bernardino, California and are used widely by personnel agencies around
                      the country. Referred to as the ‗Western Region Item Bank,‘ this database
                      contains more than 54,000 test questions covering 375 subject matter categories.

                      Supplemental questions. These are job-specific open-ended questions that are
                      graded using established rating criteria. They are typically part of the job posting,
                      allowing CSA to obtain applicant answers with the application. Generally, this
                      type of exam is developed for professional-level job classifications.




Cit y and Count y of Denver
                                                        Page 8
                  Performance test. These tests simulate an actual job requirement and are
                  typically administered for Equipment Operator and Heavy Equipment Operator
                  candidates. These examinations are expensive to administer and, therefore, tend
                  to be included in a limited number of exam plans.

                  Skills test. CSA also has programs available to conduct skills based testing on
                  computer applications, such as Microsoft Word, Excel and Access.

         Other test options may be developed between the Test Specialists and hiring agency
         based on need.

Equal Employment Opportunity Laws Applicable to the City’s
Recruiting Process
         As an employer, the City must abide by federal Equal Employment Opportunity (EEO)
         laws. These laws and regulations include:

                  Title VII of the Civil Rights Act of 1964 - Prohibits discrimination and other unlawful
                  employment practices based on race, color, religion, sex, or national origin;

                  The Americans with Disabilities Act (ADA) - Prohibits discrimination on the basis of
                  disability;

                  The Age Discrimination in Employment Act (ADEA) - Prohibits age discrimination in
                  employment; and

                  The Equal Pay Act (EPA) - Prohibits sex-based wage discrimination as part of the
                  Fair Labor Standards Act.

          While CSA personnel take many steps to comply with Equal Employment Opportunity
          Commission (EEOC) regulations, the concept of adverse impact is directly related to
          issues discussed in this report. Specifically, the EEOC has Uniform Guidelines that
          incorporate a single set of principles which are designed to assist employers in
          eliminating employment practices which discriminate on the grounds of race, color,
          religion, sex, and national origin. These guidelines provide a framework for determining
          the proper use of tests and other selection procedures. This framework particularly
          addresses adverse impact which is defined as ―a substantially different rate of selection
          in hiring, promotion, or other employment decision which works to the disadvantage of
          members of a race, sex, or ethnic group.‖5




5
 Equal Employment Opportunity Commission (EEOC), Uniform Guidelines on Employee Selection Procedures (1978), 43 FR
38295 (August 25, 1978), 29 CFR Part 1607, Section 16B, http://www.uniformguidelines.com/uniformguidelines.html (accessed
April 7, 2009).
                                                                                                     O f f ice of t he A udit or
                                                            Page 9
 CSA Has Recently Experienced Organizational Change
            A new Executive Director joined CSA at the end of 2007 and initiated an extensive
            reorganization beginning in 2008. This greatly impacted the Recruiting Section as well as
            CSA‘s other divisions. Recruiters‘ responsibilities were expanded and most Recruiting
            support staff was eliminated. Further, CSA hired a new Recruiting Supervisor in mid-2008.
            A customer service survey completed as part of audit work indicated that since these
            major changes have taken place, agencies have seen improvement. This audit seeks to
            identify areas for additional improvement and add to the progress already made by CSA
            leadership.

            In 2006, CSA Recruiting began phasing in NEOGOV, an online recruiting management
            system. Recruiters now use NEOGOV to recruit for all agencies in the City. In 2008, CSA
            launched NEOGOV Online Hiring Center, an additional software package that allows
            the hiring agencies to utilize NEOGOV in their part of the recruiting process.




 SCOPE
            The scope of the audit was to examine CSA‘s recruiting processes and practices.
            Consequently, it focused primarily on the Recruiting Section within CSA's Workforce
            Management Division. Audit work also included evaluating the CSA Board and the
            duration of City agencies‘ portion of the hiring process. The audit period extended from
            January 1, 2008 through December 31, 2008.




 OBJECTIVE
            Our objective was to assess the following aspects of the CSA recruiting process:

                      Does Recruiting have clearly articulated goals and a suitable plan for
                      accomplishing them?

                      Is the process producing timely results in terms of screening applicants and filling
                      open positions?

                      Are components of the process consistently implemented, so that the process is
                      objective and fair?

                      Does management have accurate, useful, and timely information that can be
                      used to monitor and improve the process, as well as effective controls for
                      ensuring accountability?




Cit y and Count y of Denver
                                                      P a g e 10
METHODOLOGY
         Evidence gathering and analysis techniques used to meet audit objectives included, but
         were not limited to:

                   Interviewing CSA personnel and personnel with human resources responsibilities
                   at other City agencies;

                   Attending a NEOGOV training course;

                   Attending CSA Board meetings;

                   Documenting relevant CSA policies and practices;

                   Analyzing the City Charter, the Denver Revised Municipal Code, CSA Rules, and
                   federal requirements related to recruiting and hiring;

                   Tracking new job postings for 45 days including both business days and
                   nonworking days;

                   Reviewing job postings, position qualifications, job testing, pre-employment
                   screenings, and CSA employee documentation practices;

                   Benchmarking against other municipalities regarding their recruiting and hiring
                   practices;6

                   Attending applicant examinations conducted by CSA;

                   Testing various recruitment-related practices;

                   Collecting and analyzing NEOGOV data; and

                   Evaluating NEOGOV usage and capabilities.




6
 Auditors selected the following benchmark jurisdictions due to the similarity of their population and total staff size to
Denver’s: Long Beach, CA; Portland, OR; Fulton County, GA; and Oklahoma City, OK. Some of these jurisdictions also utilize
NEOGOV.
                                                                                                         O f f ice of t he A udit or
                                                             P a g e 11
 FINDING
 Opportunities Exist to Improve the Recruiting Function
            Audit work identified opportunities to improve all four aspects of the recruiting function
            examined—planning, timeliness, consistency, and oversight. The weaknesses identified in
            CSA‘s recruiting process inhibit the efficiency and effectiveness of the process and limit
            accountability. In summary:

                      CSA currently lacks both a clear definition of what the merit system seeks to
                      accomplish and a strategic plan that establishes goals and helps the agency
                      adjust to future developments;

                      CSA‘s recruiting and hiring process appears to take longer than other jurisdictions,
                      and CSA‘s ability to monitor its performance is limited by gaps and inconsistencies
                      in the data it collects;

                      CSA personnel are not consistent in how they document or perform their
                      recruiting activities, and weak controls such as poor security over tests create
                      situations that potentially threaten fair and open competition for City positions;
                      and

                      Accountability is hampered by the lack of sound performance measures, limited
                      data, and ineffective tools for both evaluating employees‘ performance and
                      reviewing the timeliness and quality of recruiting efforts.

 Lack of a Clearly Defined Merit System and a Weak Strategic Plan
 Hinder CSA’s Effectiveness
            Audit work determined that the components of the City‘s merit system are not
            specifically defined. As a result, key stakeholders do not have a consistent understanding
            of what the merit system should encompass. This lack of clarity has implications for how
            the recruiting process is carried out. In addition, the absence of a comprehensive
            strategic plan limits CSA‘s ability to adjust to future developments.

 CSA Has Not Clearly Defined the Components of a Merit System

            Although the principles of the merit system are the foundation of CSA's existence, CSA
            does not have a specific definition for the City‘s merit system or a consistent view of what
            its components should be. This results in differing interpretations of the merit system
            throughout the City. As part of audit work, auditors asked 16 key personnel to define the
            merit system, including three in CSA management, all five Board members, and eight
            Human Resource (HR) representatives from various City agencies. No more than 50% of
            the respondents gave the same general definition and two did not recognize the term
            merit system. Table 1 summarizes the interviewees‘ responses.



Cit y and Count y of Denver
                                                      P a g e 12
                          Table 1: Merit System Definitions Provided by Survey Respondents

                                 “Merit System” means:                          No. of             % of Responses*
                                                                              Responses

               Fair and Competitive Hiring                                           8                     50.0%

               Testing or Ranking to Hire Based on Skills                            7                     43.8%

               Pay for Performance or Fair Pay                                       6                     37.5%

               Consistent/Fair Rules and Employee                                    5                     31.3%
               Treatment

               Opposite of “Spoils” or Political Patronage                           2                     12.5%

               Don’t Know                                                            2                     12.5%

               *Auditors obtained 16 responses. Multiple answers were accepted so totals do not equal
               100%.




         This lack of clear and consistent focus has implications for how the system is administered
         and what it accomplishes. For example:

                   Recruiting personnel reported the merit system is the reason testing is required of
                   all candidates. However, developing and administering a test adds time and
                   costs to the recruiting process and is not necessarily needed to conform to merit
                   system principles. Audit work found three of the four benchmarking cities
                   reported adherence to the merit system, and none of these cities require all
                   candidates be tested as part of their recruiting process.

                   In January 2009, the Board approved reducing the minimum amount of time a
                   job announcement can be posted from five to two business days. This might
                   jeopardize the merit principle of fair and open competition. For example, the
                   federal government recommends a minimum posting of five days. Additionally,
                   two of the four benchmark jurisdictions require posting a job announcement for a
                   minimum of five business days, and one requires a minimum posting of 7 business
                   days.7

         Without a clear understanding and definition of what a merit system means, CSA is
         limited in its ability to determine whether current recruiting processes reflect merit
         principles, whether some processes such as testing are even necessary, and whether a
         merit system is effective or necessary.




7 The fourth city requires a minimum posting of 3 days, but typically posts job opportunities for two weeks.
                                                                                                          O f f ice of t he A udit or
                                                              P a g e 13
 Lack of a Comprehensive Strategic Plan Limits CSA‘s Ability to Adjust for Future
 City Demands

            Audit work found CSA does not have a comprehensive strategic plan. Although CSA
            does have information typically contained in a strategic plan, it is located in three
            separate documents. This limits the organization‘s ability to use the information to plan
            effectively for the future.

            A study conducted by the International Public Management Association for Human
            Resources (IPMA-HR) indicates the absence of a strategic plan reduces a human
            resource organization‘s ability to function effectively. The study states, "The failure to have
            a strategic plan in place suggests that the organization approaches HR management in
            a fragmented and reactive fashion that, by its very nature, is less effective and more
            costly than working within a thoughtful design."8 The study recognizes that developing a
            strategic plan requires commitment from agency management. According to the study,
            "If organization leadership does not recognize value in HR, strategic planning may not be
            a visible priority, and thus, not required or mandated by leadership. Strategic planning
            requires that HR have the staff, time and energy needed to complete a planning process
            regardless of how simple or broad the resulting plan." 9 Without a comprehensive strategic
            plan, CSA is in a less advantageous position to incorporate new ideas or strategies into
            the City‘s HR efforts.

            One strategy currently utilized by the City is branding. Branding, a fairly new concept in
            the public sector, has been used by private-sector companies as a way to sell products.
            According to the IPMA-HR, ―Branding is strategic and often depends on consistency in
            messaging, communications and standards. This same concept can be applied to
            public sector organizations seeking to establish themselves as more than just an entity
            tasked with basic service delivery. Public sector organizations are beginning to recognize
            the importance of reputation or how the organization is seen through the eyes of a
            potential employee. Branding is about taking charge of that reputation, defining the
            benefits of working in a public sector environment and marketing these factors to
            potential employees.‖10 A review of CSA‘s branding strategy revealed materials to
            include:

                      Recruiting banners used at job fairs identifying the City logo;

                      Pamphlets on training, organizational development, and job opportunities;

                      Brochures promoting the City and County of Denver as a great place to work;
                      and

                      Full-page color print outs used for executive recruitments.




 8
   EquaTerra, “Recruiting and Staffing in the Public Sector: Results from the IPMA-HR Research Series”, International Public
 Management Association for Human Resources, http://ipma-hr.org/content.cfm?pageid=72 (accessed June 3, 2009).
 9
   Ibid.
 10
    Ibid.
Cit y and Count y of Denver
                                                               P a g e 14
         CSA also utilizes electronic formats offering candidates the ability to receive information
         regarding available positions by e-mail and text
         messaging. Additionally, CSA advertises on
                                                              CSA uses Twitter and Facebook to
         social media sites such as Twitter.com and
                                                                 promote City job openings.
         Facebook.com.

         However, although CSA uses a variety of
         strategies to promote the City, it appears Recruiters may not always be aware of or
         actively utilizing them. CSA‘s branding strategy was not mentioned in interviews
         conducted with Recruiting staff. Furthermore, staff in charge of career fairs and university
         outreach stated CSA does not have a branding strategy. If Recruiting staff is not
         consistently utilizing the branding tools available, the City risks losing qualified candidates
         that are unaware of the benefits of working in the public sector, or who may not have
         considered the City and County of Denver for employment.

The City’s Recruiting and Hiring Process Does Not Produce Timely
Results
         A key issue for a recruiting function is whether it succeeds in placing new employees in
         positions as expeditiously as possible. The most qualified candidates are generally desired
         by many employers, so as the length of time to hire increases, the opportunity to hire the
         most qualified decreases. Audit work found that, relative to other jurisdictions, CSA‘s
         recruiting process takes longer. A lengthy recruiting and hiring process can negatively
         affect overall customer satisfaction. There are steps CSA can take to improve the overall
         efficiency of the process.

Available Data Indicate Other Jurisdictions Process Job Applicants More Quickly

         An audit sample found the average time to fill a position in 2008 was 79.9 days.11
         Compared against two industry standards, the City‘s process is slower.

                   Comparison with other jurisdictions that use the same recruiting software – In 2006,
                   IPMA-HR issued a benchmarking study that surveyed public-sector hiring
                   agencies using NEOGOV, the online recruiting software CSA began using in
                   2006.12 This study found the mean number of days reported for ―notification of
                   vacancy until reports to work‖ (which represents an earlier starting time point and
                   therefore a longer timeframe than was measured in the audit sample) was 49
                   days, or nearly 31 days shorter than CSA‘s average time.

                   Comparison with federal standards – The federal government has an 80-day
                   standard for its hiring process. While this standard may appear very close to the
                   City‘s average of 79.9 days, closer comparison shows CSA‘s process is actually
                   slower in several respects. First, the federal government starts its measurement


11
   This is the number of days from when a Recruiter first enters recruitment information in NEOGOV (after step 4 on Figure 1,
page 7) and the new employee reports to work.
12
   International Public Management Association for Human Resources (IPMA-HR), “2006 Recruitment and Selection
Benchmarking,” IPMA-HR, http://ipma-hr.org/content.cfm?pageid=42 (accessed January 20, 2009).
                                                                                                         O f f ice of t he A udit or
                                                             P a g e 15
                      earlier in the process—the day the request to fill a position is received, and CSA
                      does not measure recruiting preparation time. If auditors had measured CSA‘s
                      processing time from this earlier point, the average processing time would be
                      greater. Second, the federal hiring time includes 14 days to advertise the job,
                      compared to the minimum of two for CSA, and time for completing an extensive
                      security background check the City does not perform. This indicates other parts
                      of CSA‘s process are taking much longer.

 Customer Service is Negatively Impacted by Timeliness Issues

            Audit work found weaknesses identified in CSA‘s recruiting processes and practices
            negatively impact customer satisfaction. CSA is primarily a customer service agency with
            a variety of customers, including hiring agencies, potential job candidates, as well as
            current City employees. Audit work included a survey of the City‘s hiring agencies. While
            most agencies reported they are fairly satisfied with the relationship and service they
            receive from CSA, several agencies also felt the recruiting process is too complicated or
            overly stringent. In a customer satisfaction survey completed by CSA, employees who
            were less than completely satisfied were asked to offer suggestions for improvement to
            make the selection process better. Of those who responded, 52% stated they would like
            to see a faster process.

 Options Exist for Making the Process More Timely

            Audit work identified several steps that could be taken to shorten the recruiting process.

                      Eliminating or compressing steps – Audit work identified some steps in the
                      recruiting process that are not mandated by law or Rule and could potentially be
                      eliminated. One example is the current requirement that the City‘s Budget and
                      Management Office (BMO) verifies available funding prior to initiating recruiting
                      activities. However, the hiring agency should be able to determine whether
                      adequate funding is available. Even if this step is retained, there are ways to
                      modify and compress the process. One of the benchmark jurisdictions begins
                      recruiting as soon as the recruiting agency is notified that a position is open,
                      rather than waiting for BMO‘s approval. This allows the City to begin its advertising
                      and screening efforts while funding is approved. When Recruiting is notified the
                      position can be filled, it is able to send its certification list to the hiring agency
                      within one day.

                      Identifying ways to increase efficiency by the hiring agency – Audit testing
                      showed the hiring agency‘s portion of the hiring process comprised
                      approximately 45-50 days of the 79.9 day average, or about 60% of the total
                      time. While this portion of the process is largely outside of CSA‘s control, they are
                      currently not monitoring this time. By measuring and addressing this time, CSA
                      could pursue strategies for working with City agencies to decrease it.

                      Shortening the process for certain types of positions – CSA does not have formal
                      policies and procedures in place to allow agencies to fill positions when there is
                      an immediate need. For example, two of the four benchmark jurisdictions allow
Cit y and Count y of Denver
                                                       P a g e 16
                  individual agencies to hire their own on-call (temporary) employees. Allowing
                  City agencies to bypass CSA by directly hiring or utilizing a temporary agency to
                  fill on-call positions could meet their needs quickly while reducing CSA‘s
                  workload. Additionally, a 2000/2001 study conducted by IPMA/NASPE identified
                  immediate job offer processes for hard to fill positions as a best practice.13

Inconsistencies and Problems Executing CSA’s Processes Hinder
Objectivity in the City’s Recruiting Efforts
         Audit work found CSA lacks policies and procedures to help prevent discrepancies and
         potential problems in the recruiting process. Detailed examination of the recruiting
         process showed numerous inconsistencies in how CSA personnel conducted and carried
         out various steps of the process. In addition, gaps and lapses in some procedures create
         situations in which fair and open competition for positions is potentially affected.

CSA Personnel Execute the Process Inconsistently

         Rule 3 serves as the primary policies and procedures for Recruiting. Audit work showed
         that while Rule 3 does provide some guidance, it is inadequate on its own because it is
         broad and vague. Recruiting has no
         additional written policies and procedures,             Recruiting processes are
         and the result is incomplete or inconsistent                  inconsistent.
         recruiting practices. Specifically, the following
         inconsistencies were identified:

         Key Recruiting Steps Not Documented by Recruiters – Audit testing indicated CSA
         Recruiters document key steps inconsistently. Rule 3 does not specifically address which
         steps are to take place, and when auditors requested written policies and procedures,
         they found the steps were not written. The Recruiting Supervisor reported the following
         procedures should occur:

                When an approval to fill form is required by the Budget and Management Office,
                Recruiters should attach the completed form in NEOGOV;

                Recruiters should be using at least two subject matter experts (SMEs), or when an
                expert from the agency grades supplemental questions asked of applicants. There
                are no guidelines designating when it is appropriate for Recruiters to grade the
                questions or when a SME should be used;

                Recruiters should complete a service level agreement for any recruitment that has
                not been conducted by that specific Recruiter before. It is not required for high-
                volume recruitments or for citywide mass recruiting (i.e. Administrative Support
                Assistant). Any time an agreement is completed, it should be attached in
                NEOGOV; and



13
 International Public Management Association/ National Association for State Personnel Executives, “2000/2001 IPMA/NASPE
HR Benchmarking Project: Best Practices”, IPMA-HR, http://ipma-hr.org/content.cfm?pageid=380 (accessed January 20, 2009).
                                                                                                    O f f ice of t he A udit or
                                                          P a g e 17
                   During a typical recruitment, Recruiters should notify applicants when they do not
                   meet the minimum qualifications, when they are deemed eligible, when they are
                   referred to the hiring agency and when the position is filled. This is automatically
                   documented in NEOGOV.

            To determine if these steps were performed, auditors tested two recruitments for each of
            the ten Recruiters. The testing compared the procedures identified by the Recruiting
            Supervisor with recruitment data from NEOGOV. In order for management to adequately
            ensure requirements are fulfilled, each of these recruiting steps should be documented in
            NEOGOV. As shown in Table 2, documentation was not present the majority of the time.

                               Table 2: Required Recruiting Steps Compared With NEOGOV Data

                              Recruitment Documentation                    Number of Times
                                                                          Documentation Not
                                                                          Present in NEOGOV

                      Attachment of approval to fill form                         8 of 18*

                      Evidence provided that more than one subject               15 of 18**
                      matter expert was used

                      Service Level Agreement Attached                            16 of 20

                      Applicants notified at all appropriate stages of            17 of 20
                      recruitment

                      *Two recruitments tested were exempt from this requirement.
                      **Two recruitments did not use supplemental questions as a test
                      mechanism.


            These findings indicate Recruiters are inconsistent when entering information into
            NEOGOV.

            Additional audit work found some data from the time to hire sample had to be excluded
            due to inaccuracy and incompleteness. Specifically, 11 requisitions that auditors tried to
            verify had to be excluded due to errors found (i.e. the person recorded as hired for that
            position was actually hired for another position). Furthermore, auditors found variations in
            how Recruiters recorded recruitments using an existing certified list. In some cases, little to
            no data was recorded about these particular recruitments which made it difficult to
            determine who was hired when. NEOGOV has the potential to provide useful information
            that can be used for monitoring and overseeing the efficiency and effectiveness of the
            recruiting process, but this potential cannot be tapped if basic information entered into
            the system is incomplete and inaccurate.

            No Procedures for Deciding When to Eliminate Recruiting Steps – Depending on the
            circumstances of a recruitment, the recruiting process may be legitimately shortened
            without violating merit principles. For example, if an agency is hiring for an Administrative

Cit y and Count y of Denver
                                                       P a g e 18
         Support Assistant and a certified list is already available, Recruiters would not need to
         complete many of the steps in the workflow process. Similarly, if the agency hired the
         same staff position on many occasions, Recruiters may be able to eliminate the hiring
         agency strategy meeting. When Recruiters are legitimately able to adjust the recruiting
         process, agencies may be able to fill their open positions faster. While CSA provided
         instances when the recruiting process may be shortened, there are no written policies
         identifying when steps may be eliminated or who has the authority to eliminate certain
         steps.14

         Inconsistency in Creating Eligible Lists – Recruiters could be formulating eligible lists
         differently because Rule 3 is unclear as to when applicants may be placed on the
         eligible list. According to Rule 3, an eligible candidate is ―an applicant for a vacant
         position who meets the criteria required for placement on a list‖.15 ―Criteria‖ is not
         defined, which means it is subject to interpretation. For example, the eligible list could
         include applicants that only meet minimum qualifications, or it could include those who
         meet minimum qualifications and pass applicable tests.

         Inconsistency in Utilizing Special and Preferred Qualifications – Recruiters gave
         inconsistent answers as to when special and preferred qualifications can be used in
         screening, eliminating and ranking applicants. Specifically, only six Recruiters reported
         they could screen and eliminate applicants based on special qualifications. CSA Rule 3 is
         clear that Recruiters may screen for special qualifications. Further, six Recruiters reported
         preferred qualifications could not be used in ranking applicants for the certified list. CSA
         Rules do not address the use of preferred qualifications.

         Using special and preferred qualifications to the fullest extent possible is important in
         order to ensure Recruiters are providing the best customer service to agencies. Recruiters
         and agencies do not have the ability to amend minimum qualifications. CSA‘s
         Classifications Division reported minimum qualifications are broad by design, as they
         must apply to whole job classes. Further, Classifications reported special and preferred
         qualifications should be used when possible in order for agencies to get the best
         qualified candidate for the position. It appears from Recruiter interview responses that
         these qualifications are being inconsistently or inaccurately used. In the case of preferred
         qualifications, it is unclear how these relate to the certified list of candidates agencies
         choose from.

         Additionally, the Recruiting Supervisor reported to auditors that special qualifications may
         not be used for some entry-level positions, which contradicts audit work. When special
         and preferred qualifications are not used to the fullest extent it can increase applicant
         numbers and workload, eligible lists could be formulated differently, and it could reduce
         the number of highly qualified candidates.




14
  Examples include recurring recruitments and recruiting for parks, seasonal and labor positions.
15
  City and County of Denver Career Service Board Personnel Rules, Rule 3-5(C): Selection Definitions, Effective May 4, 2007,
http://denvergov.org/HumanResources/RulesandPolicies/tabid/432517/Default.aspx (accessed June 10, 2009).
                                                                                                          O f f ice of t he A udit or
                                                              P a g e 19
            Inconsistency in Developing Supplemental Test Questions – Recruiters and Test Specialists
            develop supplemental questions individually and questions may vary from recruiter to
            recruiter, even for the same job classification. Further, the established CSA review process
            for supplemental questions is spread throughout several documents and does not
            appear to be clearly monitored. As a result, questions may not be developed, graded
            and documented in a consistent manner.

 Identified Control Weaknesses Potentially Affect Fair and Open Competition for
 Positions

            In some aspects of the recruiting process the lack of sufficient policies and procedures
            creates situations that could jeopardize the fairness and openness of competition for
            positions. These instances were as follows:

            Certified Lists Can Remain Open for Lengthy Periods and Be Used for Multiple
            Recruitments – Audit work found that candidates are being hired after they are
            technically no longer eligible due to a flaw in Rule 3. Rule 3 places an expiration date on
            the ―eligible‖ list, but not on the ―certified‖ list.16 CSA sends a certified list to the hiring
            agency and the agency can hire from this list for an undetermined length of time, even
            when the eligible list may have lapsed.

            For example, auditors identified a candidate who was made eligible and whose name
            was sent to the agency via the certified list on August 31, 2007. Because the eligible list
            expires after 90 days, this candidate technically became ineligible for consideration on
            November 29, 2007. However, the candidate was hired on May 27, 2008, roughly nine
            months after being made eligible. The hiring agency was not in violation of Rule 3, since
            there is no expiration date for the certified list. Contacting candidates when sufficient
            time has elapsed could negatively impact the City‘s reputation and compromises its
            ability to hire quality candidates. The lack of closure on the certified list circumvents the
            intent to hire in a timely manner.

            The lack of an expiration date on the certified list allows it to be reused. Although there is
            an interest in efficiency when using a certified list several times, there needs to be
            balance. An expiration date on one list and not the other creates imbalance between
            using an existing certified list and conducting a new recruitment.

            Audit work showed the practice of reusing certified lists may be extensive. When auditors
            attempted to review recruitments in September 2008 for time to hire information, only 17
            positions could be tested, even though there were approximately 70 requisitions created
            in that month. This was mainly due to most new hires coming from an existing certified list,
            meaning a new recruitment was not conducted for the requisition.17 In other words, only
            17 recruitments were conducted (that could be verified) out of 70 requisitions created


 16
    For an explanation of these two lists, see “Overview of CSA’s Recruiting Process” in the Introduction and Background section
 of this report.
 17
    In some cases, auditors had to exclude the requisition, not because the hiring occurred off an existing certified list, but
 because too much data was missing due to Recruiters entering information into NEOGOV inconsistently.
Cit y and Count y of Denver
                                                               P a g e 20
that month. This calls into question the balance between conducting full scale
recruitments versus using an existing certified list to fill positions. When existing lists are
used too much, at some point it will impact the quality of people hired.

Weaknesses in Security Over Tests May Compromise Results – Audit work revealed
significant weaknesses in the physical security of test books. Auditors conducted an
inventory of the test books stored in CSA‘s test vault and identified three discrepancies:

      Eight of 22 (36%) tests counted did not match the Test Specialists‘ master inventory
      list;

      The vault contained test books for three positions that were not on the master
      inventory list; and

      The master inventory listed two types of test books that were not in the vault.

These discrepancies indicate Recruiters do not consistently use the Test Content Check
Out Log or Test Administration Security Control Record to sign out test books from
inventory. In addition to limiting CSA‘s ability to manage test inventory, it is difficult to
determine which Recruiter removed test books from the vault.

Besides finding discrepancies in the number of test books on hand, auditors also
identified other inventory or security weaknesses that further inhibit Recruiting‘s ability to
manage test inventory:

      Tests that are destroyed or are no longer in use are not accounted for on the
      master inventory list;

      CSA Personnel were unable to identify the number of tests originally printed, so no
      historic (original) inventory numbers were maintained for comparison. The first
      documented inventory of the test vault was completed by CSA in January 2009;

      Recruiting has not established a standard number of test books to be printed for
      each examination, making it more difficult to determine whether the number on
      hand is complete. For example, auditors counted 49 test books for one test,
      matching the number on the master inventory list. However, it is more likely a round
      number, such as 50 books, was originally printed; and

      A security issue exists related to access to the vault where tests are kept. A key to
      the vault is available to all Recruiting staff, and Recruiting does not have a process
      to check the key in and out.

Weaknesses in the physical security of test books increase the risk of exam results being
compromised. This could potentially allow under or unqualified applicants to pass exams.
Because qualified applicants are ranked on the certified list according to test score,
certain applicants may appear more qualified than they are, making it more difficult for
hiring agencies to determine and hire the best candidate.



                                                                             O f f ice of t he A udit or
                                         P a g e 21
            These weaknesses are more significant when considering the time needed to take
            corrective action. Recruiting personnel reported that when a written test is lost or stolen,
            they develop a new test for that position. The creation of a new test may take between
            two to six months, depending on the type of questions contained in the test and the time
            commitments required by other projects in Recruiting.

            Increased Legal and Public Relations Liability Related to Possible Identity Theft – Audit
            work found supplemental test books for one position contained valid social security
            numbers. Access to the vault, combined with availability of personal information such as
            names and social security numbers, could potentially lead to identity theft and a
            corresponding lack of trust from the clients CSA serves.

            Pass Points May Unintentionally Screen Out Qualified Candidates – Audit work
            determined Recruiting‘s use of pass points may increase the risk of similarly qualified
            candidates being inconsistently removed from further consideration. 18 Specifically, the
            candidate‘s test score, regardless of the pass point, determines their rank. However,
            utilizing a pass point may eliminate statistically similar candidates from further
            consideration.19

            Furthermore, although adverse impact may not be entirely removed, CSA‘s current use
            of fixed pass points may open the City to claims of discrimination. Test Specialists began
            assigning fixed pass points to each test with the implementation of NEOGOV. The prior
            automated software allowed Test Specialists to run comprehensive reports that helped
            them account for and remove adverse impact by adjusting the pass point. However,
            Test Specialists no longer run these types of reports due to NEOGOV‘s limited reporting
            capabilities.

 Accountability Is Hampered by Limited Management Information
 and Poor Oversight Mechanisms
            Audit work determined CSA has little ability to monitor and report on its performance. It is
            generally not collecting the kinds of measures needed to accomplish this task effectively.
            Further, in several instances when it did report information about its performance in order
            for the Board to make key decisions about the recruiting process, the information was
            inaccurate. Likewise, key oversight mechanisms are weak, including performance
            evaluations of employees and quality control reviews of the agency‘s work.

 Most Available Information is Not Useful in Assessing Efficiency and Effectiveness

            Performance Indicators Are Inconsistent and of Limited Use – Audit work determined
            there are two ways in which the current performance measurement system limits the
            ability to assess the efficiency and effectiveness of the recruiting function. First,


 18
   The minimum score candidates must achieve on the test to continue in the recruitment process.
 19
   For example, in a test with a pass point of 65 and an expected error rate of 3 points, one candidate’s score of 64 is statistically
 the same as another candidate’s score of 67; however, the candidate who scores 64 will be screened out for not passing the
 test.
Cit y and Count y of Denver
                                                                 P a g e 22
monitoring and collection of performance information is inconsistent. Second, the
performance indicators monitored yield little information about efficiency and
effectiveness, and the indicators do not align with best practices.

CSA‘s collection of performance information is sporadic. According to the Recruiting
Supervisor, performance information is not produced regularly and is mostly prepared for
outside meetings or other purposes. This limits Recruiting‘s ability to conduct trend
analyses and identify progress or emerging problems. The lack of consistent performance
monitoring creates difficulties when using any of this information in decision making.
Moreover, internal accountability is difficult to achieve if information is being collected
sporadically and only for outside use.

When CSA does collect and report performance information, it tends to focus on agency
workload. Specifically, CSA Recruiting has reported on a limited basis the following
performance indicators:

       number of job postings;

       applications received;

       applications per posting;

       testing sessions held; and

       positions filled.

These indicators provide little information about the agency‘s efficiency or effectiveness.
In contrast to the kinds of information CSA collects and reports, best practices research
indicates the following types of indicators provide better information about an agency‘s
performance:

       time to hire (including measuring the time each step takes);

       cost per hire;

       time spent per recruitment;

       volume of applications;

       turnover rates;

       completion of probation; and

       qualified applicants per posting.

CSA has reported in the past on two of these indicators—volume of applications and
time to hire—but only on a limited basis. The limited focus on such indicators reduces
CSA‘s and the Board‘s ability to analyze the efficiency and effectiveness of Recruiting
activities. For example, the cost of recruiting is unknown. CSA does not know how much
time Recruiters spend on each recruitment or the total cost to hire an employee.
Although performance indicators quantifying cost are only a piece of what should be
                                                                         O f f ice of t he A udit or
                                        P a g e 23
            monitored according to best practices, cost is important in helping to determine
            Recruiting efficiency and effectiveness. Weaknesses in current performance measures
            impede CSA‘s ability to determine whether the City‘s resources are being wisely and
            effectively spent. The limited information also reduces CSA‘s ability to plan strategically
            for the future. The effectiveness of any plan or steps to improve functions is diminished
            without information demonstrating how goals will be accomplished.

 Available Information is Often Incorrect or Reported Inaccurately

            Audit work identified several instances of inaccurate reporting. These involved
            performance information in various CSA reports, and the content of presentations to the
            CSA Board.

            Issues Identified With Data Submitted in CSA Reports – Audit work identified key problems
            in the creation of two different reports submitted to the City.

                      In its 2008 annual report, CSA reported the average number of days to fill a
                      position was 29.5 days. While audit work suggests this figure represents the time it
                      takes to create a certified list, worded this way, the number appears to represent
                      the time it takes until the new employee reported for work.

                      Beyond the annual report, CSA reports on its success in meeting bonus goals.20
                      CSA reported it met its 2008 goals of creating a certified list for non-exempt
                      (hourly) and exempt (salary) positions within 30 and 45 days, respectively.
                      Specifically, CSA reported taking an average of 27.4 days to create a certified list
                      for non-exempt positions and 32.1 days to create a certified list for exempt
                      positions. Auditors‘ attempts to recreate these figures produced slightly higher
                      averages.

            Auditors attempted to review CSA‘s calculations but were unable to do so because CSA
            did not document their methodology for data verification or save the supporting data.
            Consequently, auditors performed a separate, more detailed analysis of a sample of
            recruitments conducted during the same period covered in CSA‘s reports.21 This analysis
            indicated a higher average time to create the certified list than CSA reported.

            Table 3 compares CSA figures to figures calculated by auditors. Specifically, the bonus
            goal for creating a certification list for non-exempt positions is 30 days. CSA reported an
            average time of 27.4 days, which qualified for the bonus, while auditors‘ sample showed
            an average time of 30.5 days, which would not have qualified for the bonus. The
            discrepancy resulting from this audit work highlights the importance of the data in
            NEOGOV, and the importance of CSA ensuring data used for reporting is accurate and
            reproducible.



 20
    Bonus goals are annual performance measures established by agencies. The City pays bonuses to employees when agencies
 meet their bonus goals.
 21
    CSA’s figures were for the time period January – October 2008. Auditors tested a sample from every other month in the same
 time period.
Cit y and Count y of Denver
                                                             P a g e 24
                                              Table 3: Measurement to Fill Positions

                         Measurement                                CSA Reported                Auditor’s Office
                                                                                                 Calculation

               ANNUAL REPORT: Average Time to fill                     29.5 Days22                  79.9 Days
               a Position


               BONUS GOALS: Time to create a                            27.4 Days                    30.5 Days
               certification list for non-exempt
               positions

               BONUS GOALS: Time to create a                            32.1 Days                    35.4 Days
               certification list for exempt positions




         Additionally, audit work revealed the methods CSA used to calculate its averages
         produced inaccurate and unreliable results. For example, CSA used a pre-programmed
         NEOGOV report which pulls the first date CSA refers candidates to the agency (the end
         date of CSA‘s bonus goal calculation), not the date the actual candidate hired was
         referred. Audit work identified several instances where the report generated a date
         much earlier than the hired candidate‘s actual date referred – possibly skewing CSA‘s
         average calculations. For example, auditors identified a case where the NEOGOV report
         pulled July 10, 2008 as the date the candidate was referred to the agency, but upon
         verification, the candidate hired for this position was actually referred to the agency on
         September 24, 2008. Therefore, it is possible the date used for calculation in this case was
         more than two months shorter than the actual date the candidate was hired. The only
         way auditors were able to obtain 100% reliable data was to verify individual records by
         comparing the date referred in NEOGOV to the hire date in PeopleSoft.23 CSA did not
         consistently take this step in conducting the analysis for its reports and there is no way for
         auditors to know when CSA verified referred dates.

         Problems with this report may be eliminated if CSA utilized custom NEOGOV reports. CSA
         is currently making minimal use of this capability. Overall, accountability could be
         improved if CSA takes greater advantage of this function.

         Inaccurate Information Presented to the Board – CSA Management presented a large
         model of the recruiting workflow process to the Board to support a Rule change, but
         reported to auditors that the flowchart was inaccurate. On January 15, 2009, CSA
         requested a change to Rule 3-21 regarding the minimum required days for posting job
         positions. The previous Rule required that job postings appear for a minimum of 5 business


22
   This figure may represent the time to create a certified list. However, CSA was unable to provide documentation to
substantiate whether it represents time to create a certified list or time to fill.
23
   PeopleSoft is the City’s human resource and accounting software system.
                                                                                                         O f f ice of t he A udit or
                                                             P a g e 25
            days, whereas the new Rule reduced that to a minimum of two business days. CSA
            reported that the Rule change was necessary to limit the number of applications
            because processing time increases as the volume of applications increases.

            As CSA stated in this public hearing, a day would count towards the total only if the
            posting appeared for the full day. For example, if a job was posted on a Monday
            afternoon, Monday would not count as a day. This requirement would ensure that the
            posting appeared all day Tuesday and all day Wednesday with the job closing at
            midnight on Wednesday. Board members approved the change but requested that CSA
            update them on how many times jobs were posted for less than five days (Quick Posts),
            which supervisors requested the shorter postings, which positions they were for, and the
            impact of shortening the time.

            On March 19, 2009, CSA presented some of the requested information related to the
            impact of the Rule change. Audit work determined CSA did not report all of the
            requested information, and presented information that was inaccurate. Specifically:

                      CSA reported three Quick Posts had been made since the January Rule change.
                      However, audit work determined that a total of 12 positions were posted for less
                      than five days as of March 19; and

                      CSA presented the ―average applicants per position‖ figure which auditors could
                      not successfully recreate. It should be noted that the problems in re-creating this
                      data could be due to a recent archiving project in NEOGOV, but without the
                      original methodology used to create these figures it cannot be verified that this is
                      the case.

            Reporting must be accurate in order for key stakeholders to hold Recruiting accountable
            for its performance. Inaccurate information diminishes the ability of stakeholders to
            evaluate the overall effectiveness of Recruiting. Additionally, accurate information is
            necessary to determine whether CSA is adhering to the merit system.

 Performance Evaluations Are Vague and Not Timely

            Audit work identified several ways in which CSA management‘s review of staff
            performance provided limited accountability. First, the document most commonly used
            to discuss staff performance (the duty matrix) does not adequately define roles,
            responsibilities, standards, and expectations for Recruiters. The duty matrix contains a
            general breakdown of duties for which Associate, Senior and Recruiting Specialists are
            responsible. However, this matrix is deficient in several respects as an accountability tool:

                       The matrix does not contain any definitions or explanation of what each duty
                       entails. For example, the matrix lists both customer service and client
                       management—terms that, without further definition, can be interpreted
                       differently and confused easily. It is difficult to hold Recruiters accountable for
                       unclear duty assignments.

                       The matrix lacks an explanation of how it should be applied or how performance
                       will be measured. All Recruiters interviewed had negative comments about the

Cit y and Count y of Denver
                                                      P a g e 26
                    duty matrix, and seven of ten Recruiters stated the matrix was not an accurate
                    breakdown of duties. Without clarifying documents, the matrix is an ineffective
                    management tool.

         A second problem with staff accountability is that while better evaluation tools exist, they
         are not used to assess most Recruiters‘ performance. The duty matrix was the only
         standardized Recruiter performance evaluation tool provided to auditors which was
         used consistently in 2008.

         A third problem is none of the CSA Recruiting staff received their annual performance
         evaluations, called Performance Evaluation Plan Report (PEPR), in a timely manner.
         Specifically, ten of the 13 (77%) Recruiting staff members received a forced evaluation
         on their most recent PEPR.24 At the time of testing, the remaining three staff had been
         employed less than one year, so a formal evaluation was not due. Performance
         appraisals provide several important benefits for employees, including feedback on
         performance and areas for performance improvement. They also benefit the employer
         by helping improve employees‘ understanding of job responsibilities and standards of
         performance, thereby providing management with a means to hold them accountable.

         Data obtained from CSA indicated the timeliness of performance evaluations is an issue
         citywide. It appears the City‘s framework to hold supervisors accountable for completing
         PEPRs in a timely manner is inadequate. Audit work found that the State of Colorado has
         laws in place to hold supervisors accountable when they do not complete evaluations.
         These include progressive discipline for each year the evaluation is not completed,
                                                    ranging from suspension (unpaid days off) to
                                                     demotion.25 The Rules state that supervisors
                Timeliness of performance
                                                     failing to complete PEPRs within 30 days of the
             evaluations is a citywide problem.
                                                     due date ―may be subject to discipline.‖26 This
                                                    rule is vague about what kind of disciplinary
         actions may be used and leaves this responsibility up to the agencies. The agencies are
         not required to do anything. While CSA does not have the authority to insist on timely
         PEPRs citywide, it does have the authority to request a citywide Rule change to address
         these problems.

Quality Control and Oversight Processes Need Strengthening

         Audit work found existing quality controls over the recruiting process are vague and
         largely undocumented. Recruiting has developed a general reporting structure for
         quality review of Recruiters‘ work, but no way to monitor whether reviews occur.
         Recruiting management was able to list some items that should be checked during
         quality reviews, such as spelling and content accuracy of job announcements, but could
         not provide any written documentation of such reviews or the requirements. The lack of


24
   A forced evaluation is when City employees do not receive their formal annual evaluation within 30 days of the due date. As a
result, the employee is automatically rated as “Successful” and given the corresponding pay increase.
25
   C.R.S. § 24-50-104(1)(c.5) (2009).
26
   City and County of Denver Career Service Board Personnel Rules, Rule 13-61 (C4): Merit Date, Effective January 1, 2008,
http://denvergov.org/HumanResources/RulesandPolicies/tabid/432517/Default.aspx (accessed June 10, 2009).
                                                                                                          O f f ice of t he A udit or
                                                              P a g e 27
            a well defined and documented quality control process limits CSA‘s knowledge of
            performance, their ability to hold individual personnel accountable, and increases the
            risk of inaccuracies.

            Further, audit work found there is a lack of quality controls in the testing process. For
            example, CSA does not appear to perform consistent quality reviews of exams to ensure
            testing effectively measures applicant competency. Additionally, although Recruiting
            had some policies and procedures related to checking written exams out and back in
            for testing applicants, audit work found recruiting personnel were not adhering to them.
            The absence of quality controls in the testing process is a significant problem because it
            increases the risk of litigation related to unfair testing procedures.

 CSA Governance Can be Enhanced

            CSA was specifically established as an independent agency to limit inappropriate
            political influence, but because of this independence, the CSA Board should actively
            provide oversight and hold CSA accountable for carrying out the agency‘s mission and
            accomplishing its goals.

            Action is needed on multiple fronts to strengthen the City‘s recruiting function. The
            preceding sections have detailed a number of issues that need addressing. In many
            respects, these issues are interrelated. For example, while better accountability
            mechanisms are needed, these mechanisms will not work well if good performance
            indicators are not in place. Simply conducting more performance evaluations and
            quality reviews will accomplish little if there are no indicators that measure what the
            process is accomplishing. However, even the best performance indicators are of limited
            use if employees are not consistently entering data into the NEOGOV system or making
            use of the system‘s reporting capabilities.




 RECOMMENDATIONS
            The following recommendations are designed to help CSA and its Board address issues
            identified by audit work. Recommendations are grouped to reflect the sequence of
            topics discussed in the report, but in some cases recommendations made under one
            heading may help address matters under other headings as well.

 Establishing Strategic Direction and Planning

            1. To better establish strategic direction and planning for the recruiting function, CSA, in
            consultation with the Board, should take the following steps:

                      Establishing a clear, actionable definition of the merit system and making it a part
                      of agency documents—for example, in the purpose statement or the
                      introductory statement of the CSA Rules. The definition of the merit system should
                      be detailed enough to identify when Rules are appropriate and how it
                      specifically applies to all personnel issues;

Cit y and Count y of Denver
                                                      P a g e 28
             Consider increasing the minimum job posting time from two to five business days
             to align with the federal government and other benchmarking jurisdictions;

             Developing, in a single document, a strategic plan that includes measurable
             goals and reportable performance measures that can be tracked over time; and

             Ensure all Recruiters are aware of materials and tools available to promote CSA‘s
             branding strategy.

Improving the Timeliness of Recruiting

      2. CSA should evaluate the full recruiting and hiring process in an effort to identify steps
      where efficiencies could be realized. These steps could include:

             Identifying steps that hiring agencies can take to shorten their portion of the hiring
             process;

             Pursuing rule changes that allow for quick hiring, such as authorizing agencies to
             hire their own on-call positions or utilize temp agencies; and

             Starting initial recruitment steps at the time the hiring agency decides to fill an
             open position rather than waiting until the Budget and Management Office
             determines funding for the position is available.

Making Policies and Procedures More Thorough and Consistent

      3. To ensure greater consistency in the recruiting process and integrity in data collection,
      CSA should create written policies and procedures that cover the following:

             Explaining when Recruiters may eliminate steps in the recruiting process;

             Specifying the documentation to be attached in NEOGOV to ensure consistent
             and complete recruiting records;

             Specifying procedures for developing eligible and certified lists;

             Specifying when subject matter experts should be used in applicant testing,
             including how many should be used;

             Specifying what information should be consistently entered and how to enter it
             into NEOGOV, including recruitments that utilize an existing certified list;

             Clarifying when and how it is appropriate to use special or preferred
             qualifications and promote usage by the hiring agency, including:

                 o   Using special qualifications to establish clear and concise requirements for
                     the open position and help limit the number of applications from
                     candidates that clearly do not meet the qualifications needed for the
                     position.

                 o   Ranking eligible candidates based on preferred qualifications for the
                     certified list, in conjunction with test scores; and



                                                                                  O f f ice of t he A udit or
                                              P a g e 29
                      Clearly documenting the development of supplemental test questions and
                      specifying that supplemental questions are standardized for like job postings, to
                      the degree possible.

            4. CSA should seek a Rule change establishing an expiration date for certified lists to
            ensure that these lists are not used beyond the expiration of the eligible lists.

            5. To better ensure the security of test books, CSA should take the following steps:

                      Perform an inventory of the test vault in order to create a baseline for future
                      inventories, establish a set number of test books for each examination, and
                      maintain documentation of test books that are destroyed;

                      Consider replacing manual test books with computer based or video based tests.
                      Video based tests have been shown to produce a more positive response from
                      candidates, and reduce adverse impact;

                      Eliminate pass points on written tests. By eliminating the pass point, each eligible
                      candidate has the opportunity to move forward in the hiring process; and

                      To address test security and pass point issues, CSA could also consider eliminating
                      testing from the recruiting process, or keeping testing for positions only when
                      necessary, such as for positions that may require physical demonstration of a
                      needed skill.

 Strengthening Accountability

            6. To better assess the efficiency and effectiveness of the recruiting function, CSA should
            develop specific performance measures that coincide with those identified by best
            practices. At a minimum, these measures should include time to hire, turnover rates, and
            cost per hire. Other important measures and steps include probation passing rates, time
            spent per recruitment and analyzing the various components of the recruiting process to
            identify which parts are taking longer. CSA should then incorporate performance data
            findings into management practices.

            7. To ensure information for performance measures and reports is accurate and
            consistent over time, CSA should take the following steps:

                      Ensure the methodology used to produce data is documented and can be
                      reproduced. Additionally, backup data behind figures reported publicly should
                      be retained to support such figures;

                      Verify all information before presenting it to the Board or reporting it to City
                      officials; and

                      Measure them consistently over time.

            8. To improve personnel performance evaluation, CSA should:



Cit y and Count y of Denver
                                                        P a g e 30
       Develop clear, detailed, and measurable roles and responsibilities for Recruiting
       personnel. This should include Recruiters as well as HR and Testing Specialists.
       Duties should be better defined for each level of Recruiter, including
       expectations for how each duty should be performed and how performance will
       be measured; and

       Identify specific disciplinary actions to hold management accountable when
       PEPRs are not completed in a timely manner.

9. To strengthen monitoring of job processes, CSA should develop quality controls, to
include:

       Documentation on what should be reviewed, who should review it, at what
       points in the recruiting process reviews should happen, how often it should occur
       and how it will be monitored;

       Monitoring compliance with policies and procedures; and

       A specific quality control process for ensuring data reliability and accuracy.

10. To help analyze and interpret information about the recruiting process, CSA should
increase its use of NEOGOV‘s custom reporting tools. At a minimum, create reports for
job posting data and requisition life cycle data to ensure figures reported are accurate
and useful.




                                                                          O f f ice of t he A udit or
                                       P a g e 31
 APPENDIX A
 City Positions Not in the Career Service (Charter § 9.1.1 E)
      The Career Service shall comprise all employees of the City and their positions except:
      (i) elected officers;
      (ii) members of the Mayor's cabinet;
      (iii) the Director of Excise and Licenses;
      (iv) up to fifty employees appointed to serve at the pleasure of the Mayor in positions
      specifically designated or created by the Mayor in any department or agency of the City
      under the direct control of the Mayor;
      (v) county court judges and magistrates;
      (vi) members of the Classified Service of the Police and Fire Departments, the Police Chief if
      not a member of the Classified Service, and the Undersheriff;
      (vii) attorneys and part-time employees employed by the District Attorney, other employees
      of the District Attorney excluded from the Career Service and placed in an alternate merit
      personnel system pursuant to state law, and up to ten employees appointed to serve at the
      pleasure of the District Attorney in positions specifically designated or created by the District
      Attorney in the District Attorney's office;
      (viii) certified public accountants employed by the Auditor and up to five employees
      appointed to serve at the pleasure of the Auditor in positions specifically designated or
      created by the Auditor in the Auditor's Office;
      (ix) employees of the Denver Art Museum, the Denver Museum of Nature and Science, the
      Denver Zoological Gardens, and the Denver Botanical Gardens;
      (x) persons retained on a contractual basis to perform professional or technical services for
      limited periods of time;
      (xi) employees of the City Council, Library Commission, Civil Service Commission, Board of
      Adjustment, and Denver Water; and
      (xii) any hearing officers and up to two employees in positions specifically designated or
      created by the Career Service Board, appointed to serve at the pleasure of the Board.
      (xiii) any employee appointed to serve at the pleasure of the mayor for the purpose of
      monitoring internal investigations and disciplinary actions in the Department of Safety, and
      any employees appointed by the monitor to serve at the pleasure of the monitor. The
      appointment of any monitor by the mayor pursuant to this or any other provision of the
      charter shall require confirmation by the city council.
      (xiv) the Director of Elections and no more than one other employee in a position specifically
      designated or created by the Clerk and Recorder, appointed to serve at the pleasure of the
      Clerk and Recorder. Any employee of the Denver Election Commission as of July 16, 2007
      and formerly excepted from the Career Service pursuant to this section shall retain his or her
      position as an employee of the Clerk and Recorder if the employee qualifies to retain the
      position in accordance with the rules of the Career Service Board.




Cit y and Count y of Denver
                                                   P a g e 32
AGENCY RESPONSE




                               O f f ice of t he A udit or
                  P a g e 33
Cit y and Count y of Denver
                              P a g e 34
             O f f ice of t he A udit or
P a g e 35
Cit y and Count y of Denver
                              P a g e 36
             O f f ice of t he A udit or
P a g e 37
Cit y and Count y of Denver
                              P a g e 38
             O f f ice of t he A udit or
P a g e 39
Cit y and Count y of Denver
                              P a g e 40
             O f f ice of t he A udit or
P a g e 41
Cit y and Count y of Denver
                              P a g e 42
             O f f ice of t he A udit or
P a g e 43

								
To top