Belzon metric

Document Sample
Belzon metric Powered By Docstoc
					                 Performance Based Logistics
                      Metrics Workshop
                      SOLE Conference
                       15-16 May 2007

                                        Terri Schwierling
                                        Chief, PBL & Analysis Branch
                                        Maint Spt Div, Maint Dir
                                        IMMC, AMCOM

15-16 May 2007
                 Performance Based Logistics
                       Metric Workshop
                     Topics of Discussion
     • Metric Policy and Guidance
     • Metrics Relevant to Warfighter
     • Metric Selection
     • Metric Assessment
     • Metric Weights
     • Summary

15-16 May 2007
                      Life Cycle Sustainment Outcome

     • Materiel Availability (Key Performance Parameter)
       – % of Entire Population Operational
     • Materiel Reliability (Key System Attribute)
       – Mean Time Between Failure
     • Ownership Cost (Key System Attribute)
       – O&S Costs Associated With Materiel Readiness
     • Mean Down Time
       – Average Total Downtime Required to Restore Asset
         to Operational Condition

* JROC Established Materiel Readiness/Sustainment Key Performance Parameter
  & Key System Attributes (DUSD (L&MR) Memorandum Dated 10 Mar 07)
     15-16 May 2007
                           PBL Policy and Guidance

                     Performance Based Metrics and Reporting
Overarching Metrics                       Most Frequently Used Metrics              Reporting
Operational Availability         Operational Availability                      Semiannual Report
Mission Reliability              Operational Readiness        Inherent Only:      DA and AMC
Cost Per Unit Usage              Fill Rate                     Reliability     Automated System TBD
Logistics Footprint              Customer Wait Time            Obsolescence
Logistics Response Time          Backorder Reduction

                                              Areas to Consider
                    Measurable Metric
                    Each Metric Does Not Require A Separate IPT
                    Data Collection
                    Automated Reporting Data Base Impact to Resources
                    Multiple Sub-Metric Options

   15-16 May 2007
                                                         PBL Metrics
                                      Define the process
                                • Determine customers, inputs,
                                  outputs, value-added
                                • Use walkthrough to achieve
                                  common understanding

                                                Measure process performance
                                           •   Define metrics and identify data
                                                                                      Iterate for
                                                                                                      UAVS PMO PBL Metrics
                                           •   Determine baseline performance
                                           •   Diagnose performance drivers
                                           •   Provide reports and feedback
                                                                                                      Metric             Max Rating
                                                            Improve the process
                                                                                                    System Status        85% or Higher
                                                       • Establish goals
                                                       • Develop improved process                   Readiness (SSR)
                                                       • Implement changes
                                                                                                    Customer Wait Time   90% or Higher
                     Ao                                                                             (CWT)

                           OR                                                                       Logistics            12/1 or Higher
                                                                                                    Maintenance Ratio
                                                         What are we Buying ?                       (LMR)
             CWT     FSR        SSR
                                                                                                    Field Service         Satisfactory
                                                                                                    Performance (FSR)
      MTTR        MTBF     MTBEFF         LMR

Performance Metrics Pyramid

 15-16 May 2007
      Intelligent Workload Allocation:
         Flexibility & Unity of Effort
PSI Workscope:
• Contractor Managed Supply
• Contractor Managed
  Maintenance Support
• Field Support Representatives
• Sustainment Engineering
                                      Readiness and
• Brigade Integration Team (BIT)        Availability
• Deployment Support
     (CONUS/OCONUS Trng Exercises,
     OIF/OEF, etc)

       Scope has to match the desired outcomes.
15-16 May 2007
                                             SAMPLE PBL Metrics
                                              to ORD Crosswalk
              Performance Based                                               PBL Metrics Crosswalk to
            Product Support Metrics                                                    ORD
• SSR – System Status Readiness                                            • Ao = Availability of 85%
               Defined as   Total Time – Down Time (at Sub system Level)
                                 Total Time                                            SSR and CWT

• CWT – Customer Wait Time                                                 • Mean Time Between System Abort
                                                                              20 Hrs Threshold – 57 Hrs Objective
               Defined as   Total Req’s - # of Unsuccessfully Filled
                                 Total Req’s                                           SSR and LMR
• FSR - Field Service Representative Quotient                              • Mean Time to Repair (MTTR)
               Defined as    Customer Satisfaction quotients
                             evaluated via CSAP Report                          .5 hrs (AVUM)/ 2.0 hrs (AVIM)

• LMR - Logistics Maintenance Ratio                                                 SSR, CWT, and FSR

               Defined as    Total Operating Hours                         • Operational Readiness (non-ORD)
                        # of Unscheduled Maintenance Actions                 of 90%
                                                                                     SSR, CWT, and FSR

  15-16 May 2007
                           Scoring Methodology

•       Performance Metrics are weighted as follows:
                        SSR                       50%
                        CWT                       25%
                        FSR                       20%
                        LMR                       5%

•       A Wrap up/Composite Score is calculated to determine incentive award (Fee),
        on a quarterly basis, as follows:

                     (SSR x .50) + (CWT x .25) + (FSR x .20) + (LMR x .05)

    15-16 May 2007
                                 Incentive Model
                          Minimum Acceptable Performance
1.       The Composite Wrap Up Score is calculated and Fee is then determine via the
         following Table found in Section H of the PBL Contract.
                                                    Wrap Up    Min   Incentive   Total Incentive
2.       As an example, the PSI achieved a           100.0%   3.0%     9.0%          12.0%
                                                      99.0%   3.0%     8.9%          11.9%
         Composite Wrap Up Score of 91% –             98.0%   3.0%     8.8%          11.8%
                                                      97.0%   3.0%     8.7%          11.7%
         the incentive would be 3 % (MIN) +           96.0%   3.0%     8.5%          11.5%
         7.2 % (INC) = 10.2 %                         95.0%   3.0%     8.3%          11.3%
                                                      94.0%   3.0%     8.1%          11.1%
                                                      93.0%   3.0%     7.8%          10.8%
                                Plus                  92.0%   3.0%     7.5%          10.5%
                  Score .91                           91.0%
                                                      89.0%   3.0%     6.6%          9.6%
                                                      88.0%   3.0%     6.3%          9.3%
                                                      87.0%   3.0%     6.0%          9.0%
                               Minus                  86.0%   3.0%     5.7%          8.7%

                  Score .84                           85.0%
                                                      83.0%   3.0%     4.6%          7.6%
3.       In this example, the PSI missed the          82.0%   3.0%     4.2%          7.2%
                                                      81.0%   3.0%     3.8%          6.8%
         SSR and CWT scoring a Composite              80.0%   3.0%     3.4%          6.4%
         Wrap up Score of 84%. The                    79.0%   3.0%     3.0%          6.0%
                                                      78.0%   3.0%     2.6%          5.6%
         incentive would be 8.0 % due to              77.0%   3.0%     2.0%          5.0%
                                                      76.0%   3.0%     1.5%          4.5%
         missing the PBL metric requirements.         75.0%   3.0%     1.0%          4.0%

     15-16 May 2007
                 Effective Metrics, Performance
                 Management, and Continuous
                      Improvement Culture
PBL Process Review - 1st QTR FY 06
Focused on the processes required to achieve the PBL Metrics:
    • Scrutinized the processes used to maintain PBL Shop Stock .
    • Determined quantities of items in the ROR [Repair of Repairables]
      cycle and identified the processes currently used to move items
      expeditiously through the depot’s ROR line.
    • Reviewed all aspects of FSR Utilization (Training, Management,
      Technical Documentation, etc.)
    • Reviewed the current processes used to reduce the Mishap Rate and
      identified ways to enhance these efforts.
    • Identified and quantified Integrated Material Management Activities as
      they relate to all aspects of the PBL Effort.
    • Developed new Metrics that can reduce the Mishap Rate and improve
      process efficiency !

It’s really not about what you say. It’s about what
15-16 May 2007         you do.
                    System Status Readiness
                  (No change to current Metric)
       SSR= Total Time (TT) – Down Time (DT)
                     Total Time (TT)
       • Sub-systems that count against NMC criteria include:
                 3 of the 4 AVs
                 2 GCSs
                 2 GDTs
                 1 PGCS
                 1 PGDT
                 2 TALS
                 1 LAU
                 2 of the 4 RVT’s

        • A fielded system consists of 4 AVs. The fourth AV is considered a spare AV
        and shall not be included in SSR DT calculations for the first 30 days after the
        fourth AV, as a spare, is used to replace a NMC AV. After 30 days, if an
        additional AV becomes NMC that AV will be included in DT calculations unless
        the PSI has added back a fourth spare AV to the unit.

15-16 May 2007
             Current Metric Customer Wait Time (CWT)
                    Goal 90% Weighted at 25%
CWT= Total Number of Soldier Requisitions – Late Solider Requisitions (LR)
                   Total Number of Requisitions
                                         Terms And Conditions
                   Contractor Shall fill 90% of unit requisitions per the following CWT:
                   Priority Designator (PD)      CONUS (Days)            OCONUS (Days)
                        01 thru 03                    3                          7
                        04 thru 08                    7                         10
                        09 thru 15                    10                        15

                    CWT begins on the day the requisition is submitted by a soldier either
                    through the FSR or the Standard Army Retail Supply System (SARSS)
                    when it is implemented as evidenced by the document number. When
                    Government personnel, as evidence by the carriers shipping document,
                    receive the part CWT calculations end.

                    A LR is a soldier requisition that is not filled within the number of
                    days delineated in the table based upon the unit location (CONUS/OCONUS)
  15-16 May 2007
                      Customer Wait Time (CWT)

CWT= Total Number of Soldier Requisitions – Late Solider Requisitions (LR)
                   Total Number of Requisitions

            PROS                                  CONS
   Desired Result- Responsiveness       Unit Relies on FSR to Submit Requisition

   Measurable                           FSR Relies on Depot to Submit Requisition

   Data Collection Process In place     Delay with FSR closing out Requisition

   Known out come                       Misuse of Priority Designator

                                        Does not take into account demands filled
                                          thru repair process

                                        Government Transportation

 15-16 May 2007
                                     New Metric
                            Customer Response Time (CRT)
   CRT = Combination of Requisition Processing and Repair of Repairables (ROR)

                   Considerations             Recommendation for Supply and ROR
Goals                                       Count all Requisitions
    Meet Customer Requirements                 User
    Improve Inventory Control Process          FSR
    Improve Repair of Repairables Process      Depot
Measurable – Yes                            Implement Automated Inventory Control Process
Data Collection Process In Place – Yes      Update LMI Data Base with Provisioning Type Data
Achievable – Yes                               Address all Lead Times
Weight Factor Adjustment                       Recoverability Codes
Impacts                                     Refresher Training on UPAS
   Improve System Status Readiness          Reduce all ROR lead times by 20%
   Cost                                     Track all ROR actions in UPAS
   Funding for Long Lead Items              Implement Tracking Process for Unserviceables
   Competing Resources with Production      Identify Direct Exchange Items
                                            Subcontractor/Vendor Support Agreements
                                               Flow Metrics to subs
                                               Conduct Subcontractor/Vendor Conference
  15-16 May 2007
                              New Metric Consideration
                        Customer Response Time (CRT) – What we think it is !
     CRT = Combination of Requisition Processing and Repair of Repairables (ROR)
               Requisition Processing                                       ROR Completions

 • Include only requisitions for LRU/SRU’s                     • Determine Top 15 ROR Items (by Quantity/Cost)
   (Short list – few consumables)                                OEM Action to Provide List NLT Wed at 0830
  Decision made not to use LRU/SRU or RPSTL.                    Govt will review list during telecon
  Recommendation is to include all field requisitions          • Recommend a short list of top 15 Items to put a
 • Include only requisitions coming from                         metric against
   the ‘field’ do not count requisitions                       • Establish Standard Repair Time
   coming from OEM Depot                                        OEM to provide standard repair time for each item on the ROR list
 • Include all other Priority 01 thru 03                        above. Govt will review during the telecon and determine concurrence
                                                                or non-concurrence.
  Decision made to include all priorities.
  Notes: Clock starts date/time of requisition. Clock stops
                                                               • Track these ROR actions in UPAS using
   when shipment tracking number is entered (which means         Standard Repair Time for Due out Date
   customs clearance is received). Success is 90% completion    Notes: Clock starts when item transfers into ROR warehouse (UPAS)
   fill on time.
                                                                and is assigned a PO or MO. Standard Repair Time completion date is
                                                                entered into the UPAS data base at the same time item is transferred to
                                                                ROR warehouse. Success is 90% completion of repair with in the
                                                                established standard repair time.

CRT= Total Number of Requisitions and ROR’s – Late Requisitions and ROR’s
                    Total Number of Requisitions and ROR’s
   15-16 May 2007
                   Result of Customer Response Time (CRT) Discussions
                                        2 Metrics
                     Customer Wait Time and Depot Maintenance Ratio
                Customer Wait Time (CWT)                     Depot Maintenance Ratio (DMR)

CWT= Total Number of Requisitions – Late Requisitions         DMR=     Total Hours
          Total Number of Requisitions                            # of Open Maintenance Actions

   • Includes all field requisitions                    • Maintenance Actions Per Flight Hour
   • Includes all Priorities                            • Improve ROR TAT and Reduce Backlog
   • Success is 90% completion fill on time             • Includes all Depot Level Maintenance
   • Incorporate Data Field to track                      actions not a specific list.
     complete Requisition TAT                           • Actions: - Establish Ratio
                                                                   - Starting point (snap shot of data)
                                                                   which should include all depot
                                                                   ROR backlog
                                                                   - Run Data Analysis

       15-16 May 2007
                                          Current Metric
                                  Logistics Maintenance Ratio
                                 Goal 8 thru 12:1 Weighted at 5%
            LMR=                        Total Hours
                            # of Unscheduled Maintenance Actions (UMA)

                                      Terms And Conditions
                 The contractor shall maintain a LMR of 8 through 12:1

                 TH= Total System Flight Hours. All fielded system TM shall be summed into a
                 Quarterly total.

                 UMA= Maintenance actions that are required to return the systems or subsystem
                 to a mission capable status due to component failure beyond the routine preventive
                 maintenance actions defined in the UAVS TMs. UMAs include stand alone UMAs
                 and, in some case of the parent child MAs, only the subordinate UMAs will be counted.

                 The LMR shall be calculated using both actual data and data obtained by analysis when
                 Recommended engineering changes have not been funded for implementation.

15-16 May 2007
                        Logistics Maintenance Ratio

                 LMR=               Total Hours
                        # of Unscheduled Maintenance Actions (UMA)

          PROS                                         CONS
  Measurable                                  Engineering Services currently not included
                                                  in PBL Contract
  Data Collection Process In place
                                              Requires up front investment

15-16 May 2007
                              New Metric Consideration
                            Logistics Maintenance Ratio (LMR)
                       LMR=                     Total Hours
                                    # of Unscheduled Maintenance Actions (UMA)
                   Considerations                               Recommendations
Goals                                                  Include Engineering Services in PBL
   Reduce Mishaps                                      Cost Study/Trade Analysis
   Reduce Unscheduled Maintenance Actions              Include Mishap Investigation in PBL
   Improve Reliability
   Improve Maintainability
Data Collection Process
Adjust Weight Factor
   Cost Sharing Associated with
        Mechanical Failures
   System Status Readiness

  15-16 May 2007
                                   Current Metric
                         Field Service Representative (FSR)
                         Goal Satisfactory Weighted at 20%
                       FSR = Field Service Representative Quotient
                        Defined as Customer Satisfaction Quotient
                                      Terms And Conditions
                 The parties agree that the contractor shall achieve a minimum
                 Satisfactory FSR rating:

                 The FSR Customer Assessment Report (FCA) will be used to determine
                 the FSR quarterly rating. All FCA reports will be summed and a straight
                 Average will be used to determine FSR rating.

                 Ratings on the FCA report are 5=Outstanding
                                               2=Below Average

                 The contractor shall only retain FSR Teams that maintain a performance
                 rating of “Satisfactory or better on the FCA appraisal”
15-16 May 2007
                 Field Service Representative (FSR)

                FSR = Field Service Representative Quotient
     Defined as Customer Satisfaction Quotient evaluated via CSAP Report

          PROS                                     CONS
  Customer Feedback                       Subjective not Quantitative

  Measurable                              Limited feedback from OIF Units

  Data Collection Process In place        Minimum measurement of FSR performance

  Subjective                              FSR have limited control of unit operations

  FSR Represent OEM and GOV PMO

15-16 May 2007
                       New Metric Consideration
            Field Service Representative FSR Accountability

                             FSR Performance = Mishap Reduction

            Considerations                              Recommendations
Goals                                          Enhance FSR Training
      FSR Accountability                       Measure FSR by UPAS Transactions
      Reduce FSR Footprint                     Measure FSR by Mishap per flight hour
      Minimize and Reduce Mishaps              Improve MOA with Gaining Units
      Mentor Unit                              Reduce FSR Footprint
      Increase Data Collection via UPAS          Collocated Unit
Measurable                                       Division FSR Support
Data Collection Process
Ability to influence unit/ unit leadership
Incentivize FSR that take on Challenging
      System Status Readiness
      FSR Personnel Turn Over/Transfer to
         another unit
   15-16 May 2007
                 Reliability Growth Rate (RGR)
•  Identify Reliability Growth Rate Slope over a
  two year period with quarterly goals
• Government would like to see a cost share on AV
  mishaps that fail above the RGR goal
• OEM interested in a cost share against the incentives
• Actions:
        – OEM RGR Slope under review by GOV PMO engineering staff.
        – Need final agreement on RGR
        – Establish desired weight of this metric and cost share approach

15-16 May 2007
                      Cost Sharing Clause
g. Cost Incentives

The parties hereto agree that the final incentive fee earned by the performance incentive process
stated above will be reduced by $.50 for each dollar the actual cost (In accordance with the
contractor's latest monthly project fiscal report) exceeds the target cost of the contract until the
minimum fee shown in the fee pool shown in paragraph d above is reached. The fee earned from the
performance incentive process above will be increased by $.50 for each dollar the final cost is less
than target cost until a maximum fee of 12 percent (The maximum fee shown in the fee pool in
paragraph d plus an additional one percent of the target cost) is reached.

  15-16 May 2007
                    Alignment to Known
                   Warfighter Requirements
                                                                        ORD Requirements and
    UAVS PBL Metrics Definitions                                       Corresponding PBL Metrics
• System Status Readiness (SSR)                                    • Ao = Availability of 85%
  Defined as      Total Time – Down Time (at Sub system Level)
                           Total Time
                                                                            SSR, CWT, and RGR

• Customer Wait Time (CWT)                                         • Mean Time Between System Abort
                                                                        20 Hrs Threshold – 57 Hrs Objective
  Defined as      Total Req’s - # of Unsuccessfully Filled
                           Total Req’s
                                                                                SSR and RGR
• Depot Maintenance Ratio (DMR)                                    • Mean Time to Repair (MTTR)
   Defined as Total Flight Hours current QTR
           # Open Depot Maintenance Actions
                                                                          .5 hrs (AVUM)/ 2.0 hrs (AVIM)

                                                                             SSR and CWT
• Reliability Growth Rate (RGR)
                                                                   • Operational Readiness
  Defined as Performance against a Reliability Growth Curve             (non-ORD) of 90%

                                                                             SSR, CWT, and DMR
         Colors show where
         PBL metrics                                         Support          ORD Requirements!
 15-16 May 2007
                    Metric Weights
                                  Current           Recommended ?
– System Status Readiness (SSR)   50 %                  50%

– Customer Wait Time (CWT)        25%                   10%
– Depot Maintenance Ratio (DMR)                         10%
– Reliability Growth Rate (RGR)   25% (FSR & LMR)        30%

15-16 May 2007
                                Scoring Methodology

        •        Performance Metrics are weighted as follows:
                            SSR             45%
                               RGR                 30%
                               CWT                   5%
                               DMR                 20%

        •        A Incentive Score is calculated to determine incentive award
                 (Fee), on a quarterly basis as follows:
                 (SSR x .45) + (RGR x .30) + (CWT x .05) + (DMR x .20)

                  To facilitate the change from Cost contracts to Fixed / Incentive
                 Fee type contracts – The FY 06 efforts will review collected actual
                  cost during each scoring meeting to determine how much cost
                   can be quantified and compared to the definitized cost target
15-16 May 2007
                 RAW Metric Performance Score Table
                         RAW Metric Performance Scores
                                                                            Incentive Score
                                                         RGR Score (Rate       Points (ISP)
 CWT Score         SSR score            DMR score             Target)
 100             99-100            >50              <9% of target           120
 99              97-98             41-49            10% - 29% of target     115
 98              94-96             35-40            30% - 49% of target     110
 97              92-93             29-34            50% - 69% of target     105
 95-96           90-91             25-28            70% - 79% of target     100
 93-94           88-89             21-24            80% - 89% of target     95
 91-92           86-87             19-20            90% - 99% of target     90
 89-90           84-85             17-18            100% - 109% of target   85
 86-88           81-83             15-16            110% - 119% of target   83
 83-85           76-80             13-14            120% - 139% of target   80
 81-82           71-75             11-12            140% - 159% of target   75
 76-80           66-70             10               160% - 189% of target   70
 71-75           61-65             9                190% - 219% of target   60
 66-70           56-60             8                220% - 269% of target   50
15-16 May 2007
 61-65           51-55             7                270% - 319% of target   40
                       Determination of Incentive
                        Based on Performance
The Composite Incentive Score (IS) is calculated and Fee is then determined
via the following Table found in Section H of the PBL Contract.

      Example 1: The PSI exceeded the           Incentive Score   Incentive
                                                       100          12.0%
      metrics and the Incentive Score is 93.            99          11.0%
      Therefore, the incentive would be                 98          10.5%
                                                        97          10.0%
      8.0%.                                             96           9.5%
                                                        95           9.0%
                             Exceeded                   94           8.5%
                              Metrics                   93           8.0%
                                                        92           7.5%
             IS = 93                                    91           7.0%
                                                        90           6.5%
                                                        89           6.0%
                                                        88           5.8%
                                                        87           5.6%
                          Achieved                      86           5.4%
                           Metrics                      85           5.2%
                                                        84           5.0%
            IS = 85                                     83           4.8%
                                                        82           4.6%
                                                        81           4.4%
                                                        80           4.2%
                                                        79           4.0%
                                                        78           3.8%
      Example 2: The PSI achieved all                   77           3.6%

      metrics – the IS score is 85. The                 76
      incentive would be 5.2%.                          74           3.0%

  15-16 May 2007
• Metric selection must support the scope
  of the PBL program and the desired
• Specific details in the metric calculations
  must be clearly defined and measureable
• Constant improvement process to achieve
  performance outcomes

15-16 May 2007

Shared By: