Step-by-Step Instructions

Document Sample
Step-by-Step Instructions Powered By Docstoc
					                               WFA Step-by-Step Instructions
                for the 2011 INDEPENDENT REVIEW Workbook Preparation
Introduction     The most important product you will deliver as an Examiner is your scorebook—your
                 written evaluation of an Award applicant. Scorebooks are the basis for Feedback
                 Reports that ultimately end up in the hands of Award applicants. Feedback Reports
                 provide applicants with a Criteria-based assessment of their strengths and
                 opportunities for improvement (OFIs). These instructions take you through a process for
                 completing your Independent Review. The observations you develop at this stage will
                 become inputs for a team of Examiners to use in developing a Consensus Scorebook.
Before You        Schedule around 40 hours to complete your workbook. Please track your time.
Begin             Review the materials in your evaluation packet and complete the Conflict of
                     Interest Form. Send back to WFA immediately
                  Make sure your login to WFAdepot is active
                    Your team leader will post Independent Review Worksheets with the key factor
                    worksheet completed in wfadepot.

Steps                Review the Criteria booklet.
(Details for         Read the application thoroughly before you begin writing!
each step are        Your team leader will provide a customized Independent Review workbook
on the               Review the list of key factors provided by your team lead.
following            Develop a list of initial impressions that will be part of the Consensus Key Themes
pages.)              Read and begin to evaluate each assigned Process Item (Categories 1–6).
                     Analyze the applicant’s response to each Process Item. On an Independent
                       Review Worksheet, record strength and gap/OFI statements in order of their
                       importance.
                     Determine priority statements, and bold the text on the worksheet.
                     Read and begin to evaluate each assigned Results Item (Category 7).
                     Analyze the applicant’s response to each Results Item. On an Independent
                       Review Worksheet, record strength and gap/OFI statements in order of their
                       importance.
                     Determine priority statements, and bold the text on the worksheet.
                     Review all of your Independent Review Worksheets.
                     Record the information requested on the front cover of the workbook.
                     Upload your workbook to the secure WFA depot, www.wfadepot.com




                                                                        Step-by-Step Independent Review   1
Detailed Steps   Use the instructions in the following table to complete your scorebook. STEPS are
                 general; ACTIONS include details and notes.




   STEPS                                               ACTIONS
Review the       Review these sections of the Criteria booklet (Business/Nonprofit, Education, or Health
Criteria         Care) used by the applicant:
booklet.             Core Values and Concepts
                     Criteria requirements
                     Criteria Response Guidelines
                     Scoring System and Scoring Guidelines
                 (Note: Also review the changes page in the booklet, as the 2009–2010 update included a
                 major revision of the Criteria.)
Read the               Read the entire application, highlighting and/or noting the applicant’s key
application.        processes—the methods used to address Item requirements—and noting results
                    you would expect to see related to these processes.

Review list of
key factors.            Definition: A “key factor” is an attribute of an organization or its
                        environment that influences the way the organization operates and the
                        key challenges it faces. Examiners use key factors to focus their
                        assessments on what is important to the applicant.

                    Read the Criteria questions for Items P.1 and P.2.
                    Read the applicant’s Organizational Profile, which responds to those questions.
                    Review the Key Factors Worksheet already filled in by your team lead for your
                     assigned applicant
                       For a Key Factors Worksheet Sample, see the Examiner Learning Resource
                       Center (www.baldrige.nist.gov/Examiner_Resources.htm).
Develop a list    Record your initial overall impressions of the applicant. You might record
of initial         perceptions or observations that recur throughout the application, across processes
impressions.       and results, reflecting major strengths, opportunities, or vulnerabilities. During
                   Consensus Review, these impressions may evolve into key themes.
Read and
begin to
evaluate each
Process Item
(Categories 1–
6).




                                           PROCESS ITEM EVALUATION
                        For each Item, complete the following six-step evaluation process, and
                        record statements and evidence on the Independent Review Worksheet.
                                                                          Step-by-Step Independent Review   2
STEPS                                         ACTIONS
        1. Read the Criteria Item to refresh your understanding of the requirements.

        2. Review the relevant key factors provided on the Independent Review
           Worksheet.
               — Review the four to six key factors provided that would most influence the
                 applicant’s response to the Item requirements. These key factors are a subset
                 of those listed on the KF worksheet and may even be a subset of one key
                 factor (e.g., one strategic challenge that is relevant to the Item rather than all
                 strategic challenges). As you read and work with the application add any
                 other key factors that you observe. This does not happen frequently but
                 sometimes they emerge in the text of the application rather than in the OP, or
                 the intent to apply information.
        3. Read the relevant section of the application.
        4. Identify the processes or approach the applicant uses to meet Item
           requirements.
        5. Flag, mark up, and/or take notes as needed.
               — On the application, note measures/indicators you expect to see reported in
                 Category 7—Results.
               — Note ideas, threads, or patterns that recur in multiple Items or Categories.
               — As you take notes, record the applicant’s own words rather than using your
                 own words.
        6. Develop statements of the applicant’s strengths and gaps/OFIs for the Item,
           listing them in order of their importance to the applicant.
               — In the first column, record the number(s) of the relevant key factors for each
                 strength and gap/OFI statement.
               — In the second column, state the applicant’s strengths or gaps/OFIs for this
                 Item, based on the applicant’s key processes, approach, or methods (e.g., how
                 senior leaders deploy vision and values).
               — In the third column, record the evidence you found for each strength or
                 gap/OFI (e.g., using quarterly newsletters and monthly meetings).




                                                                   Step-by-Step Independent Review   3
   STEPS                                             ACTIONS
Note the          In the next four columns, check which of the evaluation factors—approach,
evaluation         deployment, cycles of learning, and integration—your statement addresses.
factors
addressed by   Approach (A)
each of your          Definition: “Approach” refers to the methods used by an organization
strength and          to address Process Item requirements. Approach includes the
gap/OFI               appropriateness of the methods to the Item requirements and to the
statements.           organization’s operating environment.

                      — Is the approach systematic (i.e., with repeatable steps, inputs, outputs, time
                        frames)?
                      — Is there evidence that the approach is effective, i.e., meets its intended
                        purpose, addresses what is being asked for, and meets the organization’s
                        needs?
                      — Is this approach (or collection of approaches) a key organizational process? Is
                        the approach important to the applicant’s overall performance?
               Deployment (D)
                      Definition: “Deployment” refers to the extent to which an approach is
                      applied in addressing Process Item requirements. Deployment is
                      evaluated on the basis of the breadth and depth of the application of the
                      approach to relevant work units throughout the organization.

                      — Is deployment addressed?
                      — What evidence is presented that the approach is used in one, some, or all
                        appropriate work units, facilities, locations, shifts, organizational levels, and
                        so forth?
                      — Is the approach something that has been in use for a long time or newly
                        introduced?




                                                                         Step-by-Step Independent Review    4
   STEPS                                              ACTIONS
Note the        Learning (L)
evaluation
                       Definition: “Learning” refers to new knowledge or skills acquired
factors
                       through evaluation, study, experience, and innovation.
addressed by
each of your           — Has the approach been evaluated and improved? If it has, was the evaluation
strength and             and improvement fact based and systematic (e.g., was it regular, recurring,
gap/OFI                  data driven)? (Examples of improvements are helpful. Look for them.)
statements             — Is there evidence of organizational learning (i.e., evidence that the learning
(continued).             from this approach is shared with other organizational units/other work
                         processes)?
                       — Is there evidence of innovation and refinement from organizational analysis
                         and sharing (e.g., evidence that the learning is actually used to drive
                         innovation and refinement)?
                Integration (I)
                    Definition: As a process evaluation factor, “integration” covers the range
                    from organizational “alignment” of approaches in the lower scoring ranges
                    to “integration” of approaches in the higher ranges.
                    “Alignment” refers to the consistency of plans, processes, information,
                    resource decisions, actions, results, and analyses to support key
                    organization-wide goals. It requires the use of complementary measures
                    and information for planning, tracking, analysis, and improvement at three
                    levels: the organization level, the key process level, and the work unit
                    level.
                    “Integration” refers to the harmonization of plans, processes, information,
                    resource decisions, actions, results, and analyses to support key
                    organization-wide goals. Effective integration goes beyond alignment and
                    is achieved when the individual components of a performance
                    management system operate as a fully interconnected unit.

                       — How well is the approach aligned with the organizational needs the applicant
                         identifies in other Criteria Items and the Organizational Profile?
                       — Is there evidence that the applicant uses complementary measures and
                         information for planning, tracking, analysis, and improvement on three levels:
                         the organizational level, the key process level, and the department or work-
                         unit level?
                       — How well is the approach integrated with organizational needs?
                           Examples of needs are generally listed as key factors: strategic
                           challenges, objectives, and related action plans; organizational
                           mission, vision, and goals; strategic advantages; key processes and
                           measures; key customer/market segments and requirements; and
                           workforce groups and requirements.

Indicate the       Indicate the Item Area(s) addressed by each of your statements (e.g., 1.1a[2]).
Criteria Item
Reference.


                                                                          Step-by-Step Independent Review   5
   STEPS                                              ACTIONS
Determine          Review your statements.
priority           Bold the text of priority strengths or gaps/OFIs—those that are most significant to
statements,         your evaluation of the applicant. (Bolding these observations will help your
and bold the        teammates understand what you thought was most important as they synthesize
text.               comments during Consensus Review.)

Read and
begin to
evaluate each                             RESULTS ITEM EVALUATION
Results Item           For each Item, complete the following six-step Item evaluation
(Category 7).          process, and record statements and evidence on the Independent
                       Review Worksheet.


                1. Read the Criteria Item to refresh your understanding of the requirements.
                2. Review the most relevant key factors on the Independent Review Worksheet.
                       — Review the four to six key factors provided (and numbered) that would most
                         influence the applicant’s response to the Item requirements. These key factors
                         are a subset of those listed on the KF worksheet and may even be a subset of
                         one key factor (e.g., one strategic challenge that is relevant to the Item rather
                         than all strategic challenges). As you read and work with the application add
                         any other key factors that you observe. This does not happen frequently but
                         sometimes they emerge in the text of the application rather than in the OP, or
                         the intent to apply information.
                3. Read the relevant section of the application.
                4. Highlight, flag, mark up, and/or take notes as needed.
                5. Refer to the measures or indicators for which you expected to see results.
                       — Note ideas, threads, or patterns that recur in multiple Items or Categories.
                       — Check for any expected measures that you noted during your review of
                         Categories 1–6.
                6. Develop statements of the applicant’s strengths and gaps/OFIs for the Item,
                   listing them in order of their importance to the applicant.
                       — Review all figures individually and group them as appropriate as you develop
                         your statements.
                       — In the first column, record the number(s) of the relevant key factors for each
                         strength and gap/OFI statement.
                       — In the second column, state the applicant’s strengths or gaps/OFIs for this
                         Item, based on your analysis of the results.
                       — In the third column, record the evidence you found for each strength or
                         gap/OFI (e.g., Figures 7.4-1, 7.4-2).
                           You do not need to write a statement for every figure in the application.
                           Instead, focus on figures that are responsive to the Criteria requirements
                           and address important customer, product and service, market, process,
                           and action-plan performance requirements identified in the
                           Organizational Profile and in Process Items, or identify the results that
                           are missing in these important areas.
                                                                          Step-by-Step Independent Review   6
   STEPS                                             ACTIONS
Note the       In the next four columns, check which of the evaluation factors—levels, trends,
evaluation     comparisons, and integration—you observed.
factors
addressed by   Performance Levels (Le)
each of your
strength and          Definition: “Performance levels” refer to numerical information that
gap/OFI               places or positions an organization’s results and performance on a
statements.           meaningful measurement scale. Performance levels permit evaluation
                      relative to past performance, projections, goals, and appropriate
                      comparisons.
                      — What levels are provided?
                      — Is the measurement scale meaningful?
                      — Are key results missing?
               Trends (T)

                      Definition: “Trends” refer to numerical information that shows the
                      direction and rate of change for an organization’s results. A minimum
                      of three data points generally is needed to begin to ascertain a trend.
                      — Are trends provided for few, many, or most Areas addressed in the Item
                        requirements?
                      — Is the interval between measures or frequencies appropriate?
                      — Are the trends positive, negative, or flat?
                      — What is the rate of change (slope of the trend)?
                      — Do the trends demonstrate little, some, or much breadth in the applicant’s
                        improvement efforts (i.e., how widely are they deployed and shared)?
                      — Are significant variations in trends explained in the text of the application?




                                                                        Step-by-Step Independent Review   7
   STEPS                                                  ACTIONS
Note the           Comparisons (C)
evaluation
                          Definition: “Comparisons” refer to how the applicant’s results
factors
                          compare with the results of other organizations. Comparisons can be
addressed by
                          made to the results of competitors, organizations providing similar
each of your
                          products and services, industry averages, or best-in-class organizations.
strength and
                          The maturity of the organization should help determine what
gap/OFI
                          comparisons are most relevant.
statements
(continued).              — Are comparisons provided?
                          — Are the comparisons to key competitors, industry sector averages, or best-in-
                            class organizations?
                          — How does the applicant compare against these other organizations?
                   Integration (I)
                          Definition: “Integration” refers to the extent to which results measures
                          (often through segmentation) address important customer, product and
                          service, market, process, and action-plan performance requirements
                          identified in the Organizational Profile and in Process Items; include
                          valid indicators of future performance; and are harmonized across
                          processes and work units to support organization-wide goals.
                          — To what extent do results link to key factors and Process Items?
                          — Are results segmented appropriately (e.g., by key customer, patient, or student
                            segment; employee type; process/education program or service; or geographic
                            location) to help the applicant improve?

Indicate the          Indicate the Area(s) addressed by each of your statements (e.g., 7.5a, b).
Item
Reference.
Determine             Review your statements.
priority              Make sure you have noted any results that are missing for areas of importance.
observations,         Bold the text priority strengths or gaps/OFIs—those that are most significant to your
and bold the           evaluation of the applicant.
text.
Record the
                      Record the information requested on the front cover of the workbook, including the
information
                       time you spent completing it.
requested on
the front cover.
Upload your           Upload your workbook to the secure Web site (www.wfadepot.com)
scorebook.




                                                                              Step-by-Step Independent Review   8

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:10/5/2012
language:English
pages:8