LESSONS LEARNED

Document Sample
LESSONS LEARNED Powered By Docstoc
					                               LESSONS LEARNED

Agency Name:                  Department of Licensing
Project Name:                 HP3000 Replatforming
Project Level:                Level 3
Date Presented to ISB:        March 6, 2008


Project Description: The Department of Licensing (DOL) successfully completed the
second of three projects that move agency computer applications off mainframe
computers and onto DOL’s State-compliant Windows server-based platform.
The HP 3000 Replatforming project migrated the Vehicles Field System (VHS) and
Heating Oil (HOAP) applications onto DOL servers as web-based systems during the
2005-2007 biennium. This was a time-critical change because Hewlett-Packard had
announced a December 2006 end-of-life for the HP3000 line of computers and software.
Without HP support, a computer or software failure would cause a state-wide shutdown
of all title and registration transactions for vehicles and vessels and halt the daily
collection of $2-3 million dollars of State revenue. A failure would also directly impact
the 39 County Auditor offices and 140+ subagent offices that get income from these
transactions. Fortunately during the project, Hewlett-Packard extended the end-of-life
date from December 2006 to December 2008.
A secondary goal of this project was to remove the over-night delay in moving title and
registration information from the HP computer to the agency’s data repository.
“Immediate Updates” of the repository provide value to law enforcement, businesses and
government agencies by delivering up-to-the-minute information on vehicles and vessels
ownership and registration status. Immediate Updates also allows the public to complete
their business during one versus two visits to a licensing office when they need to make
more than one transaction per vehicle or vessel.
Other project goals included improving agency responsiveness to mandated licensing rule
changes, reducing the number of different computer platforms being supported and
positioning the agency to improve customer services via expansion of E-Government.
The project was completed $132,000 below the original ISB-approved $7.85 million
budget, with the bulk of the work completed on-schedule. The deployment of the
“Immediate Update” enhancement was delayed three months, partly because of the delay
in stabilizing the replatformed VFS application and partly at the request of the users who
needed DOL to avoid making this change during the busy summer months. The project
provided innovative user training via an instruction booklet illustrated with copies of
computer screen plus a PowerPoint-based presentation with voice-over. Users
appreciated this approach because it allowed them to review and repeat the training at
times convenient to them. DOL’s training cost was $47,000 out of $259,000 provided in
a separate training budget.




                                          Page 1
Original Project Scope: Re-Host DOL’s HP3000 Vehicle Field System to Microsoft
.NET environment.


Did the project deliver the functionality that was intended?        Yes      No

         If no, please describe.


Original Budget: $7.85 million       Original Schedule: June 2007
Final Cost:      $7.72 million       Completed: September 2007


Success Factors:

Executive Support: Does the Executive Sponsor have a global view of the project, set the
agenda, arrange the funding, and articulate the project’s overall objectives? Is the
Executive Sponsor actively involved, an ardent supporter, responsive, and accountable
for the project’s success?



User Involvement: Do the primary users have good communications skills allowing them
to clearly explain business processes in detail to the IT organization? Are the primary
users trained to follow project management protocols? Are the users realists and aware
of the limitations of the project?


Experienced Project Manager: Does the Project Manager possess technology and
business knowledge, judgment, negotiation, good communication, and organization
skills? Does the Project Manager possess soft skills, such as diplomacy and time
management? Is the Project Manager able to communicate with executives as well as
business representatives? Is the Project Manager able to say “no”?


Project Organization
•   This project started out with two project sponsors: one from the business and one
    being the CIO. The business sponsor was not an aggressive owner of what or how
    the project was doing, and soon all sponsorship responsibilities defaulted to the CIO
    and the project became “a computer project.” It is recommended that each project
    have only one sponsor and that that person be from the agency function that benefits
    from or needs the output of the project.
•   The CIO should remain a key member of the Project Steering Committee and
    continue to support the Project Management and IS side of the project work.




                                          Page 2
•   The CIO provided strong and effective support of the Project Manager. This was
    particularly important when dealing with Fujitsu relationship and contract
    deliverables issues.
•   The CIO also was most helpful in encouraging the project team when project
    challenges peaked.
•   In addition to the “normal” planning, tracking, controlling, reporting and leadership
    responsibilities, the Project Manager must work with IS management and staff such
    that groups that now provide services from isolated functional teams, instead join into
    the project team as collaborative partners. Groups that are especially critical to
    include in this collaborative project team culture are the Configuration Management
    group and the Network Support Services (LAN Support / Infrastructure) group (NSS).
•   Clear documentation of project team roles and responsibilities helped the project team
    smoothly and effectively work together and perform the required work. This also
    helped smooth transitions when there were changes in the staff filling team roles.
•   It might have helped if Work Group Leads had been reminded more often of their
    personal accountability for Work Group deliverables.
•   The Project’s Business Liaison should be physically located with the project team
    and assigned to report directly to the Project Manager. This person is responsible for
    establishing and strengthening two-way communications between the business and
    the project. Each day this individual will be getting information to or from the correct
    person on the business side. The role also is responsible for setup and leadership of
    activities involving business-side management or end-users, such as User Training,
    and User Acceptance. This person also creates project newsletters and high-level
    status updates for the business managers, user support staff and application users.
•   Involve Business Liaison in approval and tracking of deliverables.
•   A Project Test Lead is the second-most critical member of the project, next to the
    Project Manager. This person must have intimate hands-on experience with the
    business area and the applications into which the results of the project are to be
    provided. As with the Project Business Liaison, this person must be located on-site
    and report directly to the Project Manager.
•   Each project must have one person assigned as the Technical Lead / Technical
    Expert. This person will ensure (1) architecture of the new application, (2) the
    building of environments for Dev, QA (and Production) are completed just in time for
    use by project staff, (3) communications and coordination of work between the
    Project and the LAN Support team so that work executes smoothly and follows the
    associated Configuration Management and Network Design / Documentation
    procedures, and (4) Lead the Deployment Work Group to produce the deployment
    checklist and collect the detailed installation instructions and scripts, then lead the
    deployment activities..
•   The Technical Lead should also be responsible for setting up, executing, and
    reporting on Stress and Application Performance testing.



                                          Page 3
•   If possible, the project team should be located physically near to the impacted users,
    and / or the business staff who support those users. (Travel time between locations
    can be a significant loss of work time.)
•   For larger projects, break the team into Work Groups who are responsible for
    specific deliverables. Each Work Group is to have a responsible Lead who is a
    member of the Project Core Team.
•   Each member of the Core Team is directly responsible for keeping their work on
    schedule. That will likely include leading the clearing of all hurdles (technical and
    people issues) that would otherwise impede progress on the Work Group’s tasks.
•   When any Work Group task is stopped, that fact is to be immediately communicated
    to the Project Manager, but responsibility to fix that work stoppage remains with the
    Work Group Lead. The Project Manager may choose to assist, but will not take-
    over this responsibility.
•   Projects that include batch jobs need to provide staff (Developers, NOT the Technical
    Expert) to setup and load these jobs and calendaring on the development
    environment (DEV) such that it is compliant with Agency standards. The knowledge
    to do this cannot continue to reside just with one person within NSS. Once correctly
    operating in DEV, the configuration management services group (CMSG) can easily
    and safely promote it to QA, and NSS can promote it from QA to Production.
•   Make sure Work Group Leads understand they are responsible for saving into the
    electronic and paper repositories all records of their Work Group’s activities. Every
    Decision must be logged into SPLAT, or other electronic repository. Records must
    be kept up-to-date (not more than five days old).
•   Having the business’s most-experience user-support staff as the Test Team resulted
    in creating comprehensive and innovative test scenarios and scripts. Having an
    experienced Testing Lead from the business ensured both high quality leadership of
    testing work, and that test planning and test data preparation would reflect all the
    important business scenarios.
•   The IS Liaison for the business and the Project’s Business Liaison need to stay in
    close communication and support each other during the project. Both must stay
    equally informed, and because they work at different levels and in different ways with
    the business, they form a powerful team in representing the project while being very
    inclusive of business needs and issues. Both are required and neither has the
    bandwidth to do the work of the other.
•   The Project Core Team should consist of the Project Manager, Business Liaison,
    External QA, a Technical Lead, and Work Group Leads. The Project Manager uses
    the weekly Core Team meeting to provide information and to collect information that
    will drive decision making. It is recommended each Core Team attendee report
    weekly on (a) accomplishments during the past week, (b) hurdles to current progress,
    and (c) worries about future issues.




                                          Page 4
Clear Business Objectives: Are the project objectives clearly defined and understood
throughout the organization? Is the project measured against these objectives regularly
to provide an opportunity for early recognition and correction of problems, justification
for resources and funding, and preventive planning on future projects? Does the agency
understand the impact of the proposed business process changes?


Minimized Scope: Is project scope realistic and able to be accomplished within the
identified project duration? Is it measured and managed regularly to eliminate scope
creep?

See Formal Methodology.

Responsive Business Requirements Process: Does the project employ a responsive
requirements process that manages requirements issues quickly and without major
conflicts? Requirements management is the process of identifying, documenting,
communicating, tracking, and managing project requirements as well as changes to those
requirements. This is an ongoing process and must stay in lockstep with the development
process.


Standard Infrastructure: Has the project established a standard technology infrastructure
that includes operational and organizational protocols? Is the infrastructure commonly
understood and regularly assessed?


Formal Methodology: Does the project follow a formal methodology that provides a
realistic picture of the project and the resource commitments? Are steps and procedures
reproducible and reusable thereby maximizing project-wide consistency?

Communication Plan
•   A clear escalation path and contact information should be provided for each work
    area and technical topic. Typically this will be the Work Group Lead.
•   One of the most important communication links within the project is between the
    Business Liaison and the Test Team Lead.
•   Including an experienced person from the business as Business Liaison, responsible
    for two-way communication with the Project, in addition to IS Managers to provide
    two-way communication between the Project and IS developers, saved time and
    strengthened partnerships.
•   Provide regular, high-level status updates (accomplished, next steps) to all business
    stakeholders and users by a one-page newsletter with an eye-catching graphic
    associated with each topic. If it’s short and attractive, it will get read. If just text, it
    won’t get read.



                                             Page 5
•   Engage the user and user support staff in reviews of the user interface and in User
    Acceptance, and even in testing if they are interested. We did a period with of Tester-
    for-a-Week volunteers that was very successful in spreading real information about
    the changes that were coming.
•   Good involvement of the Vehicles Services staff in the project activities during
    immediate update and personalized specialty plate phase (IU+PSP) made things go
    smoother than they did for the Replatforming pre- and post-deployment activities.
•   The Project Manager and Business Liaison should take every opportunity to attend
    meetings and user conferences and to present the project’s objectives and timeline.
    Building relationships with individual users, user support staff, and the DOL
    managers was key to having everyone pulling together when we deployed.
•   Make sure to use SPLAT (issue tracker) to record EVERY decision made, the date,
    and who was involved. They MUST be recorded on the date they are made or you’ll
    loose track of them. The Work Group Lead, if not otherwise assigned, is responsible
    for recording decisions made in his/her Workgroup. The Project Manager or assistant
    is responsible for recording decisions made in general meetings and discussions. The
    Business Liaison is responsible for recording decisions made by the business. That
    way all decisions are recorded in one easily-accessible place. Depending on email for
    that repository does NOT work as there are too many places to look.
•   It is important to have a solid post-deployment communication plan for reporting
    status of post-deployment defects and activities taking place to fix those problems.
    This is especially important when money is involved and users will wonder what
    actions they need to take and what documentation they will have to pass an audit.
    The communication plan must address what and how information will be reported
    both inside and outside the Agency. We did not do well with this initially following
    the VFS deployment, but quickly fixed those problems.
•   Having communications about the post-deployment status sent to field offices, not
    only by the Project via normal vehicles mailboxes, but also by electronic letters from
    the Agency Director, was important in calming external fears and concerns.
•   Daily post-deployment status reports to the Project Steering Committee also helped
    disseminate accurate information in quantitative terms. This was required to combat
    reports that made problems appear much bigger than they actually were.
•   It is critical to provide everyone with up-to-date status on application defects so those
    who reported defects know their issue has been logged and some perspective on when
    it may be fixed. They also need to see the relative priority given to their request
    compared to other defects and can understand why it may take time to get their issue
    fixed. Use of an intranet site was suggested as a good way to make that information
    available.

Project Information Management
•   SPLAT was DOL IS’s tracking and reporting application. This function is now
    migrating to SharePoint, so staff are now learning to use this new tool.



                                           Page 6
•   Assignment of prioritized Action Items was an effective way to guide busy
    developers through work needed by the Project.
•   Testers, Developers and Project Managers need a way to view EVERY change to
    the infrastructure made by NSS. At least four times during the project, three-to-
    five days were lost trying to troubleshoot why a program, that had been working, all
    of a sudden stopped working. In all cases, NSS reported they had not made any
    change, and each time it was proven that someone in NSS did make the change that
    broke the application.
•   The structure of project repositories on drive P: needs improvement to better align
    with the information collected during the project. The structure should allow faster
    retrieval of the needed information.
•   A much better approach is needed to store email messages and their attachments into
    the project repository. The current Save-As process is extremely time-consuming.

Scope Control
•   The Project Manager’s leadership was critical in forcing a very clear vision and scope
    statement for the Immediate Update enhancement. Without these constraints, the risk
    of scope creep is a high risk.
•   Task estimates need to be given more buffer against external impacts on the project
    from other project’s need for resources and the impact of legislative-mandated
    changes upon the project. With no restrictions on Legislative impact, these added 30
    percent to the time required just to do the testing.
•   The business’s Personalized Specialty Plate (PSP) design team did an excellent job
    preparing the SRS (specifications). This was a VERY complex change that required
    a huge number of design decisions. The PSP specs were very stable due to this
    team’s involvement of all stakeholders in the design and review process.
•   If specification documents are found to be unclear or ambiguous, work should stop
    and the specifications should be fixed immediately via collaborative reviews. Scope
    impacts on the project need to be reported from this work, but will be less than
    spreading out this clarification over many weeks.

Change Management
•   The RFP must inform the vendor that no work on a Change Request may be done
    until it receives the required sign-offs by DOL.
•   The vendor is to provide a not-to-exceed bid for the work described in each change
    request. If the work is completed in less time, the vendor is to charge DOL only the
    actual hours or actual cost of the change.
•   Changes requiring up to 12 hours of effort from the contracted Change Request
    Buffer may be approved by the Project Manager.
•   Changes above 12 hours of effort or a cost to the project requires signature approval
    by the DOL Chief Information Officer.


                                          Page 7
•   DOL may request the change be completed by a certain date, in which case the
    vendor is to respond with the impact on other project tasks assigned to them in order
    to meet that date. Within 5 business days, DOL will either select one of those options
    or negotiate a different approach.

Configuration Management (Version Control)
•   The Project Manager must ensure the Dev and QA environments are setup at the
    beginning of the project and that the associated areas are established within the
    Configuration Management tool (VSS/TFS). All developers are to be reminded use
    of DOL’s configuration management and code promotion processes are required.
•   Areas within the configuration management tool must be setup at the beginning of the
    project to also hold the deployment checklist, instructions and scripts.
•   There continues to be significant staff resistance to use centralized configuration
    management (version control) for software. This is caused in part by the
    relinquishing of code control to a third party that feels to developers like a hurdle
    rather than a helpful safeguard. The configuration management procedures appear to
    be still evolving. They do not yet fill the needs for the dynamic multi-concurrent-
    project work environments of DOL and are just gaining procedures to support
    development interruptions from Legislative-mandated and changing business needs.
•   Currently, code held by the Configuration Management Services Group (CMSG) is
    not routinely nor reliably updated to reflect changes made via emergency releases.
    Similarly, when emergency releases don’t work correctly, fixes to these releases are
    applied on the fly and are an additional risk to having the checked-in code reflect the
    executables in Production. Additional procedures, education and commitment are
    needed to make this a stable process.
•   Configuration Management processes were evolving during this project. Many code
    revision management problems would have been avoided with the procedures that are
    now in place.
•   Having one member of the Configuration Management group dedicated to processing
    project requests was effective and important. This CMSG representative became
    familiar with how the project was managing code and this allowed her to be pro-
    active and helpful.

Design & Development Process
•   All architectural and functional designs must go through a peer review by DOL staff.
    No exceptions! Time must be put in the schedule and Quarterly Plan for this activity.
•   DOL staff are responsible for taking the time and asking questions such that they
    fully understand what they are approving. This may require perseverance when
    dealing with aggressive vendors and when staff do not understand the technology
    being applied. DOL staff’s signatures on peer reviews MUST indicate that staff fully
    understand the design being presented and approve it as being an approach for which
    they are personally willing to take ownership.


                                          Page 8
•   Peer reviews of Fujitsu’s replatforming designs took place too late (they evolved
    some bad solutions that were not reviewed with DOL staff). The problems were not
    evident until code was being delivered. The vendor was most reluctant to have DOL
    staff review their decisions early on, and the DOL Project Manager should have been
    more aggressive in not allowing that.
•   DOL IS needs written instructions on how to setup a batch environment. Currently
    there is only one person who really understands how batch needs to be setup and how
    calendaring of jobs must be defined.
•   All requests for database changes must include a statement to “Restore Permissions”
    following the change. Without this, permissions for application programs and users
    will be lost.
•   In making change requests to NSS, simply requesting replacement of an .exe or .dll
    on the server frequently requires you also request the server be “refreshed” to activate
    that change.
•   Developers should have the knowledge to setup their own batch environments in Dev,
    and then have it approved by LAN Support, so batch setup does not become this
    stressful, critical-path, late-project activity.

Testing
The Test Lead and Test Team for this project were exceptional.

Tester Selection
    •   To do adequate testing, the testers MUST come from the business area using the
        application, and have hands-on experience using the application. This is required
        for identification of the test scripts that must be run. Developing test scripts from
        design specifications can be done, but will risk missing tests that can only be
        known when the changes being tested are put into the context of the business
        workflow. For replatforming projects, there are no application design
        specifications from which to establish a test plan.
    •   The Test Team did a GREAT job at identifying test cases. This was possible
        because of their in-depth knowledge of the VFS application from years of hands-
        on experience in the field and in field user support.

Test Procedures
    •   The Test Team used a highly-effective peer review process for Test Plans and
        Test Scripts. The reviews went step-by-step through each script, and often also
        considered what was happening to the data during the test.
    •   The Testing Lead’s experience allowed her to provide very accurate estimates of
        testing times.
    •   The Test Lead from DOL and the Test Lead from Fujitsu established an excellent
        system for tracking test scripts. This is also be used on the Border Crossing
        project.


                                           Page 9
•   To get the testing work completed on time, as many as six concurrent testing
    environments were active at a time. While there was considerable push-back
    from CMSG and LAN Support to having parallel testing environments, and it
    required careful management to reconfigure environments and update the data in
    each environment, the HP Project staff did this well and the needed productivity
    from the parallel environments was achieved.
•   It was critical to have one person within the Test Team track code deliveries from
    the vendors and track what version of code and what test data were active in each
    of the testing environments. This person also initiated and tracked all
    configuration and pointer changes that needed to be made with changes installed
    to each environment. CMSG did not consider this tracking and management
    within their scope of responsibilities and LAN Support has no procedure for
    tracking the current state of each environment.
•   If Test Scripts are to be re-usable they must (1) be kept up-to-date with all
    application changes (scheduled and emergency releases), and (2) have scripts to
    automatically establish the initial data conditions required to run the test.
•   To prepare test scripts to test changes (Legislative-mandates or internal requests),
    the Test Team must be provided with a narrative of what each change modifies in
    the workflow, calculations, and user interaction with the screen. A description of
    the program change is not sufficient.
•   The Test Team needs to be promptly informed when Change Requests are
    submitted to developers to modify behavior of programs because test scripts and
    test data frequently must be modified to test the change.
•   On an application like VFS with many data-controlled paths, at least 100 percent
    more time needs to be scheduled for running through more of the different
    business scenarios. During the UAR Project, it was sufficient to create test
    scenarios to verify all the programs at least once. Scenarios were designed like
    strings threading through beads. When all the beads were included on one or
    more strings, the needed tests were defined. For the HP Project, data changed at
    one node actually changes the behavior of the programs that followed, so each
    program needs to be tested through many paths, not just one. Testing must include
    all values the navigation-controlling variables may assume.
•   Contention for testing environments and staff for batch and end-to-end testing
    required the original end-to-end test plan be “chopped up” such that the workflow
    was more complicated and took longer than originally planned. This was made
    worse by the amount of retesting imposed on the testers due to bugs in the
    delivered code. The schedule needed a bigger contingency buffer at this point so
    that so many testing activities did not have to be stacked upon each other.
•   During the stabilization of the replatformed VFS, many of the bugs reported
    couldn’t be reproduced by the Test Team. We were never sure if a prior bug fix
    took care of the problem or if Fujitsu had fixed more than the bug but only
    reported fixing the original defect. Lack of open, candid information exchange



                                      Page 10
       from Fujitsu to the Test Team was one of the factors that caused the protracted
       stabilization period.

Batch Job Setup & Testing
   •   Batch Testing setup and execution took four times as long as originally estimated
       because (a) setup of all the environments the batch jobs needed to interface with
       was more complex than anticipated, (b) only one environment at a time can be
       pointed to the QA environment with interfaces to other applications and externals,
       so batch testing and online testing had to be scheduled at different times of the
       day
   •   Establishing bug-free batch job calendaring (daily job schedule) took much longer
       than planned. The current job flow diagram is a critical and required tool to do
       this, but it needs improvement because many mistakes appear to still be made and
       the process is somewhat a trial-and-error process.
   •   Batch and End-to-end testing took between four hours and all day because of the
       old, slow servers that were provided for Testing (both batch and on-line). The
       Project Manger should have purchased new fast servers for this activity as soon as
       the impact on testing was surfaced. Unfortunately, the slowness of the test
       servers was not raised early on as a problem.
   •   Batch testing will need to have on-line scripts (activities) executed to prepare data
       for the batch runs. This must be considered when putting batch testing on the
       project schedule because it may interrupt and add to other time-sensitive work
       being performed by the Test Team. Likewise the schedule must reflect that the
       Test Team will be called upon to evaluate reports and screen values generated by
       the batch jobs to verify the batch job performed the expected work.
   •   Be sure to make data backups of conditions at the start of batch tests and at many
       mid-points during each batch test. That way, if the test needs to be restarted, a
       minimum of time is required to do data conditioning for that restart.
   •   Plan testing of batch jobs to begin with daily jobs, then move to weekly,
       monthly, quarterly, and yearly so data is being built for the next round of testing
       as tests are run.
   •   When doing batch testing, make what might be considered excessive numbers of
       data backups to ensure if data needs to be re-constructed for this testing the
       amount of work to do so is minimized.

Application Performance & Stress Testing
   •   Fujitsu claimed to have experts in stress testing, but after working with them this
       proved false. They hired Microsoft (as specified in the Contract) to setup and
       perform the stress testing, but the work was so tightly constrained by cost and
       time, the initial attempts were inadequate. It was not until DOL took over
       leadership of this task and hired Microsoft (rather than have Fujitsu continue to
       constrain the work) that reasonable results were obtained.



                                         Page 11
   •   Stress Testing will require test scenarios / scripts and test data from the Test
       Team, but must be planned and executed by the Technical Lead as the Stress
       Testing requires technical knowledge of the server environments to set up, as well
       as technical knowledge to understand the technical metrics reported by the test
       tools.
   •   When preparing to perform a Stress Test, make sure to reboot the servers to clear
       any problems (such as memory) that may remain from a prior test, and thus affect
       the new test results.

User Acceptance
   •   DOL had a hard time getting folks from the field to participate in the User
       Acceptance process. This was made more difficult each time the dates for this
       work were changed.
   •   User Acceptance made effective use of the HLB PC Training Room to provide
       reviewers hands-on experience with the on-line screens.
   •   Batch processes were not included in the User Acceptance, but copies of all
       reports were provided in book form along with a description of the data and
       situation that led to the printing of each report shown in the book.
   •   User acceptance reviews by DOL staff and field staff were held on different days,
       knowing in advance that the questions would likely be different.

User Shakedown Testing
   •   Every new application and significant change to an application should go through
       a User Shakedown such that the associated deployment requires little more than a
       replacement of test data with production data in the databases.
   •   User Shakedown is (a) to allow the users to verify usability of the change about to
       be deployed, (b) a chance to break the application by intentionally doing things
       incorrectly to ensure the system traps and reports mistakes, and (c) verify the
       application continues to perform well under load. THESE TESTS ARE
       CRITICAL TO HAVING POST-DEPLOYMENT STABILITY.
   •   The tiny field user involvement in the VFS Replatforming User Shakedown is
       directly one cause for the extensive and protracted time required for post-
       deployment stabilization. Authorization to deploy to Production without a
       thorough User Shakedown causes a HIGH risk to the project.
   •   All user organizations should have been required to do extensive User Shakedown
       testing, both to ensure performance and quality of the code to be deployed and to
       get first-had experience using the application prior to fielding requests for help
       from users.
   •   For the VFS application, allow for deployment to the Education Mode site well
       before deployment, so then users can have time for hands-on experience that (a)
       allows them to verify their understanding of how to use the new features work as



                                        Page 12
        preparation for effective use when the changes go-live; and (b) provides field
        testing (user shakedown) of the application changes by users prior to the go-live.
    •   For the IU+PSP deployment, the Education mode could not be activated prior to
        deployment because the Test Team was still testing application and only one
        environment could be set up with links between VFS, VHS, external (test)
        interfaces and the batch system.
    •   The above comment means all testing of the application must be scheduled and
        required to be complete well before User Acceptance and User Shakedown are
        planned to start.
    •   Fixes to bugs need to be tested during a different part of the day than users are
        doing User Shakedown activities because both activities cannot co-exist. (Again,
        only one environment can be active at a time with the requisite links.)
    •   Very few users participated in even minimal use of the application during the
        shakedown period. DOL had to force users to even attempt to logon so we could
        verify all users had their access set properly. Had users done this testing we
        would have known the replatformed VFS wasn’t really ready for deployment.
    •   Active user participation in a real shakedown should be required for all
        replatforming transitions, and is recommended for any other deployment of a
        significant application or application change. There is no substitute for actual use
        of the application as a readiness test prior to the official deployment.




Reliable Estimates: Does the project have a history of providing realistic estimates? Are
the current estimates reasonable and reliable?

Project Schedule
•   It was very helpful to make each contracted project deliverable a milestone on the
    project schedule. Flexibility can be achieved by listing the milestone at the same
    indent level as the summary task for work required fro the milestone.
•   Insist all project participants report weekly the hours they spent on each assigned task
    during the previous week and the new estimate of hours-to-complete, made for fairly
    accurate status tracking. We did not allow reports of Percent Complete, but
    calculated that value from the participant’s input. Reports of percent complete from
    staff typically prove to be incorrect.
•   Recruiting and hiring paperwork and processes take weeks, and typically a month.
    Project Mangers need to take this long duration into account to anticipate and start
    building project teams and obtain contracts well before the resources are needed.
    (This is very different from private industry where high-quality staff or contractors
    can be on-site and working within one week.)
•   High stress and human cost of forcing IU+PSP into a three-month window should not
    be repeated. This required 310 hours of over-time for the testers, immediately


                                          Page 13
    following 373 hours of over-time the previous quarter needed to test stabilization of
    the replatformed VFS.
•   Rework and re-testing added to the project by Legislative-mandated changes was
    very disruptive. Legislative changes should have been blocked during a project of
    this complexity.
•   We under-estimated the time to establish and test the Batch functionality during both
    the VFS Replatforming and Immediate Update phases. In both cases it took four
    times longer than pessimistically estimated. If there is even moderate complexity in
    the batch jobs and their calendaring (scheduling), a one month estimate is likely in the
    ball park.
•   DOL IS staff that are making application changes that require they upgrade their
    knowledge and skills to contemporary technology, did not appear to be motivated to
    aggressively work to achieve the required knowledge and skills before needing to
    actually make code changes. While this appears somewhat a natural reaction, it
    means considerable time must be planned for the Project Manager and staff
    supervisors to provide continuous tracking of progress and mentoring. It might have
    been better to have given these developers a new-technology assignment before the
    project to get them over the learning curve and give them confidence in their use of
    the new tools and skills.
•   Time needs to be built into work schedules for “thinking,” “evaluating,” and
    “investigating” and for upgrading technical skills.
•   Time needs to be available within each project for the Project Manager to develop
    project-specific procedures and standards, as well as to write white-papers to clarify
    both technical and project issues.
•   Tasks in the project schedule should be grouped in association with each deliverable
    being produced. Tasks should not be grouped by who is performing the tasks because
    this makes it difficult to see all the work required to complete a deliverable.
•   Make sure the project schedule includes “buffers” for unknown negative impacts on
    the workflow. There should also be an overall project buffer placed at the end of the
    project.
•   The Project Manager should identify, as part of the development of the Project
    Charter, if installation and testing the application (or application change) at DOL’s
    Enterprise Disaster Recovery Center is within the required scope.
•   Resource needs and identification of when during the project those resources will be
    needed should be an OUTPUT of the Project Schedule development.
•   To keep testing on schedule, the Test Lead needs to have quick access to the Project’s
    Technical Lead so that needed environment and database changes will be made
    quickly.
•   Batch testing is dependent on having the online programs and data sources that feed
    data to batch processing fully operational before batch testing can begin.



                                          Page 14
•   Defining the required sequence of batch jobs and how to restart batch processes at the
    point they may fail takes more time to work out than was expected. This is
    especially true when moving from one scheduling software to another.
•   Project Managers need to work with the IS Managers to learn the true Windows .net
    skill and knowledge of the assigned staff. If staff are in transition from supporting a
    mainframe application to supporting a server-based application, task estimates should
    be double checked by others and the estimates adjusted based on technical knowledge
    and hands-on experience of the staff in transition.
•   Estimates for Batch setup and testing need to be carefully established by developers
    and LAN Support TIDAL experts. These estimates need to include buffers that
    represent the level of batch setup and testing experience of the developers who lead
    and do that testing. Also factor in how long the batch jobs take to run because testing
    requires many increments and must typically be run outside of normal work hours, so
    you may be able to do only one testing cycle per day.


Skilled Staff: Has the project properly identified the required competencies, the required
level of experience and expertise for each identified skill, the number of resources needed
within the given skill, and when these will be needed? Are the staff available and
assigned to the project? Soft skills are equally important when identifying competencies.

People Issues
•   Resistance to change requires persistent, continued person contact with vendors and
    DOL staff to maintain relationships, and provide positive reinforcement by the
    Project Manager. This takes considerable time and needs to be planned for in the
    project schedule and done on a daily basis.
•   Close tracking, status reporting, and feedback of progress achieved needs to be
    reported to all project staff as well as DOL management. Reporting task status by
    person is a good thing as long as the planning estimates were provided by those doing
    the work (so were as accurate as possible) and individuals are not perpetually
    reported as either early or late.
•   The IS Division needs to significantly improve the technical maturity of its staff in
    the application of contemporary technologies, methodologies and tool. Managers
    need to continue analyzing staff skills and experience to make forward-looking
    training and experience goals.
•   When staff are assigned to a project, the supervisor of those individuals must get the
    Project Manager’s approval for every leave and training request. During this project,
    supervisors approved requested leave when that staff member was on a time-critical
    task. This happened because the individuals requesting leave frequently did not tell
    their supervisor the impact their leave would have on the project. In other cases, the
    supervisor required staff to attend training without first checking with the Project
    Manager, with similar results. What made this worse, the Project Manager did not



                                          Page 15
    even have advanced notification, which otherwise might have allowed the leave
    impact to be mitigated.
•   Everyone on the project must be made aware they are personally responsible for
    meeting agreed-upon deadlines. If you are dependent on someone else to complete
    your task, you must (a) give them reasonable notices of when they will need to work
    on your task, (b) be given any information and code to do that work before they need
    to start work, (c) have clear instructions of the work to be done, the expected
    deliverable(s), and the required delivery date. You, not they, are responsible for
    meeting your due dates. “I’m waiting on --- is NOT an acceptable excuse.”
•   DOL IS staff are using analysis, design, and development techniques of twenty years
    ago. This is impeding their productivity in both development and troubleshooting.
    Foundational techniques in the .net world, such as object oriented analysis & design
    are not yet in use and there seems so much pressure to develop systems and changes
    that I see no progress in growing these skills use of modern techniques within the
    staff. Further, as these methods are not in use within DOL, even the contractors being
    hired do not use these tools as they would be difficult to integrate with the existing
    code being modified.
•   Anticipated, late delivery on any task needs to be immediately brought to the Project
    Manager’s attention.
•   Opportunities to complete work early should be taken, unless other issues would
    make this inefficient for the project or another project.
•   When someone is to do work for you at a specific time in the project, the objective is
    to give them a clear description of the task well before they need to do the work, then
    provide a two-week, one-week, three-day and one-day count-down on when you will
    provide the input they need to start work.
•   If getting an answer or action from another person is urgent, DO NOT rely on email
    to communicate that fact. Call them or visit them to ensure you both understand
    when the work will be completed and agree on that timing. Keep the Project
    Manager informed.
•   Staff should not get frustrated when someone does not promptly reply to their emails.
    Either call or visit the individual and find out what you need to know. (This will
    lower your own stress.)
•   It is critical that the Technical Lead, supported by the Project Manager, communicate
    regularly with the Infrastructure teams to ensure that environments needed by the
    Project are fully operations and available to the project team when needed. It is
    recommended that Project-NSS status checks happen at least weekly, and escalate to
    daily, as deadline approach. The execution of changes to Project environments must
    similarly be executed when agreed. (During the HP Project, NSS’s Just-in-time
    environment deliveries and environment changes were provided late so many times,
    the resulting testing delays so frustrating to Fujitsu; it essentially nullified DOL’s
    contractual leverage to force Fujitsu to do work on schedule.)




                                         Page 16
Contract Negotiation and Management: Is the project using resources experienced in
contract negotiations? Does the project organization include a resource whose sole
function is contract management?


Request for Proposal (RFP)
RFP Preparation
   •   RFP preparation was rushed to completion at a time when the DOL managers
       who needed to participate were overwhelmed with other critical projects. As a
       consequence, sufficient thoughtful review and consideration was not put into the
       RFP’s content and organization. Wording and missing information in the RFP
       (that became part of the contract) was the main and on-going source of problems
       with the selected vendor.
   •   The RFP was written by a contractor who helped DOL prepare excellent RFPs in
       the past, but was left to her own too much when collecting and organizing the
       information. Mistakes in earlier RFPs were repeated rather than avoided. The
       write-up of the required deliverables was out of touch with the needs of this
       project.
   •   The timeline and time estimates stated in the RFP were unrealistic. When the
       DOL and vendor Project Managers prepared the initial schedule, the work
       extended three months longer under optimal considerations. On top of that, the
       RFP required the vendor to convert and integrate Legislative-mandated changes
       imposed during the project. It was not made clear in the RFP that this would
       likely require a significant amount of rework and retesting by the vendor. In fact,
       this added two months of re-conversion and four months of re-testing. The
       incorrect estimates and the imposed rework had a major, negative impact on the
       DOL-Vendor relationship, making the job for the Project Manager most difficult.
   •   The incorrect timeline in the RFP and the amount of rework required by the
       Legislative changes added Change Request costs to the project amounting to
       $857,035 (37 percent over the initial $2,290,000 contract.
   •   The RFP included many “mandatory” items for which the vendor had to mark
       they would comply or not comply. Most of these items were defined by lists of 4
       to 20+ conditions that all had to be met. Vendors complained about this “strong
       arm” approach which forced them to agree to comply with details that they did
       not agree with, or risk being eliminated. To be fair and get more honest
       proposals, these multi-part items should have been “scored” rather than being
       forced into “comply” or “not comply” status.
   •   Although the impact was too small to affect the scoring totals, some items should
       have been put into the Management versus Technical categories of the RFP. The
       rushed publication did not allow these mistakes to be fixed.




                                         Page 17
Project Requirements
   •   Work must be done BEFORE the RFP is created to establish quantitative metrics
       that define the baseline performance for normal and stressed operations as well as
       define the performance-under-load targets that must be achieved in quantitative
       values.
   •   A good requirement was the vendor’s Project Manager and all key members of
       the vendor’s team be on-site throughout the contract. This is a significant help in
       ensuring good vendor-DOL technical communication. It also facilitates
       monitoring and tracking of vendor-effort being extended on the project.
   •   The RFP and Contract did not adequately protect DOL and the Project Manager
       from poor quality work products by Fujitsu. This added extra work and
       responsibility onto the DOL Project Manager and a litany of conflicts with the
       vendor who focused on keeping his costs within the fixed bid of the contract.
          o The RFP must specify the required deliverable quality, the metrics by
            which it will be judged, the form of the delivery (code as source code, not
            executables), documents (Word document, Excel Spreadsheet, Visio
            Diagram, etc.) and how the delivery is to be made.
          o All deliverables listed in the contract are not considered “delivered” until
            DOL has held a formal review, at which the appropriate vendor technical
            experts must be present. The DOL staff identified in the RFP for that
            deliverable review must formally sign-off acceptance of the deliverable.
            The vendor, not DOL, is responsible for achieving that sign-off promptly,
            but must allow DOL a minimum of five business days per deliverable
            review.
          o The RFP must specify the metrics by which the quality of the code
            produced will be judged. For example: post-delivery defect counts, time-
            to-stabilize, readability and maintainability of code, compliance with MS
            best practices, performance under load as observed on user workstations,
            etc.
          o RFP designers should consider placing penalties on the vendor if
            deliverables are not approved by DOL within a number (3?) approval
            review cycles. This should be deducted immediately from the holdback
            amount available to the vendor at the end of the project.
   •   The RFP must remind the vendor DOL owns all work done under the contract, so
       the vendor must supply any finished and in-progress work products upon request
       by DOL. No exceptions!
   •   The RFP should ask the vendor to provide a written answer to the questions:
       “What are the top five product quality considerations for this project? What is the
       vendor going to do to deliver the highest quality work products? What metrics
       will the vendor use to track and report product quality? This then could be a topic
       for contract negotiations if DOL considered the answers inadequate.




                                         Page 18
   •   The RFP should specify where in the project (project phase or association to
       another deliverable) each deliverable will be provided to DOL. This will keep
       delivery of deliverables in the sequence intended by DOL. The sequence of
       deliverables is critical when those deliverables are quality gates for work to
       follow.
   •   Put in the RFP that DOL expects the vendor to share, at least weekly, through
       formal status reports not only the accomplishments of the vendor, but the
       problems (technical and otherwise) being experienced by the vendor and the risk
       these problems pose to the project. DOL will collaborate with the vendor when
       possible to suggest or help solve these problems as a partnership toward achieving
       project success.
   •    The vendor MUST use the tracking tool used by DOL for bugs, action items,
       decisions, risks, etc. as their primary tracking and reporting instrument. If they
       choose to use a different tool internally, all information in that alternative tool
       must be recorded by the vendor into DOL’s tool on the same day and with the
       same completeness of information as entered into the vendor’s tracking tool. (No
       exceptions!)
   •   Payments need to be based on the value to DOL of the deliverable, not the amount
       of effort required by the vendor to produce the deliverable.

Description of Work
   •   In replatforming and code conversion projects, the RFP must emphasize the
       vendor’s responsibility is to replicate the behavior of the application on its prior
       computing platform and language, not just the logic of the code.
   •   Specify that the replatformed code is to perform at least as quickly as it did on the
       prior platform, be easy to maintain, and follow DOL’s design standards for the
       target language. Performance must be defined in quantitative metrics such as
       screen-to-screen transition time (possibly for every screen transition), allowed
       delay between user action and start of printing, total batch job execution time, etc.
       Throughput should be defined by a measurable metric such as total completed
       business transactions per minute. The load at which these targets must be
       achieved must also be defined by a measurable metric such as transaction counts
       submitted per minute for each pertinent type, active concurrent users per minute,
       etc.
   •   Specifics on the nature and quality of training and training materials to be
       prepared by the vendor need to be as specific as possible. The RFP should ask the
       potential vendor to also suggest improvements on the requested training approach
       and specified training deliverables. Request the cost impact of their proposed
       training changes.
   •   The RFP should require the vendor to work with DOL to ensure: the application
       will work within DOL’s environment, the computer environments are configured
       according to DOL standards, so the application will perform as needed.



                                          Page 19
   •   The vendor is responsible for achieving application stability (no aborts, timeouts,
       or database deadlocks) and performance (screen-to-screen transition times, delays
       before print starts, times to complete data transfers) under loads that match or
       exceed quantitative values defined within the RFP. That means planning the
       stress test and measuring the existing performance must be completed BEFORE
       the RFP is prepared.
   •   The RFP must state the vendor is responsible for ensuring their code follows
       Microsoft best-practices so that the application achieves performance targets. It
       must also state the application must manage memory, temporary files, and
       database space so performance and application stability is maintained over the
       long-term life of the application.
   •   The RFP must indicate the applications will live in an environment where Load
       Balancers will manage the application’s execution over many servers and
       database activities execute on clustered configurations. The vendor is responsible
       for ensuring efficient and effective management of such resources by their
       application code.
   •   DOL-required architectural design constraints must be collected from the DOL IS
       Architecture team and either included in or attached to the RFP.
   •   The RFP should be much more specific in defining the vendor’s responsibilities in
       setting up the batch processing environment and job calendaring (scheduling).
       This and the batch testing defaulted to DOL staff when this responsibility was not
       made clear within the RFP.
   •   The vendor was responsible for leading the testing activities and collaborating
       with the DOL Test Lead and DOL Tester to do that work. Half way through the
       project, she left the project. The project was too far along and in too intense a
       stage to have the vendor provide a new test lead, so the vendor Project Manger
       took on the responsibility. In practice he was only a testing status clerk, and real
       test leadership responsibility transferred to the DOL Test Lead. From hind-sight,
       the vendor Project Manger should not have been allowed to act as Test Lead and
       the vendor should have been forced by DOL to provide an even higher-qualified
       test lead to learn and take-over Testing Leadership. While the DOL Test Lead did
       an exceptional job, she assumed a very stressful responsibility and test planning
       mistakes made by the vendor. This also left batch testing in the lurch until DOL
       staff took that over. (It may be that the vendor realized that they didn’t have the
       knowledge to plan and lead the batch testing.)

Vendor Participation in Quality Assurance
   •   Quality assurance activities (e.g. design reviews, code reviews) the vendor will be
       required to participate in, and take action from, must be spelled out. Simply
       stating the vendor will comply with DOL SDLC and QA activities is NOT
       specific enough to be measurable or enforceable.
   •   The RFP needs to state the vendor must aggressively and pro-actively seek-out
       defects and fix them. This includes looking for additional locations within the


                                         Page 20
       code that the same or similar bug may exist and fixing all occurrences of the
       defect, not just the one reported. Fujitsu was not always proactive in seeking out
       and fixing potential problems in their code. They only fixed what DOL proved to
       them was defective. This is not acceptable vendor behavior.
   •   Define how defects will be counted. Include how “Can’t reproduce” and
       “Training issue” are included, or not. Need to make sure vendor is motivated to
       not ignore or just declare issues “Can’t Reproduce.”
   •   Fujitsu was not open in discussing their code conversion process or the resulting
       code. The Project Manager and External QA contractor should have insisted the
       vendor participate in all design and code review processes with DOL staff from
       the start of the project.
   •   The RFP needs to state when the vendor fixes reported defects, they must
       document, in a fair amount of detail, everything they changed to effect the fix.
       (Fujitsu refused to do this, not wanting to report the extent of their mistakes.)
       However, this information is critical to the planning of re-testing of the
       corrections, and may show that re-testing of more processes than just the fixed are
       required. Having such information will also be helpful in future troubleshooting
       by DOL when we are doing application maintenance.

Deliverables:
   •   The deliverables in the RFP were copied from the prior UAR RFP rather than
       from what evolved during UAR to be the needed deliverables. The list further
       stated the vendor was to provide a PowerPoint presentation with each deliverable,
       which is not a value-adding or required action. The needed deliverables include
       formal DOL sign-off on (a) the vendor’s analysis and confirmation of the work to
       be done, coupled with a MS Project schedule; (b) the proposed architectural
       design as it will integrate into DOL’s infrastructure; (c) the specific code
       conversion, error reporting and event tracking that will take place, with emphasis
       on DOL buy-off on the plan details before any work is done; (d) identification of
       all development environments, the QA environment, and interface architectures
       and designs that will be required, and the promotion workflow and workflow
       timing; (e) DOL acceptance of code reviews, including online, batch, services,
       and batch calendaring (f) testing plan, scripts, and conditioned data; (g) technical
       documentation to assist for DOL’s maintenance of the application (h) user
       training materials and delivery; (i) data conversion and management; (j) Weekly
       status reporting and project schedule updates that are integrated with input from
       DOL and other vendors; (k) detailed installation instructions and scripts; and (l)
       post-deployment support. Details of these deliverables need to be put into the
       RFP to maintain quality control through the replatforming process to comply with
       DOL standards and to ensure acceptable quality of the work product resulting
       from that process.
   •   The RFP must make it clear sign-offs for Architecture Reviews, Design Reviews,
       and Code Reviews are contract deliverables. The vendor assumes all risk if they
       proceed with work that has not been signed-off at the previous step.


                                         Page 21
•   If the vendor chooses to change an approved architecture or design, that change
    must be reviewed and approved by DOL.
•   When a vendor is required to convert code and or data via an automated tool,
    specify they will be required to provide auditable proof the conversion is accurate
    and complete. The penalty of not doing so will be forfeiture of xx% of the
    holdback.
•   If DOL does not have staff available with sufficient knowledge, experience, and
    skills to evaluate architecture, design, and code quality, or can’t do so within five
    days of each associated delivery by the vendor, specify in the RFP this may be
    contracted to Microsoft or other experts. Then do so.
•   Without quantitative metrics, stress and performance goals have no “teeth.” The
    RFP should specify the following Performance & Stress Test results:
       o The quantitative, existing baseline performance under normal and stress
          loads
       o The level of stress to be applied in quantitative terms
       o The required application performance under load that must be delivered
       o The method by which performance must be measured
       o There must be no database locks or timeouts
       o There must be no application timeouts or aborts
       o In general, users must experience screen-to-screen transitions that are sub-
          second; however, some screens are known to have heavy processing load
          and thus transitions for these screens must match or exceed that of the
          existing system.
       o Additionally, no application performance degradation may negatively
          impact the user’s workflow processing speed
       o Queries to the databases must specify “NOLOCK” to avoid transactions
          from being blocked
•   Reports on Stress and Performance Testing must contain both performance graphs
    produced by the Stress Test tool AND a narrative interpretation of the results
    must be a deliverable. The RFP must specify the minimum metrics to be reported,
    the stress levels to be applied, and the performance levels that must be achieved.
•   Stress Tests should NOT be planned or run by the vendor who did the coding and
    code replatforming, and must be done under DOL’s control.
•   The RFP should state the level of testing that the vendor must perform and quality
    of testing results (no bugs?) the vendor must achieve before they hand-off code to
    the test team. . Fujitsu’s criterion was the screen should appear more-or-less
    correctly and the program must compile without errors. This led to weeks more
    than planned of bug fixes and re-testing due to screens and functionality aborting
    or not working correctly.




                                       Page 22
Vendor Selection
•   The interview, followed by a required live (not PowerPoint) demo, was again an
    excellent evaluation approach. It showed clear differences between vendors that were
    not apparent from the vendor proposals.
•   The code conversion the vendors are required to demo as part of the selection
    interview must be representative of the types of code to be converted and the level of
    difficulty of the project. The Heating Oil (HOAP) replatforming that served as the
    pilot stage for the HP 3000 Replatforming Project was too simple to judge the
    vendor’s process or skills at doing the VFS replatforming. For example, the pilot did
    not include any electronic interfaces with which the Fujitsu code must interact, while
    there were over 100 interfaces in the VFS project conversion.
•   It was not a good idea change the RFP after it had been published. The change
    specified a different vendor do the interface replacement part of the replatforming.
    Although the interface vendor did an excellent job, it required the primary vendor and
    the interface vendor to work in close collaboration and this added a risk to the project.
    This change in the RFP was made so late, several vendors who were preparing
    proposals had already worked out their bid for that work, and had included it with
    their proposals.

Contract Terms & Conditions
    During the contract negotiations, DOL required Fujitsu to have Microsoft do quality
    reviews of and approve all code conversion approaches to be implemented by Fujitsu.
    (DOL realized that it did not have staff with the bandwidth and technical expertise at
    the time to do such reviews, but DOL needed to ensure Fujitsu replatformed the DOL
    applications such that effective and efficient code was delivered.) Fujitsu reported
    that the Microsoft reviews were done, but refused to supply documentation to the
    effect. (They said that providing such information was not an appropriate
    requirement by DOL for a fixed bid contract.) Subsequent reviews of the code by
    Microsoft during Stress Testing, done at DOL’s expense, showed many serious
    design flaws that Microsoft said had been reported to Fujitsu early in the project. This
    fact brings into question whether Fujitsu actually took action on Microsoft findings.
    Fujitsu wound up having to fix most of the Microsoft-reported problems to achieve
    code stability and adequate performance of their code under load.
•   If DOL wants a vendor to review and guide the quality of work performed by another
    vendor, DOL needs to contract directly with the vendor doing the reviews, not have
    the developer sub-contract that quality check.
•   Include in the contract a reasonable number of no-charge, Change Request hours for
    small-to-medium effort changes; this continue to be a valuable and effective approach
    to minimizing change request overhead for small-cost issues. This project had a 500
    hour buffer.
•   Partial payment for systems must be contingent on achieving at least ten consecutive
    post-deployment days without a new bug being identified. The amount of the
    associated payment should be significant so the vendor is driven to complete


                                          Page 23
    stabilization quickly. With this constraint, Fujitsu worked very hard to achieve
    stability during the Unisys Replatforming Project. Without this constraint, Fujitsu
    notably did little to speed achievement of stability. In fact, they did not apply the
    needed resources and blamed DOL continuously for not providing sufficient
    information to find reported bugs. Without the constraint there was no proactive
    effort by the vendor to fix defects.
•   Start the “warranty period” only after the ten consecutive post-deployment bug-free
    days. During the warranty period which the vendor is required to provide quick-
    response (1 hour response, 8 hour correction) of any additional bugs that are found.
    This requires that the vendor’s technical experts from the project must remain
    available and have the top priority assignment of responding to DOL defect reports.
    They may or may not remain on-site at DOL. The warranty period should have
    duration of 60 to 90 calendar days. A final vendor payment of the remaining 20
    percent holdback is made at the end of the warranty period.
•   If deployment does not take place so that the warranty period will be completed by
    end-of-biennium, the associated holdback payment should be forfeited.
•   If the warranty period is not completed by the end-of-biennium, DOL will determine
    the portion of the associated hold-back payment to be made to the vendor.
•   The contract boiler-plate states that because the work is being done under a public
    contract all documents and information produced under the contract are public
    property, and the vendor may not withhold any such information. However, because
    of the fixed-price nature of the contract, information related to the vendor’s internal
    costs and payments to sub-contracts may not be required to be disclosed by other
    laws. There needs to be a clause in the contract that says the vendor will promptly
    (this may need definition) deliver to DOL, uncensored and unmodified, any and all
    technical information produced by or supporting the vendor’s execution of the
    contracted work. I recommend this because of Fujitsu’s refusal to disclose the
    amount of code converted by automated tools vs. manually, and their refusal to
    release Microsoft’s comments and suggestions on the Fujitsu code conversion design
    and resulting VB.net code.
•   Put into the RFP criteria for making payments for completion of code testing steps,
    not the delivery of code.
•   Holdback amount should be large enough to put some teeth into the vendor
    completing work promptly. The 20 percent holdback used in the UAR and HP3000
    projects seemed to provide minimal leverage.



Implementation: Has the project developed a reasonable plan for implementation? Are
the duration and amount of user training adequate?




                                          Page 24
Training
User Training
   •   Do more PowerPoint with voice-over style training. Field office users were very
       appreciative of the user training provided by this project. It put the ability to
       replay the training under the control of each individual as well as provide a hands-
       on environment (EDUCation Mode) in which to practice before the application
       “goes live”.
   •   Brad Benfield has equipment and the “professional voice” to record the voice-
       overs
   •   Make booklets of detailed feature instructions for each change / enhancement in
       addition to PowerPoint based training.
   •   Delivery of BOTH the PowerPoint training and the detailed booklet was
       considered by the field as much more beneficial and longer-lasting way to deliver
       training than the previous, time-consuming on-site training by DOL staff.
   •   It might be helpful to DOL to provide a knowledge / skill test with this training to
       give users feedback on how much of the critical information and skills that they
       actually mastered from this training.
   •   Involvement of the Test Team (with their experience using the modified
       application) both in preparing user training materials and delivering training (e.g.
       implications and interpretation of Exceptions when IU went live) was very
       effective for the User Support teams.
   •   Make sure to actively explore with users the impacts on them from changes made
       to the application. Talk through each screen that they use. Talking in terms of
       database changes is too abstract for users to see potential issues.

Technical Training
   •   IS staff did not rate the technical training by Fujitsu to be complete or very
       effective. The technical trainer was reported by DOL staff to not clearly
       understand the application and wasn’t able to supply answers to DOL developers’
       questions about how business functions were implemented. He also deferred
       questions and then never provided answers. The follow-up session with Fujitsu’s
       lead developer was more helpful.
   •   Part of the problem with the technical training came from the fact that key
       documentation by Fujitsu wasn’t provided until well after the technical training
       was delivered.
   •   Part of the DOL staff’s problems with the technical training was due to the fact
       that DOL staff hadn’t had enough hands-on experience working in the Windows
       .net environment. They had received training in .net tools, but not sufficient
       hands-on experience. Their work had remained primarily focused on the HP
       environment. They did not at the time fully understand how to manage code or do
       troubleshooting in the Windows environment.


                                         Page 25
   •   Although all DOL staff that are now supporting the replatformed VFS application
       learned the needed skills by the end of the project, the technical training lost value
       due to DOL staff’s lack of meaningful .net maintenance and troubleshooting
       experience.
   •   Documentation by DOL is now needed on how to change the VFS EDUCation
       mode to allow users hands-on experience with the to-be version of VFS prior to
       deployment into Production. (See also “User Shakedown” in this document.)
       Following deployment the EDUCation mode will match Production to allow
       training /retraining of field staff.

Deployment Planning
• A Work Group with a Lead other than the Project Manger should be established three
  months before deployment to do the detailed deployment planning and to ensure all
  details are carefully reviewed. The goal should be no change in the Deployment
  Checklist or the associated detailed installation instructions be made on deployment
  weekend. It takes too much effort and time to plan the environment details, establish
  a checklist, develop detailed instructions and scripts, and have at least four rounds of
  detailed reviews and revisions for this work to be lead by the Project Manager.
• As each deployment detail is documented, the associated back-out instruction and
  scripts must be prepared. It is difficult and time-consuming to prepare back-out
  instructions later.
• Involving the Test Team (with their hands-on experience with the modified
  application) in preparation of business user support staff was most effective.
• The activity of creating the Deployment Checklist and then using it to track activities
  is still a best practice that should be scaled for and used with every project.
• Addition of the matrix that maps the relationship between checklist items,
  deployment instruction documents, and deployment scripts was a very helpful
  addition during the IU+PSP deployment for tracking deployment readiness. This
  matrix also helped LAN Support review and execute the deployment instructions.
• Supplying detailed deployment instructions to LAN Support for review in June when
  the deployment was not until September did not result in an earlier or more thorough
  review. The Project Manager must facilitate those reviews to make them happen.
• Developers, not LAN Support, must make sure all the required changes in web.cfg
  and machine.cfg are documented for the deployment.
• One of the checklist reviews should focus on the task dependencies. Make sure (1)
  all dependencies are identified, and (2) if there are opportunities to do any task
  earlier, that is also identified. One finding is that we would have been able to start
  some of the Operational Readiness testing sooner, and thus finish sooner.
• The Deployment Checklist and all detailed instructions and scripts must be finalized
  two weeks before deployment.




                                          Page 26
• DOL Network Support Services (NSS) requested the Deployment Checklist have hot
  links to the referenced detailed instruction documents and installation / roll-back
  scripts. This allows them to more quickly get to the needed information and execute
  installation scripts.
• NSS wants to get the detailed deployment instructions early so they have time to
  review them, and in some cases re-organize them into group actions of one type (e.g.
  modifying a web.cfg file) so they can see if there are conflicting instructions and can
  do tasks more efficiently on deployment day. The Deployment Workgroup Lead
  (supported by the Project Manager) must insist all deployment materials are ready
  early enough to have three or more reviews with NSS.
• When the replatformed VFS application was deployed, NSS staff took hours longer
  than planned to do their tasks because they decided on that day they needed to set
  things up differently than had been done over prior weeks. This caused a high risk to
  the deployment’s success. This will not be repeated.
•   The lead developers associated with the application(s) being deployed should be
    physically present at HLB in the NSS area to answer any questions that come up by
    the NSS team during deployment.
•   DOL will pursue automating all of the application deployment activities. This
    includes both the installation and any back-out steps. Automation of application
    installations has been common practice for many years and should also be used
    within DOL to avoid risks of human mistakes. It would allow the deployments to go
    faster, without risk of mistakes that happen when staff is under stress and tired.
    Deployment automation should be built to be re-usable and adaptable for each change
    to the application; having a simple mechanism for selecting which of the components
    can remain as-is and which must be replaced.
•   The Test Lead’s verification that TREC, Liaison, and USS all understood the changes
    being deployed with IU+PSP immediately before that deployment resulted in those
    groups being able to correctly and effectively support users and their questions.
    Training provided by the Test Lead also covered how to process the new exception
    reports created by the application and the trouble reporting process that would be in
    effect the week immediately following deployment. This just-in-time training was
    very effective and resulted in prompt and smooth issue handling following
    deployment.
•   It is important to treat users calling in to report post-deployment problems with
    courtesy and respect. We had complaints on several instances the response was, “We
    already know about that problem,” without a “Thank you for reporting that.” DOL
    support staff and mangers need to be reminded just before deployment to (a) be
    sensitive to the user community as people in needing post-deployment help, and (b)
    as customers offering information as partners willing to collaborate on fixing
    problems.
•   If there are application security features within the application, make sure DOL’s
    Network and Application Security staff are trained in the post-deployment support
    planning.


                                         Page 27
•   Go / No-go Decisions: The Project Manager should supply daily status updates to the
    Steering Committee until he/she can recommend and gain “Go” approval.
•   Just before deployment, put out a quick notice to all users, user support staff, and
    business managers reminding them of the date, what to expect (what’s changing), and
    how to report problems and get help.
•   A “dress rehearsal” for the deployment should be included in the project schedule to
    ensure that checklist, instructions and scripts work. In this case an opportunity to do
    a deployment dress rehearsal occurred when the EDUCation mode version of VFS
    went live on a Pre-Production server to support User Shakedown.

Post-Deployment
•   When planning post-deployment support activities, procedures, and communications,
    start by creating a vivid vision of the worse-case situation. Plan actions and
    communications to address that level of problems. This proved very helpful in
    identifying a solid strategy that worked well and could easily be scaled back as it
    became clear the post-deployment was going to be smoother than expected.
•   Those who provide user support for the new / modified application should have had at
    least two weeks of hands-on use of the applications prior to deployment.
•   Relocating some of the testers into the user support work areas at HLB immediately
    following deployment, let the Testers extensive experience with the application be
    immediately available to that staff. This improved the confidence of the support staff
    in working with users and got the correct information to the users when they called
    with questions or to report problems.
•   Use the intranet site to keep everyone internally and in the field informed on the
    status of reported bugs. (Requires someone to maintain.)
•   Test Lead holds daily, early a.m. status review with TREC, Liaison, and USS.
•   Publish Daily Status Update for agency executives and the project Steering
    Committee. List Activities & Summarize Bug status in short, understandable terms.
•   PDSI and DOL IS staff provided great support and collaboration with the Test Team
    during the IU+PSP work, both before and after deployment.
•   The Test Team learned during the replatformed VFS post-deployment period how
    critical it was to provide Fujitsu with detailed information about how to reproduce
    reported bugs. Fujitsu didn’t understand the business situations in which the bugs
    were appearing, so they didn’t have an adequate starting point for troubleshooting.
•   Fujitsu did not put in the effort to effectively collaborate with the Test Team and this
    led to it taking almost four months to stabilize the Replatformed VFS.
•   Transactions reporting failures were frequently not the location of the defect. It is
    important when capturing failure information to learn what the user was doing
    immediately prior to the failure was reported. The actual problem was likely in that
    prior transaction.



                                          Page 28
•   There was not enough communications between the Project and those supporting
    users (TREC-Liaison-USS) immediately prior to and immediately following the VFS
    deployment. This was corrected during the IU+PSP deployment and made for a very
    effective user-support partnership for IU+PSP because of that change.



Additional Items:


Additional Project Processes to Recommend
•   Take the time as soon as possible to make the project schedule as comprehensive as
    possible and use it to estimate when and how many resources will be needed.
    Communicate that need in person to all impacted.
•   Update the project schedule weekly.
•   Keep attempting to get effort-based (rather than duration-based) estimates for tasks
    from staff and vendors. Keep attempting to manage using Earned Value. We will
    likely have to use “hours” as the “value” rather than “cost’ as fixed-bid contractors
    would not want to divulge cost.
•   Keep the whole project team recording decisions into the project’s tracking tool. (We
    could have done much better here, but were still better than most projects.)
•   Our tracking of deliverables-based contracts, hourly contracts, mixed deliverables /
    hourly contracts, and all associated documentation was very good. This may become
    more complex when using the iECMS computerized tracking system over
    customizable Excel spreadsheets.
•   Schedule monthly Steering Committee meetings AFTER the 15th of the month (AFRS
    close date) so you can report the “final” values from the prior month.
•   Using the weekly Core Team meetings to do the non-technical problem solving and
    decision making with the key business representatives established a strong level of
    business participation that also improved communications both to and from business
    functions.
•   The Project Manager was effective in getting NSS out of the loop at critical times in
    the testing, such as negotiating for the Test Team to be able to do their own database
    backups and restores. A backup or restore typically took two hours to complete
    because of the formal request and approval process. But 10 to 20 such changes were
    required on some testing days. By temporarily allowing testers to directly run batch
    jobs that did these repetitive backups and restores in the QA environment, no laps in
    security was opened yet considerable time was saved.
•   Not having a Technical Lead with excellent knowledge of Windows internals and the
    .net Foundation until late in the project was a considerable hurdle to good
    communications and collaboration with NSS. The representative originally supplied



                                          Page 29
    by NSS to fill that role did not have sufficient technical knowledge and
    communication skills.
•   Creating and using a Deployment Checklist with detailed installation instructions and
    scripts continued to be a best practice, especially for deployments as complex as those
    associated with application replatforming.
•   Hold user interface reviews with application users as soon as a few screens are
    ready. Repeat two-to-three times to identify required changes as early as possible to
    minimize development re-work. Get formal sign-off from those doing the review.
    Be sure to include representatives from field offices if they are application users.
    (This sign-off should be a project schedule milestone / deliverable and managed
    within the Training Work Group.)
•   SWAT Teams that included most IS managers plus business managers were quickly
    formed when major issues surfaced. One team was formed immediately following
    the VFS deployment. A second was formed when we observed a dramatic slowdown
    of VFS that was impacting users. These relatively short-lived teams made sure the
    resources and the focus was placed on fixing these problems. They were very
    effective. The daily status reports distributed by the Project Manager on SWAT
    Team actions and progress helped all parts of the Agency and the user community
    understand what was being done, established a strong problem-solving partnership
    and reduced apprehension about the problem.
•   As DOL IS staff develop technical knowledge, skills, and maturity with the now-
    standard .net platform there will be problems that must be quickly escalated to outside
    expert help. Troubleshooting the VFS slowdown was one such case. Internally we
    collected as much information as we could and then brought in Microsoft, Right!
    Systems, and others. DOL staff are mastering the new technologies and making great
    strides in finding solutions. There is important growing confidence in our internal
    ability to use and gain the productivity boost from the new methods and tools. We
    are learning from these experts, but will continue to need expert advice to quickly
    solve complex technical problems as we mature technically.




                                         Page 30

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:11
posted:10/9/2011
language:English
pages:30