Backup and Recovery by jennyyingdi

VIEWS: 30 PAGES: 17

									            Team for Texas Service Delivery




State of Texas Data Center Services
   Policies and Procedures Manual


        8.3.1 Backup and Recovery




                                      Total Pages: 17
                               Document Version: 2.0
                             Last Revised: 05/26/2009
State of Texas DCS Policies and Procedures Manual                                                          Backup and Recovery




Table of Contents
8.3.1    Backup and Recovery .............................................................................................. 3
  8.3.1.1 Description.............................................................................................................. 3
  8.3.1.2 Scope ..................................................................................................................... 3
  8.3.1.3 Objectives ............................................................................................................... 4
  8.3.1.4 DIR and Team for Texas Roles and Responsibilities .............................................. 6
  8.3.1.5 High-level Interface Flow and Narrative .................................................................. 9
  8.3.1.6 Policies ..................................................................................................................13
  8.3.1.7 Interface Information ..............................................................................................16




8.3.1 St of TX Backup and Recovery                                                                                     Page 2 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                      Backup and Recovery




8.3.1           Backup and Recovery
This section provides an overview of the services and interfaces related to Backup and Recovery for DIR,
DIR Customers, and Team for Texas.



8.3.1.1 Description
Backup and Recovery includes normal operational retrieval of offline data, data file backups, and
restoration of lost data. This also includes planning, testing, and implementing procedures and standards
required to provide normal Services in the event of any failures, including routine operational retrieval or
restoration needs and Disaster Recovery.
Backup and Recovery describes how Team for Texas addresses DIR and DIR Customer data backup
requirements, including off-site vaulting of data, cataloging off-site content, retrieving backup tapes,
restoring data from the backups, and retaining the capability to read files on long-term retention (Archival)
media.
Backup and Recovery is closely associated with and often receives requests that requires 8.3.5 External
Storage Media Management to satisfy the backup and recovery requirements of DIR Customers. Backup
and Recovery includes strategic management of the backup application that executes removable storage
media functions, and External Storage Media Management includes the day-to-day operational aspects of
providing physical removable storage media support. Refer to 8.3.5 External Storage Media Management
for more information.



8.3.1.2 Scope
Backup and Recovery includes the following activities:
       Providing backup and recovery Services to facilitate restoration of applications and
        associated data
       Verifying that backups and recoveries are completed without errors within 24 hours
       Reporting weekly on backup type, age of latest restore, and success and failure of scheduled
        backups
       Reporting monthly on backup Service Level Agreement (SLA) attainments
       Retaining success and failure data for 13 consecutive months
       Testing backups quarterly to confirm data can be restored
       Handling maintenance activities such as adding, changing, or upgrading items in the backup
        and recovery environment and handling new requirements
       Analyzing the technical environment
       Developing a Backup and Recovery Plan
       Verifying the Backup and Recovery Plan on an annual basis
       Developing requirements, procedures, and standards for off-site storage, in consultation with
        DIR Customers, and obtaining approval from DIR
       Minimizing Third Party off-site storage requests by converting to alternative data transmission
        methods (such as file transfer protocol - FTP)




8.3.1 St of TX Backup and Recovery                                                               Page 3 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                     Backup and Recovery




      Maintaining the logical integrity of data shipped to off-site storage by maintaining the
       accuracy of the backup application database
      Defining backup, restore, and retention requirements for operating systems and Third Party
       Software
      Cleaning and maintaining Equipment to minimize problems and outages at intervals
       prescribed by the manufacturer
      Directing DIR Customer personnel that perform external storage media mounts, per the
       Policies and Procedures Manual (PPM), at DIR facilities not staffed by Team for Texas
       personnel
      Recopying external storage media as necessary to support minimization of read/write errors
      Initializing new external storage media
      Maintaining and updating procedures that govern recycling and rotation of external storage
       media, external storage media management, and external storage media retention periods,
       with attention to auditing
      Recycling media regularly, managing media replacement, and recopying media to provide
       data integrity and quality
      Disposing of retired media in an environmentally sound manner after purging any DIR or DIR
       Customer data, using tools or processes which meet guidelines, prior to disposing of the
       media. Refer to 4.9 Asset Management for more information.
      Operating and supporting the media library and library management system, maintaining a
       database cataloging the archival system for the media library
      Maintaining an existing inventory control system to properly manage external storage media
       in storage, and preparing media for shipment to the remote computer recovery centers
      Maintaining the integrity of the external storage media library system
      Monitoring external storage media:
          Monitoring external storage media for reliability and minimization of read and write errors
           during the entire period of retention
          Monitoring and reporting on external storage media usage
          Monitoring external storage media in case of malfunction and initiating corrective action in
           accordance with established procedures
      Complying with DIR, state, and federal requirements related to the monitoring, maintaining,
       and reporting of external storage media, including those described in Exhibit 17A of the
       Agreement, and PPM sections 4.9 Asset Management and 4.17 Record Retention



8.3.1.3 Objectives
The objectives of Backup and Recovery are to:
      Create, maintain, and execute Backup and Recovery Plans
      Perform system and DIR Customer data backup and recovery as required and in accordance
       with DIR and DIR Customer standards and policies
      Perform nightly backups on all DIR Customer-identified systems
      Provide disk backup nightly, as appropriate to the Applications, with off-site storage for
       business continuity purposes


8.3.1 St of TX Backup and Recovery                                                                Page 4 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                     Backup and Recovery




      Perform backup and recovery procedures in response to security violations that result in lost
       or damaged information
      Perform backup and recovery procedures to assist Disaster Recovery and Business
       Continuity Services
      Establish priorities for backup and recovery procedures in Disaster Recovery and Business
       Continuity situations, based on the priorities established by DIR and DIR Customers
      In connection with the Server and Mainframe Services, provide and maintain backups, file
       recovery capabilities, and historical files of data and Software (including source code) utilized
       to process data. Additional responsibilities include:
          Adhering to the time periods specified for backups and recovery by DIR Customers in the
           Disaster Recovery plan or the Runbooks
          Using magnetic media, or other media approved by DIR and accessible to DIR/DIR
           Customers
      Provide for salvage and reclamation of media where possible and/or destruction of
       unrecoverable media in accordance with DIR guidelines and Standards
      Complete DIR Customer defined batch processing and backups in the correct sequence and
       in a timely manner according to the process and appropriate Service Levels
      Provide for automated or manual scheduling of backups
      Provide for scheduling of automated or manual shutdown and restarting of systems required
       for backups
      Provide backups and restores of data in a timely manner
      Define backup, restore, and retention requirements for operating systems and Third Party
       Software
      Retrieve on a quarterly basis randomly selected data files or datasets as a test and verify that
       the data can be restored in a usable fashion from the backup media
      Restore required data in a timely manner according to the process and appropriate Service
       Levels
      Assume responsibility for implementing DIR Customer system data backup requirements,
       which include periodic off-site vaulting of data on media and cataloging off-site content
      Provide conversions to automated backups to minimize or eliminate the need for remote
       personnel to perform normally scheduled backups
      Comply with physical specifications, retention periods, and security; provide compliance
       reports
      Restore lost, deleted, or damaged data files or datasets upon request of Authorized Users




8.3.1 St of TX Backup and Recovery                                                              Page 5 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                   Backup and Recovery




8.3.1.4 DIR and Team for Texas Roles and
        Responsibilities
The DIR, DIR Customer, and Team for Texas roles involved in Backup and Recovery, and the
responsibilities associated with those roles, are listed below.
Please note:
       Key responsibilities include those listed for each role
       Roles are meant as logical groupings of tasks. They are not meant to match any particular
        organizational structure or formal job roles
       Several roles may be performed by the same individual
       A role may be split up among several individuals



Requester (DIR, DIR Customer, or Team for Texas)
The Requester is responsible for the following:
       Submitting backup requests and recovery requests to the Team for Texas Backup and
        Recovery Team, through the Support Center
       Verifying that each request contains the following information necessary to complete the
        request:
        For a backup request:
           Information about the Requester as entered in the originating Remedy record, including
            name, location, and contact method
           Location of the data objects to be backed up
           Requested schedule and retention for backing up new data objects
           Whether the request is for a one-time backup or an ongoing backup
        For a recovery request:
           Information about the Requester as entered in the originating Remedy record, including
            name, location, and contact method
           Target for where the data objects should be recovered
           Requested timeframe for the recovery
        For quarterly backup integrity testing:
           Information about the Requester as entered in the originating Remedy record, including
            name, location, and contact method
           Files to be restored, in accordance with the Integrity Testing policy
           Location where the data files are to be restored




8.3.1 St of TX Backup and Recovery                                                           Page 6 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                    Backup and Recovery




Team for Texas Backup and Recovery Analyst
The Team for Texas Backup and Recovery Analyst is responsible for:
      Analyzing requests for new backup and recovery support
      Developing backup and recovery test and implementation plans (not specific to Disaster
       Recovery plans)
      Implementing and testing new solutions
      Serving as a representative for backup and recovery change control
      Generating requirements for tool changes
      Investigating new technologies to improve backup and recovery, including increased
       automation of routine backups
      Setting up the backup and recovery infrastructure in all Team for Texas-managed data
       centers
      Developing standards for off-site storage and vaulting
      Assuming operational responsibilities for all external storage media management functions
      Maintaining procedures for monitoring, rotating, recycling, reusing, and disposing of external
       storage media. Refer to Exhibit 17A of the Agreement, and PPM sections 4.9 Asset
       Management and 4.17 Record Retention for more information.
      Following DIR/DIR Customer priorities for backup and recovery in Disaster Recovery and
       Business Continuity situations
      Defining requirements for backup, restore, and retention for operating systems and Third
       Party Software
      Defining and publishing media labeling standards



Team for Texas Backup and Recovery Operator
The Team for Texas Backup and Recovery Operator is responsible for:
      Starting and running backup and recovery applications on the system according to batches,
       schedules, or requests
      Performing backup and recovery activities in a timely manner, including daily checklists
      Monitoring job results
      Verifying that backups and recoveries completed without errors within 24 hours
      Testing backups quarterly to confirm data can be restored in a usable fashion
      Reporting weekly on backup type, age of latest restore, and success and failure of scheduled
       backups
      Reporting monthly on backup SLA attainments
      Retaining success and failure data for 13 consecutive months
      Preparing and submitting routine server and mainframe backups to off-site storage vault
       locations




8.3.1 St of TX Backup and Recovery                                                            Page 7 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                   Backup and Recovery




      Following operational procedures that are supplied
      Performing maintenance activities:
          Cleaning and maintaining Equipment used in backup and recovery
          Maintaining backup configurations
          Maintaining supported Backup Application Version level n, n-1
          Maintaining activity logs
      Using approved and accessible media
      Following media labeling standards
      Preparing and managing external storage media
      Monitoring, rotating, recycling, reusing, and disposing of external storage media
      Directing DIR Customer personnel that perform external storage media mounts at DIR
       facilities not staffed by Team for Texas personnel
      Operating and supporting the media library and library management system
      Maintaining the accuracy of the backup application database in the media library
      Maintaining an existing inventory control system for external storage media in storage




8.3.1 St of TX Backup and Recovery                                                              Page 8 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                                                                                     Backup and Recovery




8.3.1.5 High-level Interface Flow and Narrative
The following high-level Backup and Recovery interface flow illustrates the interfaces between DIR and
Team for Texas service delivery personnel.
 DIR Customer, or
 Team for Texas)
 Requester (DIR,




                                                                                                                 5
                                                                                                                                                                    18
                                                                                                            Contact DIR
                                                              1                                                                                                 Receive
                                                                                                             Customer
                                                            Support                                                                                            Notification
                                                                                                          Representative for
                                                            Center                                                                                            Request was
                                                                                                            Backup and
                                                                                                                                                               Completed
                                                                                                             Recovery
   Team for Texas Backup and




                                                                                        3                         4
       Recovery Analyst




                                                               2
                                                                                   Requester       No     Advise Requester
                                                        Analyze Request
                                                                                    Entitled?             Whom to Contact


                                                                                      Yes


                                                                                                                  6
                                                                                                              Backup or
                                                                                                              Recovery?


                                                                                                    Recovery              Backup




                                                                           8
          Team for Texas Backup and Recovery Operator




                                                                                                      7
                                                                       External
                                                                                                 Removable                     12
                                                                        Storage
                                                                                        Yes         Media                 Update Backup
                                                                         Media
                                                                                                  Involved?                 Schedule
                                                                      Management
                                                                                                     (1)
                                                                          (1)
                                                                                                     No

                                                                                                                                                     14
                                                                                                                                13
                                                                                                                                                  External
                                                                                                                            Removable
                                                                                                     9                                             Storage
                                                                                                                               Media      Yes
                                                                                                Recover Data                                        Media
                                                                                                                             Involved?
                                                                                                                                                 Management
                                                                                                                                (2)
                                                                                                                                                     (2)
                                                                                                                                No


                                                                                                    10                          15
                                                              11
                                                                                                 Recovery                  Perform New
                                                           Incident                   No
                                                                                                 Successful                 Backup as
                                                         Management
                                                                                                     ?                      Scheduled

                                                                                                    Yes


                                                                                                                               16                                17
                                                                                                                             Backup                       Notify Requester
                                                                                                                  No                       Yes
                                                                                                                            Successful                    Backup/Recovery
                                                                                                                                ?                          was Successful




8.3.1 St of TX Backup and Recovery                                                                                                                                 Page 9 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                    Backup and Recovery




Backup and Recovery Interface Narrative
 Objective:        To successfully copy required data to alternate media, and to recover data on the
                   Team for Texas-managed systems
 Roles:            ▪   Requester (DIR, DIR Customer, or Team for Texas)
                   ▪   Team for Texas Backup and Recovery Analyst
                   ▪   Team for Texas Backup and Recovery Operator
 Prerequisites:    ▪   Procedural documentation provided
                   ▪   Contact list included in documentation
                   ▪   Custom scripts and programs supplied
 Inputs:           ▪   Backup request (scheduled or special)
                   ▪   Recovery request, including items to be recovered
                   ▪   Details of data files or datasets to be recovered
 Controls:         Backup and Recovery Plans
 Outputs:          ▪   Backed up data files or datasets
                   ▪   Recovered data files or datasets ready for use

The Backup and Recovery interface narrative includes the steps outlined below.
       Role            Step                                  Description
    Requester           1     Support Center
    (DIR, DIR                 Requests that are not already scheduled via an automated tool start at the
  Customer, or                Support Center and are entered in the Remedy tool. Requests for backup
 Team for Texas)              and recovery can be routed through 4.2.1 Incident Management, 4.5
                              Request Management, or 4.8 Change Management.
                              Submit the following information for a backup request:
                              ▪ Information about the Requester as entered in the originating Remedy
                                 tool record, including name, location, and contact method
                              ▪ Location of the data files or datasets to be backed up
                              ▪ Requested schedule and retention for backing up new data files or
                                 datasets
                              ▪ Whether the request is for a one-time backup or an ongoing backup
                              Submit the following information for a recovery request:
                              ▪ Information about the Requester as entered in the originating Remedy
                                 record, including name, location, and contact method
                              ▪ Target for where the data files or datasets should be recovered
                              ▪ Requested timeframe for the recovery
  Team for Texas        2     Analyze Request
   Backup and                 Receive request and analyze for actions required. Review Requester
    Recovery                  entitlement.
     Analyst
                        3     Requester Entitled?
                              Determine or reconfirm that the Requester is entitled to request the
                              particular backup or recovery service they are requesting.
                              ▪ If Yes, proceed to Backup or Recovery?
                              ▪ If No, proceed to Advise Requester Whom to Contact
                        4     Advise Requester Whom to Contact
                              Direct the Requester to the appropriate contact to submit a backup or
                              recovery request.



8.3.1 St of TX Backup and Recovery                                                           Page 10 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                  Backup and Recovery




       Role           Step                                  Description
    Requester          5     Contact DIR Customer Representative for Backup and Recovery
    (DIR, DIR                Contact the individual DIR Customer Representative entitled to submit
  Customer, or               requests for backups and recoveries.
 Team for Texas)             Return to Support Center.
  Team for Texas       6     Backup or Recovery?
   Backup and                ▪ If Recovery, proceed to Removable Media Involved? (1)
    Recovery                 ▪ If Backup, proceed to Update Backup Schedule
     Analyst
  Team for Texas       7     Removable Media Involved? (1)
   Backup and                Determine if the recovery activity requires an External Storage Media
    Recovery                 Management activity.
     Operator                ▪ If Yes, proceed to External Storage Media Management (1)
                             ▪ If No, proceed to Recover Data
                       8     External Storage Media Management (1)
                             Invoke 8.3.5 External Storage Media Management to handle the tape or
                             external storage media requirements of the recovery activity.
                       9     Recover Data
                             Recover the necessary data files or datasets. The source is the media
                             where the items were backed up. The source may be another computer
                             on the network or physical media such as tape or optical disk. The media
                             may need to be retrieved from off-site storage.

                      10     Recovery Successful?
                             ▪ If Yes, proceed to Notify Requester Backup/Recovery was
                                Successful
                             ▪ If No, proceed to Incident Management
                      11     Incident Management
                             Invoke 4.2.1 Incident Management to handle the issue.
                             Return to Backup or Recovery?
                      12     Update Backup Schedule
                             Integrate the new backup into the existing schedule of backups.
                      13     Removable Media Involved? (2)
                             Determine if the backup activity requires an External Storage Media
                             Management activity.
                             ▪ If Yes, proceed to External Storage Media Management (2)
                             ▪ If No, proceed to Perform New Backup as Scheduled
                      14     External Storage Media Management (2)
                             Invoke 8.3.5 External Storage Media Management to handle the tape or
                             external storage media requirements of the backup activity.
                      15     Perform New Backup as Scheduled
                             Execute the backup as scheduled.




8.3.1 St of TX Backup and Recovery                                                         Page 11 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                   Backup and Recovery




       Role           Step                                  Description
  Team for Texas      16     Backup Successful?
   Backup and                ▪ If Yes, proceed to Notify Requester Backup/Recovery was
    Recovery                    Successful
     Operator                ▪ If No, proceed to Incident Management

                      17     Notify Requester Backup/Recovery was Successful
                             Inform the Requester that data was backed up or recovered as requested.
                             If 8.3.5 External Storage Media Management was invoked to send or
                             retrieve tape or external storage media, invoke 8.3.5 External Storage
                             Media Management again as necessary to return the tape or external
                             storage media to its appropriate safe location.
    Requester         18     Receive Notification Request was Completed
    (DIR, DIR                If a recovery, check the recovered data to verify that the request was
  Customer, or               completed and that the data is usable.
 Team for Texas)             Return to Support Center for record closure.
                             The Backup and Recovery procedure is complete.




8.3.1 St of TX Backup and Recovery                                                          Page 12 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                     Backup and Recovery




8.3.1.6 Policies

Entitlement
Confirm that the request is from an Authorized User who is authorized to request a backup or restore of
the particular data. A person may be entitled to service but not be entitled to make this request. Although
initial entitlement analysis is performed by the Support Center, a second level of entitlement review
occurs within Backup and Recovery to reconfirm that the Requester is authorized. Due to the potential
impact of an unwarranted or unscheduled restore, and the cost associated with unnecessary
procurement, this secondary entitlement is performed.



Data Media
For all mainframes, servers, and locations, external media must be procured by Team for Texas. The
various media used for creating backups have different life expectancies, or allowable number of “writes”
while still remaining within the specifications. The media must be tracked and removed from service when
it has reached the end of its use.
Recovery time is the elapsed time from receiving the request and completing the second entitlement to
the time the data is recovered and ready for use.
Each DIR Customer determines the retention period, the number of extra copies retained, and the
classes of information. The numbers of copies of a deleted data file or dataset and their retention periods
also must be determined for classes of information. The data owner is required to analyze and classify
data files or datasets so the retention period can be assessed. Team for Texas will comply with all DIR
Customer record retention policies as provided by the DIR Customer.



Data Backups
Data that is locally attached and accessible through the file systems or databases on Team for Texas-
managed servers and mainframes in case of a loss of data are backed up based on the following policies:
       For all systems, incremental backups of configuration data and file systems are taken as
        specified. Incremental backups of application data are taken as specified. Incremental
        backups adhere to each DIR Customer’s backup schedule. Any modifications to this
        schedule must be requested through 4.8 Change Management.
       The schedule of executing backups are initially documented and furnished upon request. Any
        modifications to the standard backup schedule is updated and furnished upon request.
       All new requests for ongoing backups must be initiated through 4.8 Change Management.
        After the new request has been reviewed and approved, the new backup will be included in
        the backup schedule.
       One-time backups must also be initiated through 4.5 Request Management. Such requests
        are stored on separate media as identified in the change request.
       Backups are tested quarterly to confirm data can be restored. This is a test of Backup System
        integrity, as opposed to an application disaster recovery test or bare metal restore. The DIR
        Customer may choose the files for this integrity test, in accordance with the Integrity Testing
        policy.




8.3.1 St of TX Backup and Recovery                                                             Page 13 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                       Backup and Recovery




       The listings of file and directory names backed up, the date and time of backup, the size of
        the directory file, and the location of backup media are maintained by the Backup and
        Recovery Team.
       Backup media is properly labeled, documented, and tracked via the media tracking tool.
       All emergency backup requests must go through the Support Center.



Successful Backup
A backup job is considered successful if it completes with zero critical errors. Failed backup jobs may be
rerun, and if the rerun completes with zero critical errors within the backup measurement window, the
backup is considered successful. Additionally, for a Full backup to be successful, the number of bytes
transferred must also be greater than zero.
Noncritical errors are established by mutual agreement between DIR and Team for Texas. A critical error
is any error that has not been agreed by DIR and Team for Texas to be noncritical. For File Open Errors,
if there is an appropriate tool available and in use that captures open files, then the backup of open files
are included as part of the back up. If there is not an appropriate tool available and in use for the capture
of open files, then open file errors are noncritical.



Backup Escalation
If backups have not completed successfully within the backup window, an incident record with impact 3,
urgency 3, must be opened stating the reason for the backup failure/miss. If on the next day, a successful
backup is still not accomplished, then the incident record should be updated to impact 2, urgency 2. If
after the third day the backups are still failing, and no data has been backed up for 3 consecutive days,
then the incident record should be updated to a 1 impact, 1 urgency. As necessary, a problem record will
be created related to the incident record.
As necessary, the Team for Texas Backup and Recovery Operator attends Priority 1 bridge calls to
update the Service Delivery Manager (SDM), Team for Texas management, and DIR with the status for
resolution of the backup issue. Once a successful backup is completed, the incident record is closed. The
problem record, however, remains open, and updates are made to it as problem determination is
performed in accordance with 4.3 Problem Management.
Recurring backup failures due to known issues (for example, capacity issues) are addressed with the
SDM and the DIR Customer, who look at the current backup schedule and approach, and identify any
gaps or adjustments that may be needed. The backup escalation of such incidents may be stopped by
the Backup and Recovery team leads who have authorization to do so. Stopping of the backup escalation
must be documented in the Remedy record.




8.3.1 St of TX Backup and Recovery                                                              Page 14 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                     Backup and Recovery




Data Recovery
Team for Texas recovers data files or datasets as requested. In the event of data corruption or when
there is a need for a previous data file or dataset, it is recovered, as necessary, by Team for Texas. All
recovery efforts are to be consistent with the established DIR Customer data retention policies. Full
recovery from a system failure may require a joint effort between the DIR Customer and Team for Texas
depending on the type of failure. Requests for data recoveries for data files or datasets residing on server
or mainframe staging and production environments must be pre-approved in accordance with 4.8 Change
Management.
Data recovered from the requested backup copy with an abnormal recovery process may cause the
timeline to vary. Team for Texas confirms with the DIR Customer when the recovery is complete.
All recovery requests must go through the Support Center.



Labeling
All external storage media and tapes must be physically and logically labeled with contents, dates, and
DIR Customer information.



Integrity Testing
On each calendar quarter, the DIR Customer Requester will submit the files to be restored for the backup
integrity test. The Requester may choose any 5 files that are scheduled for backup for the restoration
integrity test, of which one may be a database. When a database file is chosen, the database and all
required database elements (log files, etc) are considered as one unit, and count as one file. No more
than one database file may be chosen during a given integrity test period.


Critical Service Levels and Key Measurements
The Critical Service Levels and Key Measurements related to the Backup and Recovery procedure as
defined in Exhibit 3, Attachment 3-B of the Agreement are:
       1.3.1 Successful Backup
       1.3.2 Successful Recoveries
For specific details on each Critical Service Level and Key Measurement, refer to the Agreement (Exhibit
3, Attachment 3-B) or to 4.4 Service Level Measurements and Reporting, which includes references to
the Service Level Matrix, Critical Service Level Specifications Document, and Key Measurement
Specifications Document.




8.3.1 St of TX Backup and Recovery                                                             Page 15 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                     Backup and Recovery




8.3.1.7 Interface Information
Interface Information describes both DIR-initiated interfaces with Team for Texas and the Team for
Texas-initiated interfaces with DIR associated with Backup and Recovery.


Submit a Backup or Recovery Request
To submit a request for a backup or recovery of data:
       All interfaces between DIR, DIR Customers, and Team for Texas for the purposes of
        requesting Backup and Recovery services are initiated via the Support Center
       Refer to 8.2.5 Support Center, 4.5 Request Management, 4.2.1 Incident Management, and
        8.3.5 External Storage Media Management for more information



Submit Files for Backup Integrity Testing
The procedure for the initiation of backup integrity testing is as follows:
       Backup and Recovery Team directs the opening of Remedy incident records for each DIR
        Customer by Backup and Recovery Operators no later than the 10th day of the first month
        each quarter.
       An automated e-mail from Remedy is sent to the SDM and DIR Customer Representative on
        the 10th day of the first month each quarter, notifying them that the quarterly restore test
        process is starting on the 15th day of the month. The e-mail provides a list of incident records
        by DIR Customer and includes a request for DIR Customers to update their respective
        restore test incident records with the data requested, or with a waiver by the last day of that
        month. (Note: Manual notification in this step is provided by e-mail until Remedy automation
        is in place.)
       If there is no DIR Customer update to the incident record by the last day of the first month
        each quarter, the Backup and Recovery Operator closes the incident record, noting the non-
        response.
       DIR Customers that choose to waive the restore test in lieu of past restores completed will
        update the restore test incident record, documenting the waiver request by the last day of that
        first month. Once waived, the Backup and Recovery Operator closes the incident as
        successful for that quarter and notes the waiver.
       For DIR Customers that respond by the last day of that first month via an update of the
        restore test incident record with the requested data, appropriate Backup and Recovery
        Operators are assigned to perform the restore tests of the DIR Customer-identified files from
        the most recent backup to DIR Customer-identified locations.




8.3.1 St of TX Backup and Recovery                                                             Page 16 of 17
May 26, 2009
State of Texas DCS Policies and Procedures Manual                                    Backup and Recovery




      Backup and Recovery Operator performs the restore test and requests that the DIR
       Customer Requester performs file validation and confirms restore success within the incident
       record. If confirmed, the Backup and Recovery Operator closes the incident as successful.
      If the restore is not successful, the Backup and Recovery Operator performs problem
       determination, and if not immediately resolved, opens a problem record to resolve the restore
       problem. The Backup and Recovery Operator closes the incident as unsuccessful.
      Restore test results are fully documented in each DIR Customer incident record, and each
       incident record will include the following status in the resolution detail: Agency No Response,
       Agency Test Waived, Restore Test Successful, Restore Test Unsuccessful.




8.3.1 St of TX Backup and Recovery                                                           Page 17 of 17
May 26, 2009

								
To top