Exploiting Tivoli Storage Manager Backup Technologies

Document Sample
Exploiting Tivoli Storage Manager Backup Technologies Powered By Docstoc
					Exploiting Tivoli Storage
Manager Backup
Technologies
J.P. (Jim) Smith
IBM

August 14, 2008
Session 5677
Trademarks and Disclaimers

The following are trademarks of the International Business Machines Corporation in the United States and/or other countries. For a complete
list of IBM Trademarks, see www.ibm.com/legal/copytrade.shtml:        ADD TRADEMARKS FROM DOCUMENT THAT APPLY

The following are trademarks or registered trademarks of other companies:
Java and all Java based trademarks and logos are trademarks of Sun Microsystems, Inc., in the United States and other countries or both
Microsoft, Windows, Windows NT and the Windows logo are registered trademarks of Microsoft Corporation in the United States, other
countries, or both.
Intel, Intel logo, Intel Inside, Intel Inside logo, Intel Centrino, Intel Centrino logo, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium
are trademarks or registered trademarks
of Intel Corporation or its subsidiaries in the United States and other countries.
UNIX is a registered trademark of The Open Group in the United States and other countries or both.
Linux is a trademark of Linus Torvalds in the United States, other countries, or both.
Other company, product, or service names may be trademarks or service marks of others.
NOTES:
Any performance data contained in this document was determined in a controlled environment. Actual results may vary significantly and are
dependent on many factors including system hardware configuration and software design and configuration. Some measurements quoted in
this document may have been made on development-level systems. There is no guarantee these measurements will be the same on
generally-available systems. Users of this document should verify the applicable data for their specific environment.
IBM hardware products are manufactured from new parts, or new and serviceable used parts. Regardless, our warranty terms apply.
Information is provided “AS IS” without warranty of any kind.
All customer examples cited or described in this presentation are presented as illustrations of the manner in which some customers have
used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics will vary depending
on individual customer configurations and conditions.
This publication was produced in the United States. IBM may not offer the products, services or features discussed in this document in other
countries, and the information may be subject to change without notice. Consult your local IBM business contact for information on the
product or services available in your area.




                                                                                                                                                      2
Trademarks and Disclaimers (continued)
NOTES:
All statements regarding IBM's future direction and intent are subject to change or withdrawal without notice, and represent goals and
objectives only.
Information about non-IBM products is obtained from the manufacturers of those products or their published announcements. IBM has not
tested those products and cannot confirm the performance, compatibility, or any other claims related to non-IBM products. Questions on the
capabilities of non-IBM products should be addressed to the suppliers of those products.
Prices are suggested US list prices and are subject to change without notice. Starting price may not include a hard drive, operating system
or other features. Contact your IBM representative or Business Partner for the most current pricing in your geography.
                  Any proposed use of claims in this presentation outside of the United States must be reviewed by local IBM country counsel
prior to such use.

                The information could include technical inaccuracies or typographical errors. Changes are periodically made to the
information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in
the product(s) and/or the program(s) described in this publication at any time without notice.

              Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve
as an endorsement of those Web sites. The
              materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM makes no representation or warranty regarding third-party products or services including those designated as ServerProven,
ClusterProven or BladeCenter Interoperability Program products. Support for these third-party (non-IBM) products is provided by non-IBM
Manufacturers.
IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of this document does not
give you any license to these patents. Send license inquires, in writing, to IBM Director of Licensing, IBM Corporation, New Castle Drive,
Armonk, NY 10504-1785 USA.




                                                                                                                                                3
The Speaker


              • 15 years with
                ADSM/TSM service
                and development
              • Currently TSM
                Backup-Archive and
                Space Management
                Architect
              smithjp@us.ibm.com
Disclaimer

• This presentation describes potential future enhancements to
  IBM Tivoli Storage Manager

• Information in this presentation does not constitute a
  commitment to deliver the described enhancements or to do so
  in a particular timeframe

• IBM reserves the right to change product plans, features, and
  delivery schedules according to business needs and
  requirements
Introduction

• Myriad of backup techniques in TSM
• Probably more to come in the future
• One size does not fit all
• Why can’t you (TSM) decide for me?
Goals

• Discuss various backup techniques offered by TSM Backup-
  Archive Client
   • What you get vs. what you give up with different technologies and
     variations
   • What are your decision points
• Provide quantitative advice on evaluating different technologies
  and variations
These Are Not Goals of this Presentation

• Discuss how to implement
   • Not a tutorial in option syntax
   • No dsm.opt or dsm.sys examples !

• Discussion of other backup or archive considerations
• In-depth discussion of recovery and recovery considerations
  (RPO, RTO, etc).
Traditional Incremental v. Progressive
Incremental
   ✛    Long restore time                               No unnecessary backups
   ✛    Poor media utilization                          Client data consolidated on a
   ✛    Lots of copies of unchanged data                few tapes
   ✛    High network utilization                        Lower network utilization
   ✛    Higher impact to application server             Low impact to application
   Full + Incremental          Full + Differential   Progressive incremental


       Full Backup               Full Backup          Base Backup


       Incremental               Incremental          Incremental
                                                                         A     B

       Incremental               Differential         Incremental


       Full Backup               Differential         Automated
                                                      Tape
                                                      reclamation
       Incremental               Full Backup
                                                     Incremental
       Incremental
                           Other Backup              ➼Tape reclamation             TSM
                           Products                  ➼Collocation
                                                                                         9
Agenda

• Backup techniques
   • What was available before ADSM/TSM came on the scene
   • TSM Progressive incremental backup model and its derivatives
   • Image backup
• How to determine which backup technique best fits
   • What problem are you trying to solve
   • Windows
   • Information on other operating systems not covered in presentation
     but information included in slides
Agenda

• Backup techniques
   • What was available before ADSM/TSM came on the scene
   • TSM Progressive incremental backup model and its derivatives
   • Image backup
• How to determine which backup technique best fits
   • What problem are you trying to solve
   • Windows
   • Other operating systems
Progressive Incremental Backup


         3
                            1                                  2



                                              2                      DB



                            4



 1. Client queries server for current view of file system
 2. Server returns list of files for entire file system (db query)
 3. Client scans and compares local file system
 4. Client creates transactions of files for backup
Progressive Incremental

• aka Full Progressive Incremental or Full Incremental or
  Incremental “Forever”
• What you get
   • Single-instance store
   • Easier restore; no concept of applying incremental or differential changes
     to full
   • Update attribute only
   • Include/exclude
   • Copy Frequency
   • Policy
Progressive Incremental

• Default technology in TSM
• Hot spots:
   • Memory
     • Number of active backup versions
     • Length of file name
   • Run time
     • Biggest factor is file system scan time
Progressive Incremental Derivatives

• Memory efficient backup
• Memory efficient disk caching
• Virtual mount points
• Incremental by date
• File lists
• Journal-based backups
Memory Efficient


          3
                               1    5                                 2



                                                    2                     DB



                               4


1.   Client queries server for current view of single directory
2.   Server returns list of files in directory (db query)
3.   Client scans and compares local file system
4.   Client creates transactions of files for backup
5.   Client queries server for current view of next directory, etc.
Memory Efficient Backup

• What you get
   • Smaller memory footprint for backup
• What you don’t get
   • Memory savings in directories with large number of objects
• What you give up
   • Slightly (or greatly) more network back-and-forth You also give up on fast
     backup runtime, the increase can be substantial.
• When to use this
   • Incremental backups running out of memory
   • Can get rough estimate on memory usage by examining directory with
     largest # of files
Memory Efficient Disk Caching


        3
                           1                                  2



                                             2                      DB



                           4



1. Client queries server for current view of file system
2. Server returns list of files for entire file system (db query)
3. Client scans and compares local file system by exploiting local
   disk to augment memory
4. Client creates transactions of files for backup
Memory Efficient Disk Caching

• How it works
   • Same as progressive incremental except …
   • B-A client can store TSM Server db queries on disk instead of in memory
• What you get
   • Even smaller memory footprint if regular memoryefficient backup doesn’t reduce
     memory
• What you give up
   • Processing time ? Compare disk i/o to paging
• When to use this
   • Incremental backup running out of memory and memoryefficient backup not
     sufficient Need individual file restore
   • Journal based backup technology is not available on your platform (note you might
     need this just to get the first backup done for journal based backups).
Virtual Mount Point




       /home




                                              /home/1 /home/   /home/2
                                                      2
•   Large file system logically partitioned
•   Managed as separate filespaces on TSM server
Virtual Mount Point

• How this works
   • Instead of mapping entire file system to single namespace on TSM Server
     (filespace), user can define mapping of file system branches to namespace
• What you get
   • Progressive incremental only has to work against subset of file system thus reducing
     memory requirements
• What you don’t get
   • Relief from directories with large number of objects
• What you give up
   • Centralized management of file system on TSM Server
   • Virtual mount points are static and can’t be migrated
• When to use this
   • Large, balanced (number of directories and objects) UNIX and Linux file systems
     can be efficiently divided
Incremental by Date


        3
                          1



                                           2                 DB



                          4



1. Client queries server last backup of entire file system
2. Server returns timestamp of last backup (end) of entire file
   system
3. Client scans and compares local file system
4. Client creates transactions of files for backup
Incremental By Date
•   What you get
     •   Reduced time determining which files have changed
     •   Removing processing time on server to query database
     •   Removing network traffic to communicate results
•   What you give up
     •   Flexibility on scope of backup operation; you must be doing backups of the entire file system
     •   Changes to files that don’t affect date are not captured (attribute, mode, ACL)
     •   Rename, copy and move operations
     •   Handling of deleted files from local file system not reflected in TSM Server database
     •   Policy rebinding
     •   You are not giving up scanning of local file system
     •   No free pass for having client and server clocks out of synch
     •   Maybe not able to use when client and server are in different time zones.
     •   Copy frequency control
•   When to use this
     •   Not meeting backup windows and …
     •   File system changes purely additive or changes, e.g., code repository or data that is basically “additive” (think of
         data like your family photos)
     •   Can be effective when doing weekly (or periodic) full incremental backups
     •   Read Chapter 9 in the User’s Guide for more critical information on this option
File List Backup
              2



application



    1


                                                         DB
                         3




1. Application creates list of files for backup
2. Application passes list of files to client
3. Client creates transactions of files for backup (selective
   or incremental)
File Lists

• What you get
   • Selective backup eliminates the query of the TSM Server db and scan of local file
     system
   • Optionally use grouping constructs
• What you give up
   • Time – you have to find some mechanism to feed the file list
   • Implicit file specifications – file list will not accept wild cards or directory recursion
• When to use this
   • Not meeting backup windows and …
   • File system changes are predictable, e.g., an application is creating the changes and
     it would be easy to interface into the application to get that list of changes
Journal Based Backup


  journal
            2
     1


                                                DB
                     3




  1. Journal monitors file system for changes
  2. Client queries journal for file system changes
  3. Client creates transactions of files for backup
Journal Based Backup
•   What you get
     •   Time to determine which files have changed is greatly reduced
•   What you don’t get
     •   Dismissed from doing a full progressive incremental backup (full) against a file system on some periodic basis
          •   Initial backup to enable the journal
          •   If a discrepancy is found between journal and TSM Server database
          •   If the velocity of changing files is high enough to cause a notification buffer overrun
          •   If the journal service is stopped and restarted
          •   In general, it is a good idea to do periodic progressive incremental backups
•   When to use this
     •   Not meeting backup window and …
     •   Small number of files (< 1,000,000) and small number of changes between backups (<1,000,000)
     •   Large number of objects (<10,000,000) with 10-15% change rate try JBB (should emphasize a low change
         velocity, too… large numbers of changes over a short timeframe can cause notification buffer overflows)
     •   Large number of objects + large number of changes will stress JBB … not a good fit
Adaptive Sub-file Backup

             3



  cache                                                             1

                                1

                                                                        DB



                                2    3


  1.   Client identifies the file is a candidate for backup either through
       selective backup or progressive incremental backup examination
  2.   Client creates transactions of files for backup
  3.   During data movement, client determines if there is enough
       information stored in the local cache to determine which sub-file parts
       have changed
Adaptive Sub-file Backup

• What you get
   • Faster throughput
• What you give up
   •   Local cache space (in the tens of megabytes)
   •   A bit of CPU
   •   Restore time (due to restore of base + delta)
   •   Possible out of space conditions if disk space is tight during restore due to
       how file is reconstructed from base + delta?
• When to use this
   • Network constrained
   • Small file profiles (limited to 2 GB)
Agenda

• Backup techniques
   • What was available before ADSM/TSM came on the scene
   • TSM Progressive incremental backup model and its derivatives
   • Image backup
• How to determine which backup technique best fits
   • What problem are you trying to solve
   • Windows
   • Other operating systems
Image Backup




                                               DB
                    1




 1. Client sends logical block image of file
    system to server
Image Backup

• No mapping between file and block
• Off-line and static backups vs. on-line backups
• What you get
   • Faster backups
   • No scan time to determine what has changed
   • Faster overall data movement
• What you give up
   • Ability to restore individual files directly from TSM Server
   • You need to restore image to alternate location and pull data directly using
     Explorer or equivalent.
   • Caution on using system volumes for DR or BMR
Image Backup
 • Backup at logical volume level
 • Ability to take image + incremental backups to provide single file restore
 • Available for AIX, Windows, Linux x86, Solaris and HP
   Off-line image backup
 • Volume mounted read-only
 • Exploited best by FlashCopy operations (e.g., off-line image is from a FC target)
 • On-line image (dynamic) backup
 • Volume left on-line
 • Fuzzy backup of changing data
 • On-line image backup using snapshot (Windows / Linux x86 LVM2 / AIX JFS2)
 • Volume left on-line
 • Image backup taken at single point-in-time
 • Windows – used blocks only
Image + Incremental by Date


                                  1
           4
                                  2


                                                                         DB
                                                        3



                                  5


 1.   Upon dsmc backup image command client sends logical block image of file
      system to server
 2.   Upon subsequent dsmc backup image –mode=incremental operation client
      queries server last backup of entire file system
 3.   Server returns timestamp of last backup of entire file system
 4.   Client scans and compares local file system
 5.   Client creates transactions of files for backup
Image + Incremental by Date Restore
Scenario



                            1



                                                               DB
                            2



                            3


  1. Client request incremental image restore
  2. Server returns the base image
  3. Server returns additional files which need to be applied to the
     base image to satisfy the recovery point
Image + Incremental By Date

• What you get
  • Benefit of image backup plus some individual file protection of files that are
    changing

• What you give up
  • See Incremental by Date
  • No reconciliation of deleted file
Image + Progressive Incremental Backup


                                    1
            4
                                                                        3
                                    2



                                                                            DB
                                                           3




                                    5

  1.   Upon dsmc backup image command client sends logical block image of file
       system to server
  2.   Upon subsequent dsmc incremental backup client queries server for current
       view of file system
  3.   Server returns list of files for entire file system (db query)
  4.   Client scans and compares local file system
  5.   Client creates transactions of files for backup
Image + Progressive Incremental Restore
Scenario


                                1



                                2
                                                                         DB

                                3
                                           x

                                4

  1.   Client request incremental image restore
  2.   Server returns the base image
  3.   Server returns additional files which need to be applied to the base
       image to satisfy the recovery point
  4.   Server optionally returns the list of files which need to be deleted from
       the base image to satisfy the recovery point
Image + Progressive Incremental

• What you get
  • Benefit of image backup restore for improving RTO
  • Benefit of progressive incremental backups for better
    RPO

• What you give up
  • Additional time to take periodic image backups
  • Storage space at TSM Server
Agenda

• Backup techniques
   • What was available before ADSM/TSM came on the scene
   • TSM Progressive incremental backup model and its derivatives
   • Image backup
• How to determine which backup technique best fits
   • What problem are you trying to solve
   • Windows
What Problem Are You Trying to Solve?

• Briefly touch on RTO and RPO
• Progressive Incremental backup is still most comprehensive
  backup
   • Best method to detect any type of file change
   • Individual file restore
• $64,000 question is “Why can’t you use progressive incremental
  backup?”
   • Memory
   • Backup window
Agenda

• Backup techniques
   • What was available before ADSM/TSM came on the scene
   • TSM Progressive incremental backup model and its derivatives
   • Image backup
• How to determine which backup technique best fits
   • What problem are you trying to solve
   • Windows
Windows – In Your Toolkit


 Progressive incremental backup

 Journal based backup

 Virtual mount points

 Adaptive sub-file backup

 Image backup (on-line and
 “used block only”)
Windows – Attacking Memory Problems

• You need to solve the memory problem first
   • JBB will have to be enabled and fall-back to a progressive incremental
     model at some point
   • Incremental by date needs to use a periodic full progressive incremental at
     some point …
• Is memoryefficient yes going to work
   • Find biggest directory bottleneck
      • Most objects in single directory * 300 < 2 GB ?
         • Then use memoryefficient yes
      • Most objects in single directory * 300 > 2 GB ?
         • Memoryefficient yes not going to solve the problem
            • Use memoryefficient diskcache or
            • Break-up files system or
            • Use image incremental

• If memoryefficient backup is going to help, now consider JBB to further
  reduce backup window
Windows – Attacking the Backup
Window

• Consider JBB
  • JBB will have to fall back to progressive incremental model at some point
     • Buffer Overrun Safety Valve
  • How often is this going to happen in your environment
  • How often can you tolerate this
Now What?

• How often are you seeing buffer overruns?
   • More often then backup window
     • you will never effectively be using the journal
   • Never
     • Woohoo! JBB is a good fit
   • Sometimes
     • You have to decide if it is acceptable

• Don’t forget you can tune JBB’s exclude list and eliminate
  unnecessary updates
Image Backup

• Consider using image
   • Can’t resolve the memory problems
   • Too many changes (> 30%) for JBB or progressive incremental
     backup
   • Most of the file system is small files (ave. < 1 mb)
   • Need faster recovery time then file-level restore can provide
• Try to keep the file change rate < 30% when using image and
  incremental combinations
Windows

• Other thoughts
  • Compression … if you have relatively new CPU and no hardware
    compression why not
  • Take care of your exclude lists
     • Aggressive use of EXCLUDE.DIR where possible
  • Open File Support
     • Are there files that are open that you need protected, e.g., ntuser.dat on
       workstations
     • … and can you not get the files off-line in some other manner, e.g.,
       presched command
Conclusion

• Myriad of backup techniques in TSM and more to come
• One size does not fit all
• Start with progressive incremental backup as it is still most
  comprehensive backup
• Move to other progressive incremental or image techniques as
  the need arises
Performance data

BACKUP INFORMATION


                               50
                   13-Aug-08
Special Notices
Disclaimer
The performance data contained in this presentation was measured in a controlled environment. Results obtained in
other operating environments may vary significantly depending on factors such as system workload and configuration.
Accordingly, this data does not constitute a performance guarantee or warranty.
References in this presentation to IBM products, programs, or services do not imply that IBM intends to make these
available in all countries in which IBM operates. Any reference to an IBM licensed program in this document is not
intended to state or imply that only IBM programs may be used. Any functionally equivalent program may be used
instead.
Trademarks and Registered Trademarks
The following terms are trademarks of International Business Machines Corporation in the United States, other
countries, or both:
              AIX
              IBM
              Rational
              Tivoli
              WebSphere
Other company, product, and service names may be trademarks or service marks of others.
Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States,
other countries, or both.
UNIX is a registered trademark of The Open Group in the United States and other countries.
Improved incremental backup memory usage
Incremental backup, 1 million files workload, Win2003 server, 1 WinXP client, LAN, TCP/IP, Disk
storage pool
IBM x360, 4-way 1.6GHz Xeon MP server, 6 GB, IBM PC 1-way 3GHz Pentium 4 client, 512 MB,
Compression No, Encryption No, Storage pool CRC No, Bufpoolsize 131072, Txngroupmax 256,
TXNBytelimit 2097152, NTFS filespace
                                                                                                              Elapsed time (seconds)
                               Description
                                                                      0% changed data                           5% changed data                       20% changed data
 Full incremental backup, memoryeff diskcachem, default db key size        3138                                             4102                                    5742
 Full incremental backup, memoryeff diskcachem, optimal db key size        2002                                             2850                                    4631
 Full incremental backup, memoryefficient no                               939                                              1620                                    4320

 Full incremental backup, memoryefficient yes                              813                                              1329                                    2479

 Incremental-by-date incremental backup                                    707                                              1384                                    4699
 Journal-based incremental backup                                           2                                               1995                                    6133

                                                                                                               Tivoli Storage Manager Version 5.4.0
       The disk cache method writes server                                                                            Incremental Backup Performance
                                                                                                                      Windows XP Client (3GHz, 512 MB RAM)

       inventory data to a client disk file                                                       7000
                                                                                                                          1 million 1KB files workload

                                                                                                                                                      Full incremental backup,
                                                                                                                                                      memoryefficientbackup diskcachemethod,

       The disk cache method uses less                                                            6000                                                default db key size
                                                                                                                                                      Full incremental backup,
                                                                                                                                                      memoryefficientbackup diskcachemethod,
                                                                                                  5000

       than 20 MB of memory                                                                                                                           optimal db key size
                                                                             Elapsed time (sec)




                                                                                                  4000                                                Full incremental backup,
                                                                                                                                                      memoryefficientbackup no


       The disk cache method is generally                                                         3000
                                                                                                                                                      Full incremental backup,
                                                                                                                                                      memoryefficientbackup yes


       slower than other methods
                                                                                                  2000
                                                                                                                                                      Incremental-by-date incremental backup
                                                                                                  1000


                                                                                                    0                                                 Journal-based incremental backup
                                                                                                         0%           5%             20%
                                                                                                              Percent changed data
Improved incremental backup memory usage

Incremental backup, 1 million files workload, Win2003 server, 8 WinXP clients, LAN, TCP/IP, Disk storage pool
IBM x360, 4-way 1.6GHz Xeon MP server, 6 GB, IBM PC 1-way 3GHz Pentium 4 clients, 512 MB, Compression No,
Encryption No, Storage pool CRC No, Bufpoolsize 131072, Txngroupmax 256, TXNBytelimit 2097152, NTFS filespace
                                                                                                              Elapsed time (seconds)
                                  Description
                                                                      0% changed data                            5% changed data                        20% changed data
 Full incremental backup, memoryeff diskcachem, default db key size        4142                                             5520                                    8551
 Full incremental backup, memoryeff diskcachem, optimal db key size        2969                                             4168                                    8120
 Full incremental backup, memoryefficient no                               1975                                             3436                                    7241
 Full incremental backup, memoryefficient yes                              1086                                             2177                                    4763
 Incremental-by-date incremental backup                                    812                                              2189                                    5420
 Journal-based incremental backup                                           4                                               2074                                    5746


       For all tests measured with 8 WinXP                                                                     Tivoli Storage Manager Version 5.4.0
                                                                                                                      Incremental Backup Performance
                                                                                                                     8 Windows XP Clients (3GHz, 512 MB RAM)
       clients, the disk cache method was                                                         9000
                                                                                                                           1 million 1KB files workload



       slower than the 'memoryefficientbackup
                                                                                                                                                      Full incremental backup,
                                                                                                                                                      memoryefficientbackup diskcachemethod,
                                                                                                  8000
                                                                                                                                                      default db key size


       yes' method                                                                                7000                                                Full incremental backup,
                                                                                                                                                      memoryefficientbackup diskcachemethod,
                                                                                                                                                      optimal db key size

                                                                             Elapsed time (sec)
                                                                                                  6000


       For 5% changed data, this elapsed time
                                                                                                                                                      Full incremental backup,
                                                                                                  5000                                                memoryefficientbackup no

                                                                                                  4000

       difference was measured at +1991
                                                                                                                                                      Full incremental backup,
                                                                                                  3000                                                memoryefficientbackup yes

                                                                                                  2000

       seconds, or 91% longer                                                                     1000
                                                                                                                                                      Incremental-by-date incremental backup



                                                                                                    0                                                 Journal-based incremental backup
                                                                                                         0%           5%             20%
                                                                                                              Percent changed data
Improved incremental backup memory usage

 Incremental backup, 21 million files workload, Win2003 server, Win2003 client, LAN, TCP/IP, Disk stgpool
 IBM x360, 4-way 1.6GHz Xeon MP server, 6 GB, IBM x346, 2-way 3.4GHz client, 4GB, Compression No, Encryption
 No, Storage pool CRC No, Bufpoolsize 131072, Txngroupmax 256, TXNBytelimit 2097152, NTFS filespace
                                                                                                                Elapsed time (seconds)
                                  Description
                                                                      0% changed data                               5% changed data                        20% changed data
 Full incremental backup, memoryeff diskcachem, default db key size           n/a                                            33699                                    35536
 Full incremental backup, memoryeff diskcachem, optimal db key size       18654                                              21678                                    24680
 Full incremental backup, memoryefficient no                              Failed                                             Failed                                   Failed
 Full incremental backup, memoryefficient yes                             13760                                              18394                                    23310
 Incremental-by-date incremental backup                                   11262                                              14679                                    20902
 Journal-based incremental backup                                                          2                                 10916                                    Failed

     The 'memoryefficient no' method fails for                                                                Tivoli Storage Manager Version 5.4.0
                                                                                                                      Incremental Backup Performance
     this workload because it needs to use                                                                        Windows 2003 (32bit) Client (3.4GHz, 4GB RAM)
                                                                                                                         21 million 1KB files workload


     more virtual memory than is available to                                                    40000

                                                                                                 35000
                                                                                                                                                       Full incremental backup,
                                                                                                                                                       memoryefficientbackup diskcachemethod,
                                                                                                                                                       default db key size

     a 32 bit client                                                                             30000
                                                                                                                                                       Full incremental backup,
                                                                                                                                                       memoryefficientbackup diskcachemethod,
                                                                                                                                                       optimal db key size

     The journal-based backup method fails                                  Elapsed time (sec)
                                                                                                 25000
                                                                                                                                                       Full incremental backup,
                                                                                                                                                       memoryefficientbackup no
                                                                                                 20000


     when there are lots of changed files                                                        15000                                                 Full incremental backup,
                                                                                                                                                       memoryefficientbackup yes



     The disk cache method with optimal key
                                                                                                 10000
                                                                                                                                                       Incremental-by-date incremental backup
                                                                                                 5000


     size is only 18% slower than the                                                               0
                                                                                                         0%           5%             20%
                                                                                                                                                       Journal-based incremental backup



     'memoryefficient yes' method at 5%                                                                       Percent changed data
 Improved incremental backup memory usage

Incremental backup, 21 million files workload, Win2003 server, Win2003 x64 client, LAN, TCP/IP, Disk stgpool
Intel 4-way 3GHz (Paxville dual core, hyperthread) server, IBM x346, 2-way 3.4GHz client, 4GB, Compression No,
Encryption No, Storage pool CRC No, Bufpoolsize 131072, Txngroupmax 256, TXNBytelimit 2097152, NTFS filespace
                                                                                                               Elapsed time (seconds)
                                  Description
                                                                      0% changed data                              5% changed data                        20% changed data
 Full incremental backup, memoryeff diskcachem, default db key size               n/a                                        42496                                   45117
 Full incremental backup, memoryeff diskcachem, optimal db key size               n/a                                        21327                                   24350
 Full incremental backup, memoryefficient no                              23652                                              27176                                   30088
 Full incremental backup, memoryefficient yes                             12724                                              16244                                   20714
 Incremental-by-date incremental backup                                           n/a                                        13681                                   17974
 Journal-based incremental backup                                                               2                            12075                                   Failed


        The disk cache method was faster than                                                                Tivoli Storage Manager Version 5.4.0
                                                                                                                     Incremental Backup Performance

        'memoryefficientbackup no' method on
                                                                                                                  Windows 2003 x64 Client (3.4GHz, 4GB RAM)
                                                                                                                        21 million 1KB files workload

                                                                                                50000

        Windows x64
                                                                                                                                                     Full incremental backup,
                                                                                                45000                                                memoryefficientbackup diskcachemethod,
                                                                                                                                                     default db key size
                                                                                                40000

        64bit OS allows larger process virtual
                                                                                                                                                     Full incremental backup,
                                                                                                                                                     memoryefficientbackup diskcachemethod,
                                                                                                35000
                                                                                                                                                     optimal db key size
                                                                           Elapsed time (sec)



        memory, but large real memory demand
                                                                                                30000                                                Full incremental backup,
                                                                                                                                                     memoryefficientbackup no
                                                                                                25000



        cause paging
                                                                                                20000                                                Full incremental backup,
                                                                                                                                                     memoryefficientbackup yes
                                                                                                15000



        The disk cache method disk I/O is more
                                                                                                10000                                                Incremental-by-date incremental backup

                                                                                                5000


        efficient than paging disk I/O                                                              0                                                Journal-based incremental backup
                                                                                                        0%           5%             20%
                                                                                                             Percent changed data
JBB – Buffer Overrun Safety Valve

BACKUP INFORMATION


                                                56
                                    13-Aug-08
JBB – Tracking Down the Buffer Overrun Safety
Valve

 • Run TSM journal daemon stand-alone (i.e., not communicating
   with the B-A client)
    • Look for buffer overflow errors in error log

 • New TSM 5.4.1.2 trace flag helps you see activity in journal
   daemon
    • Can use to tune exclude list of daemon
Creating Stand-Alone JBB Daemon
• Modify journal .ini file to journal one or more file systems
   • Can use configuration wizard (just don’t start the service).
• Modify the .ini file
   • Add trace semantics (see next page)
   • Modify communications (named pipe) so that the B-A client can’t
     talk to it
• Start the journal service
   • Alternatively run in foreground “tsmjbbd.exe i”
tsmjbbd.ini

              ;
              [JournalSettings]
              Traceflags=jbbfilemoncsv
              Tracefile=M:\jbbtrace
              JournalPipe=\\.\pipe\bogusPipe
              Errorlog=jbberror.log
              JournalDir=C:\Program Files\Tivoli\TSM\baclient
Buffer Overrun Safety Valve
(jbberror.log)


 ...
 06/04/2007 15:48:02 fifoQInsert(00A4ED04): Queue threshold reached,   thread 960
 returned from wait.
 06/04/2007 15:48:02 fifoQInsert(00A4ED04): Queue threshold reached,   thread 960
 will block until queue is under threshold.
 06/04/2007 15:48:02 fifoQInsert(00A4ED04): Queue threshold reached,   thread 960
 returned from wait.
 06/04/2007 15:48:05 psFsMonitorThread(tid 960): Notification buffer   overrun for
 monitored FS 'C:\', journal will be reset.
 06/04/2007 15:48:05 jnlDbCntrl(): Resetting journal for fs 'C:'.


 ...
Other platforms

BACKUP INFORMATION


                              61
                  13-Aug-08
Sample tsmstats.ini

[fileSystemStatistics.\\patmos\m$]
maxPath=68
totalLocalFiles=1381
totalLocalDirs=552
maxDirCount=30
maxDirName=M:\test\Hiragana
Agenda

• Backup techniques
   • What was available before ADSM/TSM came on the scene
   • TSM Progressive incremental backup model and its derivatives
   • Image backup
• How to determine which backup technique best fits
   • What problem are you trying to solve
   • Windows
   • Other operating systems
AIX – In Your Toolkit


 Progressive incremental backup

 Journal based backup

 Virtual mount points

 Adaptive sub-file backup

 Image backup (on-line in near
 future)
Linux – In Your Toolkit


 Progressive incremental backup

 Journal based backup

 Virtual mount points

 Adaptive sub-file backup

 Image backup (on-line)
UNIX – In Your Toolkit


 Progressive incremental backup

 Journal based backup

 Virtual mount points

 Adaptive sub-file backup

 Image backup (off-line)
Macintosh and NetWare – In Your Toolkit


 Progressive incremental backup

 Journal based backup

 Virtual mount points

 Adaptive sub-file backup

 Image backup
Other Platforms – Attacking Memory
Problems

• You need to solve the memory problem first
   • JBB (AIX) will have to be enabled and fall-back to a progressive incremental model
     at some point
   • Incremental by date needs to use a periodic full progressive incremental at some
     point …
• Is memoryefficient yes going to work
   • Find biggest directory bottleneck
       • Most objects in single directory * 300 < 2 GB ?
          • Then use memoryefficient yes
       • Most objects in single directory * 300 > 2 GB ?
          • Memoryefficient yes not going to solve the problem
             • Use memoryefficient diskcache or
             • Break-up files system or
             • Use image incremental

• If memoryefficient backup is going to help, now consider JBB (AIX) to further
  reduce backup window
AIX – Attacking the Backup Window

• Consider JBB
  • JBB will have to fall back to progressive incremental model at some point
     • Buffer Overrun Safety Valve
  • How often is this going to happen in your environment
  • How often can you tolerate this
JBB – Tracking Down the Buffer Overrun Safety
Valve

• Run TSM journal daemon stand-alone (i.e., not communicating
  with the B-A client)
   • Look for buffer overflow errors in error log

• New TSM 5.4.1.2 trace flag helps you see activity in journal
  daemon
   • Can use to tune exclude list of daemon
AIX - Creating Stand-Alone JBB Daemon

• Modify journal .ini file to journal one or more file systems
• Modify the .ini file
   • Add trace semantics (see next page)
   • Rename daemon so that the B-A client can’t talk to it

• Start the journal daemon
tsmjbbd.ini

              ;
              [JournalSettings]
              Traceflags=jbbfilemoncsv
              Tracefile=M:\jbbtrace
              Errorlog=jbberror.log
Now What?

• How often are you seeing buffer overruns?
   • More often then backup window
     • you will never effectively be using the journal
   • Never
     • Woohoo! JBB is a good fit
   • Sometimes
     • You have to decide if it is acceptable

• Don’t forget you can tune JBB’s exclude list and eliminate
  unnecessary updates
Image Backup – non-Windows

• Consider using image
   • Can’t resolve the memory problems
   • Too many changes (> 30%) for JBB or progressive incremental backup
   • File system is at least 60% full
      • UNIX and Linux paying for bandwidth by backing-up unused blocks
   • Where no on-line image support is available, you can unmount the file
     system
   • Most of the file system is small files (ave. < 1 mb)
   • Need faster recovery time then file-level restore can provide
• Try to keep the file change rate < 30% when using image and
  incremental combinations
Other Platforms

• Other thoughts
  • Compression … if you have relatively new CPU and no hardware
    compression why not
  • Take care of your exclude lists
     • Aggressive use of EXCLUDE.DIR where possible
  • Snapshot-based file backup (AIX near future enhancement) or
    using snapshotroot option with 3rd party snapshot provider
     • Are there files that are open that you need protected, e.g., ntuser.dat on
       workstations
     • … and can you not get the files off-line in some other manner, e.g.,
       presched command