AIX NFS Client Performance Improvements for Databases on NAS

Document Sample
AIX NFS Client Performance Improvements for Databases on NAS Powered By Docstoc
					   AIX NFS Client Performance
        Improvements for
       Databases on NAS
                           October 20, 2005


Sanjay Gulabani                         Diane Flemming
Sr. Performance Engineer                Advisory Software Engineer
Network Appliance, Inc.                 IBM Corporation
gulabani@netapp.com                     dianefl@us.ibm.com
                                                             Slide 1
                   2005 NAS Industry Conference
        Agenda
•   Why Databases on NAS?
•   AIX NFS improvements - CIO
•   Simple IO Performance Tests
•   OLTP Performance Comparison
•   Future work
•   Conclusion
•   Q&A

                                        Slide 2
         2005 NAS Industry Conference
      Databases on SAN
• Databases do block based I/O and prefer
  raw blocks
• But, most admins still put a volume
  manager and a file system: JFS2, VxFS,
  UFS over HW RAID storage
  Reason: Simplicity -- Easier backups and
  provisioning




                                        Slide 3
         2005 NAS Industry Conference
       NAS Myths
• Myth: NFS consumes more CPU
  for Databases
 Reality: Most NFS client
 performance problems with DB
 apps are due to kernel locking and
 are fixed in good NFS clients


                                       Slide 4
        2005 NAS Industry Conference
            NAS Myths
• Myth: Ethernet is slower than SAN
  Reality: Ethernet is 1Gb, FC is 2Gb.
  4Gb FC is almost here but so is
  10GbE.
  Ø   Ethernet is likely to takeover bandwidth of FC soon;
      cost effectively
  Ø   Its easy to setup multiple wires to match FC
      bandwidth with just 1GbE, today
  Ø   Storage latencies at database layer are measured in
      msecs, differences between SAN and NAS latencies
      are in usecs
      78usec more for 8K blocks < 0.1msec difference!


                                                    Slide 5
              2005 NAS Industry Conference
       Why NAS?
• Networking is simpler
• Networking promotes sharing
• Sharing = Higher utilization (~grid
  storage)
• NAS solutions are cheaper than SAN
• No one has won against Ethernet!
• Seriously, even blocks based storage is
  moving to Ethernet (iSCSI)

                                        Slide 6
         2005 NAS Industry Conference
          Why DB on NAS?
• Complex storage management is
  offloaded from the database servers to
  storage appliances
• Performance vs. manageability
  Ø   Database requirements different than
      traditional NFS workloads: home
      directories, distributed source code
  Ø   single writer lock can be a problem



                                            Slide 7
             2005 NAS Industry Conference
     Industry Support for NAS
• Oracle On-Demand hosts ERP+CRM
  1000+ Linux servers on NetApp NFS
• Yahoo runs Database on NetApp
• Sun 2003 presentation McDougall +
  Colaco promoted NFS
• NetApp #1 Leader of NAS $1.59b(FY05)
• IBM AIX 5.3 ML-03 improvements now
• No one got fired for buying from IBM!
  (IBM resells NetApp products)
                                         Slide 8
          2005 NAS Industry Conference
          Oracle’s Austin Datacenter
• More than 15,000 x86 servers
• 3.1 Petabytes of NetApp storage
• 100s of mission-critical hosted Oracle apps
 Source: Oracle Magazine Mar/Apr 2005




                                                       Slide 9
                        2005 NAS Industry Conference
  I/O, I/O… It’s off to disk I go
• Raw logical volumes
 –   No memory management overhead
• Buffered I/O
 –   Variety of kernel services
• Direct I/O
 –   No memory management overhead
 –   defaults to synchronous accesses over NFS
• Concurrent I/O
 –   DIO plus no inode/rnode contention
                                           Slide 10
            2005 NAS Industry Conference
            NFS File Transfer
• Metrics:
  –   CPU utilization and throughput in KB/sec
• Options:
  –   Default (buffered I/O) and CIO
• AIX Client and Server
  –   p630 1.45GHz 4-way
  –   GbE 1500-byte MTU
  –   AIX 5.3 ML-03
• Demonstrates algorithm behavior
  –   Atomic, Single-Threaded
                                                 Slide 11
             2005 NAS Industry Conference
                             NFS File T ransfer
                        Sequential Reads and Writes
             120000                                              40
             100000
                                                                 30




                                                                      CPU Util (%)
Throughput




             80000
  (KB/sec)




             60000                                               20
             40000
                                                                 10
             20000
                  0                                              0
                           Seq Write             Seq Read
               Buff I/O Throughput              CIO Throughput
               Buff I/O CPU Util                CIO CPU Util


               256MB File, NFS V3, TCP, auth=sys, 32KB RPCs
                                                                            Slide 12
                         2005 NAS Industry Conference
       Simple I/O Load Generator
• Metrics
  –   CPU utilization, IOPS, throughput, etc.
• Options
  –   NFS client mount options to specify I/O mode:
      Default (buffered I/O), DIO, CIO
• NetApp Filer FAS880
  –   ONTAP 6.5.1
• AIX Client Model p630 1.45 GHz 4-way
  –   AIX 5.3 ML-03
  –   GbE 1500-byte MTU
                                                Slide 13
               2005 NAS Industry Conference
                                 SIO Throughput
                              50% Writers, 256MB file

            80000
            7 0000
            60000
            50000                                          Buff I/O
   KB/sec




            40000                                          CIO
            30000                                          DIO
            20000
            10000
                 0
                     1    2     4   8   16 32 64 128 256
                               Number of Threads



256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client
                                                                 Slide 14
                         2005 NAS Industry Conference
                             SIO Throughput
                          100% Writers, 256MB File

            60000
            50000
            40000                                         Buff I/O
   KB/sec




            30000                                         CIO
            20000                                         DIO
            10000
                0
                    1     2   4    8   16 32 64 128 256
                              Number of Threads


256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client
                                                            Slide 15
                        2005 NAS Industry Conference
                           SIO Scaled Throughput
                            using CIO, 256MB file

             200000
  KBps/CPU




             150000

             100000

              50000

                  0
                      1      2     4     8    16    32   64   128 256
                              Number of Threads               100% Writers
                                                              50% Writers


256MB File, NFS V3, TCP, auth=sys, AIX v5.3 ML-3 client
                                                                        Slide 16
                          2005 NAS Industry Conference
     NFS FT and SIO Summary
• CIO not appropriate for single threaded
  sequential read/write workloads.
• CIO aimed at workloads with higher
  concurrency such as databases.
• Significant gain in raw throughput
  performance using CIO vs. Buffered
  I/O for workloads with higher
  concurrency.
• CPU utilization illustrates issue with
  TCP socket lock contention with
  increased concurrency.
                                         Slide 17
          2005 NAS Industry Conference
OLTP Performance
• AIX 5.2 ML-5 and AIX 5.3 ML-03 (beta)
• PowerPC pSeries 650 (2 * 1.2 GHz, 4 GB
  RAM)
• Oracle 10.1
• 1GbE for NFS or iSCSI or 2Gb FCP card
• FAS 940c
• 48 Disk Spindles (4 x DS14) 144GB 10K
  RPM
• ONTAP 6.5.1

                                          Slide 18
        2005 NAS Industry Conference
Transactions on AIX 5.2, 5.3, iSCSI, FCP

                                        Chart 1: Transactions for protocols

                     16000
                                                                                 14139
                     14000
                                                 11817           12171
                     12000
      transactions




                     10000                                                                   OETs

                     8000

                     6000
                                 4710
                     4000

                     2000

                        0
                             kernel NFS 5.2   NFS 5.3* 60u   iSCSI raw 5.3*   FCP raw 5.3*
                                 @60u                             60u             60u



   OETs = Order Entry Transactions (an Oracle OLTP benchmark, approx 2:1 read to
              write ratio)
                                                                                              Slide 19
                               2005 NAS Industry Conference
               Host CPU Utilization
                      Chart 2: Transactions and cpu utilization for protocols

           16000
                                              98                                     100
                                                                        14139
           14000                                           90

                                         11817          12171            88
           12000                                                                     80


           10000                                                                                 OETs
transactions




                                                                                     60          cpu




                                                                                           cpu
               8000
                              49
               6000                                                                  40
                          4710
               4000
                                                                                     20
               2000

                 0                                                                   0
                      kernel NFS 5.2   NFS 5.3 60u   iSCSI raw 5.3 FCP raw 5.3 60u
                          @60u                           60u



                                                                                                        Slide 20
                                 2005 NAS Industry Conference
    Protocol Efficiencies
Computed using (%CPU used/ OETs) * K:
   where K is a constant such that NFS CPU cost per K
   transactions is 1.0 and relative cost for iSCSI and FCP is
   computed using the same constant K transactions.

                                               Chart 3: Relative CPU cost per protocol for fixed transactions

                                               1.2
       normalized cpu cost (lower is better)




                                                            1
                                                 1
                                                                                0.89

                                               0.8                                                  0.75
                                                                                                        relative cpu cost

                                               0.6


                                               0.4


                                               0.2


                                                0
                                                       NFS 5.3 60u        iSCSI raw 5.3 60u    FCP raw 5.3 60u
                                                                              protocol
                                                                                                                            Slide 21
                                                       2005 NAS Industry Conference
Oracle Block Read Response Time
  Not typical response time. Random workload used. Filer with more
      cache improves block response time or more sequential reads
      improve avg. block response time.

                                                            Chart 4: Oracle Block Read Response Time

                                                   35
          Block read Resp Time (lower is better)




                                                   30      29


                                                   25

                                                   20                                                   block read resp time

                                                   15                                    14
                                                                         13
                                                                                                   12
                                                   10

                                                    5

                                                   0
                                                        kernel NFS   NFS 5.3 60u iSCSI raw 5.3 FCP raw 5.3
                                                         5.2 @60u                    60u          60u
                                                                              protocol

                                                                                                                               Slide 22
                                                          2005 NAS Industry Conference
     Future Work
• Testing and analysis on systems
  with higher numbers of CPUs.
• Further investigation on socket
  lock scaling issue on AIX




                                        Slide 23
         2005 NAS Industry Conference
    Conclusions
• AIX 5.3 ML-03 with new Concurrent
  I/O ‘CIO’ mount option delivers an
  OLTP performance on NFS
  comparable to that of block-based
  protocols iSCSI and FCP
• Don’t be afraid to deploy databases
  on AIX NFS, we will support you.
• Proof of AIX NFS performance up to
  4 CPU completed.

                                       Slide 24
        2005 NAS Industry Conference
                  References
• AIX Performance with NFS, iSCSI and FCP
  Using an Oracle Database on NetApp White
  Paper @ http://www.netapp.com/library/tr/3408.pdf
• NetApp Best Practices Paper located at:
  http://www.ibm.com/servers/storage/support/nas/
• Download SIO tool
  http://now.netapp.com/NOW/download/tools/sio_ntap/index.shtml

• Improving Database Performance with AIX
  Concurrent I/O:
  http://www-03.ibm.com/servers/aix/whitepapers/db_perf_aix.pdf




                                                                  Slide 25
                   2005 NAS Industry Conference
Questions/Answers




                                 Slide 26
  2005 NAS Industry Conference