AIXv61 perf

							What’s New - AIX 610

System p Performance / AIX v6.1

Bret Olszewski, breto@us.ibm.com Dirk Michel, dirkm@us.ibm.com

3/2/2009

© 2006 IBM Corporation

What’s New – AIX 6.1

Agenda
 VMM Page Replacement – New defaults reducing the requirement for basic performance tuning  VMM File IO Pacing Enabled By Default

 Performance Tunables
– Tunables are categorized into restricted and non-restricted tunables  AIO – Dynamic AIO tuning

– AIO Fast Path for CIO
 JFS2 – Read only access to files opened with CIO  NFS – Changes to TCP scaling window, R/W size and number of biod daemons

2

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

Memory – AIX Page replacement algorithm

 

Default AIX 5.1, 5.2, 5.3 Tuning gives us: When page replacement begins to run, it selects a page type to steal based on: – If the amount of file pages is above maxclient/maxperm, file pages are chosen – If the number of file pages is between minperm and maxclient, the type is chosen based on re-paging history – If the amount of file pages is below minperm, working storage and file pages are chosen without checking for the re-paging history Re-paging history indicates if individual pages have been written to disk and read back recently – Re-paging history adds a degree of uncertainty to the selection process – If re-paging history decides to pick working storage pages, system paging may begin • This was intended as a “safety” valve, if we are too aggressive in stealing file pages, stop • But, sometimes it is triggered by bad luck – If re-paging history decides to pick file pages and many file pages are “dirty”, heavy writes to disk can occur
• This would probably happen eventually due to “sync”, so it is not always as noticeable

How much memory is caching files

100%
Pick file pages Pick file pages -orw/s pages based on recent history
Minperm=20% Maxclient = Maxperm=80%



Pick any pages 0%

Contents of system memory



With AIX 6.1, the basic algorithms remain the same, but the tuning changes

3

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

AIX v5 vs v6 VMM Page Replacement tuning
 AIX 5.2/5.3  AIX 6.1

– minperm% = 20 – maxperm% = 80 – maxclient% = 80 – strict_maxperm = 0 – strict_maxclient = 1 – lru_file_repage = 1 – page_steal_method = 0

– minperm% = 3 – maxperm% = 90 – maxclient% = 90 – strict_maxperm = 0 – strict_maxclient = 1 – lru_file_repage = 0 – page_steal_method = 1

 On AIX 6.1, no paging to the paging space will occur unless the system memory is over committed (AVM > 97%)

4

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

Implications of the new tuning defaults

 Database environments using DB2, Oracle, and Sybase should no longer require changes to the defaults!  AIX default tuning now matches SAP recommendations, again meaning no VMM tuning should be required – It is inconvenient when IBM and SAP disagree  If you do an upgrade install, your old tunings will be preserved – You may wish to undo them, but we won’t make you  This level of tune was been applied to numerous AIX 5.3 customers through field support – We are confident this was a good thing – However, we try to never change defaults in the service stream, so AIX 5.3 remains as it was

5

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

 VMM Page Replacement – New defaults reducing the requirement for basic performance tuning  VMM File IO Pacing Enabled By Default

 Performance Tunables
– Tunables are categorized into restricted and non-restricted tunables  AIO – Dynamic AIO tuning

– AIO Fast Path for CIO
 JFS2 – Read only access to files opened with CIO  NFS – Changes to TCP scaling window, R/W size and number of biod daemons

6

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

VMM File IO Pacing Enabled By Default

 IO Pacing Enabled By Default – Prevents system responsiveness issues due to large quantities of writes – Limits the maximum number of pages of I/O outstanding to a file
• Without I/O pacing a program can fill up large amounts of memory with written pages. Those “queued” I/O’s can result in long waits for other programs using the storage • Better solution than the file system write behind techniques

– New defaults
• Not very aggressive, intended to limit one or a few programs from impacting system responsiveness. Values high enough not to impact sequential write performance • maxpout = 8193 • minpout = 4096

7

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

 VMM Page Replacement – New defaults reducing the requirement for basic performance tuning  VMM File IO Pacing Enabled By Default

 Performance Tunables
– Tunables are categorized into restricted and non-restricted tunables  AIO – Dynamic AIO tuning

– AIO Fast Path for CIO
 JFS2 – Read only access to files opened with CIO  NFS – Changes to TCP scaling window, R/W size and number of biod daemons

8

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

Performance Tunables
 Tunables now in two categories  Restricted Tunables
– Should not be changed unless recommended by AIX development or development support
– Are not shown by tuning commands unless the –F flag is used – Dynamic change will show a warning message – Permanent change must be confirmed – Permanent changes will cause an error log entry at boot time

 Non-Restricted Tunable
– Can have restricted tunables as dependencies

9

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

Changing restricted tunables

Changing a restricted tunable dynamically
> ioo -o aio_sample_rate=6 Warning: a restricted tunable has been modified

A dynamic change of a restricted tunable will inform the user.

Changing a restricted tunable permanently
> ioo -po aio_sample_rate=6 Modification to restricted tunable aio_sample_rate, confirmation required yes/no

A permanent change of a restricted tunable requires a confirmation from the user. Note: The system will log changes to restricted tunable in the system error log at boot time.
10 System p Performance / Improved Out Of The Box Performance 3/1/2007 6:18 PM © 2006 IBM Corporation

What’s New – AIX 6.1

List restricted tunables
> ioo -aF aio_active = 0 aio_maxreqs = 65536 ...

...
posix_aio_minservers = 3 posix_aio_server_inactivity = 300 ##Restricted tunables aio_fastpath = 1 aio_fsfastpath = 1 aio_kprocprio = 39

aio_multitidsusp = 1
aio_sample_rate = 5 aio_samples_per_cycle = 6 j2_maxUsableMaxTransfer = 512 j2_nBufferPerPagerDevice = 512 j2_nonFatalCrashesSystem = 0 j2_syncModifiedMapped = 1 j2_syncdLogSyncInterval = 1

11

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

TUNE_RESTRICTED Error Log Entry
LABEL: IDENTIFIER: TUNE_RESTRICTED D221BD55 Thu May 24 15:05:48 2007 637 000AB14D4C00 quake O INFO Global perftune

Date/Time: Sequence Number: Machine Id: Node Id: Class: Type: WPAR: Resource Name:

Description RESTRICTED TUNABLES MODIFIED AT REBOOT Probable Causes SYSTEM TUNING User Causes TUNABLE PARAMETER OF TYPE RESTRICTED HAS BEEN MODIFIED Recommended Actions REVIEW TUNABLE LISTS IN DETAILED DATA Detail Data LIST OF TUNABLE COMMANDS CONTROLLING MODIFIED RESTRICTED TUNABLES AT REBOOT, SEE FILE /etc/tunables/lastboot.log vmo

12

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

Why you ask?

 The number of tunables in AIX had grown to a ridiculously large number – 5.3 TL06: vmo 61, ioo 27, schedo 42, noo 135, plus a few others – 6.1 vmo 29, ioo 21, schedo 15, noo 133, plus a few others  The potential combinations that exist are too huge to effectively test and document  Many of the tunables had been created to deal with very specific customers or situations which don’t apply often  This wasn’t done in a vacuum, a survey of support and recent situations was employed to identify the commonly used tunables (which remain unrestricted)

13

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

 VMM Page Replacement – New defaults reducing the requirement for basic performance tuning  VMM File IO Pacing Enabled By Default

 Performance Tunables
– Tunables are categorized into restricted and non-restricted tunables  AIO – Dynamic AIO tuning

– AIO Fast Path for CIO
 JFS2 – Read only access to files opened with CIO  NFS – Changes to TCP scaling window, R/W size and number of biod daemons

14

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

AIX 6.1 AIO Support

 Interface Changes
– All the AIO entries in the ODM and AIO smit panels have been removed – The aioo command will not longer be shipped – All the AIO tunables have current, default, minimum and maximum value that can be viewed with ioo

 AIO kernel extension loaded at system boot
– No AIO servers are active until requests are present – Extremely low impact on memory requirements with this implementation

15

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

Improvements to AIO CIO
 AIO Fast Path for CIO enabled by default
– With the fast path, the AIO server threads no longer participate in the I/O path – By removing the AIO servers from the path, we get three things
• The removal of AIO servers as any potential resource bottleneck • The reduction in path length for AIO read/write services, as less dispatching is required • Potentially better coalescing of sequential I/O requests initiated through AIO or LISTIO services

Application
Application File System

AIO Server

File System LVM

FS no Fast Path

Device Driver

 Fast Path enabled for LV and PV’s for a long time

Application File System LVM Device Driver

– No change in behavior for environments such as Oracle 10G/ASM on raw hdisks

CIO Fast Path

16

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

General improvements to AIO

 AIO servers stay active as long as they service requests

– Number of AIO server dynamically increased/reduced based on the demand of the workload – aio_server_inactivity defines after how many seconds idle time an AIO server will exit
 Changes to AIO tunables are dynamic through ioo

– Changes do not require system reboot
– minservers is changed to a per CPU tunable – maxservers is changed to 30 – maxreqs is changed to 65536

 Benefit
– No longer necessary to tune the minservers/maxservers/maxreqs as in the past

17

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

AIO Tunables

> ioo -a aio_active = 0 aio_maxreqs = 65536 aio_maxservers = 30 aio_minservers = 3 aio_server_inactivity = 300 posix_aio_active = 0 posix_aio_maxreqs = 65536 posix_aio_maxservers = 30

posix_aio_minservers = 3
posix_aio_server_inactivity = 300

18

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

AIO Restricted Tunables

> ioo -aF ...

##Restricted tunables
aio_fastpath = 1 aio_fsfastpath = 1 aio_kprocprio = 39 aio_multitidsusp = 1 aio_sample_rate = 5 aio_samples_per_cycle = 6

posix_aio_fastpath = 1
posix_aio_fsfastpath = 1 posix_aio_kprocprio = 39 posix_aio_sample_rate = 5 posix_aio_samples_per_cycle = 6

19

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

 VMM Page Replacement – New defaults reducing the requirement for basic performance tuning  VMM File IO Pacing Enabled By Default

 Performance Tunables
– Tunables are categorized into restricted and non-restricted tunables  AIO – Dynamic AIO tuning

– AIO Fast Path for CIO
 JFS2 – Read only access to files opened with CIO  NFS – Changes to TCP scaling window, R/W size and number of biod daemons

20

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

CIO Read Mode Flag
 Allows an application to open a file read only even so it is opened in concurrent IO mode by another application

– In the past, a 2nd opening of a file without CIO, would cause “demotion” which removes many of the benefits of CIO
– The 2nd read-only opening without CIO will still result in that opening having uncached reads to the file. Thus, such programs should ensure that the I/O sizes are large enough to achieve I/O efficiency  Example, a backup application can access database files in read only mode while the database has the file opened in concurrent IO mode

21

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

 VMM Page Replacement – New defaults reducing the requirement for basic performance tuning  VMM File IO Pacing Enabled By Default

 Performance Tunables
– Tunables are categorized into restricted and non-restricted tunables  AIO – Dynamic AIO tuning

– AIO Fast Path for CIO
 JFS2 – Read only access to files opened with CIO  NFS – Changes to TCP scaling window, R/W size and number of biod daemons

22

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

NFS Performance Improvements

 RFC 1323 enabled by default – Allows for TCP window scaling beyond 64K, so more one-way packets in-flight allowed between acks for large sequential transfers. We had the nfs_rfc1323 tunable before, it just wasn't enabled by default.  Increase default number of biod daemon – 32 biod daemons per NFS V3 mount point – Very slight increase in memory (<2MB) required over previous default of 4 – Enables more I/O’s to be outstanding at the same, doesn’t speed sequential operations much, but helps random access (e.g. OLTP)  Default read/write size increased to 64k for TCP connections – Was 32k previously

23

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

Implementation Considerations
AIX 5.2/3 to AIX 6.1 migration example (DB2 performance tuning)  AIX 5.2/5.3  AIX 6.1

– VMM page replacement tuning
• • • • • reduce minperm, maxperm, maxclient turn off strict_maxclient increase minfree, maxfree Enable AIO Tune minservers, maxservers and reboot

– VMM page replacement tuning
• • • NO TUNING REQUIRED NO TUNING REQUIRED Enable CIO

– AIO tuning – DB2 tuning

– AIO tuning

– DB2 tuning
• Enable CIO

24

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

Implementation Considerations (Cont’d)

Best Practices
Do not apply legacy tuning since common legacy tunables might be restricted Change restricted tunables only if recommended by AIX development or AIX development support

25

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

Implementation Considerations (Cont’d)
Problem Determination
Common problems - seen in field or lab Legacy VMM tuning results in error log entries (TUNE_RESTRICTED) Tuning scripts fail due to required confirmation for permanent changes of restricted tunables Install/tuning scripts fail due missing aio0 device Diagnostics Check AIX errpt for TUNE_RESTRICTED Check /etc/tunables/lastboot.log PERFPMR

26

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation

What’s New – AIX 6.1

Fin

 Questions?

27

System p Performance / Improved Out Of The Box Performance

3/1/2007 6:18 PM

© 2006 IBM Corporation