Introduction to XFS HEPSYSMAN 5th December 2005
Jonathan Wheeler E-Science Centre, RAL
What exactly is a filesystem ?
Organization: • Large disk space to store small data items • Use files and directories
In UNIX everything is a stream of bytes: • More complex file definition in other OS’s • Devices are also presented in the filesystem • Linux uses virtual filesystems, e.g. /proc
Jonathan Wheeler e-Science Centre, RAL
A little bit of background
Three generation of UNIX filesystems: a) v7 / sysv / coherent / minix b) ffs / ext2 c) vxfs / ext3 / jfs / xfs / reiserfs
Jonathan Wheeler e-Science Centre, RAL
Features of 3rd generation UNIX filesystems
a) b) c) d) Intent logging / journaling Flexible metadata structures Dynamic inode allocations Extents
Jonathan Wheeler e-Science Centre, RAL
History of XFS
Developed by Silicon Graphics (now SGI) for use on IRIX: • 1993 - original design • 1994 - available for IRIX v 5.3 • 2000 - released as Open Source • 2001/2002 – some Linux distributions with native XFS support • 2003 – XFS support in 2.4 kernel
Jonathan Wheeler e-Science Centre, RAL
XFS features (1)
Journaling: • but only metadata updates, not data updates. Fast Transactions: • uses efficient B+ tree structures for fast searches and rapid space allocation
Jonathan Wheeler e-Science Centre, RAL
XFS Features (2)
64-bit file system:
a) b) 64bit variables for global counters and disk addresses Large theoretical maximum filesystem size i. For Linux 2.4: 2 Terabytes. ii. For Linux 2.6 (when using 64 bit addressing on a 64 bit platform): 9 Petabytes (or the device limits) iii. For Linux 2.6 (on 32 bit platforms): 16Terabytes (even with 64 bit addressing) Maximum file size is same as filesystem size
Jonathan Wheeler e-Science Centre, RAL
c)
XFS Features (3)
Partitioned into Allocation Groups: a) each AG manages its own free space and inodes b) provides scalability and parallelism within the file system c) limits the size of the structures needed to track this information
Jonathan Wheeler e-Science Centre, RAL
XFS Features (3, continued)
d) allows many internal pointers to be 32bits e) AGs typically range in size from 0.5 to 4GB f) files and directories are not limited to a single AG.
Jonathan Wheeler e-Science Centre, RAL
XFS Features (4)
Sophisticated support utilities: a) fast mkfs (make a file system) b) dump and restore utilities for backup c) xfsrepair to fix corrupt filesystem d) xfs_fsr (XFS defragmenter) e) xfsdb (XFS debug) f) xfscheck (XFS check) g) xfs_growfs (enlarges filesystems online)
Jonathan Wheeler e-Science Centre, RAL
Why you might want to use XFS
a) Stable, mature code base (oldest journaling filesystem available on Unix) b) Very good performance for large reads or writes c) Designed for large systems d) Supports user and group quotas e) Support for Access Control Lists and Extended Attributes
Jonathan Wheeler e-Science Centre, RAL
Extended Attributes
An extended attribute is a name/value pair associated with a file. Attributes can be attached to all types of inodes: regular files, directories, symbolic links, device nodes, and so forth. Attribute values can contain up to 64KB of arbitrary binary data. See man attr for more details
Jonathan Wheeler e-Science Centre, RAL
Extended attribute namespaces
XFS implements three attribute namespaces: a) user namespace available to all users, protected by the normal file permissions b) system namespace, accessible only to privileged users and used for protected filesystem meta-data such as access control lists (ACLs) and hierarchical storage manager (HSM) file migration status c) security namespace used by security modules (SELinux)
Jonathan Wheeler e-Science Centre, RAL
Why you might not want to use XFS
a) No data journaling b) tar, cpio do not understand Extended Attributes
Jonathan Wheeler e-Science Centre, RAL
Resources
XFS/Linux homepage: http://oss.sgi.com/projects/xfs/ contains pointers to whitepapers, books, articles, etc. Public mailing list: linux-xfs@oss.sgi.com with searchable list archive SGI Bugzilla database to report any bugs in XFS for Linux
Jonathan Wheeler e-Science Centre, RAL
Comparison tests - 1
Tests were performed: • on the same filesystem on the same hardware (AMD Athlon 1666 Mhz, 500Mb) • using 3 different operating systems: i. Redhat 7.3 (2.4.20-28.7) ii. Scientific Linux version 3 (2.4.21-20.EL) iii. Scientific Linux version 4 (2.6.9-22.0.1.EL and 2.6.9.-11.EL.XFS)
Jonathan Wheeler e-Science Centre, RAL
Comparison tests - 2
Tests used different filesystem types: • RH 7.3 – ext2, ext3 (xfs not available) • SL 3 – ext2, ext3 (attempts to create xfs filesystem caused system crash requiring hardware reset) • SL 4 – ext2, ext3, xfs (default), xfs (custom: mkfs.xfs –d agsize=4g –i version=1 –I size=512 )
Jonathan Wheeler e-Science Centre, RAL
Comparison tests - 3
Tests were performed: • using bonnie++ test program (see http://www.coker.com.au/bonnie++) • using the same parameters for each test: /usr/local/sbin/bonnie++ -d /data \ -s 4096M:1024k \ -n 192:1048576:128:1024 -m csfnfs30 \ -u root 1>> outputfile 2>&1
Jonathan Wheeler e-Science Centre, RAL
Comparison tests - 4
12 tests in total, each reporting filesystem performance and CPU use The first 6 tests measure I/O intensive work: • Sequential output for character, blocked and rewrites • Sequential input for character and blocked • Random seeks
Jonathan Wheeler e-Science Centre, RAL
Comparison tests - 6
The other 6 tests measure performance for many small files using: • Sequential create, read and delete • Random create, read and delete
Jonathan Wheeler e-Science Centre, RAL
Comparsion of filesystems (Sequential output)
35000
30000
25000
Per character (Kb/sec)
20000 Series1 15000
10000
5000
0 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type
Jonathan Wheeler e-Science Centre, RAL
Comparison of filesystems (Sequential output)
35000
30000
25000
Per block (Kb/sec)
20000 Series1 15000
10000
5000
0 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type
Jonathan Wheeler e-Science Centre, RAL
Comparison of filesystems (Sequential output)
25000
20000
Rewrite (Kb/sec)
15000 Series1 10000
5000
0 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type
Jonathan Wheeler e-Science Centre, RAL
Comparsion of filesystems (Sequential input)
35000
30000
25000
Per character (Kb/sec)
20000 Series1 15000
10000
5000
0 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type
Jonathan Wheeler e-Science Centre, RAL
Comparison of filesystems (Sequential input)
100000 98000 96000 94000
Per block (Kb/sec)
92000 90000 88000 86000 84000 82000 80000 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type Series1
Jonathan Wheeler e-Science Centre, RAL
Compariosn of filesystems (Random seeks)
25
20
Number/sec
15 Series1 10
5
0 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type
Jonathan Wheeler e-Science Centre, RAL
Comparison of filesystems (Sequential create)
45 40 35 30
Create (Num/sec)
25 Series1 20 15 10 5 0 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type
Jonathan Wheeler e-Science Centre, RAL
Comparison of filesystems (Sequential create)
180 160 140 120
Read (number/sec)
100 Series1 80 60 40 20 0 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type
Jonathan Wheeler e-Science Centre, RAL
Comparison of filesystems
400
350
300
Delete (number/sec)
250
200
Series1
150
100
50
0 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type
Jonathan Wheeler e-Science Centre, RAL
Comparison of filesystems (Random create)
25
20
Create (Number/sec)
15 Series1 10
5
0 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type
Jonathan Wheeler e-Science Centre, RAL
Comparison of filesystems (Random create)
25
20
Read (Number/sec)
15 Series1 10
5
0 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type
Jonathan Wheeler e-Science Centre, RAL
Comparison of filesystems (Random create)
50 45 40 35
Delete (Number/sec)
30 25 20 15 10 5 0 73.ext2 73.ext3 sl3.ext2 sl3.ext3 sl4.ext2 sl4.ext3 sl4.xfs sl4.xfs.special Filesystem type Series1
Jonathan Wheeler e-Science Centre, RAL