Connectivity Standard for the Future Storage Network Lizhi Charlie Zhong 1. Motivation When we look at the applications for future internet, some of them will enable real time communications, like voice over IP, video internet phone or online PC gaming. But most of the applications will simply be a form of information retrieval. Net surfing, which brings Internet to where it is now, is obviously all about searching and obtaining information one wants. Video on demand allows user to view prerecorded movies through Internet. Getting more and more popular nowadays are web-based email and personal homepages on a web server. Instead of storing information in limited local resource, people start to store them on the web for easy access, including some personal information. Web based storage goes up exponentially as people start to put almost everything on the web, including digital photos and video clips from their digital camcorders, which require much more memory than plain text files people used to store on the web. The future internet can be characterized as huge amount of information delivered in lightening fast speed over long distance with high reliability and security. Needless to say, storage technology is the key for the future internet. The earlier forms of storage technologies simply connect the storage devices to a host in the same room. The maximum distance can only be 6 meters and maximum data transfer rate is 5MB/sec. The current state of art storage technology is Fibre Channel, which allows the host to be 10km away with maximum data transfer rate being 400MB/sec. The storage technologies have evolved from a single point to point connection between a host and multiple storage devices to a network that having both hosts and storage devices as its nodes. There are two ways to build a network involving storage devices. One way is to attach the storage devices to an existing network, for example Ethernet or internet. This approach is very simple to implement and storage devices can be reconfigured easily. But the transfer of the data is impacted by the other traffic in the same network, resulting in long delay and extra overhead. The other approach is to build a separate network with only the storage devices and servers who need to access the data in it. This network is often called storage area network (SAN), or backend network, compared to the normal network, which is also referred to as front-end network. The first approach is sometimes called network attached storage (NAS), which is just the reverse ordering of the letters in SAN. The SAN incurs the extra cost of building a separate network sorely for storage devices. Additionally, a storage device can not be reconfigured without bringing down the entire system. But the special purpose network separates the storage data from the other data traffic in a front-end network, so it is not subject to the erratic behavior of the front-end network and there is not as much overhead as in a front-end network. High reliability and security can also be easier to implement. Two fundamental questions arise. With the fast implementation of optical internet and emerging internet protocols supporting quality of service, which of NAS or SAN will be more prevalent? If SAN is used, can popular network technology like Ethernet be used? 2. Fibre Channel Fiber Channel (FC) was started back in 1988 by the American National Standards Institute (ANSI) Task Group X3T11. It is designed to be a common, efficient transport service independent of protocol. 2.1 Layers FC does not follow Open System Interconnect (OSI) model, but instead, the protocol has been broken into five layers: FC-0, FC-1, FC-2, FC-3, and FC-4. Each is briefly described below along with the main functions it defines. FC-0 Signaling Media specifications Receiver/Transmitter specifications FC-1 8B/10B character encoding Link maintenance FC-2 Frame format Sequence management Exchange management Flow Control Classes of Service Login/Logout Topologies Segmentation and Reassembly FC-3 Services for multiple ports on one node FC-4 Upper Layer Protocol (ULP) mapping Small Computer System Interface (SCSI) Internet Protocol (IP) High Performance Parallel Interface (HIPPI) Asynchronous Transfer Mode - Adaptation Layer 5 (ATM-AAL5) Intelligent Peripheral Interface - 3 (IPI-3) (disk and tape) Single Byte Command Code Sets (SBCCS) One can roughly think of the FC layers defining up through the Transport layer of the OSI model. 2.2 Classes of Service FC defines several communication strategies called Classes of service. The Class used greatly depends on the type of data to be transmitted. The major difference between among the Classes is the types of flow control used. Connection-oriented services Class 1 Dedicated connection, no need for buffer-to-buffer flow control, only end-to-end flow control is used Class 1 would be used when the data needs to be continuous and time critical, such as voice or video. Class 4 Virtual connections, Quality of Service, fractional bandwidth, and buffer-to- buffer flow control for each Virtual Circuit (VC). End-to-end flow control is used like in Class 1. Class 6 Multicast service through a Fabric. End-to-end flow control is used between the source and the multicast server. Connectionless services Class 2 With acknowledgment. Both buffer-to-buffer and end-to-end flow control are used in Class 2. Class 2 is more like typical LAN traffic, such as IP or FTP, where the order and timeliness of delivery is not so important. Class 3 Without acknowledgment. It only uses buffer-to-buffer flow control. It is referred to a datagram service. Class 3 would be used when order and timeliness is not so important, and when the ULP itself handles lost frames efficiently. Class 3 is the choice for SCSI. 2.3 Flow Control FC uses credit based flow control. Credit refers to the number of frames a device can receive at a time. With this type of flow control, data will not be delivered faster than the destination buffer is able to receive it. So it is congestion free. FC uses two types of flow control, buffer-to-buffer and end-to-end. 2.4 Topologies FC defines three topologies, namely Point-to-Point, Arbitrated Loop, and Fabric. A Point-to-Point topology is the simplest of the three. It consists of two and only two FC devices connected directly together. There is no sharing of the media, which allows the devices to enjoy the total bandwidth of the link. Arbitrated Loop has become the most dominant FC topology, but it is also the most complex. It's a cost-effective way of connecting up to 127 ports in a single network without the need of a Fabric switch. Unlike the other two topologies, the media is shared among the devices, limiting each device's access. Like most ring topologies, life is made easier if the devices can be connected to a central hub or concentrator. The cabling is easier to deal with, and the hub can usually determine when to insert or de-insert a device. Thus, a "bad" device or broken fiber won't keep the whole network down. FC has hubs as well. Most hubs will simply look for valid signal on a port before it will insert the port in the data path. Some hubs are "smarter", and will provide the user with information on each of its ports, allow the user to transmit from the ports, and be more selective as to when ports are inserted. The Fabric topology is used to connect many (224) devices in a cross-point switched configuration. The benefit of this topology is that many devices can communicate at the same time; the media is not shared. Of course, it also requires the purchase of a switch. 2.5 Media, speed and distance FC can run over both copper and fiber media, which is why it is called Fibre Channel rather than Fiber Channel. The transfer rates of Fibre Channel are currently (133 Mbps, 266 Mbps, 530 Mbps, and 1 Gbps). However, data rates of 2 to 4 Gbps should be available soon. It supports distance up to 10km. 2.6 FC over IP (FCIP) FC frames are encapsulated into IP packets to allow islands of FC SANs to be interconnected over IP-based networks running over very reliable datalinks. 2.7 Advantages and Disadvantages of FC Advantages: 1. FC is optimized for storage application. It supports SCSI, a traditional storage interface. Legacy SCSI storage system can also be interfaced using a FC-to-SCSI bridge. The fact that FC supports IP also allows the same optical fiber being used in a front end network where servers communicates. FC is an open storage system, and it can connect all kinds of storage system, including RAID, tape backup, tape library, CD-ROM library and JBOD (Just a Bunch of Disks). 2. FC is a proven technology. It has been adopted by the major computer systems and storage manufactures as the next technology for enterprise storage. FC is being installed at a rapid rate while issues surrounding other technologies may not be resolved before 2004 or 2005. Companies are dying for more storage and will not wait. The value of disk drives attached to FC based SANs will grow from 1.3 billion last year to 4 billion this year and 24.6 billion by 2003, according to GartnerGroup Dataquest. 3. FC uses optical fiber whose BER is less than 10^(-12). Its credit-based flow control makes it congestion free. So FC is highly reliable. 4. FC has very little transmission overhead. Most importantly, the FC protocol is specifically designed for highly efficient operation using hardware. In contrast, TCP is typically implemented in software stack. 5. FC supports multiple classes of services. It has very low latency. 6. FC based SAN is a separate network. So storage data is protected physically from outside access. 7. FC’s ability to use a single technology for storage, networks, audio/video, or to move raw data is superior to the common frame feature adopted by Ethernet. FC’s use in both networks and storage provides a price savings since same optical fiber can be shared Disadvantages: 1. FC is used only in storage area network, a very small segment of the network market. The relative smaller volume makes the cost higher. 2. FC is difficult to implement. People are very Ethernet-centric in the midsize enterprise, where Ethernet is pervasive. They are not familiar with FC installation and interoperability. FC has a new set of networking software, routers. Company need to develop expertise in FC networking and buy unfamiliar fiber-optic switches, routers and bridges. 3. A minimum of management software is available in FC. More time will be spent to support the network. 4. Improvements made by FC over TCP/IP networks, such as security and quality of service, have already been dealt with in TCP/IP networks. TCP accelerators are being developed and TCP/IP stacks can be implemented in HW. Gigabit Ethernet uses same physical layer as FC, therefore it has similar BER, while 10-Gigabit Ethernet has much lower BER. 5. Interoperability in building FC based SAN is quite an issue since components come from different suppliers. In many cases, one switch can not communicate with another, and standard bodies are still deciding what Management Information Base will be used to interface devices on a SAN network with management software. 6. FC without a doubt wins the award for the largest and most complex set of standards documents for a communications protocol. The initial effort was started back in 1988, and much of it continues to undergo significant development to this day. There are now well over 20 individual standards documents, some of which have been adopted as standards, but many remain in draft form. Unfortunately, the current trend in FC is to continue to define more and more standard documents that increase the complexity of the protocol. This is the biggest threat to its future. 7. FC is not very good for long distances with latency, dropped packets and congestion problems. FCIP has been developed to allow FC over IP-based MAN. 8. IP network is already very popular. FC is relatively new and it is hard to survive as long as IP network is good enough. FC’s interface to existing IP networks is not efficient. 9. FC will be an extra network, on top of existing Ethernet, even though it could be used as a front-end network. 10. FC requires additional hardware interface in each machine while network adapter is already there. . Gigabit Fibre Channel ATM Ethernet Technology Storage, network, Network,storage, Network, application video, clusters video video point-to-point loop Point-to-point Topologies hub, Switched hub, switched switched Baud rate 1.06 Gbps 1.25 Gbps 622 Mbps Scalability to 2.12 Gbps, 4.24 12.5 Gbps, 50 1.24 Gbps higher data rates Gbps Gbps Guaranteed Yes No No delivery Congestion data None Yes Yes loss Variable, 0- Frame size Variable, 0-2KB Fixed, 53B 1.5KB Flow control Credit Based Rate Based Rate Based Copper and Physical media Copper and Fiber Copper and Fiber Fiber Protocols Network, SCSI, Network, Network, video supported Video video Table 1 Technology comparison 3. Gigabit Ethernet Gigabit Ethernet was released in 1998 by the IEEE 802.3z Gigabit Ethernet task force. 10-Gigabit Ethernet is currently under development by IEEE 802.3 Higher Speed Study Group. Ethernet is the dominant network technology. More than 85 percent of all installed network connections were Ethernet by the end of 1997 according to IDC. And in 1998, the technology captured 86 percent of shipments. Gigabit Ethernet uses the same frame format, which allows easy migration from existing Ethernet. At the same time, it has more bandwidth and it is much more reliable. For 10-Gigabit Ethernet, it is likely only full duplex mode is supported, meaning there will be no broadcasting and no collision. Gigabit Ethernet is yet another gigabit technology available. Its high bandwidth fit nicely in the application of storage area network. Its lower cost gives it quite an edge over FC, so it is a very good candidate for storage network. Advantages: 1. Low cost: both purchase cost and support cost. 2. Maintenance is easier. Large number of people have been trained and there are vast number of management software and trouble shooting tools available. 3. Can use a single network for both server to server communication and storage 4. Scalable to higher data rates: 10 Gbps and 40 Gbps 5. Optical link is very reliable. 10^(-18) BER or lower 6. Short standardization cycle. It took approximately 13 months to go from first draft to final approval for Gigabit Ethernet. 10-Gigabit Ethernet and 40-Gigabit Ethernet are coming. 7. Longer distance 10-Gigabit Ethernet can go up to 40km. 8. Ethernet only defines up to data link layer, can add higher layers specified by open standards based on application requirement. 9.Easier to interface to existing LAN and WAN. Disadvantages: 1. Congestion loss due to rate based flow control. 2. Does not support SCSI, not backward compatible to existing storage technologies. 3. IP network invites outside access (FC free from that due to separate network). 4. Latency due to loss of data and prepackaging. 4. Storage over Gigabit Ethernet To address the requirements of storage area network, Gigabit Ethernet needs to Support SCSI or have storage interface Low latency Security Highly reliable Gigabit Ethernet only defines up to the data link layer, so it is possible to choose proper higher layers to satisfy these requirements. Due to the popularity of NAS, many storage devices have Ethernet network interface already. And it is not difficult to do so in new storage systems (the network interface is already there on the server side). Legacy SCSI storage system can be interfaced through a bridge. Mapping SCSI commands is also an option. SCSI only requires datagram service at the lower level since it handles lost frames efficiently. There is a proposal called internet SCSI (iSCSI), which is SCSI over TCP. It maps SCSI commands to TCP similar to the way they are mapped to FC. Resource Reservation Protocol (RSVP) provides bandwidth reservation. IEEE 802.q and 802.p provide virtual LAN. Both help to reduce latency. The Encapsulating Security Payload (ESP) function in Ipsec protocol can be used to both authenticate and encrypt storage data to achieve security. Gigabit Ethernet is highly reliable already, so this is not a problem. 5. Conclusion Gigabit Ethernet provides gigabit, highly reliable connectivity needed by future storage network. Paired with higher layer protocols, it can achieve the desired security and latency requirements set by storage application. Its popularity in LAN, and MAN and WAN allows it to have lower cost and better support. The truly integration of SAN with LAN, MAN and WAN will be made possible by the use of Gigabit Ethernet, 10-Gigabit Ethernet and so on.
Pages to are hidden for
"Iscsi Project Proposal"Please download to view full document