Iscsi Project Proposal by gnu15931


More Info
									     Connectivity Standard for the Future
              Storage Network

                              Lizhi Charlie Zhong
1. Motivation

When we look at the applications for future internet, some of them will enable real time
communications, like voice over IP, video internet phone or online PC gaming. But most
of the applications will simply be a form of information retrieval. Net surfing, which
brings Internet to where it is now, is obviously all about searching and obtaining
information one wants. Video on demand allows user to view prerecorded movies
through Internet. Getting more and more popular nowadays are web-based email and
personal homepages on a web server. Instead of storing information in limited local
resource, people start to store them on the web for easy access, including some personal
information. Web based storage goes up exponentially as people start to put almost
everything on the web, including digital photos and video clips from their digital
camcorders, which require much more memory than plain text files people used to store
on the web. The future internet can be characterized as huge amount of information
delivered in lightening fast speed over long distance with high reliability and security.
Needless to say, storage technology is the key for the future internet.

The earlier forms of storage technologies simply connect the storage devices to a host in
the same room. The maximum distance can only be 6 meters and maximum data transfer
rate is 5MB/sec. The current state of art storage technology is Fibre Channel, which
allows the host to be 10km away with maximum data transfer rate being 400MB/sec. The
storage technologies have evolved from a single point to point connection between a host
and multiple storage devices to a network that having both hosts and storage devices as
its nodes. There are two ways to build a network involving storage devices. One way is to
attach the storage devices to an existing network, for example Ethernet or internet. This
approach is very simple to implement and storage devices can be reconfigured easily. But
the transfer of the data is impacted by the other traffic in the same network, resulting in
long delay and extra overhead. The other approach is to build a separate network with
only the storage devices and servers who need to access the data in it. This network is
often called storage area network (SAN), or backend network, compared to the normal
network, which is also referred to as front-end network. The first approach is sometimes
called network attached storage (NAS), which is just the reverse ordering of the letters in
SAN. The SAN incurs the extra cost of building a separate network sorely for storage
devices. Additionally, a storage device can not be reconfigured without bringing down
the entire system. But the special purpose network separates the storage data from the
other data traffic in a front-end network, so it is not subject to the erratic behavior of the
front-end network and there is not as much overhead as in a front-end network. High
reliability and security can also be easier to implement.
Two fundamental questions arise. With the fast implementation of optical internet and
emerging internet protocols supporting quality of service, which of NAS or SAN will be
more prevalent? If SAN is used, can popular network technology like Ethernet be used?

2. Fibre Channel

Fiber Channel (FC) was started back in 1988 by the American National Standards Institute
(ANSI) Task Group X3T11. It is designed to be a common, efficient transport service
independent of protocol.

2.1 Layers

FC does not follow Open System Interconnect (OSI) model, but instead, the protocol has been
broken into five layers: FC-0, FC-1, FC-2, FC-3, and FC-4. Each is briefly described below along
with the main functions it defines.

       Signaling
       Media specifications
       Receiver/Transmitter specifications
       8B/10B character encoding
       Link maintenance
       Frame format
       Sequence management
       Exchange management
       Flow Control
       Classes of Service
       Login/Logout
       Topologies
       Segmentation and Reassembly
       Services for multiple ports on one node
       Upper Layer Protocol (ULP) mapping
            Small Computer System Interface (SCSI)
            Internet Protocol (IP)
            High Performance Parallel Interface (HIPPI)
            Asynchronous Transfer Mode - Adaptation Layer 5 (ATM-AAL5)
            Intelligent Peripheral Interface - 3 (IPI-3) (disk and tape)
            Single Byte Command Code Sets (SBCCS)

One can roughly think of the FC layers defining up through the Transport layer of the OSI model.

2.2 Classes of Service
FC defines several communication strategies called Classes of service. The Class used greatly
depends on the type of data to be transmitted. The major difference between among the Classes is
the types of flow control used.

Connection-oriented services

Class 1 Dedicated connection, no need for buffer-to-buffer flow control, only end-to-end flow
control is used Class 1 would be used when the data needs to be continuous and time critical,
such as voice or video.

Class 4 Virtual connections, Quality of Service, fractional bandwidth, and buffer-to-
buffer flow control for each Virtual Circuit (VC). End-to-end flow control is used like in
Class 1.

Class 6 Multicast service through a Fabric. End-to-end flow control is used between the source
and the multicast server.

Connectionless services

Class 2 With acknowledgment. Both buffer-to-buffer and end-to-end flow control are used in
Class 2. Class 2 is more like typical LAN traffic, such as IP or FTP, where the order and
timeliness of delivery is not so important.

Class 3 Without acknowledgment. It only uses buffer-to-buffer flow control. It is referred
to a datagram service. Class 3 would be used when order and timeliness is not so
important, and when the ULP itself handles lost frames efficiently. Class 3 is the choice
for SCSI.

2.3 Flow Control
 FC uses credit based flow control. Credit refers to the number of frames a device can receive at a
time. With this type of flow control, data will not be delivered faster than the destination buffer is
able to receive it. So it is congestion free. FC uses two types of flow control, buffer-to-buffer and

2.4 Topologies

FC defines three topologies, namely Point-to-Point, Arbitrated Loop, and Fabric.
 A Point-to-Point topology is the simplest of the three. It consists of two and only two FC devices
connected directly together. There is no sharing of the media, which allows the devices to enjoy
the total bandwidth of the link.

Arbitrated Loop has become the most dominant FC topology, but it is also the most complex. It's
a cost-effective way of connecting up to 127 ports in a single network without the need of a
Fabric switch. Unlike the other two topologies, the media is shared among the devices, limiting
each device's access. Like most ring topologies, life is made easier if the devices can be
connected to a central hub or concentrator. The cabling is easier to deal with, and the hub can
usually determine when to insert or de-insert a device. Thus, a "bad" device or broken fiber won't
keep the whole network down. FC has hubs as well. Most hubs will simply look for valid signal
on a port before it will insert the port in the data path. Some hubs are "smarter", and will provide
the user with information on each of its ports, allow the user to transmit from the ports, and be
more selective as to when ports are inserted.

The Fabric topology is used to connect many (224) devices in a cross-point switched
configuration. The benefit of this topology is that many devices can communicate at the same
time; the media is not shared. Of course, it also requires the purchase of a switch.

2.5 Media, speed and distance

FC can run over both copper and fiber media, which is why it is called Fibre Channel rather than
Fiber Channel. The transfer rates of Fibre Channel are currently (133 Mbps, 266 Mbps, 530
Mbps, and 1 Gbps). However, data rates of 2 to 4 Gbps should be available soon. It supports
distance up to 10km.

2.6 FC over IP (FCIP)

FC frames are encapsulated into IP packets to allow islands of FC SANs to be
interconnected over IP-based networks running over very reliable datalinks.

2.7 Advantages and Disadvantages of FC

1. FC is optimized for storage application. It supports SCSI, a traditional storage interface.
Legacy SCSI storage system can also be interfaced using a FC-to-SCSI bridge. The fact that FC
supports IP also allows the same optical fiber being used in a front end network where servers
communicates. FC is an open storage system, and it can connect all kinds of storage system,
including RAID, tape backup, tape library, CD-ROM library and JBOD (Just a Bunch of Disks).

2. FC is a proven technology. It has been adopted by the major computer systems and storage
manufactures as the next technology for enterprise storage. FC is being installed at a rapid rate
while issues surrounding other technologies may not be resolved before 2004 or 2005. Companies
are dying for more storage and will not wait. The value of disk drives attached to FC based SANs
will grow from 1.3 billion last year to 4 billion this year and 24.6 billion by 2003, according to
GartnerGroup Dataquest.

3. FC uses optical fiber whose BER is less than 10^(-12). Its credit-based flow control makes it
congestion free. So FC is highly reliable.

4. FC has very little transmission overhead. Most importantly, the FC protocol is specifically
designed for highly efficient operation using hardware. In contrast, TCP is typically implemented
in software stack.

5. FC supports multiple classes of services. It has very low latency.

6. FC based SAN is a separate network. So storage data is protected physically from outside

7. FC’s ability to use a single technology for storage, networks, audio/video, or to move raw data
is superior to the common frame feature adopted by Ethernet. FC’s use in both networks and
storage provides a price savings since same optical fiber can be shared


1. FC is used only in storage area network, a very small segment of the network market. The
relative smaller volume makes the cost higher.

2. FC is difficult to implement. People are very Ethernet-centric in the midsize enterprise, where
Ethernet is pervasive. They are not familiar with FC installation and interoperability. FC has a
new set of networking software, routers. Company need to develop expertise in FC networking
and buy unfamiliar fiber-optic switches, routers and bridges.

3. A minimum of management software is available in FC. More time will be spent to support the

4. Improvements made by FC over TCP/IP networks, such as security and quality of service, have
already been dealt with in TCP/IP networks. TCP accelerators are being developed and TCP/IP
stacks can be implemented in HW. Gigabit Ethernet uses same physical layer as FC, therefore it
has similar BER, while 10-Gigabit Ethernet has much lower BER.
5. Interoperability in building FC based SAN is quite an issue since components come from
different suppliers. In many cases, one switch can not communicate with another, and standard
bodies are still deciding what Management Information Base will be used to interface devices on
a SAN network with management software.

6. FC without a doubt wins the award for the largest and most complex set of standards
documents for a communications protocol. The initial effort was started back in 1988, and much
of it continues to undergo significant development to this day. There are now well over 20
individual standards documents, some of which have been adopted as standards, but many remain
in draft form. Unfortunately, the current trend in FC is to continue to define more and more
standard documents that increase the complexity of the protocol. This is the biggest threat to its

7. FC is not very good for long distances with latency, dropped packets and congestion problems.
FCIP has been developed to allow FC over IP-based MAN.

8. IP network is already very popular. FC is relatively new and it is hard to survive as long as IP
network is good enough. FC’s interface to existing IP networks is not efficient.

9. FC will be an extra network, on top of existing Ethernet, even though it could be used as a
front-end network.

10. FC requires additional hardware interface in each machine while network adapter is already

                                   Fibre Channel                           ATM
              Technology           Storage, network,      Network,storage, Network,
              application          video, clusters        video            video
                                   point-to-point loop
              Topologies           hub,                                    Switched
                                                          hub, switched
              Baud rate            1.06 Gbps              1.25 Gbps        622 Mbps
              Scalability to       2.12 Gbps, 4.24        12.5 Gbps, 50
                                                                           1.24 Gbps
              higher data rates    Gbps                   Gbps
                                   Yes                    No               No
              Congestion data
                                   None                   Yes              Yes
                                                          Variable, 0-
              Frame size           Variable, 0-2KB                         Fixed, 53B
              Flow control         Credit Based           Rate Based       Rate Based
                                                                           Copper and
              Physical media       Copper and Fiber       Copper and Fiber
              Protocols            Network, SCSI,                          Network,
                                                          Network, video
              supported            Video                                   video
                          Table 1 Technology comparison

3. Gigabit Ethernet
Gigabit Ethernet was released in 1998 by the IEEE 802.3z Gigabit Ethernet task force. 10-Gigabit
Ethernet is currently under development by IEEE 802.3 Higher Speed Study Group.

Ethernet is the dominant network technology. More than 85 percent of all installed network
connections were Ethernet by the end of 1997 according to IDC. And in 1998, the technology
captured 86 percent of shipments.

Gigabit Ethernet uses the same frame format, which allows easy migration from existing
Ethernet. At the same time, it has more bandwidth and it is much more reliable. For 10-Gigabit
Ethernet, it is likely only full duplex mode is supported, meaning there will be no broadcasting
and no collision.

Gigabit Ethernet is yet another gigabit technology available. Its high bandwidth fit nicely in the
application of storage area network. Its lower cost gives it quite an edge over FC, so it is a very
good candidate for storage network.


1. Low cost: both purchase cost and support cost.

2. Maintenance is easier. Large number of people have been trained and there are vast number of
management software and trouble shooting tools available.

3. Can use a single network for both server to server communication and storage

4. Scalable to higher data rates: 10 Gbps and 40 Gbps

5. Optical link is very reliable. 10^(-18) BER or lower

6. Short standardization cycle. It took approximately 13 months to go from first draft to final
approval for Gigabit Ethernet. 10-Gigabit Ethernet and 40-Gigabit Ethernet are coming.

7. Longer distance 10-Gigabit Ethernet can go up to 40km.

8. Ethernet only defines up to data link layer, can add higher layers specified by open standards
based on application requirement.

9.Easier to interface to existing LAN and WAN.


1. Congestion loss due to rate based flow control.

2. Does not support SCSI, not backward compatible to existing storage technologies.

3. IP network invites outside access (FC free from that due to separate network).

4. Latency due to loss of data and prepackaging.

4. Storage over Gigabit Ethernet
To address the requirements of storage area network, Gigabit Ethernet needs to
 Support SCSI or have storage interface
 Low latency
 Security
 Highly reliable

Gigabit Ethernet only defines up to the data link layer, so it is possible to choose proper higher
layers to satisfy these requirements. Due to the popularity of NAS, many storage devices have
Ethernet network interface already. And it is not difficult to do so in new storage systems (the
network interface is already there on the server side). Legacy SCSI storage system can be
interfaced through a bridge. Mapping SCSI commands is also an option. SCSI only requires
datagram service at the lower level since it handles lost frames efficiently. There is a proposal
called internet SCSI (iSCSI), which is SCSI over TCP. It maps SCSI commands to TCP similar
to the way they are mapped to FC.

Resource Reservation Protocol (RSVP) provides bandwidth reservation. IEEE 802.q and 802.p
provide virtual LAN. Both help to reduce latency.

The Encapsulating Security Payload (ESP) function in Ipsec protocol can be used to both
authenticate and encrypt storage data to achieve security.

Gigabit Ethernet is highly reliable already, so this is not a problem.

5. Conclusion

Gigabit Ethernet provides gigabit, highly reliable connectivity needed by future storage network.
Paired with higher layer protocols, it can achieve the desired security and latency requirements
set by storage application. Its popularity in LAN, and MAN and WAN allows it to have lower
cost and better support. The truly integration of SAN with LAN, MAN and WAN will be made
possible by the use of Gigabit Ethernet, 10-Gigabit Ethernet and so on.

To top