Your Federal Quarterly Tax Payments are due April 15th Get Help Now >>

134 by lanyuehua


									                            Arhitektura PCI Ekspres sistema

                           PCI Express System Architecture

     Naumović Ana, Milašinović Bojana, Stanić Aleksandar, and Milutinović Veljko
              School of Electrical Engineering, University of Belgrade

Sadržaj - PCI Ekspres predstavlja treću                    2. PCI vs PCI Express
generaciju magistrala koja se koristi za povezivanje
širokog spektra sistema i perifernih uređaja. Ova          It is obvious that PCI Express evolutioned from already
tehnologija omogućava značajno veće performanse i          mentioned predecessor buses. The memory, IO and
veliku brzinu prenosa informacija. Veza je tipa point-     configuration address space model remain the same.
to-point, sa dve jednosmerne linije za prenos              By maintaining the address space model, existing OSs
podataka, u oba smera po jedna. U ovom radu                and drivers software will run in a PCI Express system
pokušaćemo da objasnimo osnovne pojmove vezane             without any modifications. So, PCI Express is software
za ovu tehnologiju. U drugom delu objasnićemo              backwards compatible with PCI and PCI-X systems.
osnovne razlike između PCI i PCI Ekspres                   Also, PCI Express supports chip-to-chip interconnect
tehnologije i takođe objasnićemo arhitekturu sistema       and board-to-board interconnect via cards and
koja uključuje topologiju sistema, transakcije i           connectors. The structure of the cards and the conectors
slojevitu strukturu.                                       is very similar to the structure of the PCI cards and
Abstract - PCI Express is the third-generation
Peripheral Component Inter-connect technology for
a wide range of systems and peripheral devices. It
provides significantly higher performance, high
speed, point-to-point, dual simplex type of
connection. We will try to explain some of the basic
facts about this technology. In the second part this
paper shows some differences between PCI and PCI
Express and also, about basic system architecture of
PCI Express, which includes architecture, topology,
transactions and layers.

1. Introduction
PCI and PCI-X, as the predecessor generation of            Fig. 1. PCI system topology.
buses are multi-drop, parallel, interconnect buses.
The multi-drop way for connecting the devices in the       The PCI system (Fig. 1.) consists of a Host (CPU) bus-
system represents a structure with one main bus (in        to-PCI bus bridge, also referred to as the North bridge.
our case PCI or PCI-X) where many devices share            Associated with the North bridge is the system memory
one bus. On the other hand, PCI Express implements         bus, graphics (AGP) bus, and a 33 MHz PCI bus. I/O
a serial, point-to-point type of connection. This is one   devices share the PCI bus and are connected to it in a
of the basic differencies between PCI Express and its      multi-drop fashion. These devices are either connected
predecesor buses. Some architectural changes had to        directly to the PCI bus on the motherboard or by way
be done in order to improve bus performance, reduce        of a peripheral card plugged into a connector on the
overall system cost and take advantage of new              bus.
developments in computer design. The point-to-point        A South bridge bridges the PCI bus to the ISA bus
way doesnt mean that all the devices in the system         where slower, lower performance peripherals exist.
which have to exchange the data have to be                 Associated with the south bridge is a USB and IDE
connected directly. Multiple PCI devices are               bus. A CD or hard disk is associated with the IDE bus.
interconnected via switches, so it is obvious that we      The South bridge contains an interrupt controller (not
can connect a large number of devices togehter in a        shown) to which interrupt signals from PCI devices are
system. Currently transmission and reception data          connected. The interrupt controller is connected to the
rate is 2.5 Gbits/sec, but it is expected that frequency   CPU via an INTR signal or an APIC bus. The South
to double to 5 Gbits/sec, and even quadruple to 10         bridge is the central resource that provides the source
Gbits/sec. A serial interconnect between two devices       of reset, reference clock, and error reporting signals.
results in fewer pins per device package, which            Boot ROM exists on the ISA bus along with a Super IO
means that PCI Express chip and board design cost          chip, which includes keyboard, mouse, floppy disk
and board design complexity are reduced.
controller and serial/parallel bus controllers. The PCI    ingress ports are forwarded to their own VC buffers at
bus arbiter logic is included in the North bridge.         the egress port. These transactions are prioritized based
We will not consider here bus cycles becouse rather        on the ingress port number when being merged into a
than bus cycles we are familiar with from PCI and          common VC output buffer for delivery across the
PCI-X architectures, PCI Express encodes                   egress link. This arbitration is referred to as Port
transactions using a packet based protocol.                arbitration.
Some of the PCI Express advantages are reduced PCI         A packet transmitted by a device is received into a VC
Express chip, board design cost and board design           buffer in the receiver at the opposite end of the Link.
complexity, the after effect of using serial rather than   The receiver periodically updates the transmitter with
parallel way of transmitting.                              information regarding the amount of buffer space it has
                                                           available. The transmitter device will only transmit a
3. PCI Express Overview                                    packet to the receiver if it knows that the receiving
                                                           device has sufficient buffer space to hold the next
                                                           transaction. The protocol by which the transmitter
A PCI Express interconnect consists of either x1, x2,      ensures that the receiving buffer has sufficient space
x4, x8, x16 or x32 point-to-point Link. A Lane             available is referred to as flow control.
consists of signal pairs in each direction. A x1 Link
consists of 1 Lane or 1 differential signal pair in each
direction for a total of four signals. The Link supports   4. PCI Express Topology
a symmetric number of Lanes in each direction.
PCI Express devices employ differential drivers and        Basic components that are included in the PCI Express
receivers at each port. A positive voltage difference      Architecture is shown in the picture below (Fig. 2.).
between the D+ and D- terminals implies Logical 1.         Some basic knowledge of the elements from this image
A negative voltage difference between D+ and D-            will give us global picture of PCI Express Architecture.
implies a Logical 0. No voltage difference between         The Root Complex is the connection between CPU,
D+ and D- means that the driver is in the high-            memory and PCI Express fabric. On the root can be
impedance tristate condition, which is referred to as      connected one or more PCI Express ports, on which
the electrical-idle and low-power state of the Link.       ends are endpoint devices or switches. This kind of
PCI Express encodes transactions using a packet            connection forms so called sub-hierarchy.
based protocol. Packets are transmitted and received       The main role of root complex is to perform
serially and byte striped across the available Lanes of    transactions. The CPU is relieved from the half of
the Link. The more Lanes implemented on a Link the         transactions because the root complex took over them.
faster a packet is transmitted and the greater the         Those transactions are memory, IO and locked
bandwidth of the Link. We mentioned above that the         transactions. Also he can be a requester and a
curently bandwidth is 2.5 gbits/sec. This speed refers     completer. And, last but not least, it can transmit
to one Lane of the Link.                                   packets from one port to another.
The Quality of Service feature of PCI Express refers
to the capability of routing packets from different
applications through the fabric with differentiated
priorities. PCI Express packets contain a Traffic
Class (TC) number between 0 and 7 that is assigned
by the device's application or device driver. Packets
with different TCs can move through the fabric with
different priority, resulting in varying performances.
These packets are routed through the fabric by
utilizing virtual channel (VC) buffers. VC buffers can
be implemented in switches, endpoints and root             Fig. 2. Pci Express system topology.
complex devices. Each Traffic Class is individually
mapped to a Virtual Channel (a VC can have several         A Hierarchy means all the devices and links that are
TCs mapped to it, but a TC cannot be mapped to             connected to the root complex, either directly or via its
multiple VCs). The TC in each packet is used by the        ports, switches and bridges.
transmitting and receiving ports to determine which        A Hierarchy Domain is a fabric of devices and Links
VC buffer to drop the packet into. Switches and            that are associated with one port of the root complex.
devices are configured to arbitrate and prioritize         Endpoints are peripheral devices, such as Ethernet,
between packets from different VCs before                  USB, etc. When we talk about transactions, endpoints
forwarding. This arbitration is referred to as VC          are very important for PCI Express architecture
arbitration. In addition, packets arriving at different
because they have the role of a requester or a            PCI-X architectures, but the message transaction is
completer. They are categorized into two groups: PCI      something new. Transactions correspond to a series of
Express endpoints and legacy endpoints.                   packet transmissions between a requester and a
A Requester is a device that initiates transaction with   completer (which were earlier mentioned) and they are
a completer. It reads data from a completer or writes     categorized into non-posted transactions and posted
data in a completer. A requester can be the root          transactions. Also, because the data is not always
complex or the endpoint device.                           settled in the same packets for read and write
A Completer is a device addressed or targeted by a        tansactions, we have to make difference between those
requester. Also, as a requester, completer can be the     two.
root complex or the endpoints.                            For Non-posted transactions, a requester transmits a
A Port is an interface between a PCI Express              TLP request packet to a completer. When completer
component and the Link. It consists of differential       receives that packet, he returns a TLP completion
transmitters and receivers. An Upstream Port is a port    packet back to the requester, and by that he is telling
that points in the direction of the root complex. A       requester that he has received the requester TLP.
Downstream Port is a port that points away from the       Non-Posted Read Transactions – There are four basic
root complex. An endpoint port is an upstream port.       steps, that are included in this kind of transaction,
A root complex port(s) is a downstream port. An           between requester and completer, so this transaction
Ingress Port is a port that receives a packet. An         completes. They are:
Egress Port is a port that transmits a packet.              1. A requester transmits non posted read transaction
We can imagine the term switch as a two or more                 TLP to a completer.
virtual PCI-to-PCI bridges where each bridge is             2. A completer decodes its contain and gather the
associated with a switch port. For example, 4 port              amount of data, which is specified in the request
switch consists of 4 virtual bridges, which are                 TLP.
connected via non defined bus (in our case – bus2).         3. A completer creates single completion TLP or
One port of the switch have to be in the direction of           multiple completion TLP with data.
the root complex. The main role of the switches is to       4. That completion TLP is being sent from a
forward all kinds of transactions to the out (this              completer to a requester, so he would „know“ that
means that it has to provide packets moving into the            a completer received data.
right direction, so the packets safely arrive to the      If an error occure in a request TLP and a completer
requester/completer). Routing trought the switch is       cannot obtain data, he creates completion TLP (step 3)
based on its address, device ID or implicitly.            and instead of data he writes an error status indication.
Switches implement two arbitration mechanisms,            By that requester knows that an error occured.
port arbitration and VC arbitration, by which they        A reqester can be root complex or an endpoint device,
determine the priority with which to forward              and a completer can be, also, root complex, switch,
packets from ingress ports to egress ports. Switches      bridge or endpoint.
support locked requests.                                  Non-Posted Write Transactions - As you can presume
Each PCI Express Link is equivalent to a logical PCI      the way of transaction is basicly the same:
bus. In other words, each Link is assigned a bus            1. A requester transmits non posted write transaction
number by the bus enumerating software. A PCI                   TLP to a completer, in which he intends to write
Express endpoint is device 0 on a PCI Express Link              to.
of a given bus number. Only one device (device 0)           2. A completer decodes its contain and accepts data.
exists per PCI Express Link. The internal bus within        3. A completer creates single completion TLP
a switch that connects all the virtual bridges together         without data.
is also numbered. The first Link associated with the        4. That completion TLP is being sent from a
root complex is number bus 1. Bus 0 is an internal              completer to a requester, for confirming reception
virtual bus within the root complex.                            of write request.
                                                          If an error occure in a request TLP and a completer
5. PCI Express Transactions                               cannot accept data, he creates completion TLP (step 3)
                                                          with an error status indication. By that requester knows
PCI Express is some kind of communication between         that an error occured.
two devices. That communication includes                  As for non posted read transactions, a reqester can be
transmission and reception of packets. That packets       root complex or an endpoint device.
are usually called Transaction Layer Packets (TLPs).      Basic difference those two transactions are that for read
There are four basic categories of transactions, and      transactions data is in the completion TLP, and for
they are: memory, IO, configuration and message           write transactions data is in the request TLP.
transactions. The first three are supported in PCI and
For posted transactions, a requester transmits a TLP
request packet to a completer. But, unlike non posted     We have to make difference between these two
transactions, completer does not answers to the           portions, because of their different behavior. If we are
receiver.                                                 talking about transaction portion we are looking at the
Posted Memory Read Transactions – The transaction         packet that has been created in the transaction layer and
contains the first two steps of non posted read           located in the buffer. This packet is better known as a
transaction. The third and the fourth steps do not        Transaction Layer Packet (TLP), and he is waiting to
exist in this way of transactions, as it is already       be moved to the lower layers. Data Link layer adds to
mentioned.                                                TLP some additional information which are taking care
Posted Memory Write Transactions - The transaction        of errors. The packet waits to be moved to the lower
contains the first two steps of non posted write          layer. Physical layer encodes the packet, and it has
transaction. The third and the fourth steps do not        been sent over the Link to the analog portion of this
exist in this way of transactions, as it is already       layer (in this case that is receiving portion of the
mentioned.                                                Physical layer). If we are talking about receive portion,
Posted transactions are optimized for the best            Physical layer decodes the incoming packet and sends
performance in completing the transaction at the          what is left to the upper layers. Data link layer scans
expense of the requester not having knowledge of          for some errors in the packets and if there are no ones
successful reception of the request by the completer.     DLL sends packet to the Transaction layer. Transaction
The completer could log an error and generate an          layer shifts TLP into the buffer and converts
error message notification to the root complex. Error     information so they can serve some device core or
handling software manages the error.                      application.
Posted Message Transactions – There are a lot of          Because of the different looking of the packet within
message transactions, like from requester to a            these three layers, we have to understand how these
completer, or from the root complex to all endpoints,     packets look in the particular moment. The
or inverse. Routing to the completer is based on its      transmission begins and finishes in the transaction
address, device ID or implicitly. The completer           layer. The packet in this layer is named TLP, as it is
accepts any data that may be contained in the packet      already mentioned. TLP consists of three specific parts
and/or performs the task specified by the message.        and they are heather, data and ECRC. The first two are
                                                          been sent from the software layer or the device core,
                                                          and they symbol the core section of the TLP. Some of
                                                          the TLPs do not have data part. Heather includes
6. PCI Express Device Layers                              information such as address, TLP type, transfer size,
                                                          requester ID/completer ID, traffic class, byte enables
For PCI Express is characteristic layered architecture,   completion codes, tag and attributes. The field data is
which consists of three layers. They are Transaction,     speaking for itself. If we talk about read request TLPs,
Data Link and Physical Layer. All of them can be          the data field is empty, and it usually excluded from the
vertically divided into two portions: transmit and        TLP. ECRC or End-to-End CRC is the error that has
receive, as it is shown in the picture below (Fig. 3.).   been found in the core. ECRC is calculated in this layer
                                                          and added to the core section. It is based on the entire
                                                          TLP, from the first byte of the header to the last byte of
                                                          the data. Next layer is named Data Link layer and
                                                          packet in this layer is named Data Link Layer Packet or
                                                          DLLP. This layer adds or removes (depends of portion,
                                                          transmit or receive) two parts: sequence ID and LCRC.
                                                          Sequence ID is used by the reply mechanism (e.g. non
                                                          posted transactions). The LCRC field is used by the
                                                          neighboring receiver device at the other end of the Link
                                                          to check for CRC errors in the core section of the TLP
                                                          plus the sequence ID. As you can presume packet
                                                          within the Physical layer is called Physical Layer
                                                          Packet or PLP. This layer concatenates start and end
                                                          fields to the packet so we receive PLP. So at the
                                                          moment when packet is traveling through the Link he
                                                          looks as shown in Fig. 4.

Fig. 3. Layered architecture
                                                         Data Link Layer – this layer is responsable for CRC
                                                         generation and for error checking. Its primary function
                                                         is to ensure data integrity.
                                                         Physical Layer – it is divided into two portions: logical
                                                         and electrical. The logical part contains digital logic
                                                         associated with processing packets before transmission
                                                         on the Link, or receiving packets from the Link before
                                                         sending to the Data Link Layer. The electrical portion
                                                         is the analog interface of the Physical Layer that
                                                         connects to the Link. It consists of differential drivers
                                                         and receivers for each Lane. Packet from the Data Link
                                                         layer is settled in the logical portion.
Fig. 4. Packet traveling through the link.
                                                         In the future, it is expected PCI Express to progress.
If we want to get the whole picture about layers we      We really with the great impatiance are expecting that
have to know what the functions of these layers are.     time.
Device core/software layer – it can be root complex
core or endpoint core, such as Ethernet controller,      7. References
USB controller, etc.
Transmit portion and Receive portion we already
explained.                                               [1] Ravi Budruk, Don Anderson, and Tom Shanley,
Transaction layer – it receives information from         “PCI Express System Architecture,” Addison Wesley,
Device core. Using that information it creates TLP. It   September 04, 2003.
contains virtual channel buffers where he stores TLP.    [2] Congdon, B., Solari, E., Clark, D.,
The flow control protocol associated with these          “The Complete PCI Express Reference: Design
virtual channel buffers ensures that a remote            Implications for Hardware and Software Developers
transmitter does not transmit too many TLPs and          (Engineer to Engineer series),” Intel Press, 2003.
cause the receiver virtual channel buffers to            [3] Wilen, A., Schade, J., Thornburg, R.,
overflow. The Transaction Layer also orders TLPs         “Introduction to PCI Express: Hardware and Software
according to ordering rules before transmission. It is   Developer's Guide,” Intel Press, 2002.
this layer that supports the Quality of Service (QoS)    [4]
It supports 4 address spaces: memory address, IO         1D3E338E6F52386256E37006DFDB3
address, configuration address and message space.        [5]

To top