Too much Internet backbone?
The last issue of the now geography of these connections provides suppliers invested in the Web to support
defunct Red Herring magazine in March insight into the network’s economic a myriad of targeted customer needs.
2003—may it rest in peace—devoted its determinants. Four institutional factors shaped net-
last breaths to a cover article discussing That said, interpreting the size of back- work infrastructure in the commercial era:
the strategies of entrepreneurs who were bones and comparing the backbones in First, the commercial network developed
buying up each other’s networks at fire different regions is not straightforward. distinct cooperative institutions for
sale prices. The article seemed to imply Its presence depends on many parame- exchanging data. It retained several of the
that commercial providers had over- ters, such as population size, type of local NSF-developed public exchange points
invested in backbone capacity in the US. industry, and other facets of local but transferred their operation to associ-
Is there any economic sense in claim- demand. ations of private ﬁrms.
ing that there is too much backbone? More to the point, delivery resources Second, a few other large ﬁrms
The commercial Internet consists of are concentrated in some areas. Net- exchanged trafﬁc using private peering—a
hubs, routers, switches, points of pres- works overlap in many locations. This bilateral agreement between two ﬁrms to
ence, and a high-speed, high-capacity redundancy looks inefficient, but that is exchange data at no cost to save each of
pipe that transmits data. People some- too simple an observation. them monitoring costs—but did not dis-
times call this pipe a backbone for short. criminate on any basis other than ﬂow size.
The Internet’s backbone connects servers Features of the Internet backbone Third, many smaller ISPs paid for
operated by different Internet service The backbone for US networks grew access to larger networks, entering into
providers (ISPs). It connects city nodes out of the network built for the US Nation- long-term contracts.
and transports data over long distances, al Science Foundation by IBM and MCI. Fourth, and quite signiﬁcantly, popular
mostly via fiber optic lines of various They designed NSFnet to serve the Web sites, such as Yahoo, made deals with
speeds and capacity. research needs of universities, and it con- cache/mirror sites operated by ﬁrms such
No vendor can point to a speciﬁc piece nected several thousand universities by as Akamai, Digital Island, and Inktomi.
of fiber and call it a backbone. This con- the early 1990s. NSFnet had concentrat- These deals eliminated much of the dif-
cept is fiction—one network line looks ed much of its communications infra- ferences in performance among loca-
and functions pretty much the same as structure around a dozen supercomputer tions—at least for the most popular sites
any other line—but the ﬁction is a conve- centers. This infrastructure connected to that could afford to pay these mirror sites.
nient one. Every major vendor has a net- other major research universities in many It also eliminated much of the competitive
work with lines that run from one point to locations, resulting in a geographically differences among the backbone locations.
another. It is too much trouble to refer to concentrated transmission network with To the delight of some and the dismay
such lines as “intermediate transmission wide dispersion of access points. of others, a multitiered system emerged.
capacity devoted primarily to carrying traf- The few loci of concentration in NSFnet The tiers were associated with footprint
ﬁc from many sources to many sources. ” were quite different from the concentra- size and trafﬁc volume; they largely pro-
The backbone’s presence and structure tion that developed after commercializa- vided shorthand designations for those
provide information about which cities are tion. The commercial Web’s connectivity that carried data over long distances, and
playing the most prominent role in the did not arise from a single entity support- those that collected charges from others
Internet’s development and diffusion. The ing a single purpose. Instead, multiple for transit service.
0272-1732/03/$17.00 2003 IEEE
78 Published by the IEEE Computer Society
For example, the largest ﬁrms all From the outset, the distribution of Coordination or crazy building?
became tier 1 providers. These ﬁrms includ- backbone capacity did not perfectly emu- Commercial ﬁrms in most industries do
ed AT&T, IBM (before it sold its provider ser- late population distribution within metro- not coordinate their building plans with
vice to AT&T), MCI, UUNet (before politan regions. Seattle, Austin, and each other; the firms building the com-
WorldCom purchased both), and Sprint. Boston have a disproportionately large mercial Internet were no different. This led
Most regional and local ISPs became lower- number of connections (relative to their to replicated transmission capacity along
tier ISPs, purchasing interconnection from populations) while larger cities such as similar paths. The backbone network of
one of several national providers. Philadelphia and Detroit have dispropor- the late 1990s embodied features that
Rarely do engineers and economists tionately fewer connections. In addition, strongly reﬂected both the lack of coordi-
see the world in precisely the same way, the largest metropolitan areas are well nation and presence of dynamic incen-
but on this one instance there was gener- served by the backbone while areas such tives. Both forces arose from the absence
al agreement. My engineering friends as the rural South have few connections. of monopoly.
commented that data interchange pre- On a broad level, this design feature To be sure, the US backbone is not a
served the architectures’ end-to-end looks familiar to observers of other US monopoly. There is a multiplicity of play-
consistency—that is, it did not alter per- communications networks. As found in ers building it, competing with each other.
formance based on the user’s identity or land-based line networks, there are This competition enhances the dynamic
the application type. Stated simply, trans- economies of scale in high-capacity incentives of each player to grow quickly,
mission capacity built anywhere interact- switching equipment and the transmis- price competitively, experiment broadly,
ed with transmission capacity situated sion of signals along high-capacity routes. and tailor network services to fill cus-
anywhere else. Appropriate contracts han- In addition, there are a few (commonly tomer needs. The impatient environment
dled the costs of interconnection among employed) right-of-way pathways (along of the late 1990s provided strong incen-
ISPs and backbone ﬁrms. Facility owner- rail lines, highways, and pipelines, for tives to grow quickly, even when the
ship did not induce discrimination based example) that are available for long-dis- growth was redundant. In other words, a
on the origin or destination of the data or tance transmission lines. Thus, straight- competitive market among network
type of application. forward economic reasons dictate that providers will necessarily lead to uncoor-
Economically speaking, the preserva- each firm have only a few major trunk dinated build-outs, such as overlapping
tion of end-to-end consistency had three lines running along similar paths. footprints and other redundancies.
signiﬁcant consequences: First, it did not Remarkably, despite the entry of many Is this outcome necessarily bad? Yes and
discourage vendors from building geo- ﬁrms and the growth of the total capaci- no. To be sure, these overlaps and redun-
graphically overlapping networks. Sec- ty, maps of Internet backbone for US data dancies appear inefﬁcient after the
ond, vendors could specialize with transmission have not changed much network is built, but this is a myopic inter-
regional footprints. Third, this built-in con- from 1997 to the present. This speaks to pretation. A competitive market gives all
sistency prevented firms with national the durability in economies of scale in players strong incentives to quickly build
footprints from having big advantages in high-capacity switching and transmission, their network, price it low, and ﬁgure out
the deployment of end-to-end applica- even during a period of high growth. In how to customize it to user needs. It
tions that require low amounts of signal other words, as with many communica- should be no surprise, therefore, that the
delay, such as virtual private networks or tion networks, the present backbone net- commercial US backbone looks so
videoconferencing applications. work is a hub and feeder system with only extremely redundant. By the same token,
a few hubs. the backbone might not have been built so
Geographic dispersion of capacity Another notable feature of competitive quickly in the absence of such competition.
During the ﬁrst few years of the Inter- backbones is multiple data-exchange Related, uncoordinated investment of
net’s commercialization, a handful of US points. Choices for data exchange loca- sunken investments can potentially lead
cities or regions dominated backbone tions have persisted into the commercial to price wars in the event of overbuilding.
capacity. Speciﬁcally, San Francisco/Silicon era. To be sure, some of these points Indeed, it would be surprising if uncoor-
Valley, Chicago, New York, Dallas, Los would have arisen under any system dinated investment resulted in too little
Angeles, Atlanta, and Washington, DC, because of their central location within infrastructure during a period of sustained
contained links to the vast majority of back- the midst of trafﬁc ﬂow. For example, data demand growth, as occurred in the late
bone capacity. As of 1997 these seven exchange in New York or Washington, DC, 1990s.
areas accounted for 64.6 percent of total makes sense for trafﬁc along the US East
capacity. By 1999, even though network Coast. The same is true for Dallas or Where are we today?
capacity quintupled over the previous two Atlanta in the South, Chicago in the Mid- These are not pleasant times for back-
years, the top seven still accounted for west, and San Francisco and Los Angeles bone providers. Spring 2000 saw the
58.8 percent of total capacity. in the West. decline of ﬁnancial support for dot-coms.
MARCH–APRIL 2003 79
The 11 September terrorist attacks shook consolidation to completely eliminate
business conﬁdence in long-term invest- redundancies, this shakeout will reshape
ments. This low continued as the media the precise ownership of the backbones
publicized the WorldCom ﬁnancial scandal. carrying future data traffic flows.
This down cycle is not over, which Observers lament that much of the
leaves an open question about the long- installed ﬁber remains unlit, while those
term state of US backbone networks. with financial acumen prowl the bank-
Qwest, Level 3 Communications, Sprint, ruptcy courts like jackals after a kill, gath-
Global Crossing, WorldCom’s MCI, ering together the assets of others. The
Williams Communications, PSINet, AT&T, final ownership configuration remains
and others all invested heavily in redun- unknown as of this writing.
dant transmission capacity during the Yes, it looks crazy, but this is what you
boom. UUNet, a WorldCom division, was expect in a market with uncoordinated
the largest backbone data carrier in the investment and a period of intense
US until the scandals that lead to World- growth followed by a period of collapse.
Com’s bankruptcy. PSINet overextended That is not so notable. The notable out-
itself and had to declare bankruptcy. come is that the players built the network
Although nobody expects ownership as fast as they did.
Coming Next Issue MAY-JUNE 2003
Guest Editors Alberto Sangiovanni-Vincentelli, UC Berkeley, and
Luciano Lavagno, Politecnico di Torino
Directions and Trends in Microelectronics
Topics will include
• Electronic System-Level Design and the Automotive Industry;
• Systems on Chip: Now and in the Future;
• Interconnect Opportunities for Gigascale Integration; and
• The Evolution of ICs and Microelectromechanical Systems (MEMS) Technology.
IEEE Micro serves
80 IEEE MICRO