Debian Codegraph @

rfc3439.html

raw
1 <pre>Network Working Group R. Bush
2 Request for Comments: 3439 D. Meyer
3 Updates: <a href="./rfc1958">1958</a> December 2002
4 Category: Informational
5
6
7 <span class="h1">Some Internet Architectural Guidelines and Philosophy</span>
8
9 Status of this Memo
10
11 This memo provides information for the Internet community. It does
12 not specify an Internet standard of any kind. Distribution of this
13 memo is unlimited.
14
15 Copyright Notice
16
17 Copyright (C) The Internet Society (2002). All Rights Reserved.
18
19 Abstract
20
21 This document extends <a href="./rfc1958">RFC 1958</a> by outlining some of the philosophical
22 guidelines to which architects and designers of Internet backbone
23 networks should adhere. We describe the Simplicity Principle, which
24 states that complexity is the primary mechanism that impedes
25 efficient scaling, and discuss its implications on the architecture,
26 design and engineering issues found in large scale Internet
27 backbones.
28
29 Table of Contents
30
31 <a href="#section-1">1</a>. Introduction . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-2">2</a>
32 <a href="#section-2">2</a>. Large Systems and The Simplicity Principle . . . . . . . . . <a href="#page-3">3</a>
33 <a href="#section-2.1">2.1</a>. The End-to-End Argument and Simplicity . . . . . . . . . <a href="#page-3">3</a>
34 <a href="#section-2.2">2.2</a>. Non-linearity and Network Complexity . . . . . . . . . . <a href="#page-3">3</a>
35 <a href="#section-2.2.1">2.2.1</a>. The Amplification Principle. . . . . . . . . . . . . . . <a href="#page-4">4</a>
36 <a href="#section-2.2.2">2.2.2</a>. The Coupling Principle . . . . . . . . . . . . . . . . . <a href="#page-5">5</a>
37 <a href="#section-2.3">2.3</a>. Complexity lesson from voice. . . . . . . . . . . . . . . <a href="#page-6">6</a>
38 <a href="#section-2.4">2.4</a>. Upgrade cost of complexity. . . . . . . . . . . . . . . . <a href="#page-7">7</a>
39 <a href="#section-3">3</a>. Layering Considered Harmful. . . . . . . . . . . . . . . . . <a href="#page-7">7</a>
40 <a href="#section-3.1">3.1</a>. Optimization Considered Harmful . . . . . . . . . . . . . <a href="#page-8">8</a>
41 3.2. Feature Richness Considered Harmful . . . . . . . . . . . <a href="#page-9">9</a>
42 3.3. Evolution of Transport Efficiency for IP. . . . . . . . . <a href="#page-9">9</a>
43 <a href="#section-3.4">3.4</a>. Convergence Layering. . . . . . . . . . . . . . . . . . . <a href="#page-9">9</a>
44 <a href="#section-3.4.1">3.4.1</a>. Note on Transport Protocol Layering. . . . . . . . . . . <a href="#page-11">11</a>
45 <a href="#section-3.5">3.5</a>. Second Order Effects . . . . . . . . . . . . . . . . . . <a href="#page-11">11</a>
46 <a href="#section-3.6">3.6</a>. Instantiating the EOSL Model with IP . . . . . . . . . . <a href="#page-12">12</a>
47 <a href="#section-4">4</a>. Avoid the Universal Interworking Function. . . . . . . . . . <a href="#page-12">12</a>
48 <a href="#section-4.1">4.1</a>. Avoid Control Plane Interworking . . . . . . . . . . . . . <a href="#page-13">13</a>
49
50
51
52 <span class="grey">Bush, et. al. Informational [Page 1]</span></pre>
53 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-2" ></span>
54 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
55
56
57 <a href="#section-5">5</a>. Packet versus Circuit Switching: Fundamental Differences . . <a href="#page-13">13</a>
58 <a href="#section-5.1">5.1</a>. Is PS is inherently more efficient than CS? . . . . . . . <a href="#page-13">13</a>
59 <a href="#section-5.2">5.2</a>. Is PS simpler than CS? . . . . . . . . . . . . . . . . . . <a href="#page-14">14</a>
60 <a href="#section-5.2.1">5.2.1</a>. Software/Firmware Complexity . . . . . . . . . . . . . . <a href="#page-15">15</a>
61 <a href="#section-5.2.2">5.2.2</a>. Macro Operation Complexity . . . . . . . . . . . . . . . <a href="#page-15">15</a>
62 <a href="#section-5.2.3">5.2.3</a>. Hardware Complexity. . . . . . . . . . . . . . . . . . . <a href="#page-15">15</a>
63 <a href="#section-5.2.4">5.2.4</a>. Power. . . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-16">16</a>
64 <a href="#section-5.2.5">5.2.5</a>. Density. . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-16">16</a>
65 <a href="#section-5.2.6">5.2.6</a>. Fixed versus variable costs. . . . . . . . . . . . . . . <a href="#page-16">16</a>
66 <a href="#section-5.2.7">5.2.7</a>. QoS. . . . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-17">17</a>
67 <a href="#section-5.2.8">5.2.8</a>. Flexibility. . . . . . . . . . . . . . . . . . . . . . . <a href="#page-17">17</a>
68 <a href="#section-5.3">5.3</a>. Relative Complexity . . . . . . . . . . . . . . . . . . . <a href="#page-17">17</a>
69 <a href="#section-5.3.1">5.3.1</a>. HBHI and the OPEX Challenge. . . . . . . . . . . . . . . <a href="#page-18">18</a>
70 <a href="#section-6">6</a>. The Myth of Over-Provisioning. . . . . . . . . . . . . . . . <a href="#page-18">18</a>
71 <a href="#section-7">7</a>. The Myth of Five Nines . . . . . . . . . . . . . . . . . . . <a href="#page-19">19</a>
72 <a href="#section-8">8</a>. Architectural Component Proportionality Law. . . . . . . . . <a href="#page-20">20</a>
73 <a href="#section-8.1">8.1</a>. Service Delivery Paths . . . . . . . . . . . . . . . . . . <a href="#page-21">21</a>
74 <a href="#section-9">9</a>. Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-21">21</a>
75 <a href="#section-10">10</a>. Security Considerations . . . . . . . . . . . . . . . . . . <a href="#page-22">22</a>
76 <a href="#section-11">11</a>. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . <a href="#page-23">23</a>
77 <a href="#section-12">12</a>. References. . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-23">23</a>
78 <a href="#section-13">13</a>. Authors' Addresses. . . . . . . . . . . . . . . . . . . . . <a href="#page-27">27</a>
79 <a href="#section-14">14</a>. Full Copyright Statement. . . . . . . . . . . . . . . . . . <a href="#page-28">28</a>
80
81 <span class="h2"><a class="selflink" id="section-1" href="#section-1">1</a>. Introduction</span>
82
83 <a href="./rfc1958">RFC 1958</a> [<a href="./rfc1958" title="&quot;Architectural principles of the Internet&quot;">RFC1958</a>] describes the underlying principles of the
84 Internet architecture. This note extends that work by outlining some
85 of the philosophical guidelines to which architects and designers of
86 Internet backbone networks should adhere. While many of the areas
87 outlined in this document may be controversial, the unifying
88 principle described here, controlling complexity as a mechanism to
89 control costs and reliability, should not be. Complexity in carrier
90 networks can derive from many sources. However, as stated in
91 [<a href="#ref-DOYLE2002" title="&quot;Robustness and the Internet: Theoretical Foundations&quot;">DOYLE2002</a>], "Complexity in most systems is driven by the need for
92 robustness to uncertainty in their environments and component parts
93 far more than by basic functionality". The major thrust of this
94 document, then, is to raise awareness about the complexity of some of
95 our current architectures, and to examine the effect such complexity
96 will almost certainly have on the IP carrier industry's ability to
97 succeed.
98
99 The rest of this document is organized as follows: The first section
100 describes the Simplicity Principle and its implications for the
101 design of very large systems. The remainder of the document outlines
102 the high-level consequences of the Simplicity Principle and how it
103 should guide large scale network architecture and design approaches.
104
105
106
107
108 <span class="grey">Bush, et. al. Informational [Page 2]</span></pre>
109 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-3" ></span>
110 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
111
112
113 <span class="h2"><a class="selflink" id="section-2" href="#section-2">2</a>. Large Systems and The Simplicity Principle</span>
114
115 The Simplicity Principle, which was perhaps first articulated by Mike
116 O'Dell, former Chief Architect at UUNET, states that complexity is
117 the primary mechanism which impedes efficient scaling, and as a
118 result is the primary driver of increases in both capital
119 expenditures (CAPEX) and operational expenditures (OPEX). The
120 implication for carrier IP networks then, is that to be successful we
121 must drive our architectures and designs toward the simplest possible
122 solutions.
123
124 <span class="h3"><a class="selflink" id="section-2.1" href="#section-2.1">2.1</a>. The End-to-End Argument and Simplicity</span>
125
126 The end-to-end argument, which is described in [<a href="#ref-SALTZER" title="&quot;End-To-End Arguments in System Design&quot;">SALTZER</a>] (as well as
127 in <a href="./rfc1958">RFC 1958</a> [<a href="./rfc1958" title="&quot;Architectural principles of the Internet&quot;">RFC1958</a>]), contends that "end-to-end protocol design
128 should not rely on the maintenance of state (i.e., information about
129 the state of the end-to-end communication) inside the network. Such
130 state should be maintained only in the end points, in such a way that
131 the state can only be destroyed when the end point itself breaks."
132 This property has also been related to Clark's "fate-sharing" concept
133 [<a href="#ref-CLARK" title="&quot;The Design Philosophy of the DARPA Internet Protocols&quot;">CLARK</a>]. We can see that the end-to-end principle leads directly to
134 the Simplicity Principle by examining the so-called "hourglass"
135 formulation of the Internet architecture [<a href="#ref-WILLINGER2002" title="&quot;Robustness and the Internet: Design and evolution&quot;">WILLINGER2002</a>]. In this
136 model, the thin waist of the hourglass is envisioned as the
137 (minimalist) IP layer, and any additional complexity is added above
138 the IP layer. In short, the complexity of the Internet belongs at
139 the edges, and the IP layer of the Internet should remain as simple
140 as possible.
141
142 Finally, note that the End-to-End Argument does not imply that the
143 core of the Internet will not contain and maintain state. In fact, a
144 huge amount coarse grained state is maintained in the Internet's core
145 (e.g., routing state). However, the important point here is that
146 this (coarse grained) state is almost orthogonal to the state
147 maintained by the end-points (e.g., hosts). It is this minimization
148 of interaction that contributes to simplicity. As a result,
149 consideration of "core vs. end-point" state interaction is crucial
150 when analyzing protocols such as Network Address Translation (NAT),
151 which reduce the transparency between network and hosts.
152
153 <span class="h3"><a class="selflink" id="section-2.2" href="#section-2.2">2.2</a>. Non-linearity and Network Complexity</span>
154
155 Complex architectures and designs have been (and continue to be)
156 among the most significant and challenging barriers to building cost-
157 effective large scale IP networks. Consider, for example, the task
158 of building a large scale packet network. Industry experience has
159 shown that building such a network is a different activity (and hence
160 requires a different skill set) than building a small to medium scale
161
162
163
164 <span class="grey">Bush, et. al. Informational [Page 3]</span></pre>
165 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-4" ></span>
166 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
167
168
169 network, and as such doesn't have the same properties. In
170 particular, the largest networks exhibit, both in theory and in
171 practice, architecture, design, and engineering non-linearities which
172 are not exhibited at smaller scale. We call this Architecture,
173 Design, and Engineering (ADE) non-linearity. That is, systems such
174 as the Internet could be described as highly self-dissimilar, with
175 extremely different scales and levels of abstraction [<a href="#ref-CARLSON" title="&quot;Complexity and Robustness&quot;">CARLSON</a>]. The
176 ADE non-linearity property is based upon two well-known principles
177 from non-linear systems theory [<a href="#ref-THOMPSON" title="&quot;Nonlinear Dynamics and Chaos&quot;">THOMPSON</a>]:
178
179 <span class="h4"><a class="selflink" id="section-2.2.1" href="#section-2.2.1">2.2.1</a>. The Amplification Principle</span>
180
181 The Amplification Principle states that there are non-linearities
182 which occur at large scale which do not occur at small to medium
183 scale.
184
185 COROLLARY: In many large networks, even small things can and do cause
186 huge events. In system-theoretic terms, in large systems such as
187 these, even small perturbations on the input to a process can
188 destabilize the system's output.
189
190 An important example of the Amplification Principle is non-linear
191 resonant amplification, which is a powerful process that can
192 transform dynamic systems, such as large networks, in surprising ways
193 with seemingly small fluctuations. These small fluctuations may
194 slowly accumulate, and if they are synchronized with other cycles,
195 may produce major changes. Resonant phenomena are examples of non-
196 linear behavior where small fluctuations may be amplified and have
197 influences far exceeding their initial sizes. The natural world is
198 filled with examples of resonant behavior that can produce system-
199 wide changes, such as the destruction of the Tacoma Narrows bridge
200 (due to the resonant amplification of small gusts of wind). Other
201 examples include the gaps in the asteroid belts and rings of Saturn
202 which are created by non-linear resonant amplification. Some
203 features of human behavior and most pilgrimage systems are influenced
204 by resonant phenomena involving the dynamics of the solar system,
205 such as solar days, the 27.3 day (sidereal) and 29.5 day (synodic)
206 cycles of the moon or the 365.25 day cycle of the sun.
207
208 In the Internet domain, it has been shown that increased inter-
209 connectivity results in more complex and often slower BGP routing
210 convergence [<a href="#ref-AHUJA" title="&quot;The Impact of Internet Policy and Topology on Delayed Routing Convergence&quot;">AHUJA</a>]. A related result is that a small amount of
211 inter-connectivity causes the output of a routing mesh to be
212 significantly more complex than its input [<a href="#ref-GRIFFIN" title="&quot;What is the Sound of One Route Flapping&quot;">GRIFFIN</a>]. An important
213 method for reducing amplification is ensure that local changes have
214 only local effect (this is as opposed to systems in which local
215 changes have global effect). Finally, ATM provides an excellent
216 example of an amplification effect: if you lose one cell, you destroy
217
218
219
220 <span class="grey">Bush, et. al. Informational [Page 4]</span></pre>
221 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-5" ></span>
222 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
223
224
225 the entire packet (and it gets worse, as in the absence of mechanisms
226 such as Early Packet Discard [<a href="#ref-ROMANOV" title="&quot;Dynamics of TCP over ATM Networks&quot;">ROMANOV</a>], you will continue to carry
227 the already damaged packet).
228
229 Another interesting example of amplification comes from the
230 engineering domain, and is described in [<a href="#ref-CARLSON" title="&quot;Complexity and Robustness&quot;">CARLSON</a>]. They consider the
231 Boeing 777, which is a "fly-by-wire" aircraft, containing as many as
232 150,000 subsystems and approximately 1000 CPUs. What they observe is
233 that while the 777 is robust to large-scale atmospheric disturbances,
234 turbulence boundaries, and variations in cargo loads (to name a few),
235 it could be catastrophically disabled my microscopic alterations in a
236 very few large CPUs (as the point out, fortunately this is a very
237 rare occurrence). This example illustrates the issue "that
238 complexity can amplify small perturbations, and the design engineer
239 must ensure such perturbations are extremely rare." [<a href="#ref-CARLSON" title="&quot;Complexity and Robustness&quot;">CARLSON</a>]
240
241 <span class="h4"><a class="selflink" id="section-2.2.2" href="#section-2.2.2">2.2.2</a>. The Coupling Principle</span>
242
243 The Coupling Principle states that as things get larger, they often
244 exhibit increased interdependence between components.
245
246 COROLLARY: The more events that simultaneously occur, the larger the
247 likelihood that two or more will interact. This phenomenon has also
248 been termed "unforeseen feature interaction" [<a href="#ref-WILLINGER2002" title="&quot;Robustness and the Internet: Design and evolution&quot;">WILLINGER2002</a>].
249
250 Much of the non-linearity observed large systems is largely due to
251 coupling. This coupling has both horizontal and vertical
252 components. In the context of networking, horizontal coupling is
253 exhibited between the same protocol layer, while vertical coupling
254 occurs between layers.
255
256 Coupling is exhibited by a wide variety of natural systems, including
257 plasma macro-instabilities (hydro-magnetic, e.g., kink, fire-hose,
258 mirror, ballooning, tearing, trapped-particle effects) [<a href="#ref-NAVE" title="&quot;The influence of mode coupling on the non-linear evolution of tearing modes&quot;">NAVE</a>], as
259 well as various kinds of electrochemical systems (consider the custom
260 fluorescent nucleotide synthesis/nucleic acid labeling problem
261 [<a href="#ref-WARD" title="&quot;Custom fluorescent-nucleotide synthesis as an alternative method for nucleic acid labeling&quot;">WARD</a>]). Coupling of clock physical periodicity has also been
262 observed [<a href="#ref-JACOBSON" title="&quot;Congestion Avoidance and Control&quot;">JACOBSON</a>], as well as coupling of various types of
263 biological cycles.
264
265 Several canonical examples also exist in well known network systems.
266 Examples include the synchronization of various control loops, such
267 as routing update synchronization and TCP Slow Start synchronization
268 [<a href="#ref-FLOYD" title="&quot;The Synchronization of Periodic Routing Messages&quot;">FLOYD</a>,<a href="#ref-JACOBSON" title="&quot;Congestion Avoidance and Control&quot;">JACOBSON</a>]. An important result of these observations is that
269 coupling is intimately related to synchronization. Injecting
270 randomness into these systems is one way to reduce coupling.
271
272
273
274
275
276 <span class="grey">Bush, et. al. Informational [Page 5]</span></pre>
277 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-6" ></span>
278 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
279
280
281 Interestingly, in analyzing risk factors for the Public Switched
282 Telephone Network (PSTN), Charles Perrow decomposes the complexity
283 problem along two related axes, which he terms "interactions" and
284 "coupling" [<a href="#ref-PERROW" title="&quot;Normal Accidents: Living with High Risk Technologies&quot;">PERROW</a>]. Perrow cites interactions and coupling as
285 significant factors in determining the reliability of a complex
286 system (and in particular, the PSTN). In this model, interactions
287 refer to the dependencies between components (linear or non-linear),
288 while coupling refers to the flexibility in a system. Systems with
289 simple, linear interactions have components that affect only other
290 components that are functionally downstream. Complex system
291 components interact with many other components in different and
292 possibly distant parts of the system. Loosely coupled systems are
293 said to have more flexibility in time constraints, sequencing, and
294 environmental assumptions than do tightly coupled systems. In
295 addition, systems with complex interactions and tight coupling are
296 likely to have unforeseen failure states (of course, complex
297 interactions permit more complications to develop and make the system
298 hard to understand and predict); this behavior is also described in
299 [<a href="#ref-WILLINGER2002" title="&quot;Robustness and the Internet: Design and evolution&quot;">WILLINGER2002</a>]. Tight coupling also means that the system has less
300 flexibility in recovering from failure states.
301
302 The PSTN's SS7 control network provides an interesting example of
303 what can go wrong with a tightly coupled complex system. Outages
304 such as the well publicized 1991 outage of AT&amp;T's SS7 demonstrates
305 the phenomenon: the outage was caused by software bugs in the
306 switches' crash recovery code. In this case, one switch crashed due
307 to a hardware glitch. When this switch came back up, it (plus a
308 reasonably probable timing event) caused its neighbors to crash When
309 the neighboring switches came back up, they caused their neighbors to
310 crash, and so on [<a href="#ref-NEUMANN" title="&quot;Cause of AT&amp;T network failure&quot;">NEUMANN</a>] (the root cause turned out to be a
311 misplaced 'break' statement; this is an excellent example of cross-
312 layer coupling). This phenomenon is similar to the phase-locking of
313 weakly coupled oscillators, in which random variations in sequence
314 times plays an important role in system stability [<a href="#ref-THOMPSON" title="&quot;Nonlinear Dynamics and Chaos&quot;">THOMPSON</a>].
315
316 <span class="h3"><a class="selflink" id="section-2.3" href="#section-2.3">2.3</a>. Complexity lesson from voice</span>
317
318 In the 1970s and 1980s, the voice carriers competed by adding
319 features which drove substantial increases in the complexity of the
320 PSTN, especially in the Class 5 switching infrastructure. This
321 complexity was typically software-based, not hardware driven, and
322 therefore had cost curves worse than Moore's Law. In summary, poor
323 margins on voice products today are due to OPEX and CAPEX costs not
324 dropping as we might expect from simple hardware-bound
325 implementations.
326
327
328
329
330
331
332 <span class="grey">Bush, et. al. Informational [Page 6]</span></pre>
333 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-7" ></span>
334 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
335
336
337 <span class="h3"><a class="selflink" id="section-2.4" href="#section-2.4">2.4</a>. Upgrade cost of complexity</span>
338
339 Consider the cost of providing new features in a complex network.
340 The traditional voice network has little intelligence in its edge
341 devices (phone instruments), and a very smart core. The Internet has
342 smart edges, computers with operating systems, applications, etc.,
343 and a simple core, which consists of a control plane and packet
344 forwarding engines. Adding an new Internet service is just a matter
345 of distributing an application to the a few consenting desktops who
346 wish to use it. Compare this to adding a service to voice, where one
347 has to upgrade the entire core.
348
349 <span class="h2"><a class="selflink" id="section-3" href="#section-3">3</a>. Layering Considered Harmful</span>
350
351 There are several generic properties of layering, or vertical
352 integration as applied to networking. In general, a layer as defined
353 in our context implements one or more of
354
355 Error Control: The layer makes the "channel" more reliable
356 (e.g., reliable transport layer)
357
358 Flow Control: The layer avoids flooding slower peer (e.g.,
359 ATM flow control)
360
361 Fragmentation: Dividing large data chunks into smaller
362 pieces, and subsequent reassembly (e.g., TCP
363 MSS fragmentation/reassembly)
364
365 Multiplexing: Allow several higher level sessions share
366 single lower level "connection" (e.g., ATM PVC)
367
368 Connection Setup: Handshaking with peer (e.g., TCP three-way
369 handshake, ATM ILMI)
370
371 Addressing/Naming: Locating, managing identifiers associated
372 with entities (e.g., GOSSIP 2 NSAP Structure
373 [<a href="./rfc1629" title="&quot;Guidelines for OSI NSAP Allocation in the Internet&quot;">RFC1629</a>])
374
375 Layering of this type does have various conceptual and structuring
376 advantages. However, in the data networking context structured
377 layering implies that the functions of each layer are carried out
378 completely before the protocol data unit is passed to the next layer.
379 This means that the optimization of each layer has to be done
380 separately. Such ordering constraints are in conflict with efficient
381 implementation of data manipulation functions. One could accuse the
382 layered model (e.g., TCP/IP and ISO OSI) of causing this conflict.
383 In fact, the operations of multiplexing and segmentation both hide
384 vital information that lower layers may need to optimize their
385
386
387
388 <span class="grey">Bush, et. al. Informational [Page 7]</span></pre>
389 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-8" ></span>
390 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
391
392
393 performance. For example, layer N may duplicate lower level
394 functionality, e.g., error recovery hop-hop versus end-to-end error
395 recovery. In addition, different layers may need the same
396 information (e.g., time stamp): layer N may need layer N-2
397 information (e.g., lower layer packet sizes), and the like [<a href="#ref-WAKEMAN" title="&quot;Layering considered harmful&quot;">WAKEMAN</a>].
398 A related and even more ironic statement comes from Tennenhouse's
399 classic paper, "Layered Multiplexing Considered Harmful"
400 [<a href="#ref-TENNENHOUSE" title="&quot;Layered multiplexing considered harmful&quot;">TENNENHOUSE</a>]: "The ATM approach to broadband networking is presently
401 being pursued within the CCITT (and elsewhere) as the unifying
402 mechanism for the support of service integration, rate adaptation,
403 and jitter control within the lower layers of the network
404 architecture. This position paper is specifically concerned with the
405 jitter arising from the design of the "middle" and "upper" layers
406 that operate within the end systems and relays of multi-service
407 networks (MSNs)."
408
409 As a result of inter-layer dependencies, increased layering can
410 quickly lead to violation of the Simplicity Principle. Industry
411 experience has taught us that increased layering frequently increases
412 complexity and hence leads to increases in OPEX, as is predicted by
413 the Simplicity Principle. A corollary is stated in <a href="./rfc1925">RFC 1925</a>
414 <a href="./rfc1925#section-2">[RFC1925], section&nbsp;2</a>(5):
415
416 "It is always possible to agglutinate multiple separate problems
417 into a single complex interdependent solution. In most cases
418 this is a bad idea."
419
420 The first order conclusion then, is that horizontal (as opposed to
421 vertical) separation may be more cost-effective and reliable in the
422 long term.
423
424 <span class="h3"><a class="selflink" id="section-3.1" href="#section-3.1">3.1</a>. Optimization Considered Harmful</span>
425
426 A corollary of the layering arguments above is that optimization can
427 also be considered harmful. In particular, optimization introduces
428 complexity, and as well as introducing tighter coupling between
429 components and layers.
430
431 An important and related effect of optimization is described by the
432 Law of Diminishing Returns, which states that if one factor of
433 production is increased while the others remain constant, the overall
434 returns will relatively decrease after a certain point [<a href="#ref-SPILLMAN" title="W. J. Spillman and E. Lang">SPILLMAN</a>].
435 The implication here is that trying to squeeze out efficiency past
436 that point only adds complexity, and hence leads to less reliable
437 systems.
438
439
440
441
442
443
444 <span class="grey">Bush, et. al. Informational [Page 8]</span></pre>
445 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-9" ></span>
446 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
447
448
449 <span class="h3"><a class="selflink" id="section-3.2" href="#section-3.2">3.2</a>. Feature Richness Considered Harmful</span>
450
451 While adding any new feature may be considered a gain (and in fact
452 frequently differentiates vendors of various types of equipment), but
453 there is a danger. The danger is in increased system complexity.
454
455 <span class="h3"><a class="selflink" id="section-3.3" href="#section-3.3">3.3</a>. Evolution of Transport Efficiency for IP</span>
456
457 The evolution of transport infrastructures for IP offers a good
458 example of how decreasing vertical integration has lead to various
459 efficiencies. In particular,
460
461 | IP over ATM over SONET --&gt;
462 | IP over SONET over WDM --&gt;
463 | IP over WDM
464 |
465 \|/
466 Decreasing complexity, CAPEX, OPEX
467
468 The key point here is that layers are removed resulting in CAPEX and
469 OPEX efficiencies.
470
471 <span class="h3"><a class="selflink" id="section-3.4" href="#section-3.4">3.4</a>. Convergence Layering</span>
472
473 Convergence is related to the layering concepts described above in
474 that convergence is achieved via a "convergence layer". The end
475 state of the convergence argument is the concept of Everything Over
476 Some Layer (EOSL). Conduit, DWDM, fiber, ATM, MPLS, and even IP have
477 all been proposed as convergence layers. It is important to note
478 that since layering typically drives OPEX up, we expect convergence
479 will as well. This observation is again consistent with industry
480 experience.
481
482 There are many notable examples of convergence layer failure.
483 Perhaps the most germane example is IP over ATM. The immediate and
484 most obvious consequence of ATM layering is the so-called cell tax:
485 First, note that the complete answer on ATM efficiency is that it
486 depends upon packet size distributions. Let's assume that typical
487 Internet type traffic patterns, which tend to have high percentages
488 of packets at 40, 44, and 552 bytes. Recent data [<a href="#ref-CAIDA">CAIDA</a>] shows that
489 about 95% of WAN bytes and 85% of packets are TCP. Much of this
490 traffic is composed of 40/44 byte packets.
491
492 Now, consider the case of a a DS3 backbone with PLCP turned on. Then
493 the maximum cell rate is 96,000 cells/sec. If you multiply this
494 value by the number of bits in the payload, you get: 96000 cells/sec
495 * 48 bytes/cell * 8 = 36.864 Mbps. This, however, is unrealistic
496 since it
497
498
499
500 <span class="grey">Bush, et. al. Informational [Page 9]</span></pre>
501 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-10" ></span>
502 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
503
504
505 assumes perfect payload packing. There are two other things that
506 contribute to the ATM overhead (cell tax): The wasted padding and the
507 8 byte SNAP header.
508
509 It is the SNAP header which causes most of the problems (and you
510 can't do anything about this), forcing most small packets to consume
511 two cells, with the second cell to be mostly empty padding (this
512 interacts really poorly with the data quoted above, e.g., that most
513 packets are 40-44 byte TCP Ack packets). This causes a loss of about
514 another 16% from the 36.8 Mbps ideal throughput.
515
516 So the total throughput ends up being (for a DS3):
517
518 DS3 Line Rate: 44.736
519 PLCP Overhead - 4.032
520 Per Cell Header: - 3.840
521 SNAP Header &amp; Padding: - 5.900
522 =========
523 30.960 Mbps
524
525 Result: With a DS3 line rate of 44.736 Mbps, the total overhead is
526 about 31%.
527
528 Another way to look at this is that since a large fraction of WAN
529 traffic is comprised of TCP ACKs, one can make a different but
530 related calculation. IP over ATM requires:
531
532 IP data (40 bytes in this case)
533 8 bytes SNAP
534 8 bytes AAL5 stuff
535 5 bytes for each cell
536 + as much more as it takes to fill out the last cell
537
538 On ATM, this becomes two cells - 106 bytes to convey 40 bytes of
539 information. The next most common size seems to be one of several
540 sizes in the 504-556 byte range - 636 bytes to carry IP, TCP, and a
541 512 byte TCP payload - with messages larger than 1000 bytes running
542 third.
543
544 One would imagine that 87% payload (556 byte message size) is better
545 than 37% payload (TCP Ack size), but it's not the 95-98% that
546 customers are used to, and the predominance of TCP Acks skews the
547 average.
548
549
550
551
552
553
554
555
556 <span class="grey">Bush, et. al. Informational [Page 10]</span></pre>
557 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-11" ></span>
558 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
559
560
561 <span class="h4"><a class="selflink" id="section-3.4.1" href="#section-3.4.1">3.4.1</a>. Note on Transport Protocol Layering</span>
562
563 Protocol layering models are frequently cast as "X over Y" models.
564 In these cases, protocol Y carries protocol X's protocol data units
565 (and possibly control data) over Y's data plane, i.e., Y is a
566 "convergence layer". Examples include Frame Relay over ATM, IP over
567 ATM, and IP over MPLS. While X over Y layering has met with only
568 marginal success [<a href="#ref-TENNENHOUSE" title="&quot;Layered multiplexing considered harmful&quot;">TENNENHOUSE</a>,<a href="#ref-WAKEMAN" title="&quot;Layering considered harmful&quot;">WAKEMAN</a>], there have been a few notable
569 instances where efficiency can be and is gained. In particular, "X
570 over Y efficiencies" can be realized when there is a kind of
571 "isomorphism" between the X and Y (i.e., there is a small convergence
572 layer). In these cases X's data, and possibly control traffic, are
573 "encapsulated" and transported over Y. Examples include Frame Relay
574 over ATM, and Frame Relay, AAL5 ATM and Ethernet over L2TPv3
575 [<a href="#ref-L2TPV3" title="&quot;Layer Two Tunneling Protocol (Version 3) -- L2TPv3&quot;">L2TPV3</a>]; the simplifying factors here are that there is no
576 requirement that a shared clock be recovered by the communicating end
577 points, and that control-plane interworking is minimized. An
578 alternative is to interwork the X and Y's control and data planes;
579 control-plane interworking is discussed below.
580
581 <span class="h3"><a class="selflink" id="section-3.5" href="#section-3.5">3.5</a>. Second Order Effects</span>
582
583 IP over ATM provides an excellent example of unanticipated second
584 order effects. In particular, Romanov and Floyd's classic study on
585 TCP good-put [<a href="#ref-ROMANOV" title="&quot;Dynamics of TCP over ATM Networks&quot;">ROMANOV</a>] on ATM showed that large UBR buffers (larger
586 than one TCP window size) are required to achieve reasonable
587 performance, that packet discard mechanisms (such as Early Packet
588 Discard, or EPD) improve the effective usage of the bandwidth and
589 that more elaborate service and drop strategies than FIFO+EPD, such
590 as per VC queuing and accounting, might be required at the bottleneck
591 to ensure both high efficiency and fairness. Though all studies
592 clearly indicate that a buffer size not less than one TCP window size
593 is required, the amount of extra buffer required naturally depends on
594 the packet discard mechanism used and is still an open issue.
595
596 Examples of this kind of problem with layering abound in practical
597 networking. Consider, for example, the effect of IP transport's
598 implicit assumptions of lower layers. In particular:
599
600 o Packet loss: TCP assumes that packet losses are indications of
601 congestion, but sometimes losses are from corruption on a wireless
602 link [<a href="./rfc3115">RFC3115</a>].
603
604 o Reordered packets: TCP assumes that significantly reordered
605 packets are indications of congestion. This is not always the
606 case [<a href="#ref-FLOYD2001" title="IEEE Communications Magazine">FLOYD2001</a>].
607
608
609
610
611
612 <span class="grey">Bush, et. al. Informational [Page 11]</span></pre>
613 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-12" ></span>
614 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
615
616
617 o Round-trip times: TCP measures round-trip times, and assumes that
618 the lack of an acknowledgment within a period of time based on the
619 measured round-trip time is a packet loss, and therefore an
620 indication of congestion [<a href="#ref-KARN" title="&quot;TCP vs Link Layer Retransmission&quot;">KARN</a>].
621
622 o Congestion control: TCP congestion control implicitly assumes that
623 all the packets in a flow are treated the same by the network, but
624 this is not always the case [<a href="#ref-HANDLEY" title="&quot;On Inter-layer Assumptions (A view from the Transport Area), slides from a presentation at the IAB workshop on Wireless Internetworking&quot;">HANDLEY</a>].
625
626 <span class="h3"><a class="selflink" id="section-3.6" href="#section-3.6">3.6</a>. Instantiating the EOSL Model with IP</span>
627
628 While IP is being proposed as a transport for almost everything, the
629 base assumption, that Everything over IP (EOIP) will result in OPEX
630 and CAPEX efficiencies, requires critical examination. In
631 particular, while it is the case that many protocols can be
632 efficiently transported over an IP network (specifically, those
633 protocols that do not need to recover synchronization between the
634 communication end points, such as Frame Relay, Ethernet, and AAL5
635 ATM), the Simplicity and Layering Principles suggest that EOIP may
636 not represent the most efficient convergence strategy for arbitrary
637 services. Rather, a more CAPEX and OPEX efficient convergence layer
638 might be much lower (again, this behavior is predicted by the
639 Simplicity Principle).
640
641 An example of where EOIP would not be the most OPEX and CAPEX
642 efficient transport would be in those cases where a service or
643 protocol needed SONET-like restoration times (e.g., 50ms). It is not
644 hard to imagine that it would cost more to build and operate an IP
645 network with this kind of restoration and convergence property (if
646 that were even possible) than it would to build the SONET network in
647 the first place.
648
649 <span class="h2"><a class="selflink" id="section-4" href="#section-4">4</a>. Avoid the Universal Interworking Function</span>
650
651 While there have been many implementations of Universal Interworking
652 unction (UIWF), IWF approaches have been problematic at large scale.
653 his concern is codified in the Principle of Minimum Intervention
654 BRYANT]:
655
656 "To minimise the scope of information, and to improve the efficiency
657 of data flow through the Encapsulation Layer, the payload should,
658 where possible, be transported as received without modification."
659
660
661
662
663
664
665
666
667
668 <span class="grey">Bush, et. al. Informational [Page 12]</span></pre>
669 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-13" ></span>
670 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
671
672
673 <span class="h3"><a class="selflink" id="section-4.1" href="#section-4.1">4.1</a>. Avoid Control Plane Interworking</span>
674
675 This corollary is best understood in the context of the integrated
676 solutions space. In this case, the architecture and design
677 frequently achieves the worst of all possible worlds. This is due to
678 the fact that such integrated solutions perform poorly at both ends
679 of the performance/CAPEX/OPEX spectrum: the protocols with the least
680 switching demand may have to bear the cost of the most expensive,
681 while the protocols with the most stringent requirements often must
682 make concessions to those with different requirements. Add to this
683 the various control plane interworking issues and you have a large
684 opportunity for failure. In summary, interworking functions should
685 be restricted to data plane interworking and encapsulations, and
686 these functions should be carried out at the edge of the network.
687
688 As described above, interworking models have been successful in those
689 cases where there is a kind of "isomorphism" between the layers being
690 interworked. The trade-off here, frequently described as the
691 "Integrated vs. Ships In the Night trade-off" has been examined at
692 various times and at various protocol layers. In general, there are
693 few cases in which such integrated solutions have proven efficient.
694 Multi-protocol BGP [<a href="./rfc2283" title="&quot;Multiprotocol Extensions for BGP4&quot;">RFC2283</a>] is a subtly different but notable
695 exception. In this case, the control plane is independent of the
696 format of the control data. That is, no control plane data
697 conversion is required, in contrast with control plane interworking
698 models such as the ATM/IP interworking envisioned by some soft-switch
699 manufacturers, and the so-called "PNNI-MPLS SIN" interworking
700 [<a href="#ref-ATMMPLS" title="&quot;ATM-MPLS Interworking Migration Complexities Issues and Preliminary Assessment&quot;">ATMMPLS</a>].
701
702 <span class="h2"><a class="selflink" id="section-5" href="#section-5">5</a>. Packet versus Circuit Switching: Fundamental Differences</span>
703
704 Conventional wisdom holds that packet switching (PS) is inherently
705 more efficient than circuit switching (CS), primarily because of the
706 efficiencies that can be gained by statistical multiplexing and the
707 fact that routing and forwarding decisions are made independently in
708 a hop-by-hop fashion [[<a href="#ref-MOLINERO2002" title="&quot;TCP Switching: Exposing Circuits to IP&quot;">MOLINERO2002</a>]. Further, it is widely assumed
709 that IP is simpler that circuit switching, and hence should be more
710 economical to deploy and manage [<a href="#ref-MCK2002" title="April">MCK2002</a>]. However, if one examines
711 these and related assumptions, a different picture emerges (see for
712 example [<a href="#ref-ODLYZKO98" title="&quot;The economics of the Internet: Utility, utilization, pricing, and Quality of Service&quot;">ODLYZKO98</a>]). The following sections discuss these
713 assumptions.
714
715 <span class="h3"><a class="selflink" id="section-5.1" href="#section-5.1">5.1</a>. Is PS is inherently more efficient than CS?</span>
716
717 It is well known that packet switches make efficient use of scarce
718 bandwidth [<a href="#ref-BARAN" title="&quot;On Distributed Communications&quot;">BARAN</a>]. This efficiency is based on the statistical
719 multiplexing inherent in packet switching. However, we continue to
720 be puzzled by what is generally believed to be the low utilization of
721
722
723
724 <span class="grey">Bush, et. al. Informational [Page 13]</span></pre>
725 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-14" ></span>
726 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
727
728
729 Internet backbones. The first question we might ask is what is the
730 current average utilization of Internet backbones, and how does that
731 relate to the utilization of long distance voice networks? Odlyzko
732 and Coffman [<a href="#ref-ODLYZKO" title="&quot;Data networks are mostly empty for good reason&quot;">ODLYZKO</a>,<a href="#ref-COFFMAN" title="&quot;Internet Growth: Is there a 'Moores Law' for Data Traffic&quot;">COFFMAN</a>] report that the average utilization of
733 links in the IP networks was in the range between 3% and 20%
734 (corporate intranets run in the 3% range, while commercial Internet
735 backbones run in the 15-20% range). On the other hand, the average
736 utilization of long haul voice lines is about 33%. In addition, for
737 2002, the average utilization of optical networks (all services)
738 appears to be hovering at about 11%, while the historical average is
739 approximately 15% [<a href="#ref-ML2002" title="&quot;Optical Systems&quot;">ML2002</a>]. The question then becomes why we see
740 such utilization levels, especially in light of the assumption that
741 PS is inherently more efficient than CS. The reasons cited by
742 Odlyzko and Coffman include:
743
744 (i). Internet traffic is extremely asymmetric and bursty, but
745 links are symmetric and of fixed capacity (i.e., don't know
746 the traffic matrix, or required link capacities);
747
748 (ii). It is difficult to predict traffic growth on a link, so
749 operators tend to add bandwidth aggressively;
750
751 (iii). Falling prices for coarser bandwidth granularity make it
752 appear more economical to add capacity in large increments.
753
754 Other static factors include protocol overhead, other kinds of
755 equipment granularity, restoration capacity, and provisioning lag
756 time all contribute to the need to "over-provision" [<a href="#ref-MC2001" title="&quot;U.S Communications Infrastructure at A Crossroads: Opportunities Amid the Gloom&quot;">MC2001</a>].
757
758 <span class="h3"><a class="selflink" id="section-5.2" href="#section-5.2">5.2</a>. Is PS simpler than CS?</span>
759
760 The end-to-end principle can be interpreted as stating that the
761 complexity of the Internet belongs at the edges. However, today's
762 Internet backbone routers are extremely complex. Further, this
763 complexity scales with line rate. Since the relative complexity of
764 circuit and packet switching seems to have resisted direct analysis,
765 we instead examine several artifacts of packet and circuit switching
766 as complexity metrics. Among the metrics we might look at are
767 software complexity, macro operation complexity, hardware complexity,
768 power consumption, and density. Each of these metrics is considered
769 below.
770
771
772
773
774
775
776
777
778
779
780 <span class="grey">Bush, et. al. Informational [Page 14]</span></pre>
781 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-15" ></span>
782 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
783
784
785 <span class="h4"><a class="selflink" id="section-5.2.1" href="#section-5.2.1">5.2.1</a>. Software/Firmware Complexity</span>
786
787 One measure of software/firmware complexity is the number of
788 instructions required to program the device. The typical software
789 image for an Internet router requires between eight and ten million
790 instructions (including firmware), whereas a typical transport switch
791 requires on average about three million instructions [<a href="#ref-MCK2002" title="April">MCK2002</a>].
792
793 This difference in software complexity has tended to make Internet
794 routers unreliable, and has notable other second order effects (e.g.,
795 it may take a long time to reboot such a router). As another point
796 of comparison, consider that the AT&amp;T (Lucent) 5ESS class 5 switch,
797 which has a huge number of calling features, requires only about
798 twice the number of lines of code as an Internet core router [<a href="#ref-EICK" title="&quot;Visualizing Software Changes&quot;">EICK</a>].
799
800 Finally, since routers are as much or more software than hardware
801 devices, another result of the code complexity is that the cost of
802 routers benefits less from Moore's Law than less software-intensive
803 devices. This causes a bandwidth/device trade-off that favors
804 bandwidth more than less software-intensive devices.
805
806 <span class="h4"><a class="selflink" id="section-5.2.2" href="#section-5.2.2">5.2.2</a>. Macro Operation Complexity</span>
807
808 An Internet router's line card must perform many complex operations,
809 including processing the packet header, longest prefix match,
810 generating ICMP error messages, processing IP header options, and
811 buffering the packet so that TCP congestion control will be effective
812 (this typically requires a buffer of size proportional to the line
813 rate times the RTT, so a buffer will hold around 250 ms of packet
814 data). This doesn't include route and packet filtering, or any QoS
815 or VPN filtering.
816
817 On the other hand, a transport switch need only to map ingress time-
818 slots to egress time-slots and interfaces, and therefore can be
819 considerably less complex.
820
821 <span class="h4"><a class="selflink" id="section-5.2.3" href="#section-5.2.3">5.2.3</a>. Hardware Complexity</span>
822
823 One measure of hardware complexity is the number of logic gates on a
824 line card [<a href="#ref-MOLINERO2002" title="&quot;TCP Switching: Exposing Circuits to IP&quot;">MOLINERO2002</a>]. Consider the case of a high-speed Internet
825 router line card: An OC192 POS router line card contains at least 30
826 million gates in ASICs, at least one CPU, 300 Mbytes of packet
827 buffers, 2 Mbytes of forwarding table, and 10 Mbytes of other
828
829
830
831
832
833
834
835
836 <span class="grey">Bush, et. al. Informational [Page 15]</span></pre>
837 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-16" ></span>
838 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
839
840
841 state memory. On the other hand, a comparable transport switch line
842 card has 7.5 million logic gates, no CPU, no packet buffer, no
843 forwarding table, and an on-chip state memory. Rather, the line-card
844 of an electronic transport switch typically contains a SONET framer,
845 a chip to map ingress time-slots to egress time-slots, and an
846 interface to the switch fabric.
847
848 <span class="h4"><a class="selflink" id="section-5.2.4" href="#section-5.2.4">5.2.4</a>. Power</span>
849
850 Since transport switches have traditionally been built from simpler
851 hardware components, they also consume less power [<a href="#ref-PMC" title="&quot;The Design of a 10 Gigabit Core Router Architecture&quot;">PMC</a>].
852
853 <span class="h4"><a class="selflink" id="section-5.2.5" href="#section-5.2.5">5.2.5</a>. Density</span>
854
855 The highest capacity transport switches have about four times the
856 capacity of an IP router [<a href="#ref-CISCO">CISCO</a>,<a href="#ref-CIENA" title="&quot;CIENA Multiwave CoreDiretor&quot;">CIENA</a>], and sell for about one-third
857 as much per Gigabit/sec. Optical (OOO) technology pushes this
858 complexity difference further (e.g., tunable lasers, MEMs switches.
859 e.g., [CALIENT]), and DWDM multiplexers provide technology to build
860 extremely high capacity, low power transport switches.
861
862 A related metric is physical footprint. In general, by virtue of
863 their higher density, transport switches have a smaller "per-gigabit"
864 physical footprint.
865
866 <span class="h4"><a class="selflink" id="section-5.2.6" href="#section-5.2.6">5.2.6</a>. Fixed versus variable costs</span>
867
868 Packet switching would seem to have high variable cost, meaning that
869 it costs more to send the n-th piece of information using packet
870 switching than it might in a circuit switched network. Much of this
871 advantage is due to the relatively static nature of circuit
872 switching, e.g., circuit switching can take advantage of of pre-
873 scheduled arrival of information to eliminate operations to be
874 performed on incoming information. For example, in the circuit
875 switched case, there is no need to buffer incoming information,
876 perform loop detection, resolve next hops, modify fields in the
877 packet header, and the like. Finally, many circuit switched networks
878 combine relatively static configuration with out-of-band control
879 planes (e.g., SS7), which greatly simplifies data-plane switching.
880 The bottom line is that as data rates get large, it becomes more and
881 more complex to switch packets, while circuit switching scales more
882 or less linearly.
883
884
885
886
887
888
889
890
891
892 <span class="grey">Bush, et. al. Informational [Page 16]</span></pre>
893 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-17" ></span>
894 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
895
896
897 <span class="h4"><a class="selflink" id="section-5.2.7" href="#section-5.2.7">5.2.7</a>. QoS</span>
898
899 While the components of a complete solution for Internet QoS,
900 including call admission control, efficient packet classification,
901 and scheduling algorithms, have been the subject of extensive
902 research and standardization for more than 10 years, end-to-end
903 signaled QoS for the Internet has not become a reality.
904 Alternatively, QoS has been part of the circuit switched
905 infrastructure almost from its inception. On the other hand, QoS is
906 usually deployed to determine queuing disciplines to be used when
907 there is insufficient bandwidth to support traffic. But unlike voice
908 traffic, packet drop or severe delay may have a much more serious
909 effect on TCP traffic due to its congestion-aware feedback loop (in
910 particular, TCP backoff/slow start).
911
912 <span class="h4"><a class="selflink" id="section-5.2.8" href="#section-5.2.8">5.2.8</a>. Flexibility</span>
913
914 A somewhat harder to quantify metric is the inherent flexibility of
915 the Internet. While the Internet's flexibility has led to its rapid
916 growth, this flexibility comes with a relatively high cost at the
917 edge: the need for highly trained support personnel. A standard rule
918 of thumb is that in an enterprise setting, a single support person
919 suffices to provide telephone service for a group, while you need ten
920 computer networking experts to serve the networking requirements of
921 the same group [<a href="#ref-ODLYZKO98A" title="&quot;Smart and stupid networks: Why the Internet is like Microsoft&quot;">ODLYZKO98A</a>]. This phenomenon is also described in
922 [<a href="#ref-PERROW" title="&quot;Normal Accidents: Living with High Risk Technologies&quot;">PERROW</a>].
923
924 <span class="h3"><a class="selflink" id="section-5.3" href="#section-5.3">5.3</a>. Relative Complexity</span>
925
926 The relative computational complexity of circuit switching as
927 compared to packet switching has been difficult to describe in formal
928 terms [<a href="#ref-PARK" title="&quot;The Internet as a Complex System: Scaling, Complexity and Control&quot;">PARK</a>]. As such, the sections above seek to describe the
929 complexity in terms of observable artifacts. With this in mind, it
930 is clear that the fundamental driver producing the increased
931 complexities outlined above is the hop-by-hop independence (HBHI)
932 inherent in the IP architecture. This is in contrast to the end to
933 end architectures such as ATM or Frame Relay.
934
935 [<a id="ref-WILLINGER2002">WILLINGER2002</a>] describes this phenomenon in terms of the robustness
936 requirement of the original Internet design, and how this requirement
937 has the driven complexity of the network. In particular, they
938 describe a "complexity/robustness" spiral, in which increases in
939 complexity create further and more serious sensitivities, which then
940 requires additional robustness (hence the spiral).
941
942
943
944
945
946
947
948 <span class="grey">Bush, et. al. Informational [Page 17]</span></pre>
949 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-18" ></span>
950 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
951
952
953 The important lesson of this section is that the Simplicity
954 Principle, while applicable to circuit switching as well as packet
955 switching, is crucial in controlling the complexity (and hence OPEX
956 and CAPEX properties) of packet networks. This idea is reinforced by
957 the observation that while packet switching is a younger, less mature
958 discipline than circuit switching, the trend in packet switches is
959 toward more complex line cards, while the complexity of circuit
960 switches appears to be scaling linearly with line rates and aggregate
961 capacity.
962
963 <span class="h4"><a class="selflink" id="section-5.3.1" href="#section-5.3.1">5.3.1</a>. HBHI and the OPEX Challenge</span>
964
965 As a result of HBHI, we need to approach IP networks in a
966 fundamentally different way than we do circuit based networks. In
967 particular, the major OPEX challenge faced by the IP network is that
968 debugging of a large-scale IP network still requires a large degree
969 of expertise and understanding, again due to the hop-by-hop
970 independence inherent in a packet architecture (again, note that this
971 hop-by-hop independence is not present in virtual circuit networks
972 such as ATM or Frame Relay). For example, you may have to visit a
973 large set of your routers only to discover that the problem is
974 external to your own network. Further, the debugging tools used to
975 diagnose problems are also complex and somewhat primitive. Finally,
976 IP has to deal with people having problems with their DNS or their
977 mail or news or some new application, whereas this is usually not the
978 case for TDM/ATM/etc. In the case of IP, this can be eased by
979 improving automation (note that much of what we mention is customer
980 facing). In general, there are many variables external to the
981 network that effect OPEX.
982
983 Finally, it is important to note that the quantitative relationship
984 between CAPEX, OPEX, and a network's inherent complexity is not well
985 understood. In fact, there are no agreed upon and quantitative
986 metrics for describing a network's complexity, so a precise
987 relationship between CAPEX, OPEX, and complexity remains elusive.
988
989 <span class="h2"><a class="selflink" id="section-6" href="#section-6">6</a>. The Myth of Over-Provisioning</span>
990
991 As noted in [<a href="#ref-MC2001" title="&quot;U.S Communications Infrastructure at A Crossroads: Opportunities Amid the Gloom&quot;">MC2001</a>] and elsewhere, much of the complexity we observe
992 in today's Internet is directed at increasing bandwidth utilization.
993 As a result, the desire of network engineers to keep network
994 utilization below 50% has been termed "over-provisioning". However,
995 this use of the term over-provisioning is a misnomer. Rather, in
996 modern Internet backbones the unused capacity is actually protection
997 capacity. In particular, one might view this as "1:1 protection at
998 the IP layer". Viewed in this way, we see that an IP network
999 provisioned to run at 50% utilization is no more over-provisioned
1000 than the typical SONET network. However, the important advantages
1001
1002
1003
1004 <span class="grey">Bush, et. al. Informational [Page 18]</span></pre>
1005 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-19" ></span>
1006 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
1007
1008
1009 that accrue to an IP network provisioned in this way include close to
1010 speed of light delay and close to zero packet loss [<a href="#ref-FRALEIGH" title="&quot;Provisioning IP Backbone Networks to Support Delay- Based Service Level Agreements&quot;">FRALEIGH</a>]. These
1011 benefits can been seen as a "side-effect" of 1:1 protection
1012 provisioning.
1013
1014 There are also other, system-theoretic reasons for providing 1:1-like
1015 protection provisioning. Most notable among these reasons is that
1016 packet-switched networks with in-band control loops can become
1017 unstable and can experience oscillations and synchronization when
1018 congested. Complex and non-linear dynamic interaction of traffic
1019 means that congestion in one part of the network will spread to other
1020 parts of the network. When routing protocol packets are lost due to
1021 congestion or route-processor overload, it causes inconsistent
1022 routing state, and this may result in traffic loops, black holes, and
1023 lost connectivity. Thus, while statistical multiplexing can in
1024 theory yield higher network utilization, in practice, to maintain
1025 consistent performance and a reasonably stable network, the dynamics
1026 of the Internet backbones favor 1:1 provisioning and its side effects
1027 to keep the network stable and delay low.
1028
1029 <span class="h2"><a class="selflink" id="section-7" href="#section-7">7</a>. The Myth of Five Nines</span>
1030
1031 Paul Baran, in his classic paper, "SOME PERSPECTIVES ON NETWORKS--
1032 PAST, PRESENT AND FUTURE", stated that "The tradeoff curves between
1033 cost and system reliability suggest that the most reliable systems
1034 might be built of relatively unreliable and hence low cost elements,
1035 if it is system reliability at the lowest overall system cost that is
1036 at issue" [<a href="#ref-BARAN77" title="&quot;SOME PERSPECTIVES ON NETWORKS--PAST, PRESENT AND FUTURE&quot;">BARAN77</a>].
1037
1038 Today we refer to this phenomenon as "the myth of five nines".
1039 Specifically, so-called five nines reliability in packet network
1040 elements is consider a myth for the following reasons: First, since
1041 80% of unscheduled outages are caused by people or process errors
1042 [<a href="#ref-SCOTT" title="&quot;Making Smart Investments to Reduce Unplanned Downtime&quot;">SCOTT</a>], there is only a 20% window in which to optimize. Thus, in
1043 order to increase component reliability, we add complexity
1044 (optimization frequently leads to complexity), which is the root
1045 cause of 80% of the unplanned outages. This effectively narrows the
1046 20% window (i.e., you increase the likelihood of people and process
1047 failure). This phenomenon is also characterized as a
1048 "complexity/robustness" spiral [<a href="#ref-WILLINGER2002" title="&quot;Robustness and the Internet: Design and evolution&quot;">WILLINGER2002</a>], in which increases in
1049 complexity create further and more serious sensitivities, which then
1050 requires additional robustness, and so on (hence the spiral).
1051
1052 The conclusion, then is that while a system like the Internet can
1053 reach five-nines-like reliability, it is undesirable (and likely
1054 impossible) to try to make any individual component, especially the
1055 most complex ones, reach that reliability standard.
1056
1057
1058
1059
1060 <span class="grey">Bush, et. al. Informational [Page 19]</span></pre>
1061 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-20" ></span>
1062 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
1063
1064
1065 <span class="h2"><a class="selflink" id="section-8" href="#section-8">8</a>. Architectural Component Proportionality Law</span>
1066
1067 As noted in the previous section, the computational complexity of
1068 packet switched networks such as the Internet has proven difficult to
1069 describe in formal terms. However, an intuitive, high level
1070 definition of architectural complexity might be that the complexity
1071 of an architecture is proportional to its number of components, and
1072 that the probability of achieving a stable implementation of an
1073 architecture is inversely proportional to its number of components.
1074 As described above, components include discrete elements such as
1075 hardware elements, space and power requirements, as well as software,
1076 firmware, and the protocols they implement.
1077
1078 Stated more abstractly:
1079
1080 Let
1081
1082 A be a representation of architecture A,
1083
1084 |A| be number of distinct components in the service
1085 delivery path of architecture A,
1086
1087 w be a monotonically increasing function,
1088
1089 P be the probability of a stable implementation of an
1090 architecture, and let
1091
1092 Then
1093
1094 Complexity(A) = O(w(|A|))
1095 P(A) = O(1/w(|A|))
1096
1097 where
1098
1099 O(f) = {g:N-&gt;R | there exists c &gt; 0 and n such that g(n)
1100 &lt; c*f(n)}
1101
1102 [That is, O(f) comprises the set of functions g for which
1103 there exists a constant c and a number n, such that g(n) is
1104 smaller or equal to c*f(n) for all n. That is, O(f) is the
1105 set of all functions that do not grow faster than f,
1106 disregarding constant factors]
1107
1108 Interestingly, the Highly Optimized Tolerance (HOT) model [<a href="#ref-HOT" title="Phys. Rev. E 60">HOT</a>]
1109 attempts to characterize complexity in general terms (HOT is one
1110 recent attempt to develop a general framework for the study of
1111 complexity, and is a member of a family of abstractions generally
1112 termed "the new science of complexity" or "complex adaptive
1113
1114
1115
1116 <span class="grey">Bush, et. al. Informational [Page 20]</span></pre>
1117 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-21" ></span>
1118 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
1119
1120
1121 systems"). Tolerance, in HOT semantics, means that "robustness in
1122 complex systems is a constrained and limited quantity that must be
1123 carefully managed and protected." One focus of the HOT model is to
1124 characterize heavy-tailed distributions such as Complexity(A) in the
1125 above example (other examples include forest fires, power outages,
1126 and Internet traffic distributions). In particular, Complexity(A)
1127 attempts to map the extreme heterogeneity of the parts of the system
1128 (Internet), and the effect of their organization into highly
1129 structured networks, with hierarchies and multiple scales.
1130
1131 <span class="h3"><a class="selflink" id="section-8.1" href="#section-8.1">8.1</a>. Service Delivery Paths</span>
1132
1133 The Architectural Component Proportionality Law (ACPL) states that
1134 the complexity of an architecture is proportional to its number of
1135 components.
1136
1137 COROLLARY: Minimize the number of components in a service delivery
1138 path, where the service delivery path can be a protocol path, a
1139 software path, or a physical path.
1140
1141 This corollary is an important consequence of the ACPL, as the path
1142 between a customer and the desired service is particularly sensitive
1143 to the number and complexity of elements in the path. This is due to
1144 the fact that the complexity "smoothing" that we find at high levels
1145 of aggregation [<a href="#ref-ZHANG" title="&quot;Impact of Aggregation on Scaling Behavior of Internet Backbone Traffic&quot;">ZHANG</a>] is missing as you move closer to the edge, as
1146 well as having complex interactions with backoffice and CRM systems.
1147 Examples of architectures that haven't found a market due to this
1148 effect include TINA-based CRM systems, CORBA/TINA based service
1149 architectures. The basic lesson here was that the only possibilities
1150 for deploying these systems were "Limited scale deployments (such) as
1151 in Starvision can avoid coping with major unproven scalability
1152 issues", or "Otherwise need massive investments (like the carrier-
1153 grade ORB built almost from scratch)" [<a href="#ref-TINA" title="&quot;What is TINA and is it useful for the TelCos?&quot;">TINA</a>]. In other words, these
1154 systems had complex service delivery paths, and were too complex to
1155 be feasibly deployed.
1156
1157 <span class="h2"><a class="selflink" id="section-9" href="#section-9">9</a>. Conclusions</span>
1158
1159 This document attempts to codify long-understood Internet
1160 architectural principles. In particular, the unifying principle
1161 described here is best expressed by the Simplicity Principle, which
1162 states complexity must be controlled if one hopes to efficiently
1163 scale a complex object. The idea that simplicity itself can lead to
1164 some form of optimality has been a common theme throughout history,
1165 and has been stated in many other ways and along many dimensions.
1166 For example, consider the maxim known as Occam's Razor, which was
1167 formulated by the medieval English philosopher and Franciscan monk
1168 William of Ockham (ca. 1285-1349), and states "Pluralitas non est
1169
1170
1171
1172 <span class="grey">Bush, et. al. Informational [Page 21]</span></pre>
1173 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-22" ></span>
1174 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
1175
1176
1177 ponenda sine neccesitate" or "plurality should not be posited without
1178 necessity." (hence Occam's Razor is sometimes called "the principle
1179 of unnecessary plurality" and " the principle of simplicity"). A
1180 perhaps more contemporary formulation of Occam's Razor states that
1181 the simplest explanation for a phenomenon is the one preferred by
1182 nature. Other formulations of the same idea can be found in the
1183 KISS (Keep It Simple Stupid) principle and the Principle of Least
1184 Astonishment (the assertion that the most usable system is the one
1185 that least often leaves users astonished). [<a href="#ref-WILLINGER2002" title="&quot;Robustness and the Internet: Design and evolution&quot;">WILLINGER2002</a>] provides
1186 a more theoretical discussion of "robustness through simplicity", and
1187 in discussing the PSTN, [<a href="#ref-KUHN87" title="&quot;Sources of Failure in the Public Switched Telephone Network&quot;">KUHN87</a>] states that in most systems, "a
1188 trade-off can be made between simplicity of interactions and
1189 looseness of coupling".
1190
1191 When applied to packet switched network architectures, the Simplicity
1192 Principle has implications that some may consider heresy, e.g., that
1193 highly converged approaches are likely to be less efficient than
1194 "less converged" solutions. Otherwise stated, the "optimal"
1195 convergence layer may be much lower in the protocol stack that is
1196 conventionally believed. In addition, the analysis above leads to
1197 several conclusions that are contrary to the conventional wisdom
1198 surrounding packet networking. Perhaps most significant is the
1199 belief that packet switching is simpler than circuit switching. This
1200 belief has lead to conclusions such as "since packet is simpler than
1201 circuit, it must cost less to operate". This study finds to the
1202 contrary. In particular, by examining the metrics described above,
1203 we find that packet switching is more complex than circuit switching.
1204 Interestingly, this conclusion is borne out by the fact that
1205 normalized OPEX for data networks is typically significantly greater
1206 than for voice networks [<a href="#ref-ML2002" title="&quot;Optical Systems&quot;">ML2002</a>].
1207
1208 Finally, the important conclusion of this work is that for packet
1209 networks that are of the scale of today's Internet or larger, we must
1210 strive for the simplest possible solutions if we hope to build cost
1211 effective infrastructures. This idea is eloquently stated in
1212 [<a href="#ref-DOYLE2002" title="&quot;Robustness and the Internet: Theoretical Foundations&quot;">DOYLE2002</a>]: "The evolution of protocols can lead to a
1213 robustness/complexity/fragility spiral where complexity added for
1214 robustness also adds new fragilities, which in turn leads to new and
1215 thus spiraling complexities". This is exactly the phenomenon that
1216 the Simplicity Principle is designed to avoid.
1217
1218 <span class="h2"><a class="selflink" id="section-10" href="#section-10">10</a>. Security Considerations</span>
1219
1220 This document does not directly effect the security of any existing
1221 Internet protocol. However, adherence to the Simplicity Principle
1222 does have a direct affect on our ability to implement secure systems.
1223 In particular, a system's complexity grows, it becomes more
1224 difficult to model and analyze, and hence it becomes more difficult
1225
1226
1227
1228 <span class="grey">Bush, et. al. Informational [Page 22]</span></pre>
1229 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-23" ></span>
1230 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
1231
1232
1233 to find and understand the security implications inherent in its
1234 architecture, design, and implementation.
1235
1236 <span class="h2"><a class="selflink" id="section-11" href="#section-11">11</a>. Acknowledgments</span>
1237
1238 Many of the ideas for comparing the complexity of circuit switched
1239 and packet switched networks were inspired by conversations with Nick
1240 McKeown. Scott Bradner, David Banister, Steve Bellovin, Steward
1241 Bryant, Christophe Diot, Susan Harris, Ananth Nagarajan, Andrew
1242 Odlyzko, Pete and Natalie Whiting, and Lixia Zhang made many helpful
1243 comments on early drafts of this document.
1244
1245 <span class="h2"><a class="selflink" id="section-12" href="#section-12">12</a>. References</span>
1246
1247 [<a id="ref-AHUJA">AHUJA</a>] "The Impact of Internet Policy and Topology on
1248 Delayed Routing Convergence", Labovitz, et. al.
1249 Infocom, 2001.
1250
1251 [<a id="ref-ATMMPLS">ATMMPLS</a>] "ATM-MPLS Interworking Migration Complexities Issues
1252 and Preliminary Assessment", School of
1253 Interdisciplinary Computing and Engineering,
1254 University of Missouri-Kansas City, April 2002
1255
1256 [<a id="ref-BARAN">BARAN</a>] "On Distributed Communications", Paul Baran, Rand
1257 Corporation Memorandum RM-3420-PR,
1258 <a href="http://www.rand.org/publications/RM/RM3420">http://www.rand.org/publications/RM/RM3420</a>", August,
1259 1964.
1260
1261 [<a id="ref-BARAN77">BARAN77</a>] "SOME PERSPECTIVES ON NETWORKS--PAST, PRESENT AND
1262 FUTURE", Paul Baran, Information Processing 77,
1263 North-Holland Publishing Company, 1977,
1264
1265 [<a id="ref-BRYANT">BRYANT</a>] "Protocol Layering in PWE3", Bryant et al, Work in
1266 Progress.
1267
1268 [<a id="ref-CAIDA">CAIDA</a>] <a href="http://www.caida.org">http://www.caida.org</a>
1269
1270 [<a id="ref-CALLIENT">CALLIENT</a>] <a href="http://www.calient.net/home.html">http://www.calient.net/home.html</a>
1271
1272 [<a id="ref-CARLSON">CARLSON</a>] "Complexity and Robustness", J.M. Carlson and John
1273 Doyle, Proc. Natl. Acad. Sci. USA, Vol. 99, Suppl. 1,
1274 2538-2545, February 19, 2002.
1275 <a href="http://www.pnas.org/cgi/doi/10.1073/pnas.012582499">http://www.pnas.org/cgi/doi/10.1073/pnas.012582499</a>
1276
1277 [<a id="ref-CIENA">CIENA</a>] "CIENA Multiwave CoreDiretor",
1278 <a href="http://www.ciena.com/downloads/products/coredirector.pdf">http://www.ciena.com/downloads/products/</a>
1279 <a href="http://www.ciena.com/downloads/products/coredirector.pdf">coredirector.pdf</a>
1280
1281
1282
1283
1284 <span class="grey">Bush, et. al. Informational [Page 23]</span></pre>
1285 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-24" ></span>
1286 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
1287
1288
1289 [<a id="ref-CISCO">CISCO</a>] <a href="http://www.cisco.com">http://www.cisco.com</a>
1290
1291 [<a id="ref-CLARK">CLARK</a>] "The Design Philosophy of the DARPA Internet
1292 Protocols", D. Clark, Proc. of the ACM SIGCOMM, 1988.
1293
1294 [<a id="ref-COFFMAN">COFFMAN</a>] "Internet Growth: Is there a 'Moores Law' for Data
1295 Traffic", K.G. Coffman and A.M. Odlyzko, pp. 47-93,
1296 Handbook of Massive Data Stes, J. Elli, P. M.
1297 Pardalos, and M. G. C. Resende, Editors. Kluwer,
1298 2002.
1299
1300 [<a id="ref-DOYLE2002">DOYLE2002</a>] "Robustness and the Internet: Theoretical
1301 Foundations", John C. Doyle, et. al. Work in
1302 Progress.
1303
1304 [<a id="ref-EICK">EICK</a>] "Visualizing Software Changes", S.G. Eick, et al,
1305 National Institute of Statistical Sciences, Technical
1306 Report 113, December 2000.
1307
1308 [<a id="ref-MOLINERO2002">MOLINERO2002</a>] "TCP Switching: Exposing Circuits to IP", Pablo
1309 Molinero-Fernandez and Nick McKeown, IEEE January,
1310 2002.
1311
1312 [<a id="ref-FLOYD">FLOYD</a>] "The Synchronization of Periodic Routing Messages",
1313 Sally Floyd and Van Jacobson, IEEE ACM Transactions
1314 on Networking, 1994.
1315
1316 [<a id="ref-FLOYD2001">FLOYD2001</a>] "A Report on Some Recent Developments in TCP
1317 Congestion Control, IEEE Communications Magazine, S.
1318 Floyd, April 2001.
1319
1320 [<a id="ref-FRALEIGH">FRALEIGH</a>] "Provisioning IP Backbone Networks to Support Delay-
1321 Based Service Level Agreements", Chuck Fraleigh,
1322 Fouad Tobagi, and Christophe Diot, 2002.
1323
1324 [<a id="ref-GRIFFIN">GRIFFIN</a>] "What is the Sound of One Route Flapping", Timothy G.
1325 Griffin, IPAM Workshop on Large-Scale Communication
1326 Networks: Topology, Routing, Traffic, and Control,
1327 March, 2002.
1328
1329 [<a id="ref-HANDLEY">HANDLEY</a>] "On Inter-layer Assumptions (A view from the
1330 Transport Area), slides from a presentation at the
1331 IAB workshop on Wireless Internetworking", M.
1332 Handley, March 2000.
1333
1334 [<a id="ref-HOT">HOT</a>] J.M. Carlson and John Doyle, Phys. Rev. E 60, 1412-
1335 1427, 1999.
1336
1337
1338
1339
1340 <span class="grey">Bush, et. al. Informational [Page 24]</span></pre>
1341 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-25" ></span>
1342 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
1343
1344
1345 [<a id="ref-ISO10589">ISO10589</a>] "Intermediate System to Intermediate System
1346 Intradomain Routing Exchange Protocol (IS-IS)".
1347
1348 [<a id="ref-JACOBSON">JACOBSON</a>] "Congestion Avoidance and Control", Van Jacobson,
1349 Proceedings of ACM Sigcomm 1988, pp. 273-288.
1350
1351 [<a id="ref-KARN">KARN</a>] "TCP vs Link Layer Retransmission" in P. Karn et al.,
1352 Advice for Internet Subnetwork Designers, Work in
1353 Progress.
1354
1355 [<a id="ref-KUHN87">KUHN87</a>] "Sources of Failure in the Public Switched Telephone
1356 Network", D. Richard Kuhn, EEE Computer, Vol. 30, No.
1357 4, April, 1997.
1358
1359 [<a id="ref-L2TPV3">L2TPV3</a>] Lan, J., et. al., "Layer Two Tunneling Protocol
1360 (Version 3) -- L2TPv3", Work in Progress.
1361
1362 [<a id="ref-MC2001">MC2001</a>] "U.S Communications Infrastructure at A Crossroads:
1363 Opportunities Amid the Gloom", McKinsey&amp;Company for
1364 Goldman-Sachs, August 2001.
1365
1366 [<a id="ref-MCK2002">MCK2002</a>] Nick McKeown, personal communication, April, 2002.
1367
1368 [<a id="ref-ML2002">ML2002</a>] "Optical Systems", Merril Lynch Technical Report,
1369 April, 2002.
1370
1371 [<a id="ref-NAVE">NAVE</a>] "The influence of mode coupling on the non-linear
1372 evolution of tearing modes", M.F.F. Nave, et al, Eur.
1373 Phys. J. D 8, 287-297.
1374
1375 [<a id="ref-NEUMANN">NEUMANN</a>] "Cause of AT&amp;T network failure", Peter G. Neumann,
1376 <a href="http://catless.ncl.ac.uk/Risks/9.62.html#subj2">http://catless.ncl.ac.uk/Risks/9.62.html#subj2</a>
1377
1378 [<a id="ref-ODLYZKO">ODLYZKO</a>] "Data networks are mostly empty for good reason",
1379 A.M. Odlyzko, IT Professional 1 (no. 2), pp. 67-69,
1380 Mar/Apr 1999.
1381
1382 [<a id="ref-ODLYZKO98A">ODLYZKO98A</a>] "Smart and stupid networks: Why the Internet is like
1383 Microsoft". A. M. Odlyzko, ACM Networker, 2(5),
1384 December, 1998.
1385
1386 [<a id="ref-ODLYZKO98">ODLYZKO98</a>] "The economics of the Internet: Utility, utilization,
1387 pricing, and Quality of Service", A.M. Odlyzko, July,
1388 1998.
1389 <a href="http://www.dtc.umn.edu/~odlyzko/doc/networks.html">http://www.dtc.umn.edu/~odlyzko/doc/networks.html</a>
1390
1391
1392
1393
1394
1395
1396 <span class="grey">Bush, et. al. Informational [Page 25]</span></pre>
1397 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-26" ></span>
1398 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
1399
1400
1401 [<a id="ref-PARK">PARK</a>] "The Internet as a Complex System: Scaling,
1402 Complexity and Control", Kihong Park and Walter
1403 Willinger, AT&amp;T Research, 2002.
1404
1405 [<a id="ref-PERROW">PERROW</a>] "Normal Accidents: Living with High Risk
1406 Technologies", Basic Books, C. Perrow, New York,
1407 1984.
1408
1409 [<a id="ref-PMC">PMC</a>] "The Design of a 10 Gigabit Core Router
1410 Architecture", PMC-Sierra, <a href="http://www.pmc-sierra.com/products/diagrams/CoreRouter_lg.html">http://www.pmc-</a>
1411 <a href="http://www.pmc-sierra.com/products/diagrams/CoreRouter_lg.html">sierra.com/products/diagrams/CoreRouter_lg.html</a>
1412
1413 [<a id="ref-RFC1629">RFC1629</a>] Colella, R., Callon, R., Gardner, E. and Y. Rekhter,
1414 "Guidelines for OSI NSAP Allocation in the Internet",
1415 <a href="./rfc1629">RFC 1629</a>, May 1994.
1416
1417 [<a id="ref-RFC1925">RFC1925</a>] Callon, R., "The Twelve Networking Truths", <a href="./rfc1925">RFC 1925</a>,
1418 1 April 1996.
1419
1420 [<a id="ref-RFC1958">RFC1958</a>] Carpenter, B., Ed., "Architectural principles of the
1421 Internet", <a href="./rfc1958">RFC 1958</a>, June 1996.
1422
1423 [<a id="ref-RFC2283">RFC2283</a>] Bates, T., Chandra, R., Katz, D. and Y. Rekhter,
1424 "Multiprotocol Extensions for BGP4", <a href="./rfc2283">RFC 2283</a>,
1425 February 1998.
1426
1427 [<a id="ref-RFC3155">RFC3155</a>] Dawkins, S., Montenegro, G., Kojo, M. and N. Vaidya,
1428 "End-to-end Performance Implications of Links with
1429 Errors", <a href="https://www.rfc-editor.org/bcp/bcp50">BCP 50</a>, <a href="./rfc3155">RFC 3155</a>, May 2001.
1430
1431 [<a id="ref-ROMANOV">ROMANOV</a>] "Dynamics of TCP over ATM Networks", A. Romanov, S.
1432 Floyd, IEEE JSAC, vol. 13, No 4, pp.633-641, May
1433 1995.
1434
1435 [<a id="ref-SALTZER">SALTZER</a>] "End-To-End Arguments in System Design", J.H.
1436 Saltzer, D.P. Reed, and D.D. Clark, ACM TOCS, Vol 2,
1437 Number 4, November 1984, pp 277-288.
1438
1439 [<a id="ref-SCOTT">SCOTT</a>] "Making Smart Investments to Reduce Unplanned
1440 Downtime", D. Scott, Tactical Guidelines, TG-07-4033,
1441 Gartner Group Research Note, March 1999.
1442
1443 [<a id="ref-SPILLMAN">SPILLMAN</a>] "The Law of Diminishing Returns:, W. J. Spillman and
1444 E. Lang, 1924.
1445
1446 [<a id="ref-STALLINGS">STALLINGS</a>] "Data and Computer Communications (2nd Ed)", William
1447 Stallings, Maxwell Macmillan, 1989.
1448
1449
1450
1451
1452 <span class="grey">Bush, et. al. Informational [Page 26]</span></pre>
1453 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-27" ></span>
1454 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
1455
1456
1457 [<a id="ref-TENNENHOUSE">TENNENHOUSE</a>] "Layered multiplexing considered harmful", D.
1458 Tennenhouse, Proceedings of the IFIP Workshop on
1459 Protocols for High-Speed Networks, Rudin ed., North
1460 Holland Publishers, May 1989.
1461
1462 [<a id="ref-THOMPSON">THOMPSON</a>] "Nonlinear Dynamics and Chaos". J.M.T. Thompson and
1463 H.B. Stewart, John Wiley and Sons, 1994, ISBN
1464 0471909602.
1465
1466 [<a id="ref-TINA">TINA</a>] "What is TINA and is it useful for the TelCos?",
1467 Paolo Coppo, Carlo A. Licciardi, CSELT, EURESCOM
1468 Participants in P847 (FT, IT, NT, TI)
1469
1470 [<a id="ref-WAKEMAN">WAKEMAN</a>] "Layering considered harmful", Ian Wakeman, Jon
1471 Crowcroft, Zheng Wang, and Dejan Sirovica, IEEE
1472 Network, January 1992, p. 7-16.
1473
1474 [<a id="ref-WARD">WARD</a>] "Custom fluorescent-nucleotide synthesis as an
1475 alternative method for nucleic acid labeling",
1476 Octavian Henegariu*, Patricia Bray-Ward and David C.
1477 Ward, Nature Biotech 18:345-348 (2000).
1478
1479 [<a id="ref-WILLINGER2002">WILLINGER2002</a>] "Robustness and the Internet: Design and evolution",
1480 Walter Willinger and John Doyle, 2002.
1481
1482 [<a id="ref-ZHANG">ZHANG</a>] "Impact of Aggregation on Scaling Behavior of
1483 Internet Backbone Traffic", Sprint ATL Technical
1484 Report TR02-ATL-020157 Zhi-Li Zhang, Vinay Ribeiroj,
1485 Sue Moon, Christophe Diot, February, 2002.
1486
1487 <span class="h2"><a class="selflink" id="section-13" href="#section-13">13</a>. Authors' Addresses</span>
1488
1489 Randy Bush
1490 EMail: [email protected]
1491
1492 David Meyer
1493 EMail: [email protected]
1494
1495
1496
1497
1498
1499
1500
1501
1502
1503
1504
1505
1506
1507
1508 <span class="grey">Bush, et. al. Informational [Page 27]</span></pre>
1509 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-28" ></span>
1510 <span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span>
1511
1512
1513 <span class="h2"><a class="selflink" id="section-14" href="#section-14">14</a>. Full Copyright Statement</span>
1514
1515 Copyright (C) The Internet Society (2002). All Rights Reserved.
1516
1517 This document and translations of it may be copied and furnished to
1518 others, and derivative works that comment on or otherwise explain it
1519 or assist in its implementation may be prepared, copied, published
1520 and distributed, in whole or in part, without restriction of any
1521 kind, provided that the above copyright notice and this paragraph are
1522 included on all such copies and derivative works. However, this
1523 document itself may not be modified in any way, such as by removing
1524 the copyright notice or references to the Internet Society or other
1525 Internet organizations, except as needed for the purpose of
1526 developing Internet standards in which case the procedures for
1527 copyrights defined in the Internet Standards process must be
1528 followed, or as required to translate it into languages other than
1529 English.
1530
1531 The limited permissions granted above are perpetual and will not be
1532 revoked by the Internet Society or its successors or assigns.
1533
1534 This document and the information contained herein is provided on an
1535 "AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING
1536 TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING
1537 BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION
1538 HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
1539 MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
1540
1541 Acknowledgement
1542
1543 Funding for the RFC Editor function is currently provided by the
1544 Internet Society.
1545
1546
1547
1548
1549
1550
1551
1552
1553
1554
1555
1556
1557
1558
1559
1560
1561
1562
1563
1564 Bush, et. al. Informational [Page 28]
1565 </pre>