| 1 |
<pre>Network Working Group R. Bush |
| 2 |
Request for Comments: 3439 D. Meyer |
| 3 |
Updates: <a href="./rfc1958">1958</a> December 2002 |
| 4 |
Category: Informational |
| 5 |
|
| 6 |
|
| 7 |
<span class="h1">Some Internet Architectural Guidelines and Philosophy</span> |
| 8 |
|
| 9 |
Status of this Memo |
| 10 |
|
| 11 |
This memo provides information for the Internet community. It does |
| 12 |
not specify an Internet standard of any kind. Distribution of this |
| 13 |
memo is unlimited. |
| 14 |
|
| 15 |
Copyright Notice |
| 16 |
|
| 17 |
Copyright (C) The Internet Society (2002). All Rights Reserved. |
| 18 |
|
| 19 |
Abstract |
| 20 |
|
| 21 |
This document extends <a href="./rfc1958">RFC 1958</a> by outlining some of the philosophical |
| 22 |
guidelines to which architects and designers of Internet backbone |
| 23 |
networks should adhere. We describe the Simplicity Principle, which |
| 24 |
states that complexity is the primary mechanism that impedes |
| 25 |
efficient scaling, and discuss its implications on the architecture, |
| 26 |
design and engineering issues found in large scale Internet |
| 27 |
backbones. |
| 28 |
|
| 29 |
Table of Contents |
| 30 |
|
| 31 |
<a href="#section-1">1</a>. Introduction . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-2">2</a> |
| 32 |
<a href="#section-2">2</a>. Large Systems and The Simplicity Principle . . . . . . . . . <a href="#page-3">3</a> |
| 33 |
<a href="#section-2.1">2.1</a>. The End-to-End Argument and Simplicity . . . . . . . . . <a href="#page-3">3</a> |
| 34 |
<a href="#section-2.2">2.2</a>. Non-linearity and Network Complexity . . . . . . . . . . <a href="#page-3">3</a> |
| 35 |
<a href="#section-2.2.1">2.2.1</a>. The Amplification Principle. . . . . . . . . . . . . . . <a href="#page-4">4</a> |
| 36 |
<a href="#section-2.2.2">2.2.2</a>. The Coupling Principle . . . . . . . . . . . . . . . . . <a href="#page-5">5</a> |
| 37 |
<a href="#section-2.3">2.3</a>. Complexity lesson from voice. . . . . . . . . . . . . . . <a href="#page-6">6</a> |
| 38 |
<a href="#section-2.4">2.4</a>. Upgrade cost of complexity. . . . . . . . . . . . . . . . <a href="#page-7">7</a> |
| 39 |
<a href="#section-3">3</a>. Layering Considered Harmful. . . . . . . . . . . . . . . . . <a href="#page-7">7</a> |
| 40 |
<a href="#section-3.1">3.1</a>. Optimization Considered Harmful . . . . . . . . . . . . . <a href="#page-8">8</a> |
| 41 |
3.2. Feature Richness Considered Harmful . . . . . . . . . . . <a href="#page-9">9</a> |
| 42 |
3.3. Evolution of Transport Efficiency for IP. . . . . . . . . <a href="#page-9">9</a> |
| 43 |
<a href="#section-3.4">3.4</a>. Convergence Layering. . . . . . . . . . . . . . . . . . . <a href="#page-9">9</a> |
| 44 |
<a href="#section-3.4.1">3.4.1</a>. Note on Transport Protocol Layering. . . . . . . . . . . <a href="#page-11">11</a> |
| 45 |
<a href="#section-3.5">3.5</a>. Second Order Effects . . . . . . . . . . . . . . . . . . <a href="#page-11">11</a> |
| 46 |
<a href="#section-3.6">3.6</a>. Instantiating the EOSL Model with IP . . . . . . . . . . <a href="#page-12">12</a> |
| 47 |
<a href="#section-4">4</a>. Avoid the Universal Interworking Function. . . . . . . . . . <a href="#page-12">12</a> |
| 48 |
<a href="#section-4.1">4.1</a>. Avoid Control Plane Interworking . . . . . . . . . . . . . <a href="#page-13">13</a> |
| 49 |
|
| 50 |
|
| 51 |
|
| 52 |
<span class="grey">Bush, et. al. Informational [Page 1]</span></pre> |
| 53 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-2" ></span> |
| 54 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 55 |
|
| 56 |
|
| 57 |
<a href="#section-5">5</a>. Packet versus Circuit Switching: Fundamental Differences . . <a href="#page-13">13</a> |
| 58 |
<a href="#section-5.1">5.1</a>. Is PS is inherently more efficient than CS? . . . . . . . <a href="#page-13">13</a> |
| 59 |
<a href="#section-5.2">5.2</a>. Is PS simpler than CS? . . . . . . . . . . . . . . . . . . <a href="#page-14">14</a> |
| 60 |
<a href="#section-5.2.1">5.2.1</a>. Software/Firmware Complexity . . . . . . . . . . . . . . <a href="#page-15">15</a> |
| 61 |
<a href="#section-5.2.2">5.2.2</a>. Macro Operation Complexity . . . . . . . . . . . . . . . <a href="#page-15">15</a> |
| 62 |
<a href="#section-5.2.3">5.2.3</a>. Hardware Complexity. . . . . . . . . . . . . . . . . . . <a href="#page-15">15</a> |
| 63 |
<a href="#section-5.2.4">5.2.4</a>. Power. . . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-16">16</a> |
| 64 |
<a href="#section-5.2.5">5.2.5</a>. Density. . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-16">16</a> |
| 65 |
<a href="#section-5.2.6">5.2.6</a>. Fixed versus variable costs. . . . . . . . . . . . . . . <a href="#page-16">16</a> |
| 66 |
<a href="#section-5.2.7">5.2.7</a>. QoS. . . . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-17">17</a> |
| 67 |
<a href="#section-5.2.8">5.2.8</a>. Flexibility. . . . . . . . . . . . . . . . . . . . . . . <a href="#page-17">17</a> |
| 68 |
<a href="#section-5.3">5.3</a>. Relative Complexity . . . . . . . . . . . . . . . . . . . <a href="#page-17">17</a> |
| 69 |
<a href="#section-5.3.1">5.3.1</a>. HBHI and the OPEX Challenge. . . . . . . . . . . . . . . <a href="#page-18">18</a> |
| 70 |
<a href="#section-6">6</a>. The Myth of Over-Provisioning. . . . . . . . . . . . . . . . <a href="#page-18">18</a> |
| 71 |
<a href="#section-7">7</a>. The Myth of Five Nines . . . . . . . . . . . . . . . . . . . <a href="#page-19">19</a> |
| 72 |
<a href="#section-8">8</a>. Architectural Component Proportionality Law. . . . . . . . . <a href="#page-20">20</a> |
| 73 |
<a href="#section-8.1">8.1</a>. Service Delivery Paths . . . . . . . . . . . . . . . . . . <a href="#page-21">21</a> |
| 74 |
<a href="#section-9">9</a>. Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-21">21</a> |
| 75 |
<a href="#section-10">10</a>. Security Considerations . . . . . . . . . . . . . . . . . . <a href="#page-22">22</a> |
| 76 |
<a href="#section-11">11</a>. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . <a href="#page-23">23</a> |
| 77 |
<a href="#section-12">12</a>. References. . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-23">23</a> |
| 78 |
<a href="#section-13">13</a>. Authors' Addresses. . . . . . . . . . . . . . . . . . . . . <a href="#page-27">27</a> |
| 79 |
<a href="#section-14">14</a>. Full Copyright Statement. . . . . . . . . . . . . . . . . . <a href="#page-28">28</a> |
| 80 |
|
| 81 |
<span class="h2"><a class="selflink" id="section-1" href="#section-1">1</a>. Introduction</span> |
| 82 |
|
| 83 |
<a href="./rfc1958">RFC 1958</a> [<a href="./rfc1958" title=""Architectural principles of the Internet"">RFC1958</a>] describes the underlying principles of the |
| 84 |
Internet architecture. This note extends that work by outlining some |
| 85 |
of the philosophical guidelines to which architects and designers of |
| 86 |
Internet backbone networks should adhere. While many of the areas |
| 87 |
outlined in this document may be controversial, the unifying |
| 88 |
principle described here, controlling complexity as a mechanism to |
| 89 |
control costs and reliability, should not be. Complexity in carrier |
| 90 |
networks can derive from many sources. However, as stated in |
| 91 |
[<a href="#ref-DOYLE2002" title=""Robustness and the Internet: Theoretical Foundations"">DOYLE2002</a>], "Complexity in most systems is driven by the need for |
| 92 |
robustness to uncertainty in their environments and component parts |
| 93 |
far more than by basic functionality". The major thrust of this |
| 94 |
document, then, is to raise awareness about the complexity of some of |
| 95 |
our current architectures, and to examine the effect such complexity |
| 96 |
will almost certainly have on the IP carrier industry's ability to |
| 97 |
succeed. |
| 98 |
|
| 99 |
The rest of this document is organized as follows: The first section |
| 100 |
describes the Simplicity Principle and its implications for the |
| 101 |
design of very large systems. The remainder of the document outlines |
| 102 |
the high-level consequences of the Simplicity Principle and how it |
| 103 |
should guide large scale network architecture and design approaches. |
| 104 |
|
| 105 |
|
| 106 |
|
| 107 |
|
| 108 |
<span class="grey">Bush, et. al. Informational [Page 2]</span></pre> |
| 109 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-3" ></span> |
| 110 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 111 |
|
| 112 |
|
| 113 |
<span class="h2"><a class="selflink" id="section-2" href="#section-2">2</a>. Large Systems and The Simplicity Principle</span> |
| 114 |
|
| 115 |
The Simplicity Principle, which was perhaps first articulated by Mike |
| 116 |
O'Dell, former Chief Architect at UUNET, states that complexity is |
| 117 |
the primary mechanism which impedes efficient scaling, and as a |
| 118 |
result is the primary driver of increases in both capital |
| 119 |
expenditures (CAPEX) and operational expenditures (OPEX). The |
| 120 |
implication for carrier IP networks then, is that to be successful we |
| 121 |
must drive our architectures and designs toward the simplest possible |
| 122 |
solutions. |
| 123 |
|
| 124 |
<span class="h3"><a class="selflink" id="section-2.1" href="#section-2.1">2.1</a>. The End-to-End Argument and Simplicity</span> |
| 125 |
|
| 126 |
The end-to-end argument, which is described in [<a href="#ref-SALTZER" title=""End-To-End Arguments in System Design"">SALTZER</a>] (as well as |
| 127 |
in <a href="./rfc1958">RFC 1958</a> [<a href="./rfc1958" title=""Architectural principles of the Internet"">RFC1958</a>]), contends that "end-to-end protocol design |
| 128 |
should not rely on the maintenance of state (i.e., information about |
| 129 |
the state of the end-to-end communication) inside the network. Such |
| 130 |
state should be maintained only in the end points, in such a way that |
| 131 |
the state can only be destroyed when the end point itself breaks." |
| 132 |
This property has also been related to Clark's "fate-sharing" concept |
| 133 |
[<a href="#ref-CLARK" title=""The Design Philosophy of the DARPA Internet Protocols"">CLARK</a>]. We can see that the end-to-end principle leads directly to |
| 134 |
the Simplicity Principle by examining the so-called "hourglass" |
| 135 |
formulation of the Internet architecture [<a href="#ref-WILLINGER2002" title=""Robustness and the Internet: Design and evolution"">WILLINGER2002</a>]. In this |
| 136 |
model, the thin waist of the hourglass is envisioned as the |
| 137 |
(minimalist) IP layer, and any additional complexity is added above |
| 138 |
the IP layer. In short, the complexity of the Internet belongs at |
| 139 |
the edges, and the IP layer of the Internet should remain as simple |
| 140 |
as possible. |
| 141 |
|
| 142 |
Finally, note that the End-to-End Argument does not imply that the |
| 143 |
core of the Internet will not contain and maintain state. In fact, a |
| 144 |
huge amount coarse grained state is maintained in the Internet's core |
| 145 |
(e.g., routing state). However, the important point here is that |
| 146 |
this (coarse grained) state is almost orthogonal to the state |
| 147 |
maintained by the end-points (e.g., hosts). It is this minimization |
| 148 |
of interaction that contributes to simplicity. As a result, |
| 149 |
consideration of "core vs. end-point" state interaction is crucial |
| 150 |
when analyzing protocols such as Network Address Translation (NAT), |
| 151 |
which reduce the transparency between network and hosts. |
| 152 |
|
| 153 |
<span class="h3"><a class="selflink" id="section-2.2" href="#section-2.2">2.2</a>. Non-linearity and Network Complexity</span> |
| 154 |
|
| 155 |
Complex architectures and designs have been (and continue to be) |
| 156 |
among the most significant and challenging barriers to building cost- |
| 157 |
effective large scale IP networks. Consider, for example, the task |
| 158 |
of building a large scale packet network. Industry experience has |
| 159 |
shown that building such a network is a different activity (and hence |
| 160 |
requires a different skill set) than building a small to medium scale |
| 161 |
|
| 162 |
|
| 163 |
|
| 164 |
<span class="grey">Bush, et. al. Informational [Page 3]</span></pre> |
| 165 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-4" ></span> |
| 166 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 167 |
|
| 168 |
|
| 169 |
network, and as such doesn't have the same properties. In |
| 170 |
particular, the largest networks exhibit, both in theory and in |
| 171 |
practice, architecture, design, and engineering non-linearities which |
| 172 |
are not exhibited at smaller scale. We call this Architecture, |
| 173 |
Design, and Engineering (ADE) non-linearity. That is, systems such |
| 174 |
as the Internet could be described as highly self-dissimilar, with |
| 175 |
extremely different scales and levels of abstraction [<a href="#ref-CARLSON" title=""Complexity and Robustness"">CARLSON</a>]. The |
| 176 |
ADE non-linearity property is based upon two well-known principles |
| 177 |
from non-linear systems theory [<a href="#ref-THOMPSON" title=""Nonlinear Dynamics and Chaos"">THOMPSON</a>]: |
| 178 |
|
| 179 |
<span class="h4"><a class="selflink" id="section-2.2.1" href="#section-2.2.1">2.2.1</a>. The Amplification Principle</span> |
| 180 |
|
| 181 |
The Amplification Principle states that there are non-linearities |
| 182 |
which occur at large scale which do not occur at small to medium |
| 183 |
scale. |
| 184 |
|
| 185 |
COROLLARY: In many large networks, even small things can and do cause |
| 186 |
huge events. In system-theoretic terms, in large systems such as |
| 187 |
these, even small perturbations on the input to a process can |
| 188 |
destabilize the system's output. |
| 189 |
|
| 190 |
An important example of the Amplification Principle is non-linear |
| 191 |
resonant amplification, which is a powerful process that can |
| 192 |
transform dynamic systems, such as large networks, in surprising ways |
| 193 |
with seemingly small fluctuations. These small fluctuations may |
| 194 |
slowly accumulate, and if they are synchronized with other cycles, |
| 195 |
may produce major changes. Resonant phenomena are examples of non- |
| 196 |
linear behavior where small fluctuations may be amplified and have |
| 197 |
influences far exceeding their initial sizes. The natural world is |
| 198 |
filled with examples of resonant behavior that can produce system- |
| 199 |
wide changes, such as the destruction of the Tacoma Narrows bridge |
| 200 |
(due to the resonant amplification of small gusts of wind). Other |
| 201 |
examples include the gaps in the asteroid belts and rings of Saturn |
| 202 |
which are created by non-linear resonant amplification. Some |
| 203 |
features of human behavior and most pilgrimage systems are influenced |
| 204 |
by resonant phenomena involving the dynamics of the solar system, |
| 205 |
such as solar days, the 27.3 day (sidereal) and 29.5 day (synodic) |
| 206 |
cycles of the moon or the 365.25 day cycle of the sun. |
| 207 |
|
| 208 |
In the Internet domain, it has been shown that increased inter- |
| 209 |
connectivity results in more complex and often slower BGP routing |
| 210 |
convergence [<a href="#ref-AHUJA" title=""The Impact of Internet Policy and Topology on Delayed Routing Convergence"">AHUJA</a>]. A related result is that a small amount of |
| 211 |
inter-connectivity causes the output of a routing mesh to be |
| 212 |
significantly more complex than its input [<a href="#ref-GRIFFIN" title=""What is the Sound of One Route Flapping"">GRIFFIN</a>]. An important |
| 213 |
method for reducing amplification is ensure that local changes have |
| 214 |
only local effect (this is as opposed to systems in which local |
| 215 |
changes have global effect). Finally, ATM provides an excellent |
| 216 |
example of an amplification effect: if you lose one cell, you destroy |
| 217 |
|
| 218 |
|
| 219 |
|
| 220 |
<span class="grey">Bush, et. al. Informational [Page 4]</span></pre> |
| 221 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-5" ></span> |
| 222 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 223 |
|
| 224 |
|
| 225 |
the entire packet (and it gets worse, as in the absence of mechanisms |
| 226 |
such as Early Packet Discard [<a href="#ref-ROMANOV" title=""Dynamics of TCP over ATM Networks"">ROMANOV</a>], you will continue to carry |
| 227 |
the already damaged packet). |
| 228 |
|
| 229 |
Another interesting example of amplification comes from the |
| 230 |
engineering domain, and is described in [<a href="#ref-CARLSON" title=""Complexity and Robustness"">CARLSON</a>]. They consider the |
| 231 |
Boeing 777, which is a "fly-by-wire" aircraft, containing as many as |
| 232 |
150,000 subsystems and approximately 1000 CPUs. What they observe is |
| 233 |
that while the 777 is robust to large-scale atmospheric disturbances, |
| 234 |
turbulence boundaries, and variations in cargo loads (to name a few), |
| 235 |
it could be catastrophically disabled my microscopic alterations in a |
| 236 |
very few large CPUs (as the point out, fortunately this is a very |
| 237 |
rare occurrence). This example illustrates the issue "that |
| 238 |
complexity can amplify small perturbations, and the design engineer |
| 239 |
must ensure such perturbations are extremely rare." [<a href="#ref-CARLSON" title=""Complexity and Robustness"">CARLSON</a>] |
| 240 |
|
| 241 |
<span class="h4"><a class="selflink" id="section-2.2.2" href="#section-2.2.2">2.2.2</a>. The Coupling Principle</span> |
| 242 |
|
| 243 |
The Coupling Principle states that as things get larger, they often |
| 244 |
exhibit increased interdependence between components. |
| 245 |
|
| 246 |
COROLLARY: The more events that simultaneously occur, the larger the |
| 247 |
likelihood that two or more will interact. This phenomenon has also |
| 248 |
been termed "unforeseen feature interaction" [<a href="#ref-WILLINGER2002" title=""Robustness and the Internet: Design and evolution"">WILLINGER2002</a>]. |
| 249 |
|
| 250 |
Much of the non-linearity observed large systems is largely due to |
| 251 |
coupling. This coupling has both horizontal and vertical |
| 252 |
components. In the context of networking, horizontal coupling is |
| 253 |
exhibited between the same protocol layer, while vertical coupling |
| 254 |
occurs between layers. |
| 255 |
|
| 256 |
Coupling is exhibited by a wide variety of natural systems, including |
| 257 |
plasma macro-instabilities (hydro-magnetic, e.g., kink, fire-hose, |
| 258 |
mirror, ballooning, tearing, trapped-particle effects) [<a href="#ref-NAVE" title=""The influence of mode coupling on the non-linear evolution of tearing modes"">NAVE</a>], as |
| 259 |
well as various kinds of electrochemical systems (consider the custom |
| 260 |
fluorescent nucleotide synthesis/nucleic acid labeling problem |
| 261 |
[<a href="#ref-WARD" title=""Custom fluorescent-nucleotide synthesis as an alternative method for nucleic acid labeling"">WARD</a>]). Coupling of clock physical periodicity has also been |
| 262 |
observed [<a href="#ref-JACOBSON" title=""Congestion Avoidance and Control"">JACOBSON</a>], as well as coupling of various types of |
| 263 |
biological cycles. |
| 264 |
|
| 265 |
Several canonical examples also exist in well known network systems. |
| 266 |
Examples include the synchronization of various control loops, such |
| 267 |
as routing update synchronization and TCP Slow Start synchronization |
| 268 |
[<a href="#ref-FLOYD" title=""The Synchronization of Periodic Routing Messages"">FLOYD</a>,<a href="#ref-JACOBSON" title=""Congestion Avoidance and Control"">JACOBSON</a>]. An important result of these observations is that |
| 269 |
coupling is intimately related to synchronization. Injecting |
| 270 |
randomness into these systems is one way to reduce coupling. |
| 271 |
|
| 272 |
|
| 273 |
|
| 274 |
|
| 275 |
|
| 276 |
<span class="grey">Bush, et. al. Informational [Page 5]</span></pre> |
| 277 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-6" ></span> |
| 278 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 279 |
|
| 280 |
|
| 281 |
Interestingly, in analyzing risk factors for the Public Switched |
| 282 |
Telephone Network (PSTN), Charles Perrow decomposes the complexity |
| 283 |
problem along two related axes, which he terms "interactions" and |
| 284 |
"coupling" [<a href="#ref-PERROW" title=""Normal Accidents: Living with High Risk Technologies"">PERROW</a>]. Perrow cites interactions and coupling as |
| 285 |
significant factors in determining the reliability of a complex |
| 286 |
system (and in particular, the PSTN). In this model, interactions |
| 287 |
refer to the dependencies between components (linear or non-linear), |
| 288 |
while coupling refers to the flexibility in a system. Systems with |
| 289 |
simple, linear interactions have components that affect only other |
| 290 |
components that are functionally downstream. Complex system |
| 291 |
components interact with many other components in different and |
| 292 |
possibly distant parts of the system. Loosely coupled systems are |
| 293 |
said to have more flexibility in time constraints, sequencing, and |
| 294 |
environmental assumptions than do tightly coupled systems. In |
| 295 |
addition, systems with complex interactions and tight coupling are |
| 296 |
likely to have unforeseen failure states (of course, complex |
| 297 |
interactions permit more complications to develop and make the system |
| 298 |
hard to understand and predict); this behavior is also described in |
| 299 |
[<a href="#ref-WILLINGER2002" title=""Robustness and the Internet: Design and evolution"">WILLINGER2002</a>]. Tight coupling also means that the system has less |
| 300 |
flexibility in recovering from failure states. |
| 301 |
|
| 302 |
The PSTN's SS7 control network provides an interesting example of |
| 303 |
what can go wrong with a tightly coupled complex system. Outages |
| 304 |
such as the well publicized 1991 outage of AT&T's SS7 demonstrates |
| 305 |
the phenomenon: the outage was caused by software bugs in the |
| 306 |
switches' crash recovery code. In this case, one switch crashed due |
| 307 |
to a hardware glitch. When this switch came back up, it (plus a |
| 308 |
reasonably probable timing event) caused its neighbors to crash When |
| 309 |
the neighboring switches came back up, they caused their neighbors to |
| 310 |
crash, and so on [<a href="#ref-NEUMANN" title=""Cause of AT&T network failure"">NEUMANN</a>] (the root cause turned out to be a |
| 311 |
misplaced 'break' statement; this is an excellent example of cross- |
| 312 |
layer coupling). This phenomenon is similar to the phase-locking of |
| 313 |
weakly coupled oscillators, in which random variations in sequence |
| 314 |
times plays an important role in system stability [<a href="#ref-THOMPSON" title=""Nonlinear Dynamics and Chaos"">THOMPSON</a>]. |
| 315 |
|
| 316 |
<span class="h3"><a class="selflink" id="section-2.3" href="#section-2.3">2.3</a>. Complexity lesson from voice</span> |
| 317 |
|
| 318 |
In the 1970s and 1980s, the voice carriers competed by adding |
| 319 |
features which drove substantial increases in the complexity of the |
| 320 |
PSTN, especially in the Class 5 switching infrastructure. This |
| 321 |
complexity was typically software-based, not hardware driven, and |
| 322 |
therefore had cost curves worse than Moore's Law. In summary, poor |
| 323 |
margins on voice products today are due to OPEX and CAPEX costs not |
| 324 |
dropping as we might expect from simple hardware-bound |
| 325 |
implementations. |
| 326 |
|
| 327 |
|
| 328 |
|
| 329 |
|
| 330 |
|
| 331 |
|
| 332 |
<span class="grey">Bush, et. al. Informational [Page 6]</span></pre> |
| 333 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-7" ></span> |
| 334 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 335 |
|
| 336 |
|
| 337 |
<span class="h3"><a class="selflink" id="section-2.4" href="#section-2.4">2.4</a>. Upgrade cost of complexity</span> |
| 338 |
|
| 339 |
Consider the cost of providing new features in a complex network. |
| 340 |
The traditional voice network has little intelligence in its edge |
| 341 |
devices (phone instruments), and a very smart core. The Internet has |
| 342 |
smart edges, computers with operating systems, applications, etc., |
| 343 |
and a simple core, which consists of a control plane and packet |
| 344 |
forwarding engines. Adding an new Internet service is just a matter |
| 345 |
of distributing an application to the a few consenting desktops who |
| 346 |
wish to use it. Compare this to adding a service to voice, where one |
| 347 |
has to upgrade the entire core. |
| 348 |
|
| 349 |
<span class="h2"><a class="selflink" id="section-3" href="#section-3">3</a>. Layering Considered Harmful</span> |
| 350 |
|
| 351 |
There are several generic properties of layering, or vertical |
| 352 |
integration as applied to networking. In general, a layer as defined |
| 353 |
in our context implements one or more of |
| 354 |
|
| 355 |
Error Control: The layer makes the "channel" more reliable |
| 356 |
(e.g., reliable transport layer) |
| 357 |
|
| 358 |
Flow Control: The layer avoids flooding slower peer (e.g., |
| 359 |
ATM flow control) |
| 360 |
|
| 361 |
Fragmentation: Dividing large data chunks into smaller |
| 362 |
pieces, and subsequent reassembly (e.g., TCP |
| 363 |
MSS fragmentation/reassembly) |
| 364 |
|
| 365 |
Multiplexing: Allow several higher level sessions share |
| 366 |
single lower level "connection" (e.g., ATM PVC) |
| 367 |
|
| 368 |
Connection Setup: Handshaking with peer (e.g., TCP three-way |
| 369 |
handshake, ATM ILMI) |
| 370 |
|
| 371 |
Addressing/Naming: Locating, managing identifiers associated |
| 372 |
with entities (e.g., GOSSIP 2 NSAP Structure |
| 373 |
[<a href="./rfc1629" title=""Guidelines for OSI NSAP Allocation in the Internet"">RFC1629</a>]) |
| 374 |
|
| 375 |
Layering of this type does have various conceptual and structuring |
| 376 |
advantages. However, in the data networking context structured |
| 377 |
layering implies that the functions of each layer are carried out |
| 378 |
completely before the protocol data unit is passed to the next layer. |
| 379 |
This means that the optimization of each layer has to be done |
| 380 |
separately. Such ordering constraints are in conflict with efficient |
| 381 |
implementation of data manipulation functions. One could accuse the |
| 382 |
layered model (e.g., TCP/IP and ISO OSI) of causing this conflict. |
| 383 |
In fact, the operations of multiplexing and segmentation both hide |
| 384 |
vital information that lower layers may need to optimize their |
| 385 |
|
| 386 |
|
| 387 |
|
| 388 |
<span class="grey">Bush, et. al. Informational [Page 7]</span></pre> |
| 389 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-8" ></span> |
| 390 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 391 |
|
| 392 |
|
| 393 |
performance. For example, layer N may duplicate lower level |
| 394 |
functionality, e.g., error recovery hop-hop versus end-to-end error |
| 395 |
recovery. In addition, different layers may need the same |
| 396 |
information (e.g., time stamp): layer N may need layer N-2 |
| 397 |
information (e.g., lower layer packet sizes), and the like [<a href="#ref-WAKEMAN" title=""Layering considered harmful"">WAKEMAN</a>]. |
| 398 |
A related and even more ironic statement comes from Tennenhouse's |
| 399 |
classic paper, "Layered Multiplexing Considered Harmful" |
| 400 |
[<a href="#ref-TENNENHOUSE" title=""Layered multiplexing considered harmful"">TENNENHOUSE</a>]: "The ATM approach to broadband networking is presently |
| 401 |
being pursued within the CCITT (and elsewhere) as the unifying |
| 402 |
mechanism for the support of service integration, rate adaptation, |
| 403 |
and jitter control within the lower layers of the network |
| 404 |
architecture. This position paper is specifically concerned with the |
| 405 |
jitter arising from the design of the "middle" and "upper" layers |
| 406 |
that operate within the end systems and relays of multi-service |
| 407 |
networks (MSNs)." |
| 408 |
|
| 409 |
As a result of inter-layer dependencies, increased layering can |
| 410 |
quickly lead to violation of the Simplicity Principle. Industry |
| 411 |
experience has taught us that increased layering frequently increases |
| 412 |
complexity and hence leads to increases in OPEX, as is predicted by |
| 413 |
the Simplicity Principle. A corollary is stated in <a href="./rfc1925">RFC 1925</a> |
| 414 |
<a href="./rfc1925#section-2">[RFC1925], section 2</a>(5): |
| 415 |
|
| 416 |
"It is always possible to agglutinate multiple separate problems |
| 417 |
into a single complex interdependent solution. In most cases |
| 418 |
this is a bad idea." |
| 419 |
|
| 420 |
The first order conclusion then, is that horizontal (as opposed to |
| 421 |
vertical) separation may be more cost-effective and reliable in the |
| 422 |
long term. |
| 423 |
|
| 424 |
<span class="h3"><a class="selflink" id="section-3.1" href="#section-3.1">3.1</a>. Optimization Considered Harmful</span> |
| 425 |
|
| 426 |
A corollary of the layering arguments above is that optimization can |
| 427 |
also be considered harmful. In particular, optimization introduces |
| 428 |
complexity, and as well as introducing tighter coupling between |
| 429 |
components and layers. |
| 430 |
|
| 431 |
An important and related effect of optimization is described by the |
| 432 |
Law of Diminishing Returns, which states that if one factor of |
| 433 |
production is increased while the others remain constant, the overall |
| 434 |
returns will relatively decrease after a certain point [<a href="#ref-SPILLMAN" title="W. J. Spillman and E. Lang">SPILLMAN</a>]. |
| 435 |
The implication here is that trying to squeeze out efficiency past |
| 436 |
that point only adds complexity, and hence leads to less reliable |
| 437 |
systems. |
| 438 |
|
| 439 |
|
| 440 |
|
| 441 |
|
| 442 |
|
| 443 |
|
| 444 |
<span class="grey">Bush, et. al. Informational [Page 8]</span></pre> |
| 445 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-9" ></span> |
| 446 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 447 |
|
| 448 |
|
| 449 |
<span class="h3"><a class="selflink" id="section-3.2" href="#section-3.2">3.2</a>. Feature Richness Considered Harmful</span> |
| 450 |
|
| 451 |
While adding any new feature may be considered a gain (and in fact |
| 452 |
frequently differentiates vendors of various types of equipment), but |
| 453 |
there is a danger. The danger is in increased system complexity. |
| 454 |
|
| 455 |
<span class="h3"><a class="selflink" id="section-3.3" href="#section-3.3">3.3</a>. Evolution of Transport Efficiency for IP</span> |
| 456 |
|
| 457 |
The evolution of transport infrastructures for IP offers a good |
| 458 |
example of how decreasing vertical integration has lead to various |
| 459 |
efficiencies. In particular, |
| 460 |
|
| 461 |
| IP over ATM over SONET --> |
| 462 |
| IP over SONET over WDM --> |
| 463 |
| IP over WDM |
| 464 |
| |
| 465 |
\|/ |
| 466 |
Decreasing complexity, CAPEX, OPEX |
| 467 |
|
| 468 |
The key point here is that layers are removed resulting in CAPEX and |
| 469 |
OPEX efficiencies. |
| 470 |
|
| 471 |
<span class="h3"><a class="selflink" id="section-3.4" href="#section-3.4">3.4</a>. Convergence Layering</span> |
| 472 |
|
| 473 |
Convergence is related to the layering concepts described above in |
| 474 |
that convergence is achieved via a "convergence layer". The end |
| 475 |
state of the convergence argument is the concept of Everything Over |
| 476 |
Some Layer (EOSL). Conduit, DWDM, fiber, ATM, MPLS, and even IP have |
| 477 |
all been proposed as convergence layers. It is important to note |
| 478 |
that since layering typically drives OPEX up, we expect convergence |
| 479 |
will as well. This observation is again consistent with industry |
| 480 |
experience. |
| 481 |
|
| 482 |
There are many notable examples of convergence layer failure. |
| 483 |
Perhaps the most germane example is IP over ATM. The immediate and |
| 484 |
most obvious consequence of ATM layering is the so-called cell tax: |
| 485 |
First, note that the complete answer on ATM efficiency is that it |
| 486 |
depends upon packet size distributions. Let's assume that typical |
| 487 |
Internet type traffic patterns, which tend to have high percentages |
| 488 |
of packets at 40, 44, and 552 bytes. Recent data [<a href="#ref-CAIDA">CAIDA</a>] shows that |
| 489 |
about 95% of WAN bytes and 85% of packets are TCP. Much of this |
| 490 |
traffic is composed of 40/44 byte packets. |
| 491 |
|
| 492 |
Now, consider the case of a a DS3 backbone with PLCP turned on. Then |
| 493 |
the maximum cell rate is 96,000 cells/sec. If you multiply this |
| 494 |
value by the number of bits in the payload, you get: 96000 cells/sec |
| 495 |
* 48 bytes/cell * 8 = 36.864 Mbps. This, however, is unrealistic |
| 496 |
since it |
| 497 |
|
| 498 |
|
| 499 |
|
| 500 |
<span class="grey">Bush, et. al. Informational [Page 9]</span></pre> |
| 501 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-10" ></span> |
| 502 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 503 |
|
| 504 |
|
| 505 |
assumes perfect payload packing. There are two other things that |
| 506 |
contribute to the ATM overhead (cell tax): The wasted padding and the |
| 507 |
8 byte SNAP header. |
| 508 |
|
| 509 |
It is the SNAP header which causes most of the problems (and you |
| 510 |
can't do anything about this), forcing most small packets to consume |
| 511 |
two cells, with the second cell to be mostly empty padding (this |
| 512 |
interacts really poorly with the data quoted above, e.g., that most |
| 513 |
packets are 40-44 byte TCP Ack packets). This causes a loss of about |
| 514 |
another 16% from the 36.8 Mbps ideal throughput. |
| 515 |
|
| 516 |
So the total throughput ends up being (for a DS3): |
| 517 |
|
| 518 |
DS3 Line Rate: 44.736 |
| 519 |
PLCP Overhead - 4.032 |
| 520 |
Per Cell Header: - 3.840 |
| 521 |
SNAP Header & Padding: - 5.900 |
| 522 |
========= |
| 523 |
30.960 Mbps |
| 524 |
|
| 525 |
Result: With a DS3 line rate of 44.736 Mbps, the total overhead is |
| 526 |
about 31%. |
| 527 |
|
| 528 |
Another way to look at this is that since a large fraction of WAN |
| 529 |
traffic is comprised of TCP ACKs, one can make a different but |
| 530 |
related calculation. IP over ATM requires: |
| 531 |
|
| 532 |
IP data (40 bytes in this case) |
| 533 |
8 bytes SNAP |
| 534 |
8 bytes AAL5 stuff |
| 535 |
5 bytes for each cell |
| 536 |
+ as much more as it takes to fill out the last cell |
| 537 |
|
| 538 |
On ATM, this becomes two cells - 106 bytes to convey 40 bytes of |
| 539 |
information. The next most common size seems to be one of several |
| 540 |
sizes in the 504-556 byte range - 636 bytes to carry IP, TCP, and a |
| 541 |
512 byte TCP payload - with messages larger than 1000 bytes running |
| 542 |
third. |
| 543 |
|
| 544 |
One would imagine that 87% payload (556 byte message size) is better |
| 545 |
than 37% payload (TCP Ack size), but it's not the 95-98% that |
| 546 |
customers are used to, and the predominance of TCP Acks skews the |
| 547 |
average. |
| 548 |
|
| 549 |
|
| 550 |
|
| 551 |
|
| 552 |
|
| 553 |
|
| 554 |
|
| 555 |
|
| 556 |
<span class="grey">Bush, et. al. Informational [Page 10]</span></pre> |
| 557 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-11" ></span> |
| 558 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 559 |
|
| 560 |
|
| 561 |
<span class="h4"><a class="selflink" id="section-3.4.1" href="#section-3.4.1">3.4.1</a>. Note on Transport Protocol Layering</span> |
| 562 |
|
| 563 |
Protocol layering models are frequently cast as "X over Y" models. |
| 564 |
In these cases, protocol Y carries protocol X's protocol data units |
| 565 |
(and possibly control data) over Y's data plane, i.e., Y is a |
| 566 |
"convergence layer". Examples include Frame Relay over ATM, IP over |
| 567 |
ATM, and IP over MPLS. While X over Y layering has met with only |
| 568 |
marginal success [<a href="#ref-TENNENHOUSE" title=""Layered multiplexing considered harmful"">TENNENHOUSE</a>,<a href="#ref-WAKEMAN" title=""Layering considered harmful"">WAKEMAN</a>], there have been a few notable |
| 569 |
instances where efficiency can be and is gained. In particular, "X |
| 570 |
over Y efficiencies" can be realized when there is a kind of |
| 571 |
"isomorphism" between the X and Y (i.e., there is a small convergence |
| 572 |
layer). In these cases X's data, and possibly control traffic, are |
| 573 |
"encapsulated" and transported over Y. Examples include Frame Relay |
| 574 |
over ATM, and Frame Relay, AAL5 ATM and Ethernet over L2TPv3 |
| 575 |
[<a href="#ref-L2TPV3" title=""Layer Two Tunneling Protocol (Version 3) -- L2TPv3"">L2TPV3</a>]; the simplifying factors here are that there is no |
| 576 |
requirement that a shared clock be recovered by the communicating end |
| 577 |
points, and that control-plane interworking is minimized. An |
| 578 |
alternative is to interwork the X and Y's control and data planes; |
| 579 |
control-plane interworking is discussed below. |
| 580 |
|
| 581 |
<span class="h3"><a class="selflink" id="section-3.5" href="#section-3.5">3.5</a>. Second Order Effects</span> |
| 582 |
|
| 583 |
IP over ATM provides an excellent example of unanticipated second |
| 584 |
order effects. In particular, Romanov and Floyd's classic study on |
| 585 |
TCP good-put [<a href="#ref-ROMANOV" title=""Dynamics of TCP over ATM Networks"">ROMANOV</a>] on ATM showed that large UBR buffers (larger |
| 586 |
than one TCP window size) are required to achieve reasonable |
| 587 |
performance, that packet discard mechanisms (such as Early Packet |
| 588 |
Discard, or EPD) improve the effective usage of the bandwidth and |
| 589 |
that more elaborate service and drop strategies than FIFO+EPD, such |
| 590 |
as per VC queuing and accounting, might be required at the bottleneck |
| 591 |
to ensure both high efficiency and fairness. Though all studies |
| 592 |
clearly indicate that a buffer size not less than one TCP window size |
| 593 |
is required, the amount of extra buffer required naturally depends on |
| 594 |
the packet discard mechanism used and is still an open issue. |
| 595 |
|
| 596 |
Examples of this kind of problem with layering abound in practical |
| 597 |
networking. Consider, for example, the effect of IP transport's |
| 598 |
implicit assumptions of lower layers. In particular: |
| 599 |
|
| 600 |
o Packet loss: TCP assumes that packet losses are indications of |
| 601 |
congestion, but sometimes losses are from corruption on a wireless |
| 602 |
link [<a href="./rfc3115">RFC3115</a>]. |
| 603 |
|
| 604 |
o Reordered packets: TCP assumes that significantly reordered |
| 605 |
packets are indications of congestion. This is not always the |
| 606 |
case [<a href="#ref-FLOYD2001" title="IEEE Communications Magazine">FLOYD2001</a>]. |
| 607 |
|
| 608 |
|
| 609 |
|
| 610 |
|
| 611 |
|
| 612 |
<span class="grey">Bush, et. al. Informational [Page 11]</span></pre> |
| 613 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-12" ></span> |
| 614 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 615 |
|
| 616 |
|
| 617 |
o Round-trip times: TCP measures round-trip times, and assumes that |
| 618 |
the lack of an acknowledgment within a period of time based on the |
| 619 |
measured round-trip time is a packet loss, and therefore an |
| 620 |
indication of congestion [<a href="#ref-KARN" title=""TCP vs Link Layer Retransmission"">KARN</a>]. |
| 621 |
|
| 622 |
o Congestion control: TCP congestion control implicitly assumes that |
| 623 |
all the packets in a flow are treated the same by the network, but |
| 624 |
this is not always the case [<a href="#ref-HANDLEY" title=""On Inter-layer Assumptions (A view from the Transport Area), slides from a presentation at the IAB workshop on Wireless Internetworking"">HANDLEY</a>]. |
| 625 |
|
| 626 |
<span class="h3"><a class="selflink" id="section-3.6" href="#section-3.6">3.6</a>. Instantiating the EOSL Model with IP</span> |
| 627 |
|
| 628 |
While IP is being proposed as a transport for almost everything, the |
| 629 |
base assumption, that Everything over IP (EOIP) will result in OPEX |
| 630 |
and CAPEX efficiencies, requires critical examination. In |
| 631 |
particular, while it is the case that many protocols can be |
| 632 |
efficiently transported over an IP network (specifically, those |
| 633 |
protocols that do not need to recover synchronization between the |
| 634 |
communication end points, such as Frame Relay, Ethernet, and AAL5 |
| 635 |
ATM), the Simplicity and Layering Principles suggest that EOIP may |
| 636 |
not represent the most efficient convergence strategy for arbitrary |
| 637 |
services. Rather, a more CAPEX and OPEX efficient convergence layer |
| 638 |
might be much lower (again, this behavior is predicted by the |
| 639 |
Simplicity Principle). |
| 640 |
|
| 641 |
An example of where EOIP would not be the most OPEX and CAPEX |
| 642 |
efficient transport would be in those cases where a service or |
| 643 |
protocol needed SONET-like restoration times (e.g., 50ms). It is not |
| 644 |
hard to imagine that it would cost more to build and operate an IP |
| 645 |
network with this kind of restoration and convergence property (if |
| 646 |
that were even possible) than it would to build the SONET network in |
| 647 |
the first place. |
| 648 |
|
| 649 |
<span class="h2"><a class="selflink" id="section-4" href="#section-4">4</a>. Avoid the Universal Interworking Function</span> |
| 650 |
|
| 651 |
While there have been many implementations of Universal Interworking |
| 652 |
unction (UIWF), IWF approaches have been problematic at large scale. |
| 653 |
his concern is codified in the Principle of Minimum Intervention |
| 654 |
BRYANT]: |
| 655 |
|
| 656 |
"To minimise the scope of information, and to improve the efficiency |
| 657 |
of data flow through the Encapsulation Layer, the payload should, |
| 658 |
where possible, be transported as received without modification." |
| 659 |
|
| 660 |
|
| 661 |
|
| 662 |
|
| 663 |
|
| 664 |
|
| 665 |
|
| 666 |
|
| 667 |
|
| 668 |
<span class="grey">Bush, et. al. Informational [Page 12]</span></pre> |
| 669 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-13" ></span> |
| 670 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 671 |
|
| 672 |
|
| 673 |
<span class="h3"><a class="selflink" id="section-4.1" href="#section-4.1">4.1</a>. Avoid Control Plane Interworking</span> |
| 674 |
|
| 675 |
This corollary is best understood in the context of the integrated |
| 676 |
solutions space. In this case, the architecture and design |
| 677 |
frequently achieves the worst of all possible worlds. This is due to |
| 678 |
the fact that such integrated solutions perform poorly at both ends |
| 679 |
of the performance/CAPEX/OPEX spectrum: the protocols with the least |
| 680 |
switching demand may have to bear the cost of the most expensive, |
| 681 |
while the protocols with the most stringent requirements often must |
| 682 |
make concessions to those with different requirements. Add to this |
| 683 |
the various control plane interworking issues and you have a large |
| 684 |
opportunity for failure. In summary, interworking functions should |
| 685 |
be restricted to data plane interworking and encapsulations, and |
| 686 |
these functions should be carried out at the edge of the network. |
| 687 |
|
| 688 |
As described above, interworking models have been successful in those |
| 689 |
cases where there is a kind of "isomorphism" between the layers being |
| 690 |
interworked. The trade-off here, frequently described as the |
| 691 |
"Integrated vs. Ships In the Night trade-off" has been examined at |
| 692 |
various times and at various protocol layers. In general, there are |
| 693 |
few cases in which such integrated solutions have proven efficient. |
| 694 |
Multi-protocol BGP [<a href="./rfc2283" title=""Multiprotocol Extensions for BGP4"">RFC2283</a>] is a subtly different but notable |
| 695 |
exception. In this case, the control plane is independent of the |
| 696 |
format of the control data. That is, no control plane data |
| 697 |
conversion is required, in contrast with control plane interworking |
| 698 |
models such as the ATM/IP interworking envisioned by some soft-switch |
| 699 |
manufacturers, and the so-called "PNNI-MPLS SIN" interworking |
| 700 |
[<a href="#ref-ATMMPLS" title=""ATM-MPLS Interworking Migration Complexities Issues and Preliminary Assessment"">ATMMPLS</a>]. |
| 701 |
|
| 702 |
<span class="h2"><a class="selflink" id="section-5" href="#section-5">5</a>. Packet versus Circuit Switching: Fundamental Differences</span> |
| 703 |
|
| 704 |
Conventional wisdom holds that packet switching (PS) is inherently |
| 705 |
more efficient than circuit switching (CS), primarily because of the |
| 706 |
efficiencies that can be gained by statistical multiplexing and the |
| 707 |
fact that routing and forwarding decisions are made independently in |
| 708 |
a hop-by-hop fashion [[<a href="#ref-MOLINERO2002" title=""TCP Switching: Exposing Circuits to IP"">MOLINERO2002</a>]. Further, it is widely assumed |
| 709 |
that IP is simpler that circuit switching, and hence should be more |
| 710 |
economical to deploy and manage [<a href="#ref-MCK2002" title="April">MCK2002</a>]. However, if one examines |
| 711 |
these and related assumptions, a different picture emerges (see for |
| 712 |
example [<a href="#ref-ODLYZKO98" title=""The economics of the Internet: Utility, utilization, pricing, and Quality of Service"">ODLYZKO98</a>]). The following sections discuss these |
| 713 |
assumptions. |
| 714 |
|
| 715 |
<span class="h3"><a class="selflink" id="section-5.1" href="#section-5.1">5.1</a>. Is PS is inherently more efficient than CS?</span> |
| 716 |
|
| 717 |
It is well known that packet switches make efficient use of scarce |
| 718 |
bandwidth [<a href="#ref-BARAN" title=""On Distributed Communications"">BARAN</a>]. This efficiency is based on the statistical |
| 719 |
multiplexing inherent in packet switching. However, we continue to |
| 720 |
be puzzled by what is generally believed to be the low utilization of |
| 721 |
|
| 722 |
|
| 723 |
|
| 724 |
<span class="grey">Bush, et. al. Informational [Page 13]</span></pre> |
| 725 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-14" ></span> |
| 726 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 727 |
|
| 728 |
|
| 729 |
Internet backbones. The first question we might ask is what is the |
| 730 |
current average utilization of Internet backbones, and how does that |
| 731 |
relate to the utilization of long distance voice networks? Odlyzko |
| 732 |
and Coffman [<a href="#ref-ODLYZKO" title=""Data networks are mostly empty for good reason"">ODLYZKO</a>,<a href="#ref-COFFMAN" title=""Internet Growth: Is there a 'Moores Law' for Data Traffic"">COFFMAN</a>] report that the average utilization of |
| 733 |
links in the IP networks was in the range between 3% and 20% |
| 734 |
(corporate intranets run in the 3% range, while commercial Internet |
| 735 |
backbones run in the 15-20% range). On the other hand, the average |
| 736 |
utilization of long haul voice lines is about 33%. In addition, for |
| 737 |
2002, the average utilization of optical networks (all services) |
| 738 |
appears to be hovering at about 11%, while the historical average is |
| 739 |
approximately 15% [<a href="#ref-ML2002" title=""Optical Systems"">ML2002</a>]. The question then becomes why we see |
| 740 |
such utilization levels, especially in light of the assumption that |
| 741 |
PS is inherently more efficient than CS. The reasons cited by |
| 742 |
Odlyzko and Coffman include: |
| 743 |
|
| 744 |
(i). Internet traffic is extremely asymmetric and bursty, but |
| 745 |
links are symmetric and of fixed capacity (i.e., don't know |
| 746 |
the traffic matrix, or required link capacities); |
| 747 |
|
| 748 |
(ii). It is difficult to predict traffic growth on a link, so |
| 749 |
operators tend to add bandwidth aggressively; |
| 750 |
|
| 751 |
(iii). Falling prices for coarser bandwidth granularity make it |
| 752 |
appear more economical to add capacity in large increments. |
| 753 |
|
| 754 |
Other static factors include protocol overhead, other kinds of |
| 755 |
equipment granularity, restoration capacity, and provisioning lag |
| 756 |
time all contribute to the need to "over-provision" [<a href="#ref-MC2001" title=""U.S Communications Infrastructure at A Crossroads: Opportunities Amid the Gloom"">MC2001</a>]. |
| 757 |
|
| 758 |
<span class="h3"><a class="selflink" id="section-5.2" href="#section-5.2">5.2</a>. Is PS simpler than CS?</span> |
| 759 |
|
| 760 |
The end-to-end principle can be interpreted as stating that the |
| 761 |
complexity of the Internet belongs at the edges. However, today's |
| 762 |
Internet backbone routers are extremely complex. Further, this |
| 763 |
complexity scales with line rate. Since the relative complexity of |
| 764 |
circuit and packet switching seems to have resisted direct analysis, |
| 765 |
we instead examine several artifacts of packet and circuit switching |
| 766 |
as complexity metrics. Among the metrics we might look at are |
| 767 |
software complexity, macro operation complexity, hardware complexity, |
| 768 |
power consumption, and density. Each of these metrics is considered |
| 769 |
below. |
| 770 |
|
| 771 |
|
| 772 |
|
| 773 |
|
| 774 |
|
| 775 |
|
| 776 |
|
| 777 |
|
| 778 |
|
| 779 |
|
| 780 |
<span class="grey">Bush, et. al. Informational [Page 14]</span></pre> |
| 781 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-15" ></span> |
| 782 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 783 |
|
| 784 |
|
| 785 |
<span class="h4"><a class="selflink" id="section-5.2.1" href="#section-5.2.1">5.2.1</a>. Software/Firmware Complexity</span> |
| 786 |
|
| 787 |
One measure of software/firmware complexity is the number of |
| 788 |
instructions required to program the device. The typical software |
| 789 |
image for an Internet router requires between eight and ten million |
| 790 |
instructions (including firmware), whereas a typical transport switch |
| 791 |
requires on average about three million instructions [<a href="#ref-MCK2002" title="April">MCK2002</a>]. |
| 792 |
|
| 793 |
This difference in software complexity has tended to make Internet |
| 794 |
routers unreliable, and has notable other second order effects (e.g., |
| 795 |
it may take a long time to reboot such a router). As another point |
| 796 |
of comparison, consider that the AT&T (Lucent) 5ESS class 5 switch, |
| 797 |
which has a huge number of calling features, requires only about |
| 798 |
twice the number of lines of code as an Internet core router [<a href="#ref-EICK" title=""Visualizing Software Changes"">EICK</a>]. |
| 799 |
|
| 800 |
Finally, since routers are as much or more software than hardware |
| 801 |
devices, another result of the code complexity is that the cost of |
| 802 |
routers benefits less from Moore's Law than less software-intensive |
| 803 |
devices. This causes a bandwidth/device trade-off that favors |
| 804 |
bandwidth more than less software-intensive devices. |
| 805 |
|
| 806 |
<span class="h4"><a class="selflink" id="section-5.2.2" href="#section-5.2.2">5.2.2</a>. Macro Operation Complexity</span> |
| 807 |
|
| 808 |
An Internet router's line card must perform many complex operations, |
| 809 |
including processing the packet header, longest prefix match, |
| 810 |
generating ICMP error messages, processing IP header options, and |
| 811 |
buffering the packet so that TCP congestion control will be effective |
| 812 |
(this typically requires a buffer of size proportional to the line |
| 813 |
rate times the RTT, so a buffer will hold around 250 ms of packet |
| 814 |
data). This doesn't include route and packet filtering, or any QoS |
| 815 |
or VPN filtering. |
| 816 |
|
| 817 |
On the other hand, a transport switch need only to map ingress time- |
| 818 |
slots to egress time-slots and interfaces, and therefore can be |
| 819 |
considerably less complex. |
| 820 |
|
| 821 |
<span class="h4"><a class="selflink" id="section-5.2.3" href="#section-5.2.3">5.2.3</a>. Hardware Complexity</span> |
| 822 |
|
| 823 |
One measure of hardware complexity is the number of logic gates on a |
| 824 |
line card [<a href="#ref-MOLINERO2002" title=""TCP Switching: Exposing Circuits to IP"">MOLINERO2002</a>]. Consider the case of a high-speed Internet |
| 825 |
router line card: An OC192 POS router line card contains at least 30 |
| 826 |
million gates in ASICs, at least one CPU, 300 Mbytes of packet |
| 827 |
buffers, 2 Mbytes of forwarding table, and 10 Mbytes of other |
| 828 |
|
| 829 |
|
| 830 |
|
| 831 |
|
| 832 |
|
| 833 |
|
| 834 |
|
| 835 |
|
| 836 |
<span class="grey">Bush, et. al. Informational [Page 15]</span></pre> |
| 837 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-16" ></span> |
| 838 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 839 |
|
| 840 |
|
| 841 |
state memory. On the other hand, a comparable transport switch line |
| 842 |
card has 7.5 million logic gates, no CPU, no packet buffer, no |
| 843 |
forwarding table, and an on-chip state memory. Rather, the line-card |
| 844 |
of an electronic transport switch typically contains a SONET framer, |
| 845 |
a chip to map ingress time-slots to egress time-slots, and an |
| 846 |
interface to the switch fabric. |
| 847 |
|
| 848 |
<span class="h4"><a class="selflink" id="section-5.2.4" href="#section-5.2.4">5.2.4</a>. Power</span> |
| 849 |
|
| 850 |
Since transport switches have traditionally been built from simpler |
| 851 |
hardware components, they also consume less power [<a href="#ref-PMC" title=""The Design of a 10 Gigabit Core Router Architecture"">PMC</a>]. |
| 852 |
|
| 853 |
<span class="h4"><a class="selflink" id="section-5.2.5" href="#section-5.2.5">5.2.5</a>. Density</span> |
| 854 |
|
| 855 |
The highest capacity transport switches have about four times the |
| 856 |
capacity of an IP router [<a href="#ref-CISCO">CISCO</a>,<a href="#ref-CIENA" title=""CIENA Multiwave CoreDiretor"">CIENA</a>], and sell for about one-third |
| 857 |
as much per Gigabit/sec. Optical (OOO) technology pushes this |
| 858 |
complexity difference further (e.g., tunable lasers, MEMs switches. |
| 859 |
e.g., [CALIENT]), and DWDM multiplexers provide technology to build |
| 860 |
extremely high capacity, low power transport switches. |
| 861 |
|
| 862 |
A related metric is physical footprint. In general, by virtue of |
| 863 |
their higher density, transport switches have a smaller "per-gigabit" |
| 864 |
physical footprint. |
| 865 |
|
| 866 |
<span class="h4"><a class="selflink" id="section-5.2.6" href="#section-5.2.6">5.2.6</a>. Fixed versus variable costs</span> |
| 867 |
|
| 868 |
Packet switching would seem to have high variable cost, meaning that |
| 869 |
it costs more to send the n-th piece of information using packet |
| 870 |
switching than it might in a circuit switched network. Much of this |
| 871 |
advantage is due to the relatively static nature of circuit |
| 872 |
switching, e.g., circuit switching can take advantage of of pre- |
| 873 |
scheduled arrival of information to eliminate operations to be |
| 874 |
performed on incoming information. For example, in the circuit |
| 875 |
switched case, there is no need to buffer incoming information, |
| 876 |
perform loop detection, resolve next hops, modify fields in the |
| 877 |
packet header, and the like. Finally, many circuit switched networks |
| 878 |
combine relatively static configuration with out-of-band control |
| 879 |
planes (e.g., SS7), which greatly simplifies data-plane switching. |
| 880 |
The bottom line is that as data rates get large, it becomes more and |
| 881 |
more complex to switch packets, while circuit switching scales more |
| 882 |
or less linearly. |
| 883 |
|
| 884 |
|
| 885 |
|
| 886 |
|
| 887 |
|
| 888 |
|
| 889 |
|
| 890 |
|
| 891 |
|
| 892 |
<span class="grey">Bush, et. al. Informational [Page 16]</span></pre> |
| 893 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-17" ></span> |
| 894 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 895 |
|
| 896 |
|
| 897 |
<span class="h4"><a class="selflink" id="section-5.2.7" href="#section-5.2.7">5.2.7</a>. QoS</span> |
| 898 |
|
| 899 |
While the components of a complete solution for Internet QoS, |
| 900 |
including call admission control, efficient packet classification, |
| 901 |
and scheduling algorithms, have been the subject of extensive |
| 902 |
research and standardization for more than 10 years, end-to-end |
| 903 |
signaled QoS for the Internet has not become a reality. |
| 904 |
Alternatively, QoS has been part of the circuit switched |
| 905 |
infrastructure almost from its inception. On the other hand, QoS is |
| 906 |
usually deployed to determine queuing disciplines to be used when |
| 907 |
there is insufficient bandwidth to support traffic. But unlike voice |
| 908 |
traffic, packet drop or severe delay may have a much more serious |
| 909 |
effect on TCP traffic due to its congestion-aware feedback loop (in |
| 910 |
particular, TCP backoff/slow start). |
| 911 |
|
| 912 |
<span class="h4"><a class="selflink" id="section-5.2.8" href="#section-5.2.8">5.2.8</a>. Flexibility</span> |
| 913 |
|
| 914 |
A somewhat harder to quantify metric is the inherent flexibility of |
| 915 |
the Internet. While the Internet's flexibility has led to its rapid |
| 916 |
growth, this flexibility comes with a relatively high cost at the |
| 917 |
edge: the need for highly trained support personnel. A standard rule |
| 918 |
of thumb is that in an enterprise setting, a single support person |
| 919 |
suffices to provide telephone service for a group, while you need ten |
| 920 |
computer networking experts to serve the networking requirements of |
| 921 |
the same group [<a href="#ref-ODLYZKO98A" title=""Smart and stupid networks: Why the Internet is like Microsoft"">ODLYZKO98A</a>]. This phenomenon is also described in |
| 922 |
[<a href="#ref-PERROW" title=""Normal Accidents: Living with High Risk Technologies"">PERROW</a>]. |
| 923 |
|
| 924 |
<span class="h3"><a class="selflink" id="section-5.3" href="#section-5.3">5.3</a>. Relative Complexity</span> |
| 925 |
|
| 926 |
The relative computational complexity of circuit switching as |
| 927 |
compared to packet switching has been difficult to describe in formal |
| 928 |
terms [<a href="#ref-PARK" title=""The Internet as a Complex System: Scaling, Complexity and Control"">PARK</a>]. As such, the sections above seek to describe the |
| 929 |
complexity in terms of observable artifacts. With this in mind, it |
| 930 |
is clear that the fundamental driver producing the increased |
| 931 |
complexities outlined above is the hop-by-hop independence (HBHI) |
| 932 |
inherent in the IP architecture. This is in contrast to the end to |
| 933 |
end architectures such as ATM or Frame Relay. |
| 934 |
|
| 935 |
[<a id="ref-WILLINGER2002">WILLINGER2002</a>] describes this phenomenon in terms of the robustness |
| 936 |
requirement of the original Internet design, and how this requirement |
| 937 |
has the driven complexity of the network. In particular, they |
| 938 |
describe a "complexity/robustness" spiral, in which increases in |
| 939 |
complexity create further and more serious sensitivities, which then |
| 940 |
requires additional robustness (hence the spiral). |
| 941 |
|
| 942 |
|
| 943 |
|
| 944 |
|
| 945 |
|
| 946 |
|
| 947 |
|
| 948 |
<span class="grey">Bush, et. al. Informational [Page 17]</span></pre> |
| 949 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-18" ></span> |
| 950 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 951 |
|
| 952 |
|
| 953 |
The important lesson of this section is that the Simplicity |
| 954 |
Principle, while applicable to circuit switching as well as packet |
| 955 |
switching, is crucial in controlling the complexity (and hence OPEX |
| 956 |
and CAPEX properties) of packet networks. This idea is reinforced by |
| 957 |
the observation that while packet switching is a younger, less mature |
| 958 |
discipline than circuit switching, the trend in packet switches is |
| 959 |
toward more complex line cards, while the complexity of circuit |
| 960 |
switches appears to be scaling linearly with line rates and aggregate |
| 961 |
capacity. |
| 962 |
|
| 963 |
<span class="h4"><a class="selflink" id="section-5.3.1" href="#section-5.3.1">5.3.1</a>. HBHI and the OPEX Challenge</span> |
| 964 |
|
| 965 |
As a result of HBHI, we need to approach IP networks in a |
| 966 |
fundamentally different way than we do circuit based networks. In |
| 967 |
particular, the major OPEX challenge faced by the IP network is that |
| 968 |
debugging of a large-scale IP network still requires a large degree |
| 969 |
of expertise and understanding, again due to the hop-by-hop |
| 970 |
independence inherent in a packet architecture (again, note that this |
| 971 |
hop-by-hop independence is not present in virtual circuit networks |
| 972 |
such as ATM or Frame Relay). For example, you may have to visit a |
| 973 |
large set of your routers only to discover that the problem is |
| 974 |
external to your own network. Further, the debugging tools used to |
| 975 |
diagnose problems are also complex and somewhat primitive. Finally, |
| 976 |
IP has to deal with people having problems with their DNS or their |
| 977 |
mail or news or some new application, whereas this is usually not the |
| 978 |
case for TDM/ATM/etc. In the case of IP, this can be eased by |
| 979 |
improving automation (note that much of what we mention is customer |
| 980 |
facing). In general, there are many variables external to the |
| 981 |
network that effect OPEX. |
| 982 |
|
| 983 |
Finally, it is important to note that the quantitative relationship |
| 984 |
between CAPEX, OPEX, and a network's inherent complexity is not well |
| 985 |
understood. In fact, there are no agreed upon and quantitative |
| 986 |
metrics for describing a network's complexity, so a precise |
| 987 |
relationship between CAPEX, OPEX, and complexity remains elusive. |
| 988 |
|
| 989 |
<span class="h2"><a class="selflink" id="section-6" href="#section-6">6</a>. The Myth of Over-Provisioning</span> |
| 990 |
|
| 991 |
As noted in [<a href="#ref-MC2001" title=""U.S Communications Infrastructure at A Crossroads: Opportunities Amid the Gloom"">MC2001</a>] and elsewhere, much of the complexity we observe |
| 992 |
in today's Internet is directed at increasing bandwidth utilization. |
| 993 |
As a result, the desire of network engineers to keep network |
| 994 |
utilization below 50% has been termed "over-provisioning". However, |
| 995 |
this use of the term over-provisioning is a misnomer. Rather, in |
| 996 |
modern Internet backbones the unused capacity is actually protection |
| 997 |
capacity. In particular, one might view this as "1:1 protection at |
| 998 |
the IP layer". Viewed in this way, we see that an IP network |
| 999 |
provisioned to run at 50% utilization is no more over-provisioned |
| 1000 |
than the typical SONET network. However, the important advantages |
| 1001 |
|
| 1002 |
|
| 1003 |
|
| 1004 |
<span class="grey">Bush, et. al. Informational [Page 18]</span></pre> |
| 1005 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-19" ></span> |
| 1006 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 1007 |
|
| 1008 |
|
| 1009 |
that accrue to an IP network provisioned in this way include close to |
| 1010 |
speed of light delay and close to zero packet loss [<a href="#ref-FRALEIGH" title=""Provisioning IP Backbone Networks to Support Delay- Based Service Level Agreements"">FRALEIGH</a>]. These |
| 1011 |
benefits can been seen as a "side-effect" of 1:1 protection |
| 1012 |
provisioning. |
| 1013 |
|
| 1014 |
There are also other, system-theoretic reasons for providing 1:1-like |
| 1015 |
protection provisioning. Most notable among these reasons is that |
| 1016 |
packet-switched networks with in-band control loops can become |
| 1017 |
unstable and can experience oscillations and synchronization when |
| 1018 |
congested. Complex and non-linear dynamic interaction of traffic |
| 1019 |
means that congestion in one part of the network will spread to other |
| 1020 |
parts of the network. When routing protocol packets are lost due to |
| 1021 |
congestion or route-processor overload, it causes inconsistent |
| 1022 |
routing state, and this may result in traffic loops, black holes, and |
| 1023 |
lost connectivity. Thus, while statistical multiplexing can in |
| 1024 |
theory yield higher network utilization, in practice, to maintain |
| 1025 |
consistent performance and a reasonably stable network, the dynamics |
| 1026 |
of the Internet backbones favor 1:1 provisioning and its side effects |
| 1027 |
to keep the network stable and delay low. |
| 1028 |
|
| 1029 |
<span class="h2"><a class="selflink" id="section-7" href="#section-7">7</a>. The Myth of Five Nines</span> |
| 1030 |
|
| 1031 |
Paul Baran, in his classic paper, "SOME PERSPECTIVES ON NETWORKS-- |
| 1032 |
PAST, PRESENT AND FUTURE", stated that "The tradeoff curves between |
| 1033 |
cost and system reliability suggest that the most reliable systems |
| 1034 |
might be built of relatively unreliable and hence low cost elements, |
| 1035 |
if it is system reliability at the lowest overall system cost that is |
| 1036 |
at issue" [<a href="#ref-BARAN77" title=""SOME PERSPECTIVES ON NETWORKS--PAST, PRESENT AND FUTURE"">BARAN77</a>]. |
| 1037 |
|
| 1038 |
Today we refer to this phenomenon as "the myth of five nines". |
| 1039 |
Specifically, so-called five nines reliability in packet network |
| 1040 |
elements is consider a myth for the following reasons: First, since |
| 1041 |
80% of unscheduled outages are caused by people or process errors |
| 1042 |
[<a href="#ref-SCOTT" title=""Making Smart Investments to Reduce Unplanned Downtime"">SCOTT</a>], there is only a 20% window in which to optimize. Thus, in |
| 1043 |
order to increase component reliability, we add complexity |
| 1044 |
(optimization frequently leads to complexity), which is the root |
| 1045 |
cause of 80% of the unplanned outages. This effectively narrows the |
| 1046 |
20% window (i.e., you increase the likelihood of people and process |
| 1047 |
failure). This phenomenon is also characterized as a |
| 1048 |
"complexity/robustness" spiral [<a href="#ref-WILLINGER2002" title=""Robustness and the Internet: Design and evolution"">WILLINGER2002</a>], in which increases in |
| 1049 |
complexity create further and more serious sensitivities, which then |
| 1050 |
requires additional robustness, and so on (hence the spiral). |
| 1051 |
|
| 1052 |
The conclusion, then is that while a system like the Internet can |
| 1053 |
reach five-nines-like reliability, it is undesirable (and likely |
| 1054 |
impossible) to try to make any individual component, especially the |
| 1055 |
most complex ones, reach that reliability standard. |
| 1056 |
|
| 1057 |
|
| 1058 |
|
| 1059 |
|
| 1060 |
<span class="grey">Bush, et. al. Informational [Page 19]</span></pre> |
| 1061 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-20" ></span> |
| 1062 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 1063 |
|
| 1064 |
|
| 1065 |
<span class="h2"><a class="selflink" id="section-8" href="#section-8">8</a>. Architectural Component Proportionality Law</span> |
| 1066 |
|
| 1067 |
As noted in the previous section, the computational complexity of |
| 1068 |
packet switched networks such as the Internet has proven difficult to |
| 1069 |
describe in formal terms. However, an intuitive, high level |
| 1070 |
definition of architectural complexity might be that the complexity |
| 1071 |
of an architecture is proportional to its number of components, and |
| 1072 |
that the probability of achieving a stable implementation of an |
| 1073 |
architecture is inversely proportional to its number of components. |
| 1074 |
As described above, components include discrete elements such as |
| 1075 |
hardware elements, space and power requirements, as well as software, |
| 1076 |
firmware, and the protocols they implement. |
| 1077 |
|
| 1078 |
Stated more abstractly: |
| 1079 |
|
| 1080 |
Let |
| 1081 |
|
| 1082 |
A be a representation of architecture A, |
| 1083 |
|
| 1084 |
|A| be number of distinct components in the service |
| 1085 |
delivery path of architecture A, |
| 1086 |
|
| 1087 |
w be a monotonically increasing function, |
| 1088 |
|
| 1089 |
P be the probability of a stable implementation of an |
| 1090 |
architecture, and let |
| 1091 |
|
| 1092 |
Then |
| 1093 |
|
| 1094 |
Complexity(A) = O(w(|A|)) |
| 1095 |
P(A) = O(1/w(|A|)) |
| 1096 |
|
| 1097 |
where |
| 1098 |
|
| 1099 |
O(f) = {g:N->R | there exists c > 0 and n such that g(n) |
| 1100 |
< c*f(n)} |
| 1101 |
|
| 1102 |
[That is, O(f) comprises the set of functions g for which |
| 1103 |
there exists a constant c and a number n, such that g(n) is |
| 1104 |
smaller or equal to c*f(n) for all n. That is, O(f) is the |
| 1105 |
set of all functions that do not grow faster than f, |
| 1106 |
disregarding constant factors] |
| 1107 |
|
| 1108 |
Interestingly, the Highly Optimized Tolerance (HOT) model [<a href="#ref-HOT" title="Phys. Rev. E 60">HOT</a>] |
| 1109 |
attempts to characterize complexity in general terms (HOT is one |
| 1110 |
recent attempt to develop a general framework for the study of |
| 1111 |
complexity, and is a member of a family of abstractions generally |
| 1112 |
termed "the new science of complexity" or "complex adaptive |
| 1113 |
|
| 1114 |
|
| 1115 |
|
| 1116 |
<span class="grey">Bush, et. al. Informational [Page 20]</span></pre> |
| 1117 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-21" ></span> |
| 1118 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 1119 |
|
| 1120 |
|
| 1121 |
systems"). Tolerance, in HOT semantics, means that "robustness in |
| 1122 |
complex systems is a constrained and limited quantity that must be |
| 1123 |
carefully managed and protected." One focus of the HOT model is to |
| 1124 |
characterize heavy-tailed distributions such as Complexity(A) in the |
| 1125 |
above example (other examples include forest fires, power outages, |
| 1126 |
and Internet traffic distributions). In particular, Complexity(A) |
| 1127 |
attempts to map the extreme heterogeneity of the parts of the system |
| 1128 |
(Internet), and the effect of their organization into highly |
| 1129 |
structured networks, with hierarchies and multiple scales. |
| 1130 |
|
| 1131 |
<span class="h3"><a class="selflink" id="section-8.1" href="#section-8.1">8.1</a>. Service Delivery Paths</span> |
| 1132 |
|
| 1133 |
The Architectural Component Proportionality Law (ACPL) states that |
| 1134 |
the complexity of an architecture is proportional to its number of |
| 1135 |
components. |
| 1136 |
|
| 1137 |
COROLLARY: Minimize the number of components in a service delivery |
| 1138 |
path, where the service delivery path can be a protocol path, a |
| 1139 |
software path, or a physical path. |
| 1140 |
|
| 1141 |
This corollary is an important consequence of the ACPL, as the path |
| 1142 |
between a customer and the desired service is particularly sensitive |
| 1143 |
to the number and complexity of elements in the path. This is due to |
| 1144 |
the fact that the complexity "smoothing" that we find at high levels |
| 1145 |
of aggregation [<a href="#ref-ZHANG" title=""Impact of Aggregation on Scaling Behavior of Internet Backbone Traffic"">ZHANG</a>] is missing as you move closer to the edge, as |
| 1146 |
well as having complex interactions with backoffice and CRM systems. |
| 1147 |
Examples of architectures that haven't found a market due to this |
| 1148 |
effect include TINA-based CRM systems, CORBA/TINA based service |
| 1149 |
architectures. The basic lesson here was that the only possibilities |
| 1150 |
for deploying these systems were "Limited scale deployments (such) as |
| 1151 |
in Starvision can avoid coping with major unproven scalability |
| 1152 |
issues", or "Otherwise need massive investments (like the carrier- |
| 1153 |
grade ORB built almost from scratch)" [<a href="#ref-TINA" title=""What is TINA and is it useful for the TelCos?"">TINA</a>]. In other words, these |
| 1154 |
systems had complex service delivery paths, and were too complex to |
| 1155 |
be feasibly deployed. |
| 1156 |
|
| 1157 |
<span class="h2"><a class="selflink" id="section-9" href="#section-9">9</a>. Conclusions</span> |
| 1158 |
|
| 1159 |
This document attempts to codify long-understood Internet |
| 1160 |
architectural principles. In particular, the unifying principle |
| 1161 |
described here is best expressed by the Simplicity Principle, which |
| 1162 |
states complexity must be controlled if one hopes to efficiently |
| 1163 |
scale a complex object. The idea that simplicity itself can lead to |
| 1164 |
some form of optimality has been a common theme throughout history, |
| 1165 |
and has been stated in many other ways and along many dimensions. |
| 1166 |
For example, consider the maxim known as Occam's Razor, which was |
| 1167 |
formulated by the medieval English philosopher and Franciscan monk |
| 1168 |
William of Ockham (ca. 1285-1349), and states "Pluralitas non est |
| 1169 |
|
| 1170 |
|
| 1171 |
|
| 1172 |
<span class="grey">Bush, et. al. Informational [Page 21]</span></pre> |
| 1173 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-22" ></span> |
| 1174 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 1175 |
|
| 1176 |
|
| 1177 |
ponenda sine neccesitate" or "plurality should not be posited without |
| 1178 |
necessity." (hence Occam's Razor is sometimes called "the principle |
| 1179 |
of unnecessary plurality" and " the principle of simplicity"). A |
| 1180 |
perhaps more contemporary formulation of Occam's Razor states that |
| 1181 |
the simplest explanation for a phenomenon is the one preferred by |
| 1182 |
nature. Other formulations of the same idea can be found in the |
| 1183 |
KISS (Keep It Simple Stupid) principle and the Principle of Least |
| 1184 |
Astonishment (the assertion that the most usable system is the one |
| 1185 |
that least often leaves users astonished). [<a href="#ref-WILLINGER2002" title=""Robustness and the Internet: Design and evolution"">WILLINGER2002</a>] provides |
| 1186 |
a more theoretical discussion of "robustness through simplicity", and |
| 1187 |
in discussing the PSTN, [<a href="#ref-KUHN87" title=""Sources of Failure in the Public Switched Telephone Network"">KUHN87</a>] states that in most systems, "a |
| 1188 |
trade-off can be made between simplicity of interactions and |
| 1189 |
looseness of coupling". |
| 1190 |
|
| 1191 |
When applied to packet switched network architectures, the Simplicity |
| 1192 |
Principle has implications that some may consider heresy, e.g., that |
| 1193 |
highly converged approaches are likely to be less efficient than |
| 1194 |
"less converged" solutions. Otherwise stated, the "optimal" |
| 1195 |
convergence layer may be much lower in the protocol stack that is |
| 1196 |
conventionally believed. In addition, the analysis above leads to |
| 1197 |
several conclusions that are contrary to the conventional wisdom |
| 1198 |
surrounding packet networking. Perhaps most significant is the |
| 1199 |
belief that packet switching is simpler than circuit switching. This |
| 1200 |
belief has lead to conclusions such as "since packet is simpler than |
| 1201 |
circuit, it must cost less to operate". This study finds to the |
| 1202 |
contrary. In particular, by examining the metrics described above, |
| 1203 |
we find that packet switching is more complex than circuit switching. |
| 1204 |
Interestingly, this conclusion is borne out by the fact that |
| 1205 |
normalized OPEX for data networks is typically significantly greater |
| 1206 |
than for voice networks [<a href="#ref-ML2002" title=""Optical Systems"">ML2002</a>]. |
| 1207 |
|
| 1208 |
Finally, the important conclusion of this work is that for packet |
| 1209 |
networks that are of the scale of today's Internet or larger, we must |
| 1210 |
strive for the simplest possible solutions if we hope to build cost |
| 1211 |
effective infrastructures. This idea is eloquently stated in |
| 1212 |
[<a href="#ref-DOYLE2002" title=""Robustness and the Internet: Theoretical Foundations"">DOYLE2002</a>]: "The evolution of protocols can lead to a |
| 1213 |
robustness/complexity/fragility spiral where complexity added for |
| 1214 |
robustness also adds new fragilities, which in turn leads to new and |
| 1215 |
thus spiraling complexities". This is exactly the phenomenon that |
| 1216 |
the Simplicity Principle is designed to avoid. |
| 1217 |
|
| 1218 |
<span class="h2"><a class="selflink" id="section-10" href="#section-10">10</a>. Security Considerations</span> |
| 1219 |
|
| 1220 |
This document does not directly effect the security of any existing |
| 1221 |
Internet protocol. However, adherence to the Simplicity Principle |
| 1222 |
does have a direct affect on our ability to implement secure systems. |
| 1223 |
In particular, a system's complexity grows, it becomes more |
| 1224 |
difficult to model and analyze, and hence it becomes more difficult |
| 1225 |
|
| 1226 |
|
| 1227 |
|
| 1228 |
<span class="grey">Bush, et. al. Informational [Page 22]</span></pre> |
| 1229 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-23" ></span> |
| 1230 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 1231 |
|
| 1232 |
|
| 1233 |
to find and understand the security implications inherent in its |
| 1234 |
architecture, design, and implementation. |
| 1235 |
|
| 1236 |
<span class="h2"><a class="selflink" id="section-11" href="#section-11">11</a>. Acknowledgments</span> |
| 1237 |
|
| 1238 |
Many of the ideas for comparing the complexity of circuit switched |
| 1239 |
and packet switched networks were inspired by conversations with Nick |
| 1240 |
McKeown. Scott Bradner, David Banister, Steve Bellovin, Steward |
| 1241 |
Bryant, Christophe Diot, Susan Harris, Ananth Nagarajan, Andrew |
| 1242 |
Odlyzko, Pete and Natalie Whiting, and Lixia Zhang made many helpful |
| 1243 |
comments on early drafts of this document. |
| 1244 |
|
| 1245 |
<span class="h2"><a class="selflink" id="section-12" href="#section-12">12</a>. References</span> |
| 1246 |
|
| 1247 |
[<a id="ref-AHUJA">AHUJA</a>] "The Impact of Internet Policy and Topology on |
| 1248 |
Delayed Routing Convergence", Labovitz, et. al. |
| 1249 |
Infocom, 2001. |
| 1250 |
|
| 1251 |
[<a id="ref-ATMMPLS">ATMMPLS</a>] "ATM-MPLS Interworking Migration Complexities Issues |
| 1252 |
and Preliminary Assessment", School of |
| 1253 |
Interdisciplinary Computing and Engineering, |
| 1254 |
University of Missouri-Kansas City, April 2002 |
| 1255 |
|
| 1256 |
[<a id="ref-BARAN">BARAN</a>] "On Distributed Communications", Paul Baran, Rand |
| 1257 |
Corporation Memorandum RM-3420-PR, |
| 1258 |
<a href="http://www.rand.org/publications/RM/RM3420">http://www.rand.org/publications/RM/RM3420</a>", August, |
| 1259 |
1964. |
| 1260 |
|
| 1261 |
[<a id="ref-BARAN77">BARAN77</a>] "SOME PERSPECTIVES ON NETWORKS--PAST, PRESENT AND |
| 1262 |
FUTURE", Paul Baran, Information Processing 77, |
| 1263 |
North-Holland Publishing Company, 1977, |
| 1264 |
|
| 1265 |
[<a id="ref-BRYANT">BRYANT</a>] "Protocol Layering in PWE3", Bryant et al, Work in |
| 1266 |
Progress. |
| 1267 |
|
| 1268 |
[<a id="ref-CAIDA">CAIDA</a>] <a href="http://www.caida.org">http://www.caida.org</a> |
| 1269 |
|
| 1270 |
[<a id="ref-CALLIENT">CALLIENT</a>] <a href="http://www.calient.net/home.html">http://www.calient.net/home.html</a> |
| 1271 |
|
| 1272 |
[<a id="ref-CARLSON">CARLSON</a>] "Complexity and Robustness", J.M. Carlson and John |
| 1273 |
Doyle, Proc. Natl. Acad. Sci. USA, Vol. 99, Suppl. 1, |
| 1274 |
2538-2545, February 19, 2002. |
| 1275 |
<a href="http://www.pnas.org/cgi/doi/10.1073/pnas.012582499">http://www.pnas.org/cgi/doi/10.1073/pnas.012582499</a> |
| 1276 |
|
| 1277 |
[<a id="ref-CIENA">CIENA</a>] "CIENA Multiwave CoreDiretor", |
| 1278 |
<a href="http://www.ciena.com/downloads/products/coredirector.pdf">http://www.ciena.com/downloads/products/</a> |
| 1279 |
<a href="http://www.ciena.com/downloads/products/coredirector.pdf">coredirector.pdf</a> |
| 1280 |
|
| 1281 |
|
| 1282 |
|
| 1283 |
|
| 1284 |
<span class="grey">Bush, et. al. Informational [Page 23]</span></pre> |
| 1285 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-24" ></span> |
| 1286 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 1287 |
|
| 1288 |
|
| 1289 |
[<a id="ref-CISCO">CISCO</a>] <a href="http://www.cisco.com">http://www.cisco.com</a> |
| 1290 |
|
| 1291 |
[<a id="ref-CLARK">CLARK</a>] "The Design Philosophy of the DARPA Internet |
| 1292 |
Protocols", D. Clark, Proc. of the ACM SIGCOMM, 1988. |
| 1293 |
|
| 1294 |
[<a id="ref-COFFMAN">COFFMAN</a>] "Internet Growth: Is there a 'Moores Law' for Data |
| 1295 |
Traffic", K.G. Coffman and A.M. Odlyzko, pp. 47-93, |
| 1296 |
Handbook of Massive Data Stes, J. Elli, P. M. |
| 1297 |
Pardalos, and M. G. C. Resende, Editors. Kluwer, |
| 1298 |
2002. |
| 1299 |
|
| 1300 |
[<a id="ref-DOYLE2002">DOYLE2002</a>] "Robustness and the Internet: Theoretical |
| 1301 |
Foundations", John C. Doyle, et. al. Work in |
| 1302 |
Progress. |
| 1303 |
|
| 1304 |
[<a id="ref-EICK">EICK</a>] "Visualizing Software Changes", S.G. Eick, et al, |
| 1305 |
National Institute of Statistical Sciences, Technical |
| 1306 |
Report 113, December 2000. |
| 1307 |
|
| 1308 |
[<a id="ref-MOLINERO2002">MOLINERO2002</a>] "TCP Switching: Exposing Circuits to IP", Pablo |
| 1309 |
Molinero-Fernandez and Nick McKeown, IEEE January, |
| 1310 |
2002. |
| 1311 |
|
| 1312 |
[<a id="ref-FLOYD">FLOYD</a>] "The Synchronization of Periodic Routing Messages", |
| 1313 |
Sally Floyd and Van Jacobson, IEEE ACM Transactions |
| 1314 |
on Networking, 1994. |
| 1315 |
|
| 1316 |
[<a id="ref-FLOYD2001">FLOYD2001</a>] "A Report on Some Recent Developments in TCP |
| 1317 |
Congestion Control, IEEE Communications Magazine, S. |
| 1318 |
Floyd, April 2001. |
| 1319 |
|
| 1320 |
[<a id="ref-FRALEIGH">FRALEIGH</a>] "Provisioning IP Backbone Networks to Support Delay- |
| 1321 |
Based Service Level Agreements", Chuck Fraleigh, |
| 1322 |
Fouad Tobagi, and Christophe Diot, 2002. |
| 1323 |
|
| 1324 |
[<a id="ref-GRIFFIN">GRIFFIN</a>] "What is the Sound of One Route Flapping", Timothy G. |
| 1325 |
Griffin, IPAM Workshop on Large-Scale Communication |
| 1326 |
Networks: Topology, Routing, Traffic, and Control, |
| 1327 |
March, 2002. |
| 1328 |
|
| 1329 |
[<a id="ref-HANDLEY">HANDLEY</a>] "On Inter-layer Assumptions (A view from the |
| 1330 |
Transport Area), slides from a presentation at the |
| 1331 |
IAB workshop on Wireless Internetworking", M. |
| 1332 |
Handley, March 2000. |
| 1333 |
|
| 1334 |
[<a id="ref-HOT">HOT</a>] J.M. Carlson and John Doyle, Phys. Rev. E 60, 1412- |
| 1335 |
1427, 1999. |
| 1336 |
|
| 1337 |
|
| 1338 |
|
| 1339 |
|
| 1340 |
<span class="grey">Bush, et. al. Informational [Page 24]</span></pre> |
| 1341 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-25" ></span> |
| 1342 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 1343 |
|
| 1344 |
|
| 1345 |
[<a id="ref-ISO10589">ISO10589</a>] "Intermediate System to Intermediate System |
| 1346 |
Intradomain Routing Exchange Protocol (IS-IS)". |
| 1347 |
|
| 1348 |
[<a id="ref-JACOBSON">JACOBSON</a>] "Congestion Avoidance and Control", Van Jacobson, |
| 1349 |
Proceedings of ACM Sigcomm 1988, pp. 273-288. |
| 1350 |
|
| 1351 |
[<a id="ref-KARN">KARN</a>] "TCP vs Link Layer Retransmission" in P. Karn et al., |
| 1352 |
Advice for Internet Subnetwork Designers, Work in |
| 1353 |
Progress. |
| 1354 |
|
| 1355 |
[<a id="ref-KUHN87">KUHN87</a>] "Sources of Failure in the Public Switched Telephone |
| 1356 |
Network", D. Richard Kuhn, EEE Computer, Vol. 30, No. |
| 1357 |
4, April, 1997. |
| 1358 |
|
| 1359 |
[<a id="ref-L2TPV3">L2TPV3</a>] Lan, J., et. al., "Layer Two Tunneling Protocol |
| 1360 |
(Version 3) -- L2TPv3", Work in Progress. |
| 1361 |
|
| 1362 |
[<a id="ref-MC2001">MC2001</a>] "U.S Communications Infrastructure at A Crossroads: |
| 1363 |
Opportunities Amid the Gloom", McKinsey&Company for |
| 1364 |
Goldman-Sachs, August 2001. |
| 1365 |
|
| 1366 |
[<a id="ref-MCK2002">MCK2002</a>] Nick McKeown, personal communication, April, 2002. |
| 1367 |
|
| 1368 |
[<a id="ref-ML2002">ML2002</a>] "Optical Systems", Merril Lynch Technical Report, |
| 1369 |
April, 2002. |
| 1370 |
|
| 1371 |
[<a id="ref-NAVE">NAVE</a>] "The influence of mode coupling on the non-linear |
| 1372 |
evolution of tearing modes", M.F.F. Nave, et al, Eur. |
| 1373 |
Phys. J. D 8, 287-297. |
| 1374 |
|
| 1375 |
[<a id="ref-NEUMANN">NEUMANN</a>] "Cause of AT&T network failure", Peter G. Neumann, |
| 1376 |
<a href="http://catless.ncl.ac.uk/Risks/9.62.html#subj2">http://catless.ncl.ac.uk/Risks/9.62.html#subj2</a> |
| 1377 |
|
| 1378 |
[<a id="ref-ODLYZKO">ODLYZKO</a>] "Data networks are mostly empty for good reason", |
| 1379 |
A.M. Odlyzko, IT Professional 1 (no. 2), pp. 67-69, |
| 1380 |
Mar/Apr 1999. |
| 1381 |
|
| 1382 |
[<a id="ref-ODLYZKO98A">ODLYZKO98A</a>] "Smart and stupid networks: Why the Internet is like |
| 1383 |
Microsoft". A. M. Odlyzko, ACM Networker, 2(5), |
| 1384 |
December, 1998. |
| 1385 |
|
| 1386 |
[<a id="ref-ODLYZKO98">ODLYZKO98</a>] "The economics of the Internet: Utility, utilization, |
| 1387 |
pricing, and Quality of Service", A.M. Odlyzko, July, |
| 1388 |
1998. |
| 1389 |
<a href="http://www.dtc.umn.edu/~odlyzko/doc/networks.html">http://www.dtc.umn.edu/~odlyzko/doc/networks.html</a> |
| 1390 |
|
| 1391 |
|
| 1392 |
|
| 1393 |
|
| 1394 |
|
| 1395 |
|
| 1396 |
<span class="grey">Bush, et. al. Informational [Page 25]</span></pre> |
| 1397 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-26" ></span> |
| 1398 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 1399 |
|
| 1400 |
|
| 1401 |
[<a id="ref-PARK">PARK</a>] "The Internet as a Complex System: Scaling, |
| 1402 |
Complexity and Control", Kihong Park and Walter |
| 1403 |
Willinger, AT&T Research, 2002. |
| 1404 |
|
| 1405 |
[<a id="ref-PERROW">PERROW</a>] "Normal Accidents: Living with High Risk |
| 1406 |
Technologies", Basic Books, C. Perrow, New York, |
| 1407 |
1984. |
| 1408 |
|
| 1409 |
[<a id="ref-PMC">PMC</a>] "The Design of a 10 Gigabit Core Router |
| 1410 |
Architecture", PMC-Sierra, <a href="http://www.pmc-sierra.com/products/diagrams/CoreRouter_lg.html">http://www.pmc-</a> |
| 1411 |
<a href="http://www.pmc-sierra.com/products/diagrams/CoreRouter_lg.html">sierra.com/products/diagrams/CoreRouter_lg.html</a> |
| 1412 |
|
| 1413 |
[<a id="ref-RFC1629">RFC1629</a>] Colella, R., Callon, R., Gardner, E. and Y. Rekhter, |
| 1414 |
"Guidelines for OSI NSAP Allocation in the Internet", |
| 1415 |
<a href="./rfc1629">RFC 1629</a>, May 1994. |
| 1416 |
|
| 1417 |
[<a id="ref-RFC1925">RFC1925</a>] Callon, R., "The Twelve Networking Truths", <a href="./rfc1925">RFC 1925</a>, |
| 1418 |
1 April 1996. |
| 1419 |
|
| 1420 |
[<a id="ref-RFC1958">RFC1958</a>] Carpenter, B., Ed., "Architectural principles of the |
| 1421 |
Internet", <a href="./rfc1958">RFC 1958</a>, June 1996. |
| 1422 |
|
| 1423 |
[<a id="ref-RFC2283">RFC2283</a>] Bates, T., Chandra, R., Katz, D. and Y. Rekhter, |
| 1424 |
"Multiprotocol Extensions for BGP4", <a href="./rfc2283">RFC 2283</a>, |
| 1425 |
February 1998. |
| 1426 |
|
| 1427 |
[<a id="ref-RFC3155">RFC3155</a>] Dawkins, S., Montenegro, G., Kojo, M. and N. Vaidya, |
| 1428 |
"End-to-end Performance Implications of Links with |
| 1429 |
Errors", <a href="https://www.rfc-editor.org/bcp/bcp50">BCP 50</a>, <a href="./rfc3155">RFC 3155</a>, May 2001. |
| 1430 |
|
| 1431 |
[<a id="ref-ROMANOV">ROMANOV</a>] "Dynamics of TCP over ATM Networks", A. Romanov, S. |
| 1432 |
Floyd, IEEE JSAC, vol. 13, No 4, pp.633-641, May |
| 1433 |
1995. |
| 1434 |
|
| 1435 |
[<a id="ref-SALTZER">SALTZER</a>] "End-To-End Arguments in System Design", J.H. |
| 1436 |
Saltzer, D.P. Reed, and D.D. Clark, ACM TOCS, Vol 2, |
| 1437 |
Number 4, November 1984, pp 277-288. |
| 1438 |
|
| 1439 |
[<a id="ref-SCOTT">SCOTT</a>] "Making Smart Investments to Reduce Unplanned |
| 1440 |
Downtime", D. Scott, Tactical Guidelines, TG-07-4033, |
| 1441 |
Gartner Group Research Note, March 1999. |
| 1442 |
|
| 1443 |
[<a id="ref-SPILLMAN">SPILLMAN</a>] "The Law of Diminishing Returns:, W. J. Spillman and |
| 1444 |
E. Lang, 1924. |
| 1445 |
|
| 1446 |
[<a id="ref-STALLINGS">STALLINGS</a>] "Data and Computer Communications (2nd Ed)", William |
| 1447 |
Stallings, Maxwell Macmillan, 1989. |
| 1448 |
|
| 1449 |
|
| 1450 |
|
| 1451 |
|
| 1452 |
<span class="grey">Bush, et. al. Informational [Page 26]</span></pre> |
| 1453 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-27" ></span> |
| 1454 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 1455 |
|
| 1456 |
|
| 1457 |
[<a id="ref-TENNENHOUSE">TENNENHOUSE</a>] "Layered multiplexing considered harmful", D. |
| 1458 |
Tennenhouse, Proceedings of the IFIP Workshop on |
| 1459 |
Protocols for High-Speed Networks, Rudin ed., North |
| 1460 |
Holland Publishers, May 1989. |
| 1461 |
|
| 1462 |
[<a id="ref-THOMPSON">THOMPSON</a>] "Nonlinear Dynamics and Chaos". J.M.T. Thompson and |
| 1463 |
H.B. Stewart, John Wiley and Sons, 1994, ISBN |
| 1464 |
0471909602. |
| 1465 |
|
| 1466 |
[<a id="ref-TINA">TINA</a>] "What is TINA and is it useful for the TelCos?", |
| 1467 |
Paolo Coppo, Carlo A. Licciardi, CSELT, EURESCOM |
| 1468 |
Participants in P847 (FT, IT, NT, TI) |
| 1469 |
|
| 1470 |
[<a id="ref-WAKEMAN">WAKEMAN</a>] "Layering considered harmful", Ian Wakeman, Jon |
| 1471 |
Crowcroft, Zheng Wang, and Dejan Sirovica, IEEE |
| 1472 |
Network, January 1992, p. 7-16. |
| 1473 |
|
| 1474 |
[<a id="ref-WARD">WARD</a>] "Custom fluorescent-nucleotide synthesis as an |
| 1475 |
alternative method for nucleic acid labeling", |
| 1476 |
Octavian Henegariu*, Patricia Bray-Ward and David C. |
| 1477 |
Ward, Nature Biotech 18:345-348 (2000). |
| 1478 |
|
| 1479 |
[<a id="ref-WILLINGER2002">WILLINGER2002</a>] "Robustness and the Internet: Design and evolution", |
| 1480 |
Walter Willinger and John Doyle, 2002. |
| 1481 |
|
| 1482 |
[<a id="ref-ZHANG">ZHANG</a>] "Impact of Aggregation on Scaling Behavior of |
| 1483 |
Internet Backbone Traffic", Sprint ATL Technical |
| 1484 |
Report TR02-ATL-020157 Zhi-Li Zhang, Vinay Ribeiroj, |
| 1485 |
Sue Moon, Christophe Diot, February, 2002. |
| 1486 |
|
| 1487 |
<span class="h2"><a class="selflink" id="section-13" href="#section-13">13</a>. Authors' Addresses</span> |
| 1488 |
|
| 1489 |
Randy Bush |
| 1490 |
EMail: [email protected] |
| 1491 |
|
| 1492 |
David Meyer |
| 1493 |
EMail: [email protected] |
| 1494 |
|
| 1495 |
|
| 1496 |
|
| 1497 |
|
| 1498 |
|
| 1499 |
|
| 1500 |
|
| 1501 |
|
| 1502 |
|
| 1503 |
|
| 1504 |
|
| 1505 |
|
| 1506 |
|
| 1507 |
|
| 1508 |
<span class="grey">Bush, et. al. Informational [Page 27]</span></pre> |
| 1509 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-28" ></span> |
| 1510 |
<span class="grey"><a href="./rfc3439">RFC 3439</a> Internet Architectural Guidelines December 2002</span> |
| 1511 |
|
| 1512 |
|
| 1513 |
<span class="h2"><a class="selflink" id="section-14" href="#section-14">14</a>. Full Copyright Statement</span> |
| 1514 |
|
| 1515 |
Copyright (C) The Internet Society (2002). All Rights Reserved. |
| 1516 |
|
| 1517 |
This document and translations of it may be copied and furnished to |
| 1518 |
others, and derivative works that comment on or otherwise explain it |
| 1519 |
or assist in its implementation may be prepared, copied, published |
| 1520 |
and distributed, in whole or in part, without restriction of any |
| 1521 |
kind, provided that the above copyright notice and this paragraph are |
| 1522 |
included on all such copies and derivative works. However, this |
| 1523 |
document itself may not be modified in any way, such as by removing |
| 1524 |
the copyright notice or references to the Internet Society or other |
| 1525 |
Internet organizations, except as needed for the purpose of |
| 1526 |
developing Internet standards in which case the procedures for |
| 1527 |
copyrights defined in the Internet Standards process must be |
| 1528 |
followed, or as required to translate it into languages other than |
| 1529 |
English. |
| 1530 |
|
| 1531 |
The limited permissions granted above are perpetual and will not be |
| 1532 |
revoked by the Internet Society or its successors or assigns. |
| 1533 |
|
| 1534 |
This document and the information contained herein is provided on an |
| 1535 |
"AS IS" basis and THE INTERNET SOCIETY AND THE INTERNET ENGINEERING |
| 1536 |
TASK FORCE DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING |
| 1537 |
BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION |
| 1538 |
HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF |
| 1539 |
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. |
| 1540 |
|
| 1541 |
Acknowledgement |
| 1542 |
|
| 1543 |
Funding for the RFC Editor function is currently provided by the |
| 1544 |
Internet Society. |
| 1545 |
|
| 1546 |
|
| 1547 |
|
| 1548 |
|
| 1549 |
|
| 1550 |
|
| 1551 |
|
| 1552 |
|
| 1553 |
|
| 1554 |
|
| 1555 |
|
| 1556 |
|
| 1557 |
|
| 1558 |
|
| 1559 |
|
| 1560 |
|
| 1561 |
|
| 1562 |
|
| 1563 |
|
| 1564 |
Bush, et. al. Informational [Page 28] |
| 1565 |
</pre> |