| 1 |
<pre>Internet Engineering Task Force (IETF) S. Bensley |
| 2 |
Request for Comments: 8257 D. Thaler |
| 3 |
Category: Informational P. Balasubramanian |
| 4 |
ISSN: 2070-1721 Microsoft |
| 5 |
L. Eggert |
| 6 |
NetApp |
| 7 |
G. Judd |
| 8 |
Morgan Stanley |
| 9 |
October 2017 |
| 10 |
|
| 11 |
|
| 12 |
<span class="h1">Data Center TCP (DCTCP): TCP Congestion Control for Data Centers</span> |
| 13 |
|
| 14 |
Abstract |
| 15 |
|
| 16 |
This Informational RFC describes Data Center TCP (DCTCP): a TCP |
| 17 |
congestion control scheme for data-center traffic. DCTCP extends the |
| 18 |
Explicit Congestion Notification (ECN) processing to estimate the |
| 19 |
fraction of bytes that encounter congestion rather than simply |
| 20 |
detecting that some congestion has occurred. DCTCP then scales the |
| 21 |
TCP congestion window based on this estimate. This method achieves |
| 22 |
high-burst tolerance, low latency, and high throughput with shallow- |
| 23 |
buffered switches. This memo also discusses deployment issues |
| 24 |
related to the coexistence of DCTCP and conventional TCP, discusses |
| 25 |
the lack of a negotiating mechanism between sender and receiver, and |
| 26 |
presents some possible mitigations. This memo documents DCTCP as |
| 27 |
currently implemented by several major operating systems. DCTCP, as |
| 28 |
described in this specification, is applicable to deployments in |
| 29 |
controlled environments like data centers, but it must not be |
| 30 |
deployed over the public Internet without additional measures. |
| 31 |
|
| 32 |
Status of This Memo |
| 33 |
|
| 34 |
This document is not an Internet Standards Track specification; it is |
| 35 |
published for informational purposes. |
| 36 |
|
| 37 |
This document is a product of the Internet Engineering Task Force |
| 38 |
(IETF). It represents the consensus of the IETF community. It has |
| 39 |
received public review and has been approved for publication by the |
| 40 |
Internet Engineering Steering Group (IESG). Not all documents |
| 41 |
approved by the IESG are a candidate for any level of Internet |
| 42 |
Standard; see <a href="./rfc7841#section-2">Section 2 of RFC 7841</a>. |
| 43 |
|
| 44 |
Information about the current status of this document, any errata, |
| 45 |
and how to provide feedback on it may be obtained at |
| 46 |
<a href="https://www.rfc-editor.org/info/rfc8257">https://www.rfc-editor.org/info/rfc8257</a>. |
| 47 |
|
| 48 |
|
| 49 |
|
| 50 |
|
| 51 |
|
| 52 |
<span class="grey">Bensley, et al. Informational [Page 1]</span></pre> |
| 53 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-2" ></span> |
| 54 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 55 |
|
| 56 |
|
| 57 |
Copyright Notice |
| 58 |
|
| 59 |
Copyright (c) 2017 IETF Trust and the persons identified as the |
| 60 |
document authors. All rights reserved. |
| 61 |
|
| 62 |
This document is subject to <a href="https://www.rfc-editor.org/bcp/bcp78">BCP 78</a> and the IETF Trust's Legal |
| 63 |
Provisions Relating to IETF Documents |
| 64 |
(<a href="https://trustee.ietf.org/license-info">https://trustee.ietf.org/license-info</a>) in effect on the date of |
| 65 |
publication of this document. Please review these documents |
| 66 |
carefully, as they describe your rights and restrictions with respect |
| 67 |
to this document. Code Components extracted from this document must |
| 68 |
include Simplified BSD License text as described in Section 4.e of |
| 69 |
the Trust Legal Provisions and are provided without warranty as |
| 70 |
described in the Simplified BSD License. |
| 71 |
|
| 72 |
Table of Contents |
| 73 |
|
| 74 |
<a href="#section-1">1</a>. Introduction . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-3">3</a> |
| 75 |
<a href="#section-2">2</a>. Terminology . . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-4">4</a> |
| 76 |
<a href="#section-3">3</a>. DCTCP Algorithm . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-5">5</a> |
| 77 |
<a href="#section-3.1">3.1</a>. Marking Congestion on the L3 Switches and Routers . . . . <a href="#page-5">5</a> |
| 78 |
<a href="#section-3.2">3.2</a>. Echoing Congestion Information on the Receiver . . . . . <a href="#page-5">5</a> |
| 79 |
3.3. Processing Echoed Congestion Indications on the Sender . 7 |
| 80 |
<a href="#section-3.4">3.4</a>. Handling of Congestion Window Growth . . . . . . . . . . <a href="#page-8">8</a> |
| 81 |
<a href="#section-3.5">3.5</a>. Handling of Packet Loss . . . . . . . . . . . . . . . . . <a href="#page-8">8</a> |
| 82 |
<a href="#section-3.6">3.6</a>. Handling of SYN, SYN-ACK, and RST Packets . . . . . . . . <a href="#page-9">9</a> |
| 83 |
<a href="#section-4">4</a>. Implementation Issues . . . . . . . . . . . . . . . . . . . . <a href="#page-9">9</a> |
| 84 |
<a href="#section-4.1">4.1</a>. Configuration of DCTCP . . . . . . . . . . . . . . . . . <a href="#page-9">9</a> |
| 85 |
<a href="#section-4.2">4.2</a>. Computation of DCTCP.Alpha . . . . . . . . . . . . . . . <a href="#page-10">10</a> |
| 86 |
<a href="#section-5">5</a>. Deployment Issues . . . . . . . . . . . . . . . . . . . . . . <a href="#page-11">11</a> |
| 87 |
<a href="#section-6">6</a>. Known Issues . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-12">12</a> |
| 88 |
<a href="#section-7">7</a>. Security Considerations . . . . . . . . . . . . . . . . . . . <a href="#page-12">12</a> |
| 89 |
<a href="#section-8">8</a>. IANA Considerations . . . . . . . . . . . . . . . . . . . . . <a href="#page-13">13</a> |
| 90 |
<a href="#section-9">9</a>. References . . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-13">13</a> |
| 91 |
<a href="#section-9.1">9.1</a>. Normative References . . . . . . . . . . . . . . . . . . <a href="#page-13">13</a> |
| 92 |
<a href="#section-9.2">9.2</a>. Informative References . . . . . . . . . . . . . . . . . <a href="#page-14">14</a> |
| 93 |
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-16">16</a> |
| 94 |
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . <a href="#page-16">16</a> |
| 95 |
|
| 96 |
|
| 97 |
|
| 98 |
|
| 99 |
|
| 100 |
|
| 101 |
|
| 102 |
|
| 103 |
|
| 104 |
|
| 105 |
|
| 106 |
|
| 107 |
|
| 108 |
<span class="grey">Bensley, et al. Informational [Page 2]</span></pre> |
| 109 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-3" ></span> |
| 110 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 111 |
|
| 112 |
|
| 113 |
<span class="h2"><a class="selflink" id="section-1" href="#section-1">1</a>. Introduction</span> |
| 114 |
|
| 115 |
Large data centers necessarily need many network switches to |
| 116 |
interconnect their many servers. Therefore, a data center can |
| 117 |
greatly reduce its capital expenditure by leveraging low-cost |
| 118 |
switches. However, such low-cost switches tend to have limited queue |
| 119 |
capacities; thus, they are more susceptible to packet loss due to |
| 120 |
congestion. |
| 121 |
|
| 122 |
Network traffic in a data center is often a mix of short and long |
| 123 |
flows, where the short flows require low latencies and the long flows |
| 124 |
require high throughputs. Data centers also experience incast |
| 125 |
bursts, where many servers send traffic to a single server at the |
| 126 |
same time. For example, this traffic pattern is a natural |
| 127 |
consequence of the MapReduce [<a href="#ref-MAPREDUCE">MAPREDUCE</a>] workload: the worker nodes |
| 128 |
complete at approximately the same time, and all reply to the master |
| 129 |
node concurrently. |
| 130 |
|
| 131 |
These factors place some conflicting demands on the queue occupancy |
| 132 |
of a switch: |
| 133 |
|
| 134 |
o The queue must be short enough that it does not impose excessive |
| 135 |
latency on short flows. |
| 136 |
|
| 137 |
o The queue must be long enough to buffer sufficient data for the |
| 138 |
long flows to saturate the path capacity. |
| 139 |
|
| 140 |
o The queue must be long enough to absorb incast bursts without |
| 141 |
excessive packet loss. |
| 142 |
|
| 143 |
Standard TCP congestion control [<a href="./rfc5681" title=""TCP Congestion Control"">RFC5681</a>] relies on packet loss to |
| 144 |
detect congestion. This does not meet the demands described above. |
| 145 |
First, short flows will start to experience unacceptable latencies |
| 146 |
before packet loss occurs. Second, by the time TCP congestion |
| 147 |
control kicks in on the senders, most of the incast burst has already |
| 148 |
been dropped. |
| 149 |
|
| 150 |
[<a id="ref-RFC3168">RFC3168</a>] describes a mechanism for using Explicit Congestion |
| 151 |
Notification (ECN) from the switches for detection of congestion. |
| 152 |
However, this method only detects the presence of congestion, not its |
| 153 |
extent. In the presence of mild congestion, the TCP congestion |
| 154 |
window is reduced too aggressively, and this unnecessarily reduces |
| 155 |
the throughput of long flows. |
| 156 |
|
| 157 |
Data Center TCP (DCTCP) changes traditional ECN processing by |
| 158 |
estimating the fraction of bytes that encounter congestion rather |
| 159 |
than simply detecting that some congestion has occurred. DCTCP then |
| 160 |
scales the TCP congestion window based on this estimate. This method |
| 161 |
|
| 162 |
|
| 163 |
|
| 164 |
<span class="grey">Bensley, et al. Informational [Page 3]</span></pre> |
| 165 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-4" ></span> |
| 166 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 167 |
|
| 168 |
|
| 169 |
achieves high-burst tolerance, low latency, and high throughput with |
| 170 |
shallow-buffered switches. DCTCP is a modification to the processing |
| 171 |
of ECN by a conventional TCP and requires that standard TCP |
| 172 |
congestion control be used for handling packet loss. |
| 173 |
|
| 174 |
DCTCP should only be deployed in an intra-data-center environment |
| 175 |
where both endpoints and the switching fabric are under a single |
| 176 |
administrative domain. DCTCP MUST NOT be deployed over the public |
| 177 |
Internet without additional measures, as detailed in <a href="#section-5">Section 5</a>. |
| 178 |
|
| 179 |
The objective of this Informational RFC is to document DCTCP as a new |
| 180 |
approach (which is known to be widely implemented and deployed) to |
| 181 |
address TCP congestion control in data centers. The IETF TCPM |
| 182 |
Working Group reached consensus regarding the fact that a DCTCP |
| 183 |
standard would require further work. A precise documentation of |
| 184 |
running code enables follow-up Experimental or Standards Track RFCs |
| 185 |
through the IETF stream. |
| 186 |
|
| 187 |
This document describes DCTCP as implemented in Microsoft Windows |
| 188 |
Server 2012 [<a href="#ref-WINDOWS" title=""Data Center Transmission Control Protocol (DCTCP)"">WINDOWS</a>]. The Linux [<a href="#ref-LINUX" title=""net: tcp: add DCTCP congestion control algorithm"">LINUX</a>] and FreeBSD [<a href="#ref-FREEBSD" title=""DCTCP (Data Center TCP) implementation"">FREEBSD</a>] |
| 189 |
operating systems have also implemented support for DCTCP in a way |
| 190 |
that is believed to follow this document. Deployment experiences |
| 191 |
with DCTCP have been documented in [<a href="#ref-MORGANSTANLEY">MORGANSTANLEY</a>]. |
| 192 |
|
| 193 |
<span class="h2"><a class="selflink" id="section-2" href="#section-2">2</a>. Terminology</span> |
| 194 |
|
| 195 |
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", |
| 196 |
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and |
| 197 |
"OPTIONAL" in this document are to be interpreted as described in |
| 198 |
<a href="https://www.rfc-editor.org/bcp/bcp14">BCP 14</a> [<a href="./rfc2119" title=""Key words for use in RFCs to Indicate Requirement Levels"">RFC2119</a>] [<a href="./rfc8174" title=""Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words"">RFC8174</a>] when, and only when, they appear in all |
| 199 |
capitals, as shown here. |
| 200 |
|
| 201 |
Normative language is used to describe how necessary the various |
| 202 |
aspects of a DCTCP implementation are for interoperability, but even |
| 203 |
compliant implementations without the measures in Sections <a href="#section-4">4</a>-<a href="#section-6">6</a> would |
| 204 |
still only be safe to deploy in controlled environments, i.e., not |
| 205 |
over the public Internet. |
| 206 |
|
| 207 |
|
| 208 |
|
| 209 |
|
| 210 |
|
| 211 |
|
| 212 |
|
| 213 |
|
| 214 |
|
| 215 |
|
| 216 |
|
| 217 |
|
| 218 |
|
| 219 |
|
| 220 |
<span class="grey">Bensley, et al. Informational [Page 4]</span></pre> |
| 221 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-5" ></span> |
| 222 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 223 |
|
| 224 |
|
| 225 |
<span class="h2"><a class="selflink" id="section-3" href="#section-3">3</a>. DCTCP Algorithm</span> |
| 226 |
|
| 227 |
There are three components involved in the DCTCP algorithm: |
| 228 |
|
| 229 |
o The switches (or other intermediate devices in the network) detect |
| 230 |
congestion and set the Congestion Encountered (CE) codepoint in |
| 231 |
the IP header. |
| 232 |
|
| 233 |
o The receiver echoes the congestion information back to the sender, |
| 234 |
using the ECN-Echo (ECE) flag in the TCP header. |
| 235 |
|
| 236 |
o The sender computes a congestion estimate and reacts by reducing |
| 237 |
the TCP congestion window (cwnd) accordingly. |
| 238 |
|
| 239 |
<span class="h3"><a class="selflink" id="section-3.1" href="#section-3.1">3.1</a>. Marking Congestion on the L3 Switches and Routers</span> |
| 240 |
|
| 241 |
The Layer 3 (L3) switches and routers in a data-center fabric |
| 242 |
indicate congestion to the end nodes by setting the CE codepoint in |
| 243 |
the IP header as specified in <a href="./rfc3168#section-5">Section 5 of [RFC3168]</a>. For example, |
| 244 |
the switches may be configured with a congestion threshold. When a |
| 245 |
packet arrives at a switch and its queue length is greater than the |
| 246 |
congestion threshold, the switch sets the CE codepoint in the packet. |
| 247 |
For example, Section 3.4 of [<a href="#ref-DCTCP10" title=""Data Center TCP (DCTCP)"">DCTCP10</a>] suggests threshold marking with |
| 248 |
a threshold of K > (RTT * C)/7, where C is the link rate in packets |
| 249 |
per second. In typical deployments, the marking threshold is set to |
| 250 |
be a small value to maintain a short average queueing delay. |
| 251 |
However, the actual algorithm for marking congestion is an |
| 252 |
implementation detail of the switch and will generally not be known |
| 253 |
to the sender and receiver. Therefore, the sender and receiver |
| 254 |
should not assume that a particular marking algorithm is implemented |
| 255 |
by the switching fabric. |
| 256 |
|
| 257 |
<span class="h3"><a class="selflink" id="section-3.2" href="#section-3.2">3.2</a>. Echoing Congestion Information on the Receiver</span> |
| 258 |
|
| 259 |
According to <a href="./rfc3168#section-6.1.3">Section 6.1.3 of [RFC3168]</a>, the receiver sets the ECE |
| 260 |
flag if any of the packets being acknowledged had the CE codepoint |
| 261 |
set. The receiver then continues to set the ECE flag until it |
| 262 |
receives a packet with the Congestion Window Reduced (CWR) flag set. |
| 263 |
However, the DCTCP algorithm requires more-detailed congestion |
| 264 |
information. In particular, the sender must be able to determine the |
| 265 |
number of bytes sent that encountered congestion. Thus, the scheme |
| 266 |
described in [<a href="./rfc3168" title=""The Addition of Explicit Congestion Notification (ECN) to IP"">RFC3168</a>] does not suffice. |
| 267 |
|
| 268 |
One possible solution is to ACK every packet and set the ECE flag in |
| 269 |
the ACK if and only if the CE codepoint was set in the packet being |
| 270 |
acknowledged. However, this prevents the use of delayed ACKs, which |
| 271 |
are an important performance optimization in data centers. If the |
| 272 |
delayed ACK frequency is n, then an ACK is generated every n packets. |
| 273 |
|
| 274 |
|
| 275 |
|
| 276 |
<span class="grey">Bensley, et al. Informational [Page 5]</span></pre> |
| 277 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-6" ></span> |
| 278 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 279 |
|
| 280 |
|
| 281 |
The typical value of n is 2, but it could be affected by ACK |
| 282 |
throttling or packet-coalescing techniques designed to improve |
| 283 |
performance. |
| 284 |
|
| 285 |
Instead, DCTCP introduces a new Boolean TCP state variable, DCTCP |
| 286 |
Congestion Encountered (DCTCP.CE), which is initialized to false and |
| 287 |
stored in the Transmission Control Block (TCB). When sending an ACK, |
| 288 |
the ECE flag MUST be set if and only if DCTCP.CE is true. When |
| 289 |
receiving packets, the CE codepoint MUST be processed as follows: |
| 290 |
|
| 291 |
1. If the CE codepoint is set and DCTCP.CE is false, set DCTCP.CE to |
| 292 |
true and send an immediate ACK. |
| 293 |
|
| 294 |
2. If the CE codepoint is not set and DCTCP.CE is true, set DCTCP.CE |
| 295 |
to false and send an immediate ACK. |
| 296 |
|
| 297 |
3. Otherwise, ignore the CE codepoint. |
| 298 |
|
| 299 |
Since the immediate ACK reflects the new DCTCP.CE state, it may |
| 300 |
acknowledge any previously unacknowledged packets in the old state. |
| 301 |
This can lead to an incorrect rate computation at the sender per |
| 302 |
<a href="#section-3.3">Section 3.3</a>. To avoid this, an implementation MAY choose to send two |
| 303 |
ACKs: one for previously unacknowledged packets and another |
| 304 |
acknowledging the most recently received packet. |
| 305 |
|
| 306 |
Receiver handling of the CWR bit is also per [<a href="./rfc3168" title=""The Addition of Explicit Congestion Notification (ECN) to IP"">RFC3168</a>] (including |
| 307 |
[<a href="#ref-Err3639" title="RFC 3168">Err3639</a>]). That is, on receipt of a segment with both the CE and |
| 308 |
CWR bits set, CWR is processed first and then CE is processed. |
| 309 |
|
| 310 |
Send immediate |
| 311 |
ACK with ECE=0 |
| 312 |
.-----. .--------------. .-----. |
| 313 |
Send 1 ACK / v v | | \ |
| 314 |
for every | .------------. .------------. | Send 1 ACK |
| 315 |
n packets | | DCTCP.CE=0 | | DCTCP.CE=1 | | for every |
| 316 |
with ECE=0 | '------------' '------------' | n packets |
| 317 |
\ | | ^ ^ / with ECE=1 |
| 318 |
'-----' '--------------' '-----' |
| 319 |
Send immediate |
| 320 |
ACK with ECE=1 |
| 321 |
|
| 322 |
|
| 323 |
Figure 1: ACK Generation State Machine |
| 324 |
|
| 325 |
|
| 326 |
|
| 327 |
|
| 328 |
|
| 329 |
|
| 330 |
|
| 331 |
|
| 332 |
<span class="grey">Bensley, et al. Informational [Page 6]</span></pre> |
| 333 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-7" ></span> |
| 334 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 335 |
|
| 336 |
|
| 337 |
<span class="h3"><a class="selflink" id="section-3.3" href="#section-3.3">3.3</a>. Processing Echoed Congestion Indications on the Sender</span> |
| 338 |
|
| 339 |
The sender estimates the fraction of bytes sent that encountered |
| 340 |
congestion. The current estimate is stored in a new TCP state |
| 341 |
variable, DCTCP.Alpha, which is initialized to 1 and SHOULD be |
| 342 |
updated as follows: |
| 343 |
|
| 344 |
DCTCP.Alpha = DCTCP.Alpha * (1 - g) + g * M |
| 345 |
|
| 346 |
where: |
| 347 |
|
| 348 |
o g is the estimation gain, a real number between 0 and 1. The |
| 349 |
selection of g is left to the implementation. See <a href="#section-4">Section 4</a> for |
| 350 |
further considerations. |
| 351 |
|
| 352 |
o M is the fraction of bytes sent that encountered congestion during |
| 353 |
the previous observation window, where the observation window is |
| 354 |
chosen to be approximately the Round-Trip Time (RTT). In |
| 355 |
particular, an observation window ends when all bytes in flight at |
| 356 |
the beginning of the window have been acknowledged. |
| 357 |
|
| 358 |
In order to update DCTCP.Alpha, the TCP state variables defined in |
| 359 |
[<a href="./rfc0793" title=""Transmission Control Protocol"">RFC0793</a>] are used, and three additional TCP state variables are |
| 360 |
introduced: |
| 361 |
|
| 362 |
o DCTCP.WindowEnd: the TCP sequence number threshold when one |
| 363 |
observation window ends and another is to begin; initialized to |
| 364 |
SND.UNA. |
| 365 |
|
| 366 |
o DCTCP.BytesAcked: the number of sent bytes acknowledged during the |
| 367 |
current observation window; initialized to 0. |
| 368 |
|
| 369 |
o DCTCP.BytesMarked: the number of bytes sent during the current |
| 370 |
observation window that encountered congestion; initialized to 0. |
| 371 |
|
| 372 |
The congestion estimator on the sender MUST process acceptable ACKs |
| 373 |
as follows: |
| 374 |
|
| 375 |
1. Compute the bytes acknowledged (TCP Selective Acknowledgment |
| 376 |
(SACK) options [<a href="./rfc2018" title=""TCP Selective Acknowledgment Options"">RFC2018</a>] are ignored for this computation): |
| 377 |
|
| 378 |
BytesAcked = SEG.ACK - SND.UNA |
| 379 |
|
| 380 |
2. Update the bytes sent: |
| 381 |
|
| 382 |
DCTCP.BytesAcked += BytesAcked |
| 383 |
|
| 384 |
|
| 385 |
|
| 386 |
|
| 387 |
|
| 388 |
<span class="grey">Bensley, et al. Informational [Page 7]</span></pre> |
| 389 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-8" ></span> |
| 390 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 391 |
|
| 392 |
|
| 393 |
3. If the ECE flag is set, update the bytes marked: |
| 394 |
|
| 395 |
DCTCP.BytesMarked += BytesAcked |
| 396 |
|
| 397 |
4. If the acknowledgment number is less than or equal to |
| 398 |
DCTCP.WindowEnd, stop processing. Otherwise, the end of the |
| 399 |
observation window has been reached, so proceed to update the |
| 400 |
congestion estimate as follows: |
| 401 |
|
| 402 |
5. Compute the congestion level for the current observation window: |
| 403 |
|
| 404 |
M = DCTCP.BytesMarked / DCTCP.BytesAcked |
| 405 |
|
| 406 |
6. Update the congestion estimate: |
| 407 |
|
| 408 |
DCTCP.Alpha = DCTCP.Alpha * (1 - g) + g * M |
| 409 |
|
| 410 |
7. Determine the end of the next observation window: |
| 411 |
|
| 412 |
DCTCP.WindowEnd = SND.NXT |
| 413 |
|
| 414 |
8. Reset the byte counters: |
| 415 |
|
| 416 |
DCTCP.BytesAcked = DCTCP.BytesMarked = 0 |
| 417 |
|
| 418 |
9. Rather than always halving the congestion window as described in |
| 419 |
[<a href="./rfc3168" title=""The Addition of Explicit Congestion Notification (ECN) to IP"">RFC3168</a>], the sender SHOULD update cwnd as follows: |
| 420 |
|
| 421 |
cwnd = cwnd * (1 - DCTCP.Alpha / 2) |
| 422 |
|
| 423 |
Just as specified in [<a href="./rfc3168" title=""The Addition of Explicit Congestion Notification (ECN) to IP"">RFC3168</a>], DCTCP does not react to congestion |
| 424 |
indications more than once for every window of data. The setting of |
| 425 |
the CWR bit is also as per [<a href="./rfc3168" title=""The Addition of Explicit Congestion Notification (ECN) to IP"">RFC3168</a>]. This is required for |
| 426 |
interoperation with classic ECN receivers due to potential |
| 427 |
misconfigurations. |
| 428 |
|
| 429 |
<span class="h3"><a class="selflink" id="section-3.4" href="#section-3.4">3.4</a>. Handling of Congestion Window Growth</span> |
| 430 |
|
| 431 |
A DCTCP sender grows its congestion window in the same way as |
| 432 |
conventional TCP. Slow start and congestion avoidance algorithms are |
| 433 |
handled as specified in [<a href="./rfc5681" title=""TCP Congestion Control"">RFC5681</a>]. |
| 434 |
|
| 435 |
<span class="h3"><a class="selflink" id="section-3.5" href="#section-3.5">3.5</a>. Handling of Packet Loss</span> |
| 436 |
|
| 437 |
A DCTCP sender MUST react to loss episodes in the same way as |
| 438 |
conventional TCP, including fast retransmit and fast recovery |
| 439 |
algorithms, as specified in [<a href="./rfc5681" title=""TCP Congestion Control"">RFC5681</a>]. For cases where the packet |
| 440 |
loss is inferred and not explicitly signaled by ECN, the cwnd and |
| 441 |
|
| 442 |
|
| 443 |
|
| 444 |
<span class="grey">Bensley, et al. Informational [Page 8]</span></pre> |
| 445 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-9" ></span> |
| 446 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 447 |
|
| 448 |
|
| 449 |
other state variables like ssthresh MUST be changed in the same way |
| 450 |
that a conventional TCP would have changed them. As with ECN, a |
| 451 |
DCTCP sender will only reduce the cwnd once per window of data across |
| 452 |
all loss signals. Just as specified in [<a href="./rfc5681" title=""TCP Congestion Control"">RFC5681</a>], upon a timeout, |
| 453 |
the cwnd MUST be set to no more than the loss window (1 full-sized |
| 454 |
segment), regardless of previous cwnd reductions in a given window of |
| 455 |
data. |
| 456 |
|
| 457 |
<span class="h3"><a class="selflink" id="section-3.6" href="#section-3.6">3.6</a>. Handling of SYN, SYN-ACK, and RST Packets</span> |
| 458 |
|
| 459 |
If SYN, SYN-ACK, and RST packets for DCTCP connections have the ECN- |
| 460 |
Capable Transport (ECT) codepoint set in the IP header, they will |
| 461 |
receive the same treatment as other DCTCP packets when forwarded by a |
| 462 |
switching fabric under load. Lack of ECT in these packets can result |
| 463 |
in a higher drop rate, depending on the switching fabric |
| 464 |
configuration. Hence, for DCTCP connections, the sender SHOULD set |
| 465 |
ECT for SYN, SYN-ACK, and RST packets. A DCTCP receiver ignores CE |
| 466 |
codepoints set on any SYN, SYN-ACK, or RST packets. |
| 467 |
|
| 468 |
<span class="h2"><a class="selflink" id="section-4" href="#section-4">4</a>. Implementation Issues</span> |
| 469 |
|
| 470 |
<span class="h3"><a class="selflink" id="section-4.1" href="#section-4.1">4.1</a>. Configuration of DCTCP</span> |
| 471 |
|
| 472 |
An implementation needs to know when to use DCTCP. Data-center |
| 473 |
servers may need to communicate with endpoints outside the data |
| 474 |
center, where DCTCP is unsuitable or unsupported. Thus, a global |
| 475 |
configuration setting to enable DCTCP will generally not suffice. |
| 476 |
DCTCP provides no mechanism for negotiating its use. Thus, |
| 477 |
additional management and configuration functionality is needed to |
| 478 |
ensure that DCTCP is not used with non-DCTCP endpoints. |
| 479 |
|
| 480 |
Known solutions rely on either configuration or heuristics. |
| 481 |
Heuristics need to allow endpoints to individually enable DCTCP to |
| 482 |
ensure a DCTCP sender is always paired with a DCTCP receiver. One |
| 483 |
approach is to enable DCTCP based on the IP address of the remote |
| 484 |
endpoint. Another approach is to detect connections that transmit |
| 485 |
within the bounds of a data center. For example, an implementation |
| 486 |
could support automatic selection of DCTCP if the estimated RTT is |
| 487 |
less than a threshold (like 10 msec) and ECN is successfully |
| 488 |
negotiated under the assumption that if the RTT is low, then the two |
| 489 |
endpoints are likely in the same data-center network. |
| 490 |
|
| 491 |
[<a id="ref-RFC3168">RFC3168</a>] forbids the ECN-marking of pure ACK packets because of the |
| 492 |
inability of TCP to mitigate ACK-path congestion. <a href="./rfc3168">RFC 3168</a> also |
| 493 |
forbids ECN-marking of retransmissions, window probes, and RSTs. |
| 494 |
However, dropping all these control packets -- rather than ECN- |
| 495 |
marking them -- has considerable performance disadvantages. It is |
| 496 |
RECOMMENDED that an implementation provide a configuration knob that |
| 497 |
|
| 498 |
|
| 499 |
|
| 500 |
<span class="grey">Bensley, et al. Informational [Page 9]</span></pre> |
| 501 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-10" ></span> |
| 502 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 503 |
|
| 504 |
|
| 505 |
will cause ECT to be set on such control packets, which can be used |
| 506 |
in environments where such concerns do not apply. See |
| 507 |
[<a href="#ref-ECN-EXPERIMENTATION">ECN-EXPERIMENTATION</a>] for details. |
| 508 |
|
| 509 |
It is useful to implement DCTCP as an additional action on top of an |
| 510 |
existing congestion control algorithm like Reno [<a href="./rfc5681" title=""TCP Congestion Control"">RFC5681</a>]. The DCTCP |
| 511 |
implementation MAY also allow configuration of resetting the value of |
| 512 |
DCTCP.Alpha as part of processing any loss episodes. |
| 513 |
|
| 514 |
<span class="h3"><a class="selflink" id="section-4.2" href="#section-4.2">4.2</a>. Computation of DCTCP.Alpha</span> |
| 515 |
|
| 516 |
As noted in <a href="#section-3.3">Section 3.3</a>, the implementation will need to choose a |
| 517 |
suitable estimation gain. [<a href="#ref-DCTCP10" title=""Data Center TCP (DCTCP)"">DCTCP10</a>] provides a theoretical basis for |
| 518 |
selecting the gain. However, it may be more practical to use |
| 519 |
experimentation to select a suitable gain for a particular network |
| 520 |
and workload. A fixed estimation gain of 1/16 is used in some |
| 521 |
implementations. (It should be noted that values of 0 or 1 for g |
| 522 |
result in problematic behavior; g=0 fixes DCTCP.Alpha to its initial |
| 523 |
value, and g=1 sets it to M without any smoothing.) |
| 524 |
|
| 525 |
The DCTCP.Alpha computation as per the formula in <a href="#section-3.3">Section 3.3</a> |
| 526 |
involves fractions. An efficient kernel implementation MAY scale the |
| 527 |
DCTCP.Alpha value for efficient computation using shift operations. |
| 528 |
For example, if the implementation chooses g as 1/16, multiplications |
| 529 |
of DCTCP.Alpha by g become right-shifts by 4. A scaling |
| 530 |
implementation SHOULD ensure that DCTCP.Alpha is able to reach 0 once |
| 531 |
it falls below the smallest shifted value (16 in the above example). |
| 532 |
At the other extreme, a scaled update needs to ensure DCTCP.Alpha |
| 533 |
does not exceed the scaling factor, which would be equivalent to |
| 534 |
greater than 100% congestion. So, DCTCP.Alpha MUST be clamped after |
| 535 |
an update. |
| 536 |
|
| 537 |
This results in the following computations replacing steps 5 and 6 in |
| 538 |
<a href="#section-3.3">Section 3.3</a>, where SCF is the chosen scaling factor (65536 in the |
| 539 |
example), and SHF is the shift factor (4 in the example): |
| 540 |
|
| 541 |
1. Compute the congestion level for the current observation window: |
| 542 |
|
| 543 |
ScaledM = SCF * DCTCP.BytesMarked / DCTCP.BytesAcked |
| 544 |
|
| 545 |
2. Update the congestion estimate: |
| 546 |
|
| 547 |
if (DCTCP.Alpha >> SHF) == 0, then DCTCP.Alpha = 0 |
| 548 |
|
| 549 |
DCTCP.Alpha += (ScaledM >> SHF) - (DCTCP.Alpha >> SHF) |
| 550 |
|
| 551 |
if DCTCP.Alpha > SCF, then DCTCP.Alpha = SCF |
| 552 |
|
| 553 |
|
| 554 |
|
| 555 |
|
| 556 |
<span class="grey">Bensley, et al. Informational [Page 10]</span></pre> |
| 557 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-11" ></span> |
| 558 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 559 |
|
| 560 |
|
| 561 |
<span class="h2"><a class="selflink" id="section-5" href="#section-5">5</a>. Deployment Issues</span> |
| 562 |
|
| 563 |
DCTCP and conventional TCP congestion control do not coexist well in |
| 564 |
the same network. In typical DCTCP deployments, the marking |
| 565 |
threshold in the switching fabric is set to a very low value to |
| 566 |
reduce queueing delay, and a relatively small amount of congestion |
| 567 |
will exceed the marking threshold. During such periods of |
| 568 |
congestion, conventional TCP will suffer packet loss and quickly and |
| 569 |
drastically reduce cwnd. DCTCP, on the other hand, will use the |
| 570 |
fraction of marked packets to reduce cwnd more gradually. Thus, the |
| 571 |
rate reduction in DCTCP will be much slower than that of conventional |
| 572 |
TCP, and DCTCP traffic will gain a larger share of the capacity |
| 573 |
compared to conventional TCP traffic traversing the same path. If |
| 574 |
the traffic in the data center is a mix of conventional TCP and |
| 575 |
DCTCP, it is RECOMMENDED that DCTCP traffic be segregated from |
| 576 |
conventional TCP traffic. [<a href="#ref-MORGANSTANLEY">MORGANSTANLEY</a>] describes a deployment |
| 577 |
that uses the IP Differentiated Services Codepoint (DSCP) bits to |
| 578 |
segregate the network such that Active Queue Management (AQM) |
| 579 |
[<a href="./rfc7567" title=""IETF Recommendations Regarding Active Queue Management"">RFC7567</a>] is applied to DCTCP traffic, whereas TCP traffic is managed |
| 580 |
via drop-tail queueing. |
| 581 |
|
| 582 |
Deployments should take into account segregation of non-TCP traffic |
| 583 |
as well. Today's commodity switches allow configuration of different |
| 584 |
marking/drop profiles for non-TCP and non-IP packets. Non-TCP and |
| 585 |
non-IP packets should be able to pass through such switches, unless |
| 586 |
they really run out of buffer space. |
| 587 |
|
| 588 |
Since DCTCP relies on congestion marking by the switches, DCTCP's |
| 589 |
potential can only be realized in data centers where the entire |
| 590 |
network infrastructure supports ECN. The switches may also support |
| 591 |
configuration of the congestion threshold used for marking. The |
| 592 |
proposed parameterization can be configured with switches that |
| 593 |
implement Random Early Detection (RED) [<a href="./rfc2309" title=""Recommendations on Queue Management and Congestion Avoidance in the Internet"">RFC2309</a>]. [<a href="#ref-DCTCP10" title=""Data Center TCP (DCTCP)"">DCTCP10</a>] provides |
| 594 |
a theoretical basis for selecting the congestion threshold, but, as |
| 595 |
with the estimation gain, it may be more practical to rely on |
| 596 |
experimentation or simply to use the default configuration of the |
| 597 |
device. DCTCP will revert to loss-based congestion control when |
| 598 |
packet loss is experienced (e.g., when transiting a congested drop- |
| 599 |
tail link, or a link with an AQM drop behavior). |
| 600 |
|
| 601 |
DCTCP requires changes on both the sender and the receiver, so both |
| 602 |
endpoints must support DCTCP. Furthermore, DCTCP provides no |
| 603 |
mechanism for negotiating its use, so both endpoints must be |
| 604 |
configured through some out-of-band mechanism to use DCTCP. A |
| 605 |
variant of DCTCP that can be deployed unilaterally and that only |
| 606 |
requires standard ECN behavior has been described in [<a href="#ref-ODCTCP" title=""Improving Transmission Performance with One- Sided Datacenter TCP"">ODCTCP</a>] and |
| 607 |
[<a href="#ref-BSDCAN" title=""Extensions to FreeBSD Datacenter TCP for Incremental Deployment Support"">BSDCAN</a>], but it requires additional experimental evaluation. |
| 608 |
|
| 609 |
|
| 610 |
|
| 611 |
|
| 612 |
<span class="grey">Bensley, et al. Informational [Page 11]</span></pre> |
| 613 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-12" ></span> |
| 614 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 615 |
|
| 616 |
|
| 617 |
<span class="h2"><a class="selflink" id="section-6" href="#section-6">6</a>. Known Issues</span> |
| 618 |
|
| 619 |
DCTCP relies on the sender's ability to reconstruct the stream of CE |
| 620 |
codepoints received by the remote endpoint. To accomplish this, |
| 621 |
DCTCP avoids using a single ACK packet to acknowledge segments |
| 622 |
received both with and without the CE codepoint set. However, if one |
| 623 |
or more ACK packets are dropped, it is possible that a subsequent ACK |
| 624 |
will cumulatively acknowledge a mix of CE and non-CE segments. This |
| 625 |
will, of course, result in a less-accurate congestion estimate. |
| 626 |
There are some potential considerations: |
| 627 |
|
| 628 |
o Even with an inaccurate congestion estimate, DCTCP may still |
| 629 |
perform better than [<a href="./rfc3168" title=""The Addition of Explicit Congestion Notification (ECN) to IP"">RFC3168</a>]. |
| 630 |
|
| 631 |
o If the estimation gain is small relative to the packet loss rate, |
| 632 |
the estimate may not be too inaccurate. |
| 633 |
|
| 634 |
o If ACK packet loss mostly occurs under heavy congestion, most |
| 635 |
drops will occur during an unbroken string of CE packets, and the |
| 636 |
estimate will be unaffected. |
| 637 |
|
| 638 |
However, the effect of packet drops on DCTCP under real-world |
| 639 |
conditions has not been analyzed. |
| 640 |
|
| 641 |
DCTCP provides no mechanism for negotiating its use. The effect of |
| 642 |
using DCTCP with a standard ECN endpoint has been analyzed in |
| 643 |
[<a href="#ref-ODCTCP" title=""Improving Transmission Performance with One- Sided Datacenter TCP"">ODCTCP</a>] and [<a href="#ref-BSDCAN" title=""Extensions to FreeBSD Datacenter TCP for Incremental Deployment Support"">BSDCAN</a>]. Furthermore, it is possible that other |
| 644 |
implementations may also modify behavior in the [<a href="./rfc3168" title=""The Addition of Explicit Congestion Notification (ECN) to IP"">RFC3168</a>] style |
| 645 |
without negotiation, causing further interoperability issues. |
| 646 |
|
| 647 |
Much like standard TCP, DCTCP is biased against flows with longer |
| 648 |
RTTs. A method for improving the RTT fairness of DCTCP has been |
| 649 |
proposed in [<a href="#ref-ADCTCP" title=""Analysis of DCTCP: Stability, Convergence, and Fairness"">ADCTCP</a>], but it requires additional experimental |
| 650 |
evaluation. |
| 651 |
|
| 652 |
<span class="h2"><a class="selflink" id="section-7" href="#section-7">7</a>. Security Considerations</span> |
| 653 |
|
| 654 |
DCTCP enhances ECN; thus, it inherits the general security |
| 655 |
considerations discussed in [<a href="./rfc3168" title=""The Addition of Explicit Congestion Notification (ECN) to IP"">RFC3168</a>], although additional mitigation |
| 656 |
options exist due to the limited intra-data-center deployment of |
| 657 |
DCTCP. |
| 658 |
|
| 659 |
The processing changes introduced by DCTCP do not exacerbate the |
| 660 |
considerations in [<a href="./rfc3168" title=""The Addition of Explicit Congestion Notification (ECN) to IP"">RFC3168</a>] or introduce new ones. In particular, |
| 661 |
with either algorithm, the network infrastructure or the remote |
| 662 |
endpoint can falsely report congestion and, thus, cause the sender to |
| 663 |
reduce cwnd. However, this is no worse than what can be achieved by |
| 664 |
simply dropping packets. |
| 665 |
|
| 666 |
|
| 667 |
|
| 668 |
<span class="grey">Bensley, et al. Informational [Page 12]</span></pre> |
| 669 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-13" ></span> |
| 670 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 671 |
|
| 672 |
|
| 673 |
[<a id="ref-RFC3168">RFC3168</a>] requires that a compliant TCP must not set ECT on SYN or |
| 674 |
SYN-ACK packets. [<a href="./rfc5562" title=""Adding Explicit Congestion Notification (ECN) Capability to TCP's SYN/ACK Packets"">RFC5562</a>] proposes setting ECT on SYN-ACK packets |
| 675 |
but maintains the restriction of no ECT on SYN packets. Both these |
| 676 |
RFCs prohibit ECT in SYN packets due to security concerns regarding |
| 677 |
malicious SYN packets with ECT set. However, these RFCs are intended |
| 678 |
for general Internet use; they do not directly apply to a controlled |
| 679 |
data-center environment. The security concerns addressed by both of |
| 680 |
these RFCs might not apply in controlled environments like data |
| 681 |
centers, and it might not be necessary to account for the presence of |
| 682 |
non-ECN servers. Beyond the security considerations related to |
| 683 |
virtual servers, additional security can be imposed in the physical |
| 684 |
servers to intercept and drop traffic resembling an attack. |
| 685 |
|
| 686 |
<span class="h2"><a class="selflink" id="section-8" href="#section-8">8</a>. IANA Considerations</span> |
| 687 |
|
| 688 |
This document does not require any IANA actions. |
| 689 |
|
| 690 |
<span class="h2"><a class="selflink" id="section-9" href="#section-9">9</a>. References</span> |
| 691 |
|
| 692 |
<span class="h3"><a class="selflink" id="section-9.1" href="#section-9.1">9.1</a>. Normative References</span> |
| 693 |
|
| 694 |
[<a id="ref-RFC0793">RFC0793</a>] Postel, J., "Transmission Control Protocol", STD 7, |
| 695 |
<a href="./rfc793">RFC 793</a>, DOI 10.17487/RFC0793, September 1981, |
| 696 |
<<a href="https://www.rfc-editor.org/info/rfc793">https://www.rfc-editor.org/info/rfc793</a>>. |
| 697 |
|
| 698 |
[<a id="ref-RFC2018">RFC2018</a>] Mathis, M., Mahdavi, J., Floyd, S., and A. Romanow, "TCP |
| 699 |
Selective Acknowledgment Options", <a href="./rfc2018">RFC 2018</a>, |
| 700 |
DOI 10.17487/RFC2018, October 1996, |
| 701 |
<<a href="https://www.rfc-editor.org/info/rfc2018">https://www.rfc-editor.org/info/rfc2018</a>>. |
| 702 |
|
| 703 |
[<a id="ref-RFC2119">RFC2119</a>] Bradner, S., "Key words for use in RFCs to Indicate |
| 704 |
Requirement Levels", <a href="https://www.rfc-editor.org/bcp/bcp14">BCP 14</a>, <a href="./rfc2119">RFC 2119</a>, |
| 705 |
DOI 10.17487/RFC2119, March 1997, |
| 706 |
<<a href="https://www.rfc-editor.org/info/rfc2119">https://www.rfc-editor.org/info/rfc2119</a>>. |
| 707 |
|
| 708 |
[<a id="ref-RFC3168">RFC3168</a>] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition |
| 709 |
of Explicit Congestion Notification (ECN) to IP", |
| 710 |
<a href="./rfc3168">RFC 3168</a>, DOI 10.17487/RFC3168, September 2001, |
| 711 |
<<a href="https://www.rfc-editor.org/info/rfc3168">https://www.rfc-editor.org/info/rfc3168</a>>. |
| 712 |
|
| 713 |
[<a id="ref-RFC5562">RFC5562</a>] Kuzmanovic, A., Mondal, A., Floyd, S., and K. |
| 714 |
Ramakrishnan, "Adding Explicit Congestion Notification |
| 715 |
(ECN) Capability to TCP's SYN/ACK Packets", <a href="./rfc5562">RFC 5562</a>, |
| 716 |
DOI 10.17487/RFC5562, June 2009, |
| 717 |
<<a href="https://www.rfc-editor.org/info/rfc5562">https://www.rfc-editor.org/info/rfc5562</a>>. |
| 718 |
|
| 719 |
|
| 720 |
|
| 721 |
|
| 722 |
|
| 723 |
|
| 724 |
<span class="grey">Bensley, et al. Informational [Page 13]</span></pre> |
| 725 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-14" ></span> |
| 726 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 727 |
|
| 728 |
|
| 729 |
[<a id="ref-RFC5681">RFC5681</a>] Allman, M., Paxson, V., and E. Blanton, "TCP Congestion |
| 730 |
Control", <a href="./rfc5681">RFC 5681</a>, DOI 10.17487/RFC5681, September 2009, |
| 731 |
<<a href="https://www.rfc-editor.org/info/rfc5681">https://www.rfc-editor.org/info/rfc5681</a>>. |
| 732 |
|
| 733 |
[<a id="ref-RFC8174">RFC8174</a>] Leiba, B., "Ambiguity of Uppercase vs Lowercase in <a href="./rfc2119">RFC</a> |
| 734 |
<a href="./rfc2119">2119</a> Key Words", <a href="https://www.rfc-editor.org/bcp/bcp14">BCP 14</a>, <a href="./rfc8174">RFC 8174</a>, DOI 10.17487/RFC8174, |
| 735 |
May 2017, <<a href="https://www.rfc-editor.org/info/rfc8174">https://www.rfc-editor.org/info/rfc8174</a>>. |
| 736 |
|
| 737 |
<span class="h3"><a class="selflink" id="section-9.2" href="#section-9.2">9.2</a>. Informative References</span> |
| 738 |
|
| 739 |
[<a id="ref-ADCTCP">ADCTCP</a>] Alizadeh, M., Javanmard, A., and B. Prabhakar, "Analysis |
| 740 |
of DCTCP: Stability, Convergence, and Fairness", |
| 741 |
DOI 10.1145/1993744.1993753, Proceedings of the ACM |
| 742 |
SIGMETRICS Joint International Conference on Measurement |
| 743 |
and Modeling of Computer Systems, June 2011, |
| 744 |
<<a href="https://dl.acm.org/citation.cfm?id=1993753">https://dl.acm.org/citation.cfm?id=1993753</a>>. |
| 745 |
|
| 746 |
[<a id="ref-BSDCAN">BSDCAN</a>] Kato, M., Eggert, L., Zimmermann, A., van Meter, R., and |
| 747 |
H. Tokuda, "Extensions to FreeBSD Datacenter TCP for |
| 748 |
Incremental Deployment Support", BSDCan 2015, June 2015, |
| 749 |
<<a href="https://www.bsdcan.org/2015/schedule/events/559.en.html">https://www.bsdcan.org/2015/schedule/events/559.en.html</a>>. |
| 750 |
|
| 751 |
[<a id="ref-DCTCP10">DCTCP10</a>] Alizadeh, M., Greenberg, A., Maltz, D., Padhye, J., Patel, |
| 752 |
P., Prabhakar, B., Sengupta, S., and M. Sridharan, "Data |
| 753 |
Center TCP (DCTCP)", DOI 10.1145/1851182.1851192, |
| 754 |
Proceedings of the ACM SIGCOMM 2010 Conference, August |
| 755 |
2010, |
| 756 |
<<a href="http://dl.acm.org/citation.cfm?doid=1851182.1851192">http://dl.acm.org/citation.cfm?doid=1851182.1851192</a>>. |
| 757 |
|
| 758 |
[<a id="ref-ECN-EXPERIMENTATION">ECN-EXPERIMENTATION</a>] |
| 759 |
Black, D., "Explicit Congestion Notification (ECN) |
| 760 |
Experimentation", Work in Progress, <a href="./draft-ietf-tsvwg-ecn-experimentation-06">draft-ietf-tsvwg-ecn-</a> |
| 761 |
<a href="./draft-ietf-tsvwg-ecn-experimentation-06">experimentation-06</a>, September 2017. |
| 762 |
|
| 763 |
[<a id="ref-Err3639">Err3639</a>] RFC Errata, Erratum ID 3639, <a href="./rfc3168">RFC 3168</a>, |
| 764 |
<<a href="https://www.rfc-editor.org/errata/eid3639">https://www.rfc-editor.org/errata/eid3639</a>>. |
| 765 |
|
| 766 |
[<a id="ref-FREEBSD">FREEBSD</a>] Kato, M. and H. Panchasara, "DCTCP (Data Center TCP) |
| 767 |
implementation", January 2015, |
| 768 |
<<a href="https://github.com/freebsd/freebsd/commit/8ad879445281027858a7fa706d13e458095b595f">https://github.com/freebsd/freebsd/</a> |
| 769 |
<a href="https://github.com/freebsd/freebsd/commit/8ad879445281027858a7fa706d13e458095b595f">commit/8ad879445281027858a7fa706d13e458095b595f</a>>. |
| 770 |
|
| 771 |
[<a id="ref-LINUX">LINUX</a>] Borkmann, D., Westphal, F., and Glenn. Judd, "net: tcp: |
| 772 |
add DCTCP congestion control algorithm", LINUX DCTCP |
| 773 |
Patch, September 2014, <<a href="https://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/?id=e3118e8359bb7c59555aca60c725106e6d78c5ce">https://git.kernel.org/cgit/linux/</a> |
| 774 |
<a href="https://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/?id=e3118e8359bb7c59555aca60c725106e6d78c5ce">kernel/git/davem/net-next.git/commit/</a> |
| 775 |
<a href="https://git.kernel.org/cgit/linux/kernel/git/davem/net-next.git/commit/?id=e3118e8359bb7c59555aca60c725106e6d78c5ce">?id=e3118e8359bb7c59555aca60c725106e6d78c5ce</a>>. |
| 776 |
|
| 777 |
|
| 778 |
|
| 779 |
|
| 780 |
<span class="grey">Bensley, et al. Informational [Page 14]</span></pre> |
| 781 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-15" ></span> |
| 782 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 783 |
|
| 784 |
|
| 785 |
[<a id="ref-MAPREDUCE">MAPREDUCE</a>] |
| 786 |
Dean, J. and S. Ghemawat, "MapReduce: Simplified Data |
| 787 |
Processing on Large Clusters", Proceedings of the 6th |
| 788 |
ACM/USENIX Symposium on Operating Systems Design and |
| 789 |
Implementation, October 2004, <<a href="https://www.usenix.org/legacy/publications/library/proceedings/osdi04/tech/dean.html">https://www.usenix.org/</a> |
| 790 |
<a href="https://www.usenix.org/legacy/publications/library/proceedings/osdi04/tech/dean.html">legacy/publications/library/proceedings/osdi04/tech/</a> |
| 791 |
<a href="https://www.usenix.org/legacy/publications/library/proceedings/osdi04/tech/dean.html">dean.html</a>>. |
| 792 |
|
| 793 |
[<a id="ref-MORGANSTANLEY">MORGANSTANLEY</a>] |
| 794 |
Judd, G., "Attaining the Promise and Avoiding the Pitfalls |
| 795 |
of TCP in the Datacenter", Proceedings of the 12th USENIX |
| 796 |
Symposium on Networked Systems Design and Implementation, |
| 797 |
May 2015, <<a href="https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/judd">https://www.usenix.org/conference/nsdi15/</a> |
| 798 |
<a href="https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/judd">technical-sessions/presentation/judd</a>>. |
| 799 |
|
| 800 |
[<a id="ref-ODCTCP">ODCTCP</a>] Kato, M., "Improving Transmission Performance with One- |
| 801 |
Sided Datacenter TCP", M.S. Thesis, Keio University, 2013, |
| 802 |
<<a href="http://eggert.org/students/kato-thesis.pdf">http://eggert.org/students/kato-thesis.pdf</a>>. |
| 803 |
|
| 804 |
[<a id="ref-RFC2309">RFC2309</a>] Braden, B., Clark, D., Crowcroft, J., Davie, B., Deering, |
| 805 |
S., Estrin, D., Floyd, S., Jacobson, V., Minshall, G., |
| 806 |
Partridge, C., Peterson, L., Ramakrishnan, K., Shenker, |
| 807 |
S., Wroclawski, J., and L. Zhang, "Recommendations on |
| 808 |
Queue Management and Congestion Avoidance in the |
| 809 |
Internet", <a href="./rfc2309">RFC 2309</a>, DOI 10.17487/RFC2309, April 1998, |
| 810 |
<<a href="https://www.rfc-editor.org/info/rfc2309">https://www.rfc-editor.org/info/rfc2309</a>>. |
| 811 |
|
| 812 |
[<a id="ref-RFC7567">RFC7567</a>] Baker, F., Ed. and G. Fairhurst, Ed., "IETF |
| 813 |
Recommendations Regarding Active Queue Management", |
| 814 |
<a href="https://www.rfc-editor.org/bcp/bcp197">BCP 197</a>, <a href="./rfc7567">RFC 7567</a>, DOI 10.17487/RFC7567, July 2015, |
| 815 |
<<a href="https://www.rfc-editor.org/info/rfc7567">https://www.rfc-editor.org/info/rfc7567</a>>. |
| 816 |
|
| 817 |
[<a id="ref-WINDOWS">WINDOWS</a>] Microsoft, "Data Center Transmission Control Protocol |
| 818 |
(DCTCP)", May 2012, <<a href="https://technet.microsoft.com/">https://technet.microsoft.com/</a> |
| 819 |
en-us/library/hh997028(v=ws.11).aspx>. |
| 820 |
|
| 821 |
|
| 822 |
|
| 823 |
|
| 824 |
|
| 825 |
|
| 826 |
|
| 827 |
|
| 828 |
|
| 829 |
|
| 830 |
|
| 831 |
|
| 832 |
|
| 833 |
|
| 834 |
|
| 835 |
|
| 836 |
<span class="grey">Bensley, et al. Informational [Page 15]</span></pre> |
| 837 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-16" ></span> |
| 838 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 839 |
|
| 840 |
|
| 841 |
Acknowledgments |
| 842 |
|
| 843 |
The DCTCP algorithm was originally proposed and analyzed in [<a href="#ref-DCTCP10" title=""Data Center TCP (DCTCP)"">DCTCP10</a>] |
| 844 |
by Mohammad Alizadeh, Albert Greenberg, Dave Maltz, Jitu Padhye, |
| 845 |
Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, and Murari |
| 846 |
Sridharan. |
| 847 |
|
| 848 |
We would like to thank Andrew Shewmaker for identifying the problem |
| 849 |
of clamping DCTCP.Alpha and proposing a solution for it. |
| 850 |
|
| 851 |
Lars Eggert has received funding from the European Union's Horizon |
| 852 |
2020 research and innovation program 2014-2018 under grant agreement |
| 853 |
No. 644866 ("SSICLOPS"). This document reflects only the authors' |
| 854 |
views and the European Commission is not responsible for any use that |
| 855 |
may be made of the information it contains. |
| 856 |
|
| 857 |
Authors' Addresses |
| 858 |
|
| 859 |
Stephen Bensley |
| 860 |
Microsoft |
| 861 |
One Microsoft Way |
| 862 |
Redmond, WA 98052 |
| 863 |
United States of America |
| 864 |
|
| 865 |
Phone: +1 425 703 5570 |
| 866 |
Email: [email protected] |
| 867 |
|
| 868 |
|
| 869 |
Dave Thaler |
| 870 |
Microsoft |
| 871 |
|
| 872 |
Phone: +1 425 703 8835 |
| 873 |
Email: [email protected] |
| 874 |
|
| 875 |
|
| 876 |
Praveen Balasubramanian |
| 877 |
Microsoft |
| 878 |
|
| 879 |
Phone: +1 425 538 2782 |
| 880 |
Email: [email protected] |
| 881 |
|
| 882 |
|
| 883 |
|
| 884 |
|
| 885 |
|
| 886 |
|
| 887 |
|
| 888 |
|
| 889 |
|
| 890 |
|
| 891 |
|
| 892 |
<span class="grey">Bensley, et al. Informational [Page 16]</span></pre> |
| 893 |
<hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-17" ></span> |
| 894 |
<span class="grey"><a href="./rfc8257">RFC 8257</a> DCTCP October 2017</span> |
| 895 |
|
| 896 |
|
| 897 |
Lars Eggert |
| 898 |
NetApp |
| 899 |
Sonnenallee 1 |
| 900 |
Kirchheim 85551 |
| 901 |
Germany |
| 902 |
|
| 903 |
Phone: +49 151 120 55791 |
| 904 |
Email: [email protected] |
| 905 |
URI: <a href="http://eggert.org/">http://eggert.org/</a> |
| 906 |
|
| 907 |
|
| 908 |
Glenn Judd |
| 909 |
Morgan Stanley |
| 910 |
|
| 911 |
Phone: +1 973 979 6481 |
| 912 |
Email: [email protected] |
| 913 |
|
| 914 |
|
| 915 |
|
| 916 |
|
| 917 |
|
| 918 |
|
| 919 |
|
| 920 |
|
| 921 |
|
| 922 |
|
| 923 |
|
| 924 |
|
| 925 |
|
| 926 |
|
| 927 |
|
| 928 |
|
| 929 |
|
| 930 |
|
| 931 |
|
| 932 |
|
| 933 |
|
| 934 |
|
| 935 |
|
| 936 |
|
| 937 |
|
| 938 |
|
| 939 |
|
| 940 |
|
| 941 |
|
| 942 |
|
| 943 |
|
| 944 |
|
| 945 |
|
| 946 |
|
| 947 |
|
| 948 |
Bensley, et al. Informational [Page 17] |
| 949 |
</pre> |