Debian Codegraph @

rfc1944.html

raw
1 <pre>Network Working Group S. Bradner
2 Request for Comments: 1944 Harvard University
3 Category: Informational J. McQuaid
4 Bay Networks
5 May 1996
6
7
8 <span class="h1">Benchmarking Methodology for Network Interconnect Devices</span>
9
10 Status of This Memo
11
12 This memo provides information for the Internet community. This memo
13 does not specify an Internet standard of any kind. Distribution of
14 this memo is unlimited.
15
16 Abstract
17
18 This document discusses and defines a number of tests that may be
19 used to describe the performance characteristics of a network
20 interconnecting device. In addition to defining the tests this
21 document also describes specific formats for reporting the results of
22 the tests. <a href="#appendix-A">Appendix A</a> lists the tests and conditions that we believe
23 should be included for specific cases and gives additional
24 information about testing practices. <a href="#appendix-B">Appendix B</a> is a reference
25 listing of maximum frame rates to be used with specific frame sizes
26 on various media and <a href="#appendix-C">Appendix C</a> gives some examples of frame formats
27 to be used in testing.
28
29 <span class="h2"><a class="selflink" id="section-1" href="#section-1">1</a>. Introduction</span>
30
31 Vendors often engage in "specsmanship" in an attempt to give their
32 products a better position in the marketplace. This often involves
33 "smoke &amp; mirrors" to confuse the potential users of the products.
34
35 This document defines a specific set of tests that vendors can use to
36 measure and report the performance characteristics of network
37 devices. The results of these tests will provide the user comparable
38 data from different vendors with which to evaluate these devices.
39
40 A previous document, "Benchmarking Terminology for Network
41 Interconnect Devices" (<a href="./rfc1242">RFC 1242</a>), defined many of the terms that are
42 used in this document. The terminology document should be consulted
43 before attempting to make use of this document.
44
45
46
47
48
49
50
51
52 <span class="grey">Bradner &amp; McQuaid Informational [Page 1]</span></pre>
53 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-2" ></span>
54 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
55
56
57 <span class="h2"><a class="selflink" id="section-2" href="#section-2">2</a>. Real world</span>
58
59 In producing this document the authors attempted to keep in mind the
60 requirement that apparatus to perform the described tests must
61 actually be built. We do not know of "off the shelf" equipment
62 available to implement all of the tests but it is our opinion that
63 such equipment can be constructed.
64
65 <span class="h2"><a class="selflink" id="section-3" href="#section-3">3</a>. Tests to be run</span>
66
67 There are a number of tests described in this document. Not all of
68 the tests apply to all types of devices under test (DUTs). Vendors
69 should perform all of the tests that can be supported by a specific
70 type of product. The authors understand that it will take a
71 considerable period of time to perform all of the recommended tests
72 nder all of the recommended conditions. We believe that the results
73 are worth the effort. <a href="#appendix-A">Appendix A</a> lists some of the tests and
74 conditions that we believe should be included for specific cases.
75
76 <span class="h2"><a class="selflink" id="section-4" href="#section-4">4</a>. Evaluating the results</span>
77
78 Performing all of the recommended tests will result in a great deal
79 of data. Much of this data will not apply to the evaluation of the
80 devices under each circumstance. For example, the rate at which a
81 router forwards IPX frames will be of little use in selecting a
82 router for an environment that does not (and will not) support that
83 protocol. Evaluating even that data which is relevant to a
84 particular network installation will require experience which may not
85 be readily available. Furthermore, selection of the tests to be run
86 and evaluation of the test data must be done with an understanding of
87 generally accepted testing practices regarding repeatability,
88 variance and statistical significance of small numbers of trials.
89
90 <span class="h2"><a class="selflink" id="section-5" href="#section-5">5</a>. Requirements</span>
91
92 In this document, the words that are used to define the significance
93 of each particular requirement are capitalized. These words are:
94
95 * "MUST" This word, or the words "REQUIRED" and "SHALL" mean that
96 the item is an absolute requirement of the specification.
97
98 * "SHOULD" This word or the adjective "RECOMMENDED" means that there
99 may exist valid reasons in particular circumstances to ignore this
100 item, but the full implications should be understood and the case
101 carefully weighed before choosing a different course.
102
103 * "MAY" This word or the adjective "OPTIONAL" means that this item
104 is truly optional. One vendor may choose to include the item because
105
106
107
108 <span class="grey">Bradner &amp; McQuaid Informational [Page 2]</span></pre>
109 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-3" ></span>
110 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
111
112
113 a particular marketplace requires it or because it enhances the
114 product, for example; another vendor may omit the same item.
115
116 An implementation is not compliant if it fails to satisfy one or more
117 of the MUST requirements for the protocols it implements. An
118 implementation that satisfies all the MUST and all the SHOULD
119 requirements for its protocols is said to be "unconditionally
120 compliant"; one that satisfies all the MUST requirements but not all
121 the SHOULD requirements for its protocols is said to be
122 "conditionally compliant".
123
124 <span class="h2"><a class="selflink" id="section-6" href="#section-6">6</a>. Test set up</span>
125
126 The ideal way to implement this series of tests is to use a tester
127 with both transmitting and receiving ports. Connections are made
128 from the sending ports of the tester to the receiving ports of the
129 DUT and from the sending ports of the DUT back to the tester. (see
130 Figure 1) Since the tester both sends the test traffic and receives
131 it back, after the traffic has been forwarded but the DUT, the tester
132 can easily determine if all of the transmitted packets were received
133 and verify that the correct packets were received. The same
134 functionality can be obtained with separate transmitting and
135 receiving devices (see Figure 2) but unless they are remotely
136 controlled by some computer in a way that simulates the single
137 tester, the labor required to accurately perform some of the tests
138 (particularly the throughput test) can be prohibitive.
139
140 +------------+
141 | |
142 +------------| tester |&lt;-------------+
143 | | | |
144 | +------------+ |
145 | |
146 | +------------+ |
147 | | | |
148 +-----------&gt;| DUT |--------------+
149 | |
150 +------------+
151 Figure 1
152
153 +--------+ +------------+ +----------+
154 | | | | | |
155 | sender |--------&gt;| DUT |---------&gt;| receiver |
156 | | | | | |
157 +--------+ +------------+ +----------+
158 Figure 2
159
160
161
162
163
164 <span class="grey">Bradner &amp; McQuaid Informational [Page 3]</span></pre>
165 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-4" ></span>
166 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
167
168
169 <span class="h3"><a class="selflink" id="section-6.1" href="#section-6.1">6.1</a> Test set up for multiple media types</span>
170
171 Two different setups could be used to test a DUT which is used in
172 real-world networks to connect networks of differing media type,
173 local Ethernet to a backbone FDDI ring for example. The tester could
174 support both media types in which case the set up shown in Figure 1
175 would be used.
176
177 Two identical DUTs are used in the other test set up. (see Figure 3)
178 In many cases this set up may more accurately simulate the real
179 world. For example, connecting two LANs together with a WAN link or
180 high speed backbone. This set up would not be as good at simulating
181 a system where clients on a Ethernet LAN were interacting with a
182 server on an FDDI backbone.
183
184 +-----------+
185 | |
186 +---------------------| tester |&lt;---------------------+
187 | | | |
188 | +-----------+ |
189 | |
190 | +----------+ +----------+ |
191 | | | | | |
192 +-------&gt;| DUT 1 |--------------&gt;| DUT 2 |---------+
193 | | | |
194 +----------+ +----------+
195
196 Figure 3
197
198 <span class="h2"><a class="selflink" id="section-7" href="#section-7">7</a>. DUT set up</span>
199
200 Before starting to perform the tests, the DUT to be tested MUST be
201 configured following the instructions provided to the user.
202 Specifically, it is expected that all of the supported protocols will
203 be configured and enabled during this set up (See <a href="#appendix-A">Appendix A</a>). It is
204 expected that all of the tests will be run without changing the
205 configuration or setup of the DUT in any way other than that required
206 to do the specific test. For example, it is not acceptable to change
207 the size of frame handling buffers between tests of frame handling
208 rates or to disable all but one transport protocol when testing the
209 throughput of that protocol. It is necessary to modify the
210 configuration when starting a test to determine the effect of filters
211 on throughput, but the only change MUST be to enable the specific
212 filter. The DUT set up SHOULD include the normally recommended
213 routing update intervals and keep alive frequency. The specific
214 version of the software and the exact DUT configuration, including
215 what functions are disabled, used during the tests MUST be included
216 as part of the report of the results.
217
218
219
220 <span class="grey">Bradner &amp; McQuaid Informational [Page 4]</span></pre>
221 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-5" ></span>
222 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
223
224
225 <span class="h2"><a class="selflink" id="section-8" href="#section-8">8</a>. Frame formats</span>
226
227 The formats of the test frames to use for TCP/IP over Ethernet are
228 shown in <a href="#appendix-C">Appendix C</a>: Test Frame Formats. These exact frame formats
229 SHOULD be used in the tests described in this document for this
230 protocol/media combination and that these frames will be used as a
231 template for testing other protocol/media combinations. The specific
232 formats that are used to define the test frames for a particular test
233 series MUST be included in the report of the results.
234
235 <span class="h2"><a class="selflink" id="section-9" href="#section-9">9</a>. Frame sizes</span>
236
237 All of the described tests SHOULD be performed at a number of frame
238 sizes. Specifically, the sizes SHOULD include the maximum and minimum
239 legitimate sizes for the protocol under test on the media under test
240 and enough sizes in between to be able to get a full characterization
241 of the DUT performance. Except where noted, at least five frame
242 sizes SHOULD be tested for each test condition.
243
244 Theoretically the minimum size UDP Echo request frame would consist
245 of an IP header (minimum length 20 octets), a UDP header (8 octets)
246 and whatever MAC level header is required by the media in use. The
247 theoretical maximum frame size is determined by the size of the
248 length field in the IP header. In almost all cases the actual
249 maximum and minimum sizes are determined by the limitations of the
250 media.
251
252 In theory it would be ideal to distribute the frame sizes in a way
253 that would evenly distribute the theoretical frame rates. These
254 recommendations incorporate this theory but specify frame sizes which
255 are easy to understand and remember. In addition, many of the same
256 frame sizes are specified on each of the media types to allow for
257 easy performance comparisons.
258
259 Note: The inclusion of an unrealistically small frame size on some of
260 the media types (i.e. with little or no space for data) is to help
261 characterize the per-frame processing overhead of the DUT.
262
263 9.1 Frame sizes to be used on Ethernet
264
265 64, 128, 256, 512, 1024, 1280, 1518
266
267 These sizes include the maximum and minimum frame sizes permitted
268 by the Ethernet standard and a selection of sizes between these
269 extremes with a finer granularity for the smaller frame sizes and
270 higher frame rates.
271
272
273
274
275
276 <span class="grey">Bradner &amp; McQuaid Informational [Page 5]</span></pre>
277 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-6" ></span>
278 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
279
280
281 9.2 Frame sizes to be used on 4Mb and 16Mb token ring
282
283 54, 64, 128, 256, 1024, 1518, 2048, 4472
284
285 The frame size recommendations for token ring assume that there is
286 no RIF field in the frames of routed protocols. A RIF field would
287 be present in any direct source route bridge performance test.
288 The minimum size frame for UDP on token ring is 54 octets. The
289 maximum size of 4472 octets is recommended for 16Mb token ring
290 instead of the theoretical size of 17.9Kb because of the size
291 limitations imposed by many token ring interfaces. The reminder
292 of the sizes are selected to permit direct comparisons with other
293 types of media. An IP (i.e. not UDP) frame may be used in
294 addition if a higher data rate is desired, in which case the
295 minimum frame size is 46 octets.
296
297 9.3 Frame sizes to be used on FDDI
298
299 54, 64, 128, 256, 1024, 1518, 2048, 4472
300
301 The minimum size frame for UDP on FDDI is 53 octets, the minimum
302 size of 54 is recommended to allow direct comparison to token ring
303 performance. The maximum size of 4472 is recommended instead of
304 the theoretical maximum size of 4500 octets to permit the same
305 type of comparison. An IP (i.e. not UDP) frame may be used in
306 addition if a higher data rate is desired, in which case the
307 minimum frame size is 45 octets.
308
309 9.4 Frame sizes in the presence of disparate MTUs
310
311 When the interconnect DUT supports connecting links with disparate
312 MTUs, the frame sizes for the link with the *larger* MTU SHOULD be
313 used, up to the limit of the protocol being tested. If the
314 interconnect DUT does not support the fragmenting of frames in the
315 presence of MTU mismatch, the forwarding rate for that frame size
316 shall be reported as zero.
317
318 For example, the test of IP forwarding with a bridge or router
319 that joins FDDI and Ethernet should use the frame sizes of FDDI
320 when going from the FDDI to the Ethernet link. If the bridge does
321 not support IP fragmentation, the forwarding rate for those frames
322 too large for Ethernet should be reported as zero.
323
324 <span class="h2"><a class="selflink" id="section-10" href="#section-10">10</a>. Verifying received frames</span>
325
326 The test equipment SHOULD discard any frames received during a test
327 run that are not actual forwarded test frames. For example, keep-
328 alive and routing update frames SHOULD NOT be included in the count
329
330
331
332 <span class="grey">Bradner &amp; McQuaid Informational [Page 6]</span></pre>
333 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-7" ></span>
334 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
335
336
337 of received frames. In any case, the test equipment SHOULD verify
338 the length of the received frames and check that they match the
339 expected length.
340
341 Preferably, the test equipment SHOULD include sequence numbers in the
342 transmitted frames and check for these numbers on the received
343 frames. If this is done, the reported results SHOULD include in
344 addition to the number of frames dropped, the number of frames that
345 were received out of order, the number of duplicate frames received
346 and the number of gaps in the received frame numbering sequence.
347 This functionality is required for some of the described tests.
348
349 <span class="h2"><a class="selflink" id="section-11" href="#section-11">11</a>. Modifiers</span>
350
351 It might be useful to know the DUT performance under a number of
352 conditions; some of these conditions are noted below. The reported
353 results SHOULD include as many of these conditions as the test
354 equipment is able to generate. The suite of tests SHOULD be first
355 run without any modifying conditions and then repeated under each of
356 the conditions separately. To preserve the ability to compare the
357 results of these tests any frames that are required to generate the
358 modifying conditions (management queries for example) will be
359 included in the same data stream as the normal test frames in place
360 of one of the test frames and not be supplied to the DUT on a
361 separate network port.
362
363 11.1 Broadcast frames
364
365 In most router designs special processing is required when frames
366 addressed to the hardware broadcast address are received. In
367 bridges (or in bridge mode on routers) these broadcast frames must
368 be flooded to a number of ports. The stream of test frames SHOULD
369 be augmented with 1% frames addressed to the hardware broadcast
370 address. The frames sent to the broadcast address should be of a
371 type that the router will not need to process. The aim of this
372 test is to determine if there is any effect on the forwarding rate
373 of the other data in the stream. The specific frames that should
374 be used are included in the test frame format document. The
375 broadcast frames SHOULD be evenly distributed throughout the data
376 stream, for example, every 100th frame.
377
378 The same test SHOULD be performed on bridge-like DUTs but in this
379 case the broadcast packets will be processed and flooded to all
380 outputs.
381
382 It is understood that a level of broadcast frames of 1% is much
383 higher than many networks experience but, as in drug toxicity
384 evaluations, the higher level is required to be able to gage the
385
386
387
388 <span class="grey">Bradner &amp; McQuaid Informational [Page 7]</span></pre>
389 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-8" ></span>
390 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
391
392
393 effect which would otherwise often fall within the normal
394 variability of the system performance. Due to design factors some
395 test equipment will not be able to generate a level of alternate
396 frames this low. In these cases the percentage SHOULD be as small
397 as the equipment can provide and that the actual level be
398 described in the report of the test results.
399
400 11.2 Management frames
401
402 Most data networks now make use of management protocols such as
403 SNMP. In many environments there can be a number of management
404 stations sending queries to the same DUT at the same time.
405
406 The stream of test frames SHOULD be augmented with one management
407 query as the first frame sent each second during the duration of
408 the trial. The result of the query must fit into one response
409 frame. The response frame SHOULD be verified by the test
410 equipment. One example of the specific query frame that should be
411 used is shown in <a href="#appendix-C">Appendix C</a>.
412
413 11.3 Routing update frames
414
415 The processing of dynamic routing protocol updates could have a
416 significant impact on the ability of a router to forward data
417 frames. The stream of test frames SHOULD be augmented with one
418 routing update frame transmitted as the first frame transmitted
419 during the trial. Routing update frames SHOULD be sent at the
420 rate specified in <a href="#appendix-C">Appendix C</a> for the specific routing protocol
421 being used in the test. Two routing update frames are defined in
422 <a href="#appendix-C">Appendix C</a> for the TCP/IP over Ethernet example. The routing
423 frames are designed to change the routing to a number of networks
424 that are not involved in the forwarding of the test data. The
425 first frame sets the routing table state to "A", the second one
426 changes the state to "B". The frames MUST be alternated during
427 the trial.
428
429 The test SHOULD verify that the routing update was processed by
430 the DUT.
431
432 11.4 Filters
433
434 Filters are added to routers and bridges to selectively inhibit
435 the forwarding of frames that would normally be forwarded. This
436 is usually done to implement security controls on the data that is
437 accepted between one area and another. Different products have
438 different capabilities to implement filters.
439
440
441
442
443
444 <span class="grey">Bradner &amp; McQuaid Informational [Page 8]</span></pre>
445 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-9" ></span>
446 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
447
448
449 The DUT SHOULD be first configured to add one filter condition and
450 the tests performed. This filter SHOULD permit the forwarding of
451 the test data stream. In routers this filter SHOULD be of the
452 form:
453
454 forward input_protocol_address to output_protocol_address
455
456 In bridges the filter SHOULD be of the form:
457
458 forward destination_hardware_address
459
460 The DUT SHOULD be then reconfigured to implement a total of 25
461 filters. The first 24 of these filters SHOULD be of the form:
462
463 block input_protocol_address to output_protocol_address
464
465 The 24 input and output protocol addresses SHOULD not be any that
466 are represented in the test data stream. The last filter SHOULD
467 permit the forwarding of the test data stream. By "first" and
468 "last" we mean to ensure that in the second case, 25 conditions
469 must be checked before the data frames will match the conditions
470 that permit the forwarding of the frame. Of course, if the DUT
471 reorders the filters or does not use a linear scan of the filter
472 rules the effect of the sequence in which the filters are input is
473 properly lost.
474
475 The exact filters configuration command lines used SHOULD be
476 included with the report of the results.
477
478 11.4.1 Filter Addresses
479
480 Two sets of filter addresses are required, one for the single
481 filter case and one for the 25 filter case.
482
483 The single filter case should permit traffic from IP address
484 198.18.1.2 to IP address 198.19.65.2 and deny all other
485 traffic.
486
487 The 25 filter case should follow the following sequence.
488
489 deny aa.ba.1.1 to aa.ba.100.1
490 deny aa.ba.2.2 to aa.ba.101.2
491 deny aa.ba.3.3 to aa.ba.103.3
492 ...
493 deny aa.ba.12.12 to aa.ba.112.12
494 allow aa.bc.1.2 to aa.bc.65.1
495 deny aa.ba.13.13 to aa.ba.113.13
496 deny aa.ba.14.14 to aa.ba.114.14
497
498
499
500 <span class="grey">Bradner &amp; McQuaid Informational [Page 9]</span></pre>
501 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-10" ></span>
502 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
503
504
505 ...
506 deny aa.ba.24.24 to aa.ba.124.24
507 deny all else
508
509 All previous filter conditions should be cleared from the
510 router before this sequence is entered. The sequence is
511 selected to test to see if the router sorts the filter
512 conditions or accepts them in the order that they were entered.
513 Both of these procedures will result in a greater impact on
514 performance than will some form of hash coding.
515
516 <span class="h2"><a class="selflink" id="section-12" href="#section-12">12</a>. Protocol addresses</span>
517
518 It is easier to implement these tests using a single logical stream
519 of data, with one source protocol address and one destination
520 protocol address, and for some conditions like the filters described
521 above, a practical requirement. Networks in the real world are not
522 limited to single streams of data. The test suite SHOULD be first run
523 with a single protocol (or hardware for bridge tests) source and
524 destination address pair. The tests SHOULD then be repeated with
525 using a random destination address. While testing routers the
526 addresses SHOULD be random and uniformly distributed over a range of
527 256 networks and random and uniformly distributed over the full MAC
528 range for bridges. The specific address ranges to use for IP are
529 shown in <a href="#appendix-C">Appendix C</a>.
530
531 <span class="h2"><a class="selflink" id="section-13" href="#section-13">13</a>. Route Set Up</span>
532
533 It is not reasonable that all of the routing information necessary to
534 forward the test stream, especially in the multiple address case,
535 will be manually set up. At the start of each trial a routing update
536 MUST be sent to the DUT. This routing update MUST include all of the
537 network addresses that will be required for the trial. All of the
538 addresses SHOULD resolve to the same "next-hop". Normally this will
539 be the address of the receiving side of the test equipment. This
540 routing update will have to be repeated at the interval required by
541 the routing protocol being used. An example of the format and
542 repetition interval of the update frames is given in <a href="#appendix-C">Appendix C</a>.
543
544 <span class="h2"><a class="selflink" id="section-14" href="#section-14">14</a>. Bidirectional traffic</span>
545
546 Normal network activity is not all in a single direction. To test
547 the bidirectional performance of a DUT, the test series SHOULD be run
548 with the same data rate being offered from each direction. The sum of
549 the data rates should not exceed the theoretical limit for the media.
550
551
552
553
554
555
556 <span class="grey">Bradner &amp; McQuaid Informational [Page 10]</span></pre>
557 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-11" ></span>
558 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
559
560
561 <span class="h2"><a class="selflink" id="section-15" href="#section-15">15</a>. Single stream path</span>
562
563 The full suite of tests SHOULD be run along with whatever modifier
564 conditions that are relevant using a single input and output network
565 port on the DUT. If the internal design of the DUT has multiple
566 distinct pathways, for example, multiple interface cards each with
567 multiple network ports, then all possible types of pathways SHOULD be
568 tested separately.
569
570 <span class="h2"><a class="selflink" id="section-16" href="#section-16">16</a>. Multi-port</span>
571
572 Many current router and bridge products provide many network ports in
573 the same module. In performing these tests first half of the ports
574 are designated as "input ports" and half are designated as "output
575 ports". These ports SHOULD be evenly distributed across the DUT
576 architecture. For example if a DUT has two interface cards each of
577 which has four ports, two ports on each interface card are designated
578 as input and two are designated as output. The specified tests are
579 run using the same data rate being offered to each of the input
580 ports. The addresses in the input data streams SHOULD be set so that
581 a frame will be directed to each of the output ports in sequence so
582 that all "output" ports will get an even distribution of packets from
583 this input. The same configuration MAY be used to perform a
584 bidirectional multi-stream test. In this case all of the ports are
585 considered both input and output ports and each data stream MUST
586 consist of frames addressed to all of the other ports.
587
588 Consider the following 6 port DUT:
589
590 --------------
591 ---------| in A out X|--------
592 ---------| in B out Y|--------
593 ---------| in C out Z|--------
594 --------------
595
596 The addressing of the data streams for each of the inputs SHOULD be:
597
598 stream sent to input A:
599 packet to out X, packet to out Y, packet to out Z
600 stream sent to input B:
601 packet to out X, packet to out Y, packet to out Z
602 stream sent to input C
603 packet to out X, packet to out Y, packet to out Z
604
605 Note that these streams each follow the same sequence so that 3
606 packets will arrive at output X at the same time, then 3 packets at
607 Y, then 3 packets at Z. This procedure ensures that, as in the real
608 world, the DUT will have to deal with multiple packets addressed to
609
610
611
612 <span class="grey">Bradner &amp; McQuaid Informational [Page 11]</span></pre>
613 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-12" ></span>
614 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
615
616
617 the same output at the same time.
618
619 <span class="h2"><a class="selflink" id="section-17" href="#section-17">17</a>. Multiple protocols</span>
620
621 This document does not address the issue of testing the effects of a
622 mixed protocol environment other than to suggest that if such tests
623 are wanted then frames SHOULD be distributed between all of the test
624 protocols. The distribution MAY approximate the conditions on the
625 network in which the DUT would be used.
626
627 <span class="h2"><a class="selflink" id="section-18" href="#section-18">18</a>. Multiple frame sizes</span>
628
629 This document does not address the issue of testing the effects of a
630 mixed frame size environment other than to suggest that if such tests
631 are wanted then frames SHOULD be distributed between all of the
632 listed sizes for the protocol under test. The distribution MAY
633 approximate the conditions on the network in which the DUT would be
634 used. The authors do not have any idea how the results of such a test
635 would be interpreted other than to directly compare multiple DUTs in
636 some very specific simulated network.
637
638 <span class="h2"><a class="selflink" id="section-19" href="#section-19">19</a>. Testing performance beyond a single DUT.</span>
639
640 In the performance testing of a single DUT, the paradigm can be
641 described as applying some input to a DUT and monitoring the output.
642 The results of which can be used to form a basis of characterization
643 of that device under those test conditions.
644
645 This model is useful when the test input and output are homogenous
646 (e.g., 64-byte IP, 802.3 frames into the DUT; 64 byte IP, 802.3
647 frames out), or the method of test can distinguish between dissimilar
648 input/output. (E.g., 1518 byte IP, 802.3 frames in; 576 byte,
649 fragmented IP, X.25 frames out.)
650
651 By extending the single DUT test model, reasonable benchmarks
652 regarding multiple DUTs or heterogeneous environments may be
653 collected. In this extension, the single DUT is replaced by a system
654 of interconnected network DUTs. This test methodology would support
655 the benchmarking of a variety of device/media/service/protocol
656 combinations. For example, a configuration for a LAN-to-WAN-to-LAN
657 test might be:
658
659 (1) 802.3-&gt; DUT 1 -&gt; X.25 @ 64kbps -&gt; DUT 2 -&gt; 802.3
660
661 Or a mixed LAN configuration might be:
662
663 (2) 802.3 -&gt; DUT 1 -&gt; FDDI -&gt; DUT 2 -&gt; FDDI -&gt; DUT 3 -&gt; 802.3
664
665
666
667
668 <span class="grey">Bradner &amp; McQuaid Informational [Page 12]</span></pre>
669 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-13" ></span>
670 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
671
672
673 In both examples 1 and 2, end-to-end benchmarks of each system could
674 be empirically ascertained. Other behavior may be characterized
675 through the use of intermediate devices. In example 2, the
676 configuration may be used to give an indication of the FDDI to FDDI
677 capability exhibited by DUT 2.
678
679 Because multiple DUTs are treated as a single system, there are
680 limitations to this methodology. For instance, this methodology may
681 yield an aggregate benchmark for a tested system. That benchmark
682 alone, however, may not necessarily reflect asymmetries in behavior
683 between the DUTs, latencies introduce by other apparatus (e.g.,
684 CSUs/DSUs, switches), etc.
685
686 Further, care must be used when comparing benchmarks of different
687 systems by ensuring that the DUTs' features/configuration of the
688 tested systems have the appropriate common denominators to allow
689 comparison.
690
691 <span class="h2"><a class="selflink" id="section-20" href="#section-20">20</a>. Maximum frame rate</span>
692
693 The maximum frame rates that should be used when testing LAN
694 connections SHOULD be the listed theoretical maximum rate for the
695 frame size on the media.
696
697 The maximum frame rate that should be used when testing WAN
698 connections SHOULD be greater than the listed theoretical maximum
699 rate for the frame size on that speed connection. The higher rate
700 for WAN tests is to compensate for the fact that some vendors employ
701 various forms of header compression.
702
703 A list of maximum frame rates for LAN connections is included in
704 <a href="#appendix-B">Appendix B</a>.
705
706 <span class="h2"><a class="selflink" id="section-21" href="#section-21">21</a>. Bursty traffic</span>
707
708 It is convenient to measure the DUT performance under steady state
709 load but this is an unrealistic way to gauge the functioning of a DUT
710 since actual network traffic normally consists of bursts of frames.
711 Some of the tests described below SHOULD be performed with both
712 steady state traffic and with traffic consisting of repeated bursts
713 of frames. The frames within a burst are transmitted with the
714 minimum legitimate inter-frame gap.
715
716 The objective of the test is to determine the minimum interval
717 between bursts which the DUT can process with no frame loss. During
718 each test the number of frames in each burst is held constant and the
719 inter-burst interval varied. Tests SHOULD be run with burst sizes of
720 16, 64, 256 and 1024 frames.
721
722
723
724 <span class="grey">Bradner &amp; McQuaid Informational [Page 13]</span></pre>
725 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-14" ></span>
726 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
727
728
729 <span class="h2"><a class="selflink" id="section-22" href="#section-22">22</a>. Frames per token</span>
730
731 Although it is possible to configure some token ring and FDDI
732 interfaces to transmit more than one frame each time that the token
733 is received, most of the network devices currently available transmit
734 only one frame per token. These tests SHOULD first be performed
735 while transmitting only one frame per token.
736
737 Some current high-performance workstation servers do transmit more
738 than one frame per token on FDDI to maximize throughput. Since this
739 may be a common feature in future workstations and servers,
740 interconnect devices with FDDI interfaces SHOULD be tested with 1, 4,
741 8, and 16 frames per token. The reported frame rate SHOULD be the
742 average rate of frame transmission over the total trial period.
743
744 <span class="h2"><a class="selflink" id="section-23" href="#section-23">23</a>. Trial description</span>
745
746 A particular test consists of multiple trials. Each trial returns
747 one piece of information, for example the loss rate at a particular
748 input frame rate. Each trial consists of a number of phases:
749
750 a) If the DUT is a router, send the routing update to the "input"
751 port and pause two seconds to be sure that the routing has settled.
752
753 b) Send the "learning frames" to the "output" port and wait 2
754 seconds to be sure that the learning has settled. Bridge learning
755 frames are frames with source addresses that are the same as the
756 destination addresses used by the test frames. Learning frames for
757 other protocols are used to prime the address resolution tables in
758 the DUT. The formats of the learning frame that should be used are
759 shown in the Test Frame Formats document.
760
761 c) Run the test trial.
762
763 d) Wait for two seconds for any residual frames to be received.
764
765 e) Wait for at least five seconds for the DUT to restabilize.
766
767 <span class="h2"><a class="selflink" id="section-24" href="#section-24">24</a>. Trial duration</span>
768
769 The aim of these tests is to determine the rate continuously
770 supportable by the DUT. The actual duration of the test trials must
771 be a compromise between this aim and the duration of the benchmarking
772 test suite. The duration of the test portion of each trial SHOULD be
773 at least 60 seconds. The tests that involve some form of "binary
774 search", for example the throughput test, to determine the exact
775 result MAY use a shorter trial duration to minimize the length of the
776 search procedure, but the final determination SHOULD be made with
777
778
779
780 <span class="grey">Bradner &amp; McQuaid Informational [Page 14]</span></pre>
781 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-15" ></span>
782 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
783
784
785 full length trials.
786
787 <span class="h2"><a class="selflink" id="section-25" href="#section-25">25</a>. Address resolution</span>
788
789 The DUT SHOULD be able to respond to address resolution requests sent
790 by the DUT wherever the protocol requires such a process.
791
792 <span class="h2"><a class="selflink" id="section-26" href="#section-26">26</a>. Benchmarking tests:</span>
793
794 Note: The notation "type of data stream" refers to the above
795 modifications to a frame stream with a constant inter-frame gap, for
796 example, the addition of traffic filters to the configuration of the
797 DUT.
798
799 26.1 Throughput
800
801 Objective:
802 To determine the DUT throughput as defined in <a href="./rfc1242">RFC 1242</a>.
803
804 Procedure:
805 Send a specific number of frames at a specific rate through the
806 DUT and then count the frames that are transmitted by the DUT. If
807 the count of offered frames is equal to the count of received
808 frames, the rate of the offered stream is raised and the test
809 rerun. If fewer frames are received than were transmitted, the
810 rate of the offered stream is reduced and the test is rerun.
811
812 The throughput is the fastest rate at which the count of test
813 frames transmitted by the DUT is equal to the number of test
814 frames sent to it by the test equipment.
815
816 Reporting format:
817 The results of the throughput test SHOULD be reported in the form
818 of a graph. If it is, the x coordinate SHOULD be the frame size,
819 the y coordinate SHOULD be the frame rate. There SHOULD be at
820 least two lines on the graph. There SHOULD be one line showing
821 the theoretical frame rate for the media at the various frame
822 sizes. The second line SHOULD be the plot of the test results.
823 Additional lines MAY be used on the graph to report the results
824 for each type of data stream tested. Text accompanying the graph
825 SHOULD indicate the protocol, data stream format, and type of
826 media used in the tests.
827
828 We assume that if a single value is desired for advertising
829 purposes the vendor will select the rate for the minimum frame
830 size for the media. If this is done then the figure MUST be
831 expressed in frames per second. The rate MAY also be expressed in
832 bits (or bytes) per second if the vendor so desires. The
833
834
835
836 <span class="grey">Bradner &amp; McQuaid Informational [Page 15]</span></pre>
837 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-16" ></span>
838 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
839
840
841 statement of performance MUST include a/ the measured maximum
842 frame rate, b/ the size of the frame used, c/ the theoretical
843 limit of the media for that frame size, and d/ the type of
844 protocol used in the test. Even if a single value is used as part
845 of the advertising copy, the full table of results SHOULD be
846 included in the product data sheet.
847
848 26.2 Latency
849
850 Objective:
851 To determine the latency as defined in <a href="./rfc1242">RFC 1242</a>.
852
853 Procedure:
854 First determine the throughput for DUT at each of the listed frame
855 sizes. Send a stream of frames at a particular frame size through
856 the DUT at the determined throughput rate to a specific
857 destination. The stream SHOULD be at least 120 seconds in
858 duration. An identifying tag SHOULD be included in one frame
859 after 60 seconds with the type of tag being implementation
860 dependent. The time at which this frame is fully transmitted is
861 recorded (timestamp A). The receiver logic in the test equipment
862 MUST recognize the tag information in the frame stream and record
863 the time at which the tagged frame was received (timestamp B).
864
865 The latency is timestamp B minus timestamp A as per the relevant
866 definition frm <a href="./rfc1242">RFC 1242</a>, namely latency as defined for store and
867 forward devices or latency as defined for bit forwarding devices.
868
869 The test MUST be repeated at least 20 times with the reported
870 value being the average of the recorded values.
871
872 This test SHOULD be performed with the test frame addressed to the
873 same destination as the rest of the data stream and also with each
874 of the test frames addressed to a new destination network.
875
876 Reporting format:
877 The report MUST state which definition of latency (from <a href="./rfc1242">RFC 1242</a>)
878 was used for this test. The latency results SHOULD be reported
879 in the format of a table with a row for each of the tested frame
880 sizes. There SHOULD be columns for the frame size, the rate at
881 which the latency test was run for that frame size, for the media
882 types tested, and for the resultant latency values for each
883 type of data stream tested.
884
885
886
887
888
889
890
891
892 <span class="grey">Bradner &amp; McQuaid Informational [Page 16]</span></pre>
893 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-17" ></span>
894 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
895
896
897 26.3 Frame loss rate
898
899 Objective:
900 To determine the frame loss rate, as defined in <a href="./rfc1242">RFC 1242</a>, of a DUT
901 throughout the entire range of input data rates and frame sizes.
902
903 Procedure:
904 Send a specific number of frames at a specific rate through the
905 DUT to be tested and count the frames that are transmitted by the
906 DUT. The frame loss rate at each point is calculated using the
907 following equation:
908
909 ( ( input_count - output_count ) * 100 ) / input_count
910
911 The first trial SHOULD be run for the frame rate that corresponds
912 to 100% of the maximum rate for the frame size on the input media.
913 Repeat the procedure for the rate that corresponds to 90% of the
914 maximum rate used and then for 80% of this rate. This sequence
915 SHOULD be continued (at reducing 10% intervals) until there are
916 two successive trials in which no frames are lost. The maximum
917 granularity of the trials MUST be 10% of the maximum rate, a finer
918 granularity is encouraged.
919
920 Reporting format:
921 The results of the frame loss rate test SHOULD be plotted as a
922 graph. If this is done then the X axis MUST be the input frame
923 rate as a percent of the theoretical rate for the media at the
924 specific frame size. The Y axis MUST be the percent loss at the
925 particular input rate. The left end of the X axis and the bottom
926 of the Y axis MUST be 0 percent; the right end of the X axis and
927 the top of the Y axis MUST be 100 percent. Multiple lines on the
928 graph MAY used to report the frame loss rate for different frame
929 sizes, protocols, and types of data streams.
930
931 Note: See <a href="#section-18">section 18</a> for the maximum frame rates that SHOULD be
932 used.
933
934 26.4 Back-to-back frames
935
936 Objective:
937 To characterize the ability of a DUT to process back-to-back
938 frames as defined in <a href="./rfc1242">RFC 1242</a>.
939
940 Procedure:
941 Send a burst of frames with minimum inter-frame gaps to the DUT
942 and count the number of frames forwarded by the DUT. If the count
943 of transmitted frames is equal to the number of frames forwarded
944 the length of the burst is increased and the test is rerun. If
945
946
947
948 <span class="grey">Bradner &amp; McQuaid Informational [Page 17]</span></pre>
949 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-18" ></span>
950 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
951
952
953 the number of forwarded frames is less than the number
954 transmitted, the length of the burst is reduced and the test is
955 rerun.
956
957 The back-to-back value is the number of frames in the longest
958 burst that the DUT will handle without the loss of any frames.
959 The trial length MUST be at least 2 seconds and SHOULD be
960 repeated at least 50 times with the average of the recorded values
961 being reported.
962
963 Reporting format:
964 The back-to-back results SHOULD be reported in the format of a
965 table with a row for each of the tested frame sizes. There SHOULD
966 be columns for the frame size and for the resultant average frame
967 count for each type of data stream tested. The standard deviation
968 for each measurement MAY also be reported.
969
970 26.5 System recovery
971
972 Objective:
973 To characterize the speed at which a DUT recovers from an overload
974 condition.
975
976 Procedure:
977 First determine the throughput for a DUT at each of the listed
978 frame sizes.
979
980 Send a stream of frames at a rate 110% of the recorded throughput
981 rate or the maximum rate for the media, whichever is lower, for at
982 least 60 seconds. At Timestamp A reduce the frame rate to 50% of
983 the above rate and record the time of the last frame lost
984 (Timestamp B). The system recovery time is determined by
985 subtracting Timestamp B from Timestamp A. The test SHOULD be
986 repeated a number of times and the average of the recorded values
987 being reported.
988
989 Reporting format:
990 The system recovery results SHOULD be reported in the format of a
991 table with a row for each of the tested frame sizes. There SHOULD
992 be columns for the frame size, the frame rate used as the
993 throughput rate for each type of data stream tested, and for the
994 measured recovery time for each type of data stream tested.
995
996 26.6 Reset
997
998 Objective:
999 To characterize the speed at which a DUT recovers from a device or
1000 software reset.
1001
1002
1003
1004 <span class="grey">Bradner &amp; McQuaid Informational [Page 18]</span></pre>
1005 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-19" ></span>
1006 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1007
1008
1009 Procedure:
1010 First determine the throughput for the DUT for the minimum frame
1011 size on the media used in the testing.
1012
1013 Send a continuous stream of frames at the determined throughput
1014 rate for the minimum sized frames. Cause a reset in the DUT.
1015 Monitor the output until frames begin to be forwarded and record
1016 the time that the last frame (Timestamp A) of the initial stream
1017 and the first frame of the new stream (Timestamp B) are received.
1018 A power interruption reset test is performed as above except that
1019 the power to the DUT should be interrupted for 10 seconds in place
1020 of causing a reset.
1021
1022 This test SHOULD only be run using frames addressed to networks
1023 directly connected to the DUT so that there is no requirement to
1024 delay until a routing update is received.
1025
1026 The reset value is obtained by subtracting Timestamp A from
1027 Timestamp B.
1028
1029 Hardware and software resets, as well as a power interruption
1030 SHOULD be tested.
1031
1032 Reporting format:
1033 The reset value SHOULD be reported in a simple set of statements,
1034 one for each reset type.
1035
1036 <span class="h2"><a class="selflink" id="section-27" href="#section-27">27</a>. Security Considerations</span>
1037
1038 Security issues are not addressed in this document.
1039
1040
1041
1042
1043
1044
1045
1046
1047
1048
1049
1050
1051
1052
1053
1054
1055
1056
1057
1058
1059
1060 <span class="grey">Bradner &amp; McQuaid Informational [Page 19]</span></pre>
1061 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-20" ></span>
1062 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1063
1064
1065 <span class="h2"><a class="selflink" id="section-28" href="#section-28">28</a>. Editors' Addresses</span>
1066
1067 Scott Bradner
1068 Harvard University
1069 1350 Mass. Ave, room 813
1070 Cambridge, MA 02138
1071
1072 Phone +1 617 495-3864
1073 Fax +1 617 496-8500
1074 EMail: [email protected]
1075
1076
1077 Jim McQuaid
1078 Bay Networks
1079 3 Federal Street
1080 Billerica, MA 01821
1081
1082 Phone +1 508 436-3915
1083 Fax: +1 508 670-8145
1084 EMail: [email protected]
1085
1086
1087
1088
1089
1090
1091
1092
1093
1094
1095
1096
1097
1098
1099
1100
1101
1102
1103
1104
1105
1106
1107
1108
1109
1110
1111
1112
1113
1114
1115
1116 <span class="grey">Bradner &amp; McQuaid Informational [Page 20]</span></pre>
1117 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-21" ></span>
1118 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1119
1120
1121 Appendix A: Testing Considerations
1122
1123 <span class="h3"><a class="selflink" id="appendix-A.1" href="#appendix-A.1">A.1</a> Scope Of This Appendix</span>
1124
1125 This appendix discusses certain issues in the benchmarking
1126 methodology where experience or judgment may play a role in the tests
1127 selected to be run or in the approach to constructing the test with a
1128 particular DUT. As such, this <a href="#appendix-M">appendix M</a>UST not be read as an
1129 amendment to the methodology described in the body of this document
1130 but as a guide to testing practice.
1131
1132 1. Typical testing practice has been to enable all protocols to be
1133 tested and conduct all testing with no further configuration of
1134 protocols, even though a given set of trials may exercise only one
1135 protocol at a time. This minimizes the opportunities to "tune" a
1136 DUT for a single protocol.
1137
1138 2. The least common denominator of the available filter functions
1139 should be used to ensure that there is a basis for comparison
1140 between vendors. Because of product differences, those conducting
1141 and evaluating tests must make a judgment about this issue.
1142
1143 3. Architectural considerations may need to be considered. For
1144 example, first perform the tests with the stream going between
1145 ports on the same interface card and the repeat the tests with the
1146 stream going into a port on one interface card and out of a port
1147 on a second interface card. There will almost always be a best
1148 case and worst case configuration for a given DUT architecture.
1149
1150 4. Testing done using traffic streams consisting of mixed protocols
1151 has not shown much difference between testing with individual
1152 protocols. That is, if protocol A testing and protocol B testing
1153 give two different performance results, mixed protocol testing
1154 appears to give a result which is the average of the two.
1155
1156 5. Wide Area Network (WAN) performance may be tested by setting up
1157 two identical devices connected by the appropriate short- haul
1158 versions of the WAN modems. Performance is then measured between
1159 a LAN interface on one DUT to a LAN interface on the other DUT.
1160
1161 The maximum frame rate to be used for LAN-WAN-LAN configurations is a
1162 judgment that can be based on known characteristics of the overall
1163 system including compression effects, fragmentation, and gross link
1164 speeds. Practice suggests that the rate should be at least 110% of
1165 the slowest link speed. Substantive issues of testing compression
1166 itself are beyond the scope of this document.
1167
1168
1169
1170
1171
1172 <span class="grey">Bradner &amp; McQuaid Informational [Page 21]</span></pre>
1173 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-22" ></span>
1174 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1175
1176
1177 Appendix B: Maximum frame rates reference
1178
1179 (Provided by Roger Beeman, Cisco Systems)
1180
1181 Size Ethernet 16Mb Token Ring FDDI
1182 (bytes) (pps) (pps) (pps)
1183
1184 64 14880 24691 152439
1185 128 8445 13793 85616
1186 256 4528 7326 45620
1187 512 2349 3780 23585
1188 768 1586 2547 15903
1189 1024 1197 1921 11996
1190 1280 961 1542 9630
1191 1518 812 1302 8138
1192
1193 Ethernet size
1194 Preamble 64 bits
1195 Frame 8 x N bits
1196 Gap 96 bits
1197
1198 16Mb Token Ring size
1199 SD 8 bits
1200 AC 8 bits
1201 FC 8 bits
1202 DA 48 bits
1203 SA 48 bits
1204 RI 48 bits ( 06 30 00 12 00 30 )
1205 SNAP
1206 DSAP 8 bits
1207 SSAP 8 bits
1208 Control 8 bits
1209 Vendor 24 bits
1210 Type 16 bits
1211 Data 8 x ( N - 18) bits
1212 FCS 32 bits
1213 ED 8 bits
1214 FS 8 bits
1215
1216 Tokens or idles between packets are not included
1217
1218 FDDI size
1219 Preamble 64 bits
1220 SD 8 bits
1221 FC 8 bits
1222 DA 48 bits
1223 SA 48 bits
1224 SNAP
1225
1226
1227
1228 <span class="grey">Bradner &amp; McQuaid Informational [Page 22]</span></pre>
1229 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-23" ></span>
1230 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1231
1232
1233 DSAP 8 bits
1234 SSAP 8 bits
1235 Control 8 bits
1236 Vendor 24 bits
1237 Type 16 bits
1238 Data 8 x ( N - 18) bits
1239 FCS 32 bits
1240 ED 4 bits
1241 FS 12 bits
1242
1243 Appendix C: Test Frame Formats
1244
1245 This appendix defines the frame formats that may be used with these
1246 tests. It also includes protocol specific parameters for TCP/IP over
1247 Ethernet to be used with the tests as an example.
1248
1249 <span class="h3"><a class="selflink" id="appendix-C.1" href="#appendix-C.1">C.1</a>. Introduction</span>
1250
1251 The general logic used in the selection of the parameters and the
1252 design of the frame formats is explained for each case within the
1253 TCP/IP section. The same logic has been used in the other sections.
1254 Comments are used in these sections only if there is a protocol
1255 specific feature to be explained. Parameters and frame formats for
1256 additional protocols can be defined by the reader by using the same
1257 logic.
1258
1259 <span class="h3"><a class="selflink" id="appendix-C.2" href="#appendix-C.2">C.2</a>. TCP/IP Information</span>
1260
1261 The following section deals with the TCP/IP protocol suite.
1262
1263 C.2.1 Frame Type.
1264 An application level datagram echo request is used for the test
1265 data frame in the protocols that support such a function. A
1266 datagram protocol is used to minimize the chance that a router
1267 might expect a specific session initialization sequence, as might
1268 be the case for a reliable stream protocol. A specific defined
1269 protocol is used because some routers verify the protocol field
1270 and refuse to forward unknown protocols.
1271
1272 For TCP/IP a UDP Echo Request is used.
1273
1274 C.2.2 Protocol Addresses
1275 Two sets of addresses must be defined: first the addresses
1276 assigned to the router ports, and second the address that are to
1277 be used in the frames themselves and in the routing updates.
1278
1279 The network addresses 192.18.0.0 through 192.19.255.255 are have
1280 been assigned to the BMWG by the IANA for this purpose. This
1281
1282
1283
1284 <span class="grey">Bradner &amp; McQuaid Informational [Page 23]</span></pre>
1285 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-24" ></span>
1286 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1287
1288
1289 assignment was made to minimize the chance of conflict in case a
1290 testing device were to be accidentally connected to part of the
1291 Internet. The specific use of the addresses is detailed below.
1292
1293 C.2.2.1 Router port protocol addresses
1294 Half of the ports on a multi-port router are referred to as
1295 "input" ports and the other half as "output" ports even though
1296 some of the tests use all ports both as input and output. A
1297 contiguous series of IP Class C network addresses from
1298 198.18.1.0 to 198.18.64.0 have been assigned for use on the
1299 "input" ports. A second series from 198.19.1.0 to 198.19.64.0
1300 have been assigned for use on the "output" ports. In all cases
1301 the router port is node 1 on the appropriate network. For
1302 example, a two port DUT would have an IP address of 198.18.1.1
1303 on one port and 198.19.1.1 on the other port.
1304
1305 Some of the tests described in the methodology memo make use of
1306 an SNMP management connection to the DUT. The management
1307 access address for the DUT is assumed to be the first of the
1308 "input" ports (198.18.1.1).
1309
1310 C.2.2.2 Frame addresses
1311 Some of the described tests assume adjacent network routing
1312 (the reboot time test for example). The IP address used in the
1313 test frame is that of node 2 on the appropriate Class C
1314 network. (198.19.1.2 for example)
1315
1316 If the test involves non-adjacent network routing the phantom
1317 routers are located at node 10 of each of the appropriate Class
1318 C networks. A series of Class C network addresses from
1319 198.18.65.0 to 198.18.254.0 has been assigned for use as the
1320 networks accessible through the phantom routers on the "input"
1321 side of DUT. The series of Class C networks from 198.19.65.0
1322 to 198.19.254.0 have been assigned to be used as the networks
1323 visible through the phantom routers on the "output" side of the
1324 DUT.
1325
1326 C.2.3 Routing Update Frequency
1327
1328 The update interval for each routing protocol is may have to be
1329 determined by the specifications of the individual protocol. For
1330 IP RIP, Cisco IGRP and for OSPF a routing update frame or frames
1331 should precede each stream of test frames by 5 seconds. This
1332 frequency is sufficient for trial durations of up to 60 seconds.
1333 Routing updates must be mixed with the stream of test frames if
1334 longer trial periods are selected. The frequency of updates
1335 should be taken from the following table.
1336
1337
1338
1339
1340 <span class="grey">Bradner &amp; McQuaid Informational [Page 24]</span></pre>
1341 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-25" ></span>
1342 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1343
1344
1345 IP-RIP 30 sec
1346 IGRP 90 sec
1347 OSPF 90 sec
1348
1349 C.2.4 Frame Formats - detailed discussion
1350
1351 C.2.4.1 Learning Frame
1352 In most protocols a procedure is used to determine the mapping
1353 between the protocol node address and the MAC address. The
1354 Address Resolution Protocol (ARP) is used to perform this
1355 function in TCP/IP. No such procedure is required in XNS or
1356 IPX because the MAC address is used as the protocol node
1357 address.
1358
1359 In the ideal case the tester would be able to respond to ARP
1360 requests from the DUT. In cases where this is not possible an
1361 ARP request should be sent to the router's "output" port. This
1362 request should be seen as coming from the immediate destination
1363 of the test frame stream. (i.e. the phantom router (Figure 2)
1364 or the end node if adjacent network routing is being used.) It
1365 is assumed that the router will cache the MAC address of the
1366 requesting device. The ARP request should be sent 5 seconds
1367 before the test frame stream starts in each trial. Trial
1368 lengths of longer than 50 seconds may require that the router
1369 be configured for an extended ARP timeout.
1370
1371 +--------+ +------------+
1372 | | | phantom |------ P LAN
1373 A
1374 IN A------| DUT |------------| |------ P LAN
1375 B
1376 | | OUT A | router |------ P LAN
1377 C
1378 +--------+ +------------+
1379
1380 Figure 2
1381
1382 In the case where full routing is being used
1383
1384 C.2.4.2 Routing Update Frame
1385 If the test does not involve adjacent net routing the tester
1386 must supply proper routing information using a routing update.
1387 A single routing update is used before each trial on each
1388 "destination" port (see section C.24). This update includes
1389 the network addresses that are reachable through a phantom
1390 router on the network attached to the port. For a full mesh
1391 test, one destination network address is present in the routing
1392 update for each of the "input" ports. The test stream on each
1393
1394
1395
1396 <span class="grey">Bradner &amp; McQuaid Informational [Page 25]</span></pre>
1397 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-26" ></span>
1398 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1399
1400
1401 "input" port consists of a repeating sequence of frames, one to
1402 each of the "output" ports.
1403
1404 C.2.4.3 Management Query Frame
1405 The management overhead test uses SNMP to query a set of
1406 variables that should be present in all DUTs that support SNMP.
1407 The variables for a single interface only are read by an NMS
1408 at the appropriate intervals. The list of variables to
1409 retrieve follow:
1410
1411 sysUpTime
1412 ifInOctets
1413 ifOutOctets
1414 ifInUcastPkts
1415 ifOutUcastPkts
1416
1417 C.2.4.4 Test Frames
1418 The test frame is an UDP Echo Request with enough data to fill
1419 out the required frame size. The data should not be all bits
1420 off or all bits on since these patters can cause a "bit
1421 stuffing" process to be used to maintain clock synchronization
1422 on WAN links. This process will result in a longer frame than
1423 was intended.
1424
1425 C.2.4.5 Frame Formats - TCP/IP on Ethernet
1426 Each of the frames below are described for the 1st pair of DUT
1427 ports, i.e. "input" port #1 and "output" port #1. Addresses
1428 must be changed if the frame is to be used for other ports.
1429
1430 C.2.6.1 Learning Frame
1431
1432 ARP Request on Ethernet
1433
1434 -- DATAGRAM HEADER
1435 offset data (hex) description
1436 00 FF FF FF FF FF FF dest MAC address send to
1437 broadcast address
1438 06 xx xx xx xx xx xx set to source MAC address
1439 12 08 06 ARP type
1440 14 00 01 hardware type Ethernet = 1
1441 16 08 00 protocol type IP = 800
1442 18 06 hardware address length 48 bits
1443 on Ethernet
1444 19 04 protocol address length 4 octets
1445 for IP
1446 20 00 01 opcode request = 1
1447 22 xx xx xx xx xx xx source MAC address
1448 28 xx xx xx xx source IP address
1449
1450
1451
1452 <span class="grey">Bradner &amp; McQuaid Informational [Page 26]</span></pre>
1453 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-27" ></span>
1454 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1455
1456
1457 32 FF FF FF FF FF FF requesting DUT's MAC address
1458 38 xx xx xx xx DUT's IP address
1459
1460 C.2.6.2 Routing Update Frame
1461
1462 -- DATAGRAM HEADER
1463 offset data (hex) description
1464 00 FF FF FF FF FF FF dest MAC address is broadcast
1465 06 xx xx xx xx xx xx source hardware address
1466 12 08 00 type
1467
1468 -- IP HEADER
1469 14 45 IP version - 4, header length (4
1470 byte units) - 5
1471 15 00 service field
1472 16 00 EE total length
1473 18 00 00 ID
1474 20 40 00 flags (3 bits) 4 (do not
1475 fragment),
1476 fragment offset-0
1477 22 0A TTL
1478 23 11 protocol - 17 (UDP)
1479 24 C4 8D header checksum
1480 26 xx xx xx xx source IP address
1481 30 xx xx xx destination IP address
1482 33 FF host part = FF for broadcast
1483
1484 -- UDP HEADER
1485 34 02 08 source port 208 = RIP
1486 36 02 08 destination port 208 = RIP
1487 38 00 DA UDP message length
1488 40 00 00 UDP checksum
1489
1490 -- RIP packet
1491 42 02 command = response
1492 43 01 version = 1
1493 44 00 00 0
1494
1495 -- net 1
1496 46 00 02 family = IP
1497 48 00 00 0
1498 50 xx xx xx net 1 IP address
1499 53 00 net not node
1500 54 00 00 00 00 0
1501 58 00 00 00 00 0
1502 62 00 00 00 07 metric 7
1503
1504 -- net 2
1505
1506
1507
1508 <span class="grey">Bradner &amp; McQuaid Informational [Page 27]</span></pre>
1509 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-28" ></span>
1510 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1511
1512
1513 66 00 02 family = IP
1514 68 00 00 0
1515 70 xx xx xx net 2 IP address
1516 73 00 net not node
1517 74 00 00 00 00 0
1518 78 00 00 00 00 0
1519 82 00 00 00 07 metric 7
1520
1521 -- net 3
1522 86 00 02 family = IP
1523 88 00 00 0
1524 90 xx xx xx net 3 IP address
1525 93 00 net not node
1526 94 00 00 00 00 0
1527 98 00 00 00 00 0
1528 102 00 00 00 07 metric 7
1529
1530 -- net 4
1531 106 00 02 family = IP
1532 108 00 00 0
1533 110 xx xx xx net 4 IP address
1534 113 00 net not node
1535 114 00 00 00 00 0
1536 118 00 00 00 00 0
1537 122 00 00 00 07 metric 7
1538
1539 -- net 5
1540 126 00 02 family = IP
1541 128 00 00 0
1542 130 00 net 5 IP address
1543 133 00 net not node
1544 134 00 00 00 00 0
1545 138 00 00 00 00 0
1546 142 00 00 00 07 metric 7
1547
1548 -- net 6
1549 146 00 02 family = IP
1550 148 00 00 0
1551 150 xx xx xx net 6 IP address
1552 153 00 net not node
1553 154 00 00 00 00 0
1554 158 00 00 00 00 0
1555 162 00 00 00 07 metric 7
1556
1557 C.2.4.6 Management Query Frame
1558
1559 To be defined.
1560
1561
1562
1563
1564 <span class="grey">Bradner &amp; McQuaid Informational [Page 28]</span></pre>
1565 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-29" ></span>
1566 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1567
1568
1569 C.2.6.4 Test Frames
1570 UDP echo request on Ethernet
1571
1572 -- DATAGRAM HEADER
1573 offset data (hex) description
1574 00 xx xx xx xx xx xx set to dest MAC address
1575 06 xx xx xx xx xx xx set to source MAC address
1576 12 08 00 type
1577
1578 -- IP HEADER
1579 14 45 IP version - 4 header length 5 4
1580 byte units
1581 15 00 TOS
1582 16 00 2E total length*
1583 18 00 00 ID
1584 20 00 00 flags (3 bits) - 0 fragment
1585 offset-0
1586 22 0A TTL
1587 23 11 protocol - 17 (UDP)
1588 24 C4 8D header checksum*
1589 26 xx xx xx xx set to source IP address**
1590 30 xx xx xx xx set to destination IP address**
1591
1592 -- UDP HEADER
1593 34 C0 20 source port
1594 36 00 07 destination port 07 = Echo
1595 38 00 1A UDP message length*
1596 40 00 00 UDP checksum
1597
1598 -- UDP DATA
1599 42 00 01 02 03 04 05 06 07 some data***
1600 50 08 09 0A 0B 0C 0D 0E 0F
1601
1602 * - change for different length frames
1603
1604 ** - change for different logical streams
1605
1606 *** - fill remainder of frame with incrementing octets,
1607 repeated if required by frame length
1608
1609
1610
1611
1612
1613
1614
1615
1616
1617
1618
1619
1620 <span class="grey">Bradner &amp; McQuaid Informational [Page 29]</span></pre>
1621 <hr class='noprint'/><!--NewPage--><pre class='newpage'><span id="page-30" ></span>
1622 <span class="grey"><a href="./rfc1944">RFC 1944</a> Benchmarking Methodology May 1996</span>
1623
1624
1625 Values to be used in Total Length and UDP message length fields:
1626
1627 frame size total length UDP message length
1628 64 00 2E 00 1A
1629 128 00 6E 00 5A
1630 256 00 EE 00 9A
1631 512 01 EE 01 9A
1632 768 02 EE 02 9A
1633 1024 03 EE 03 9A
1634 1280 04 EE 04 9A
1635 1518 05 DC 05 C8
1636
1637
1638
1639
1640
1641
1642
1643
1644
1645
1646
1647
1648
1649
1650
1651
1652
1653
1654
1655
1656
1657
1658
1659
1660
1661
1662
1663
1664
1665
1666
1667
1668
1669
1670
1671
1672
1673
1674
1675
1676 Bradner &amp; McQuaid Informational [Page 30]
1677 </pre>