Debian Codegraph @

third-party/benchmark/docs/user_guide.md

raw
1 # User Guide
2
3 ## Command Line
4
5 [Output Formats](#output-formats)
6
7 [Output Files](#output-files)
8
9 [Running Benchmarks](#running-benchmarks)
10
11 [Running a Subset of Benchmarks](#running-a-subset-of-benchmarks)
12
13 [Result Comparison](#result-comparison)
14
15 [Extra Context](#extra-context)
16
17 ## Library
18
19 [Runtime and Reporting Considerations](#runtime-and-reporting-considerations)
20
21 [Setup/Teardown](#setupteardown)
22
23 [Passing Arguments](#passing-arguments)
24
25 [Custom Benchmark Name](#custom-benchmark-name)
26
27 [Calculating Asymptotic Complexity](#asymptotic-complexity)
28
29 [Templated Benchmarks](#templated-benchmarks)
30
31 [Templated Benchmarks that take arguments](#templated-benchmarks-with-arguments)
32
33 [Fixtures](#fixtures)
34
35 [Custom Counters](#custom-counters)
36
37 [Multithreaded Benchmarks](#multithreaded-benchmarks)
38
39 [CPU Timers](#cpu-timers)
40
41 [Manual Timing](#manual-timing)
42
43 [Setting the Time Unit](#setting-the-time-unit)
44
45 [Random Interleaving](random_interleaving.md)
46
47 [User-Requested Performance Counters](perf_counters.md)
48
49 [Preventing Optimization](#preventing-optimization)
50
51 [Reporting Statistics](#reporting-statistics)
52
53 [Custom Statistics](#custom-statistics)
54
55 [Memory Usage](#memory-usage)
56
57 [Using RegisterBenchmark](#using-register-benchmark)
58
59 [Exiting with an Error](#exiting-with-an-error)
60
61 [A Faster `KeepRunning` Loop](#a-faster-keep-running-loop)
62
63 ## Benchmarking Tips
64
65 [Disabling CPU Frequency Scaling](#disabling-cpu-frequency-scaling)
66
67 [Reducing Variance in Benchmarks](reducing_variance.md)
68
69 <a name="output-formats" />
70
71 ## Output Formats
72
73 The library supports multiple output formats. Use the
74 `--benchmark_format=<console|json|csv>` flag (or set the
75 `BENCHMARK_FORMAT=<console|json|csv>` environment variable) to set
76 the format type. `console` is the default format.
77
78 The Console format is intended to be a human readable format. By default
79 the format generates color output. Context is output on stderr and the
80 tabular data on stdout. Example tabular output looks like:
81
82 ```
83 Benchmark Time(ns) CPU(ns) Iterations
84 ----------------------------------------------------------------------
85 BM_SetInsert/1024/1 28928 29349 23853 133.097kB/s 33.2742k items/s
86 BM_SetInsert/1024/8 32065 32913 21375 949.487kB/s 237.372k items/s
87 BM_SetInsert/1024/10 33157 33648 21431 1.13369MB/s 290.225k items/s
88 ```
89
90 The JSON format outputs human readable json split into two top level attributes.
91 The `context` attribute contains information about the run in general, including
92 information about the CPU and the date.
93 The `benchmarks` attribute contains a list of every benchmark run. Example json
94 output looks like:
95
96 ```json
97 {
98 "context": {
99 "date": "2015/03/17-18:40:25",
100 "num_cpus": 40,
101 "mhz_per_cpu": 2801,
102 "cpu_scaling_enabled": false,
103 "build_type": "debug"
104 },
105 "benchmarks": [
106 {
107 "name": "BM_SetInsert/1024/1",
108 "iterations": 94877,
109 "real_time": 29275,
110 "cpu_time": 29836,
111 "bytes_per_second": 134066,
112 "items_per_second": 33516
113 },
114 {
115 "name": "BM_SetInsert/1024/8",
116 "iterations": 21609,
117 "real_time": 32317,
118 "cpu_time": 32429,
119 "bytes_per_second": 986770,
120 "items_per_second": 246693
121 },
122 {
123 "name": "BM_SetInsert/1024/10",
124 "iterations": 21393,
125 "real_time": 32724,
126 "cpu_time": 33355,
127 "bytes_per_second": 1199226,
128 "items_per_second": 299807
129 }
130 ]
131 }
132 ```
133
134 The CSV format outputs comma-separated values. The `context` is output on stderr
135 and the CSV itself on stdout. Example CSV output looks like:
136
137 ```
138 name,iterations,real_time,cpu_time,bytes_per_second,items_per_second,label
139 "BM_SetInsert/1024/1",65465,17890.7,8407.45,475768,118942,
140 "BM_SetInsert/1024/8",116606,18810.1,9766.64,3.27646e+06,819115,
141 "BM_SetInsert/1024/10",106365,17238.4,8421.53,4.74973e+06,1.18743e+06,
142 ```
143
144 <a name="output-files" />
145
146 ## Output Files
147
148 Write benchmark results to a file with the `--benchmark_out=<filename>` option
149 (or set `BENCHMARK_OUT`). Specify the output format with
150 `--benchmark_out_format={json|console|csv}` (or set
151 `BENCHMARK_OUT_FORMAT={json|console|csv}`). Note that the 'csv' reporter is
152 deprecated and the saved `.csv` file
153 [is not parsable](https://github.com/google/benchmark/issues/794) by csv
154 parsers.
155
156 Specifying `--benchmark_out` does not suppress the console output.
157
158 <a name="running-benchmarks" />
159
160 ## Running Benchmarks
161
162 Benchmarks are executed by running the produced binaries. Benchmarks binaries,
163 by default, accept options that may be specified either through their command
164 line interface or by setting environment variables before execution. For every
165 `--option_flag=<value>` CLI switch, a corresponding environment variable
166 `OPTION_FLAG=<value>` exist and is used as default if set (CLI switches always
167 prevails). A complete list of CLI options is available running benchmarks
168 with the `--help` switch.
169
170 <a name="running-a-subset-of-benchmarks" />
171
172 ## Running a Subset of Benchmarks
173
174 The `--benchmark_filter=<regex>` option (or `BENCHMARK_FILTER=<regex>`
175 environment variable) can be used to only run the benchmarks that match
176 the specified `<regex>`. For example:
177
178 ```bash
179 $ ./run_benchmarks.x --benchmark_filter=BM_memcpy/32
180 Run on (1 X 2300 MHz CPU )
181 2016-06-25 19:34:24
182 Benchmark Time CPU Iterations
183 ----------------------------------------------------
184 BM_memcpy/32 11 ns 11 ns 79545455
185 BM_memcpy/32k 2181 ns 2185 ns 324074
186 BM_memcpy/32 12 ns 12 ns 54687500
187 BM_memcpy/32k 1834 ns 1837 ns 357143
188 ```
189
190 ## Disabling Benchmarks
191
192 It is possible to temporarily disable benchmarks by renaming the benchmark
193 function to have the prefix "DISABLED_". This will cause the benchmark to
194 be skipped at runtime.
195
196 <a name="result-comparison" />
197
198 ## Result comparison
199
200 It is possible to compare the benchmarking results.
201 See [Additional Tooling Documentation](tools.md)
202
203 <a name="extra-context" />
204
205 ## Extra Context
206
207 Sometimes it's useful to add extra context to the content printed before the
208 results. By default this section includes information about the CPU on which
209 the benchmarks are running. If you do want to add more context, you can use
210 the `benchmark_context` command line flag:
211
212 ```bash
213 $ ./run_benchmarks --benchmark_context=pwd=`pwd`
214 Run on (1 x 2300 MHz CPU)
215 pwd: /home/user/benchmark/
216 Benchmark Time CPU Iterations
217 ----------------------------------------------------
218 BM_memcpy/32 11 ns 11 ns 79545455
219 BM_memcpy/32k 2181 ns 2185 ns 324074
220 ```
221
222 You can get the same effect with the API:
223
224 ```c++
225 benchmark::AddCustomContext("foo", "bar");
226 ```
227
228 Note that attempts to add a second value with the same key will fail with an
229 error message.
230
231 <a name="runtime-and-reporting-considerations" />
232
233 ## Runtime and Reporting Considerations
234
235 When the benchmark binary is executed, each benchmark function is run serially.
236 The number of iterations to run is determined dynamically by running the
237 benchmark a few times and measuring the time taken and ensuring that the
238 ultimate result will be statistically stable. As such, faster benchmark
239 functions will be run for more iterations than slower benchmark functions, and
240 the number of iterations is thus reported.
241
242 In all cases, the number of iterations for which the benchmark is run is
243 governed by the amount of time the benchmark takes. Concretely, the number of
244 iterations is at least one, not more than 1e9, until CPU time is greater than
245 the minimum time, or the wallclock time is 5x minimum time. The minimum time is
246 set per benchmark by calling `MinTime` on the registered benchmark object.
247
248 Furthermore warming up a benchmark might be necessary in order to get
249 stable results because of e.g caching effects of the code under benchmark.
250 Warming up means running the benchmark a given amount of time, before
251 results are actually taken into account. The amount of time for which
252 the warmup should be run can be set per benchmark by calling
253 `MinWarmUpTime` on the registered benchmark object or for all benchmarks
254 using the `--benchmark_min_warmup_time` command-line option. Note that
255 `MinWarmUpTime` will overwrite the value of `--benchmark_min_warmup_time`
256 for the single benchmark. How many iterations the warmup run of each
257 benchmark takes is determined the same way as described in the paragraph
258 above. Per default the warmup phase is set to 0 seconds and is therefore
259 disabled.
260
261 Average timings are then reported over the iterations run. If multiple
262 repetitions are requested using the `--benchmark_repetitions` command-line
263 option, or at registration time, the benchmark function will be run several
264 times and statistical results across these repetitions will also be reported.
265
266 As well as the per-benchmark entries, a preamble in the report will include
267 information about the machine on which the benchmarks are run.
268
269 <a name="setup-teardown" />
270
271 ## Setup/Teardown
272
273 Global setup/teardown specific to each benchmark can be done by
274 passing a callback to Setup/Teardown:
275
276 The setup/teardown callbacks will be invoked once for each benchmark. If the
277 benchmark is multi-threaded (will run in k threads), they will be invoked
278 exactly once before each run with k threads.
279
280 If the benchmark uses different size groups of threads, the above will be true
281 for each size group.
282
283 Eg.,
284
285 ```c++
286 static void DoSetup(const benchmark::State& state) {
287 }
288
289 static void DoTeardown(const benchmark::State& state) {
290 }
291
292 static void BM_func(benchmark::State& state) {...}
293
294 BENCHMARK(BM_func)->Arg(1)->Arg(3)->Threads(16)->Threads(32)->Setup(DoSetup)->Teardown(DoTeardown);
295
296 ```
297
298 In this example, `DoSetup` and `DoTearDown` will be invoked 4 times each,
299 specifically, once for each of this family:
300 - BM_func_Arg_1_Threads_16, BM_func_Arg_1_Threads_32
301 - BM_func_Arg_3_Threads_16, BM_func_Arg_3_Threads_32
302
303 <a name="passing-arguments" />
304
305 ## Passing Arguments
306
307 Sometimes a family of benchmarks can be implemented with just one routine that
308 takes an extra argument to specify which one of the family of benchmarks to
309 run. For example, the following code defines a family of benchmarks for
310 measuring the speed of `memcpy()` calls of different lengths:
311
312 ```c++
313 static void BM_memcpy(benchmark::State& state) {
314 char* src = new char[state.range(0)];
315 char* dst = new char[state.range(0)];
316 memset(src, 'x', state.range(0));
317 for (auto _ : state)
318 memcpy(dst, src, state.range(0));
319 state.SetBytesProcessed(int64_t(state.iterations()) *
320 int64_t(state.range(0)));
321 delete[] src;
322 delete[] dst;
323 }
324 BENCHMARK(BM_memcpy)->Arg(8)->Arg(64)->Arg(512)->Arg(4<<10)->Arg(8<<10);
325 ```
326
327 The preceding code is quite repetitive, and can be replaced with the following
328 short-hand. The following invocation will pick a few appropriate arguments in
329 the specified range and will generate a benchmark for each such argument.
330
331 ```c++
332 BENCHMARK(BM_memcpy)->Range(8, 8<<10);
333 ```
334
335 By default the arguments in the range are generated in multiples of eight and
336 the command above selects [ 8, 64, 512, 4k, 8k ]. In the following code the
337 range multiplier is changed to multiples of two.
338
339 ```c++
340 BENCHMARK(BM_memcpy)->RangeMultiplier(2)->Range(8, 8<<10);
341 ```
342
343 Now arguments generated are [ 8, 16, 32, 64, 128, 256, 512, 1024, 2k, 4k, 8k ].
344
345 The preceding code shows a method of defining a sparse range. The following
346 example shows a method of defining a dense range. It is then used to benchmark
347 the performance of `std::vector` initialization for uniformly increasing sizes.
348
349 ```c++
350 static void BM_DenseRange(benchmark::State& state) {
351 for(auto _ : state) {
352 std::vector<int> v(state.range(0), state.range(0));
353 auto data = v.data();
354 benchmark::DoNotOptimize(data);
355 benchmark::ClobberMemory();
356 }
357 }
358 BENCHMARK(BM_DenseRange)->DenseRange(0, 1024, 128);
359 ```
360
361 Now arguments generated are [ 0, 128, 256, 384, 512, 640, 768, 896, 1024 ].
362
363 You might have a benchmark that depends on two or more inputs. For example, the
364 following code defines a family of benchmarks for measuring the speed of set
365 insertion.
366
367 ```c++
368 static void BM_SetInsert(benchmark::State& state) {
369 std::set<int> data;
370 for (auto _ : state) {
371 state.PauseTiming();
372 data = ConstructRandomSet(state.range(0));
373 state.ResumeTiming();
374 for (int j = 0; j < state.range(1); ++j)
375 data.insert(RandomNumber());
376 }
377 }
378 BENCHMARK(BM_SetInsert)
379 ->Args({1<<10, 128})
380 ->Args({2<<10, 128})
381 ->Args({4<<10, 128})
382 ->Args({8<<10, 128})
383 ->Args({1<<10, 512})
384 ->Args({2<<10, 512})
385 ->Args({4<<10, 512})
386 ->Args({8<<10, 512});
387 ```
388
389 The preceding code is quite repetitive, and can be replaced with the following
390 short-hand. The following macro will pick a few appropriate arguments in the
391 product of the two specified ranges and will generate a benchmark for each such
392 pair.
393
394 <!-- {% raw %} -->
395 ```c++
396 BENCHMARK(BM_SetInsert)->Ranges({{1<<10, 8<<10}, {128, 512}});
397 ```
398 <!-- {% endraw %} -->
399
400 Some benchmarks may require specific argument values that cannot be expressed
401 with `Ranges`. In this case, `ArgsProduct` offers the ability to generate a
402 benchmark input for each combination in the product of the supplied vectors.
403
404 <!-- {% raw %} -->
405 ```c++
406 BENCHMARK(BM_SetInsert)
407 ->ArgsProduct({{1<<10, 3<<10, 8<<10}, {20, 40, 60, 80}})
408 // would generate the same benchmark arguments as
409 BENCHMARK(BM_SetInsert)
410 ->Args({1<<10, 20})
411 ->Args({3<<10, 20})
412 ->Args({8<<10, 20})
413 ->Args({3<<10, 40})
414 ->Args({8<<10, 40})
415 ->Args({1<<10, 40})
416 ->Args({1<<10, 60})
417 ->Args({3<<10, 60})
418 ->Args({8<<10, 60})
419 ->Args({1<<10, 80})
420 ->Args({3<<10, 80})
421 ->Args({8<<10, 80});
422 ```
423 <!-- {% endraw %} -->
424
425 For the most common scenarios, helper methods for creating a list of
426 integers for a given sparse or dense range are provided.
427
428 ```c++
429 BENCHMARK(BM_SetInsert)
430 ->ArgsProduct({
431 benchmark::CreateRange(8, 128, /*multi=*/2),
432 benchmark::CreateDenseRange(1, 4, /*step=*/1)
433 })
434 // would generate the same benchmark arguments as
435 BENCHMARK(BM_SetInsert)
436 ->ArgsProduct({
437 {8, 16, 32, 64, 128},
438 {1, 2, 3, 4}
439 });
440 ```
441
442 For more complex patterns of inputs, passing a custom function to `Apply` allows
443 programmatic specification of an arbitrary set of arguments on which to run the
444 benchmark. The following example enumerates a dense range on one parameter,
445 and a sparse range on the second.
446
447 ```c++
448 static void CustomArguments(benchmark::internal::Benchmark* b) {
449 for (int i = 0; i <= 10; ++i)
450 for (int j = 32; j <= 1024*1024; j *= 8)
451 b->Args({i, j});
452 }
453 BENCHMARK(BM_SetInsert)->Apply(CustomArguments);
454 ```
455
456 ### Passing Arbitrary Arguments to a Benchmark
457
458 In C++11 it is possible to define a benchmark that takes an arbitrary number
459 of extra arguments. The `BENCHMARK_CAPTURE(func, test_case_name, ...args)`
460 macro creates a benchmark that invokes `func` with the `benchmark::State` as
461 the first argument followed by the specified `args...`.
462 The `test_case_name` is appended to the name of the benchmark and
463 should describe the values passed.
464
465 ```c++
466 template <class ...Args>
467 void BM_takes_args(benchmark::State& state, Args&&... args) {
468 auto args_tuple = std::make_tuple(std::move(args)...);
469 for (auto _ : state) {
470 std::cout << std::get<0>(args_tuple) << ": " << std::get<1>(args_tuple)
471 << '\n';
472 [...]
473 }
474 }
475 // Registers a benchmark named "BM_takes_args/int_string_test" that passes
476 // the specified values to `args`.
477 BENCHMARK_CAPTURE(BM_takes_args, int_string_test, 42, std::string("abc"));
478
479 // Registers the same benchmark "BM_takes_args/int_test" that passes
480 // the specified values to `args`.
481 BENCHMARK_CAPTURE(BM_takes_args, int_test, 42, 43);
482 ```
483
484 Note that elements of `...args` may refer to global variables. Users should
485 avoid modifying global state inside of a benchmark.
486
487 <a name="asymptotic-complexity" />
488
489 ## Calculating Asymptotic Complexity (Big O)
490
491 Asymptotic complexity might be calculated for a family of benchmarks. The
492 following code will calculate the coefficient for the high-order term in the
493 running time and the normalized root-mean square error of string comparison.
494
495 ```c++
496 static void BM_StringCompare(benchmark::State& state) {
497 std::string s1(state.range(0), '-');
498 std::string s2(state.range(0), '-');
499 for (auto _ : state) {
500 auto comparison_result = s1.compare(s2);
501 benchmark::DoNotOptimize(comparison_result);
502 }
503 state.SetComplexityN(state.range(0));
504 }
505 BENCHMARK(BM_StringCompare)
506 ->RangeMultiplier(2)->Range(1<<10, 1<<18)->Complexity(benchmark::oN);
507 ```
508
509 As shown in the following invocation, asymptotic complexity might also be
510 calculated automatically.
511
512 ```c++
513 BENCHMARK(BM_StringCompare)
514 ->RangeMultiplier(2)->Range(1<<10, 1<<18)->Complexity();
515 ```
516
517 The following code will specify asymptotic complexity with a lambda function,
518 that might be used to customize high-order term calculation.
519
520 ```c++
521 BENCHMARK(BM_StringCompare)->RangeMultiplier(2)
522 ->Range(1<<10, 1<<18)->Complexity([](benchmark::IterationCount n)->double{return n; });
523 ```
524
525 <a name="custom-benchmark-name" />
526
527 ## Custom Benchmark Name
528
529 You can change the benchmark's name as follows:
530
531 ```c++
532 BENCHMARK(BM_memcpy)->Name("memcpy")->RangeMultiplier(2)->Range(8, 8<<10);
533 ```
534
535 The invocation will execute the benchmark as before using `BM_memcpy` but changes
536 the prefix in the report to `memcpy`.
537
538 <a name="templated-benchmarks" />
539
540 ## Templated Benchmarks
541
542 This example produces and consumes messages of size `sizeof(v)` `range_x`
543 times. It also outputs throughput in the absence of multiprogramming.
544
545 ```c++
546 template <class Q> void BM_Sequential(benchmark::State& state) {
547 Q q;
548 typename Q::value_type v;
549 for (auto _ : state) {
550 for (int i = state.range(0); i--; )
551 q.push(v);
552 for (int e = state.range(0); e--; )
553 q.Wait(&v);
554 }
555 // actually messages, not bytes:
556 state.SetBytesProcessed(
557 static_cast<int64_t>(state.iterations())*state.range(0));
558 }
559 // C++03
560 BENCHMARK_TEMPLATE(BM_Sequential, WaitQueue<int>)->Range(1<<0, 1<<10);
561
562 // C++11 or newer, you can use the BENCHMARK macro with template parameters:
563 BENCHMARK(BM_Sequential<WaitQueue<int>>)->Range(1<<0, 1<<10);
564
565 ```
566
567 Three macros are provided for adding benchmark templates.
568
569 ```c++
570 #ifdef BENCHMARK_HAS_CXX11
571 #define BENCHMARK(func<...>) // Takes any number of parameters.
572 #else // C++ < C++11
573 #define BENCHMARK_TEMPLATE(func, arg1)
574 #endif
575 #define BENCHMARK_TEMPLATE1(func, arg1)
576 #define BENCHMARK_TEMPLATE2(func, arg1, arg2)
577 ```
578
579 <a name="templated-benchmarks-with-arguments" />
580
581 ## Templated Benchmarks that take arguments
582
583 Sometimes there is a need to template benchmarks, and provide arguments to them.
584
585 ```c++
586 template <class Q> void BM_Sequential_With_Step(benchmark::State& state, int step) {
587 Q q;
588 typename Q::value_type v;
589 for (auto _ : state) {
590 for (int i = state.range(0); i-=step; )
591 q.push(v);
592 for (int e = state.range(0); e-=step; )
593 q.Wait(&v);
594 }
595 // actually messages, not bytes:
596 state.SetBytesProcessed(
597 static_cast<int64_t>(state.iterations())*state.range(0));
598 }
599
600 BENCHMARK_TEMPLATE1_CAPTURE(BM_Sequential, WaitQueue<int>, Step1, 1)->Range(1<<0, 1<<10);
601 ```
602
603 <a name="fixtures" />
604
605 ## Fixtures
606
607 Fixture tests are created by first defining a type that derives from
608 `::benchmark::Fixture` and then creating/registering the tests using the
609 following macros:
610
611 * `BENCHMARK_F(ClassName, Method)`
612 * `BENCHMARK_DEFINE_F(ClassName, Method)`
613 * `BENCHMARK_REGISTER_F(ClassName, Method)`
614
615 For Example:
616
617 ```c++
618 class MyFixture : public benchmark::Fixture {
619 public:
620 void SetUp(::benchmark::State& state) {
621 }
622
623 void TearDown(::benchmark::State& state) {
624 }
625 };
626
627 BENCHMARK_F(MyFixture, FooTest)(benchmark::State& st) {
628 for (auto _ : st) {
629 ...
630 }
631 }
632
633 BENCHMARK_DEFINE_F(MyFixture, BarTest)(benchmark::State& st) {
634 for (auto _ : st) {
635 ...
636 }
637 }
638 /* BarTest is NOT registered */
639 BENCHMARK_REGISTER_F(MyFixture, BarTest)->Threads(2);
640 /* BarTest is now registered */
641 ```
642
643 ### Templated Fixtures
644
645 Also you can create templated fixture by using the following macros:
646
647 * `BENCHMARK_TEMPLATE_F(ClassName, Method, ...)`
648 * `BENCHMARK_TEMPLATE_DEFINE_F(ClassName, Method, ...)`
649
650 For example:
651
652 ```c++
653 template<typename T>
654 class MyFixture : public benchmark::Fixture {};
655
656 BENCHMARK_TEMPLATE_F(MyFixture, IntTest, int)(benchmark::State& st) {
657 for (auto _ : st) {
658 ...
659 }
660 }
661
662 BENCHMARK_TEMPLATE_DEFINE_F(MyFixture, DoubleTest, double)(benchmark::State& st) {
663 for (auto _ : st) {
664 ...
665 }
666 }
667
668 BENCHMARK_REGISTER_F(MyFixture, DoubleTest)->Threads(2);
669 ```
670
671 <a name="custom-counters" />
672
673 ## Custom Counters
674
675 You can add your own counters with user-defined names. The example below
676 will add columns "Foo", "Bar" and "Baz" in its output:
677
678 ```c++
679 static void UserCountersExample1(benchmark::State& state) {
680 double numFoos = 0, numBars = 0, numBazs = 0;
681 for (auto _ : state) {
682 // ... count Foo,Bar,Baz events
683 }
684 state.counters["Foo"] = numFoos;
685 state.counters["Bar"] = numBars;
686 state.counters["Baz"] = numBazs;
687 }
688 ```
689
690 The `state.counters` object is a `std::map` with `std::string` keys
691 and `Counter` values. The latter is a `double`-like class, via an implicit
692 conversion to `double&`. Thus you can use all of the standard arithmetic
693 assignment operators (`=,+=,-=,*=,/=`) to change the value of each counter.
694
695 In multithreaded benchmarks, each counter is set on the calling thread only.
696 When the benchmark finishes, the counters from each thread will be summed;
697 the resulting sum is the value which will be shown for the benchmark.
698
699 The `Counter` constructor accepts three parameters: the value as a `double`
700 ; a bit flag which allows you to show counters as rates, and/or as per-thread
701 iteration, and/or as per-thread averages, and/or iteration invariants,
702 and/or finally inverting the result; and a flag specifying the 'unit' - i.e.
703 is 1k a 1000 (default, `benchmark::Counter::OneK::kIs1000`), or 1024
704 (`benchmark::Counter::OneK::kIs1024`)?
705
706 ```c++
707 // sets a simple counter
708 state.counters["Foo"] = numFoos;
709
710 // Set the counter as a rate. It will be presented divided
711 // by the duration of the benchmark.
712 // Meaning: per one second, how many 'foo's are processed?
713 state.counters["FooRate"] = Counter(numFoos, benchmark::Counter::kIsRate);
714
715 // Set the counter as a rate. It will be presented divided
716 // by the duration of the benchmark, and the result inverted.
717 // Meaning: how many seconds it takes to process one 'foo'?
718 state.counters["FooInvRate"] = Counter(numFoos, benchmark::Counter::kIsRate | benchmark::Counter::kInvert);
719
720 // Set the counter as a thread-average quantity. It will
721 // be presented divided by the number of threads.
722 state.counters["FooAvg"] = Counter(numFoos, benchmark::Counter::kAvgThreads);
723
724 // There's also a combined flag:
725 state.counters["FooAvgRate"] = Counter(numFoos,benchmark::Counter::kAvgThreadsRate);
726
727 // This says that we process with the rate of state.range(0) bytes every iteration:
728 state.counters["BytesProcessed"] = Counter(state.range(0), benchmark::Counter::kIsIterationInvariantRate, benchmark::Counter::OneK::kIs1024);
729 ```
730
731 When you're compiling in C++11 mode or later you can use `insert()` with
732 `std::initializer_list`:
733
734 <!-- {% raw %} -->
735 ```c++
736 // With C++11, this can be done:
737 state.counters.insert({{"Foo", numFoos}, {"Bar", numBars}, {"Baz", numBazs}});
738 // ... instead of:
739 state.counters["Foo"] = numFoos;
740 state.counters["Bar"] = numBars;
741 state.counters["Baz"] = numBazs;
742 ```
743 <!-- {% endraw %} -->
744
745 ### Counter Reporting
746
747 When using the console reporter, by default, user counters are printed at
748 the end after the table, the same way as ``bytes_processed`` and
749 ``items_processed``. This is best for cases in which there are few counters,
750 or where there are only a couple of lines per benchmark. Here's an example of
751 the default output:
752
753 ```
754 ------------------------------------------------------------------------------
755 Benchmark Time CPU Iterations UserCounters...
756 ------------------------------------------------------------------------------
757 BM_UserCounter/threads:8 2248 ns 10277 ns 68808 Bar=16 Bat=40 Baz=24 Foo=8
758 BM_UserCounter/threads:1 9797 ns 9788 ns 71523 Bar=2 Bat=5 Baz=3 Foo=1024m
759 BM_UserCounter/threads:2 4924 ns 9842 ns 71036 Bar=4 Bat=10 Baz=6 Foo=2
760 BM_UserCounter/threads:4 2589 ns 10284 ns 68012 Bar=8 Bat=20 Baz=12 Foo=4
761 BM_UserCounter/threads:8 2212 ns 10287 ns 68040 Bar=16 Bat=40 Baz=24 Foo=8
762 BM_UserCounter/threads:16 1782 ns 10278 ns 68144 Bar=32 Bat=80 Baz=48 Foo=16
763 BM_UserCounter/threads:32 1291 ns 10296 ns 68256 Bar=64 Bat=160 Baz=96 Foo=32
764 BM_UserCounter/threads:4 2615 ns 10307 ns 68040 Bar=8 Bat=20 Baz=12 Foo=4
765 BM_Factorial 26 ns 26 ns 26608979 40320
766 BM_Factorial/real_time 26 ns 26 ns 26587936 40320
767 BM_CalculatePiRange/1 16 ns 16 ns 45704255 0
768 BM_CalculatePiRange/8 73 ns 73 ns 9520927 3.28374
769 BM_CalculatePiRange/64 609 ns 609 ns 1140647 3.15746
770 BM_CalculatePiRange/512 4900 ns 4901 ns 142696 3.14355
771 ```
772
773 If this doesn't suit you, you can print each counter as a table column by
774 passing the flag `--benchmark_counters_tabular=true` to the benchmark
775 application. This is best for cases in which there are a lot of counters, or
776 a lot of lines per individual benchmark. Note that this will trigger a
777 reprinting of the table header any time the counter set changes between
778 individual benchmarks. Here's an example of corresponding output when
779 `--benchmark_counters_tabular=true` is passed:
780
781 ```
782 ---------------------------------------------------------------------------------------
783 Benchmark Time CPU Iterations Bar Bat Baz Foo
784 ---------------------------------------------------------------------------------------
785 BM_UserCounter/threads:8 2198 ns 9953 ns 70688 16 40 24 8
786 BM_UserCounter/threads:1 9504 ns 9504 ns 73787 2 5 3 1
787 BM_UserCounter/threads:2 4775 ns 9550 ns 72606 4 10 6 2
788 BM_UserCounter/threads:4 2508 ns 9951 ns 70332 8 20 12 4
789 BM_UserCounter/threads:8 2055 ns 9933 ns 70344 16 40 24 8
790 BM_UserCounter/threads:16 1610 ns 9946 ns 70720 32 80 48 16
791 BM_UserCounter/threads:32 1192 ns 9948 ns 70496 64 160 96 32
792 BM_UserCounter/threads:4 2506 ns 9949 ns 70332 8 20 12 4
793 --------------------------------------------------------------
794 Benchmark Time CPU Iterations
795 --------------------------------------------------------------
796 BM_Factorial 26 ns 26 ns 26392245 40320
797 BM_Factorial/real_time 26 ns 26 ns 26494107 40320
798 BM_CalculatePiRange/1 15 ns 15 ns 45571597 0
799 BM_CalculatePiRange/8 74 ns 74 ns 9450212 3.28374
800 BM_CalculatePiRange/64 595 ns 595 ns 1173901 3.15746
801 BM_CalculatePiRange/512 4752 ns 4752 ns 147380 3.14355
802 BM_CalculatePiRange/4k 37970 ns 37972 ns 18453 3.14184
803 BM_CalculatePiRange/32k 303733 ns 303744 ns 2305 3.14162
804 BM_CalculatePiRange/256k 2434095 ns 2434186 ns 288 3.1416
805 BM_CalculatePiRange/1024k 9721140 ns 9721413 ns 71 3.14159
806 BM_CalculatePi/threads:8 2255 ns 9943 ns 70936
807 ```
808
809 Note above the additional header printed when the benchmark changes from
810 ``BM_UserCounter`` to ``BM_Factorial``. This is because ``BM_Factorial`` does
811 not have the same counter set as ``BM_UserCounter``.
812
813 <a name="multithreaded-benchmarks"/>
814
815 ## Multithreaded Benchmarks
816
817 In a multithreaded test (benchmark invoked by multiple threads simultaneously),
818 it is guaranteed that none of the threads will start until all have reached
819 the start of the benchmark loop, and all will have finished before any thread
820 exits the benchmark loop. (This behavior is also provided by the `KeepRunning()`
821 API) As such, any global setup or teardown can be wrapped in a check against the thread
822 index:
823
824 ```c++
825 static void BM_MultiThreaded(benchmark::State& state) {
826 if (state.thread_index() == 0) {
827 // Setup code here.
828 }
829 for (auto _ : state) {
830 // Run the test as normal.
831 }
832 if (state.thread_index() == 0) {
833 // Teardown code here.
834 }
835 }
836 BENCHMARK(BM_MultiThreaded)->Threads(2);
837 ```
838
839 To run the benchmark across a range of thread counts, instead of `Threads`, use
840 `ThreadRange`. This takes two parameters (`min_threads` and `max_threads`) and
841 runs the benchmark once for values in the inclusive range. For example:
842
843 ```c++
844 BENCHMARK(BM_MultiThreaded)->ThreadRange(1, 8);
845 ```
846
847 will run `BM_MultiThreaded` with thread counts 1, 2, 4, and 8.
848
849 If the benchmarked code itself uses threads and you want to compare it to
850 single-threaded code, you may want to use real-time ("wallclock") measurements
851 for latency comparisons:
852
853 ```c++
854 BENCHMARK(BM_test)->Range(8, 8<<10)->UseRealTime();
855 ```
856
857 Without `UseRealTime`, CPU time is used by default.
858
859 <a name="cpu-timers" />
860
861 ## CPU Timers
862
863 By default, the CPU timer only measures the time spent by the main thread.
864 If the benchmark itself uses threads internally, this measurement may not
865 be what you are looking for. Instead, there is a way to measure the total
866 CPU usage of the process, by all the threads.
867
868 ```c++
869 void callee(int i);
870
871 static void MyMain(int size) {
872 #pragma omp parallel for
873 for(int i = 0; i < size; i++)
874 callee(i);
875 }
876
877 static void BM_OpenMP(benchmark::State& state) {
878 for (auto _ : state)
879 MyMain(state.range(0));
880 }
881
882 // Measure the time spent by the main thread, use it to decide for how long to
883 // run the benchmark loop. Depending on the internal implementation detail may
884 // measure to anywhere from near-zero (the overhead spent before/after work
885 // handoff to worker thread[s]) to the whole single-thread time.
886 BENCHMARK(BM_OpenMP)->Range(8, 8<<10);
887
888 // Measure the user-visible time, the wall clock (literally, the time that
889 // has passed on the clock on the wall), use it to decide for how long to
890 // run the benchmark loop. This will always be meaningful, and will match the
891 // time spent by the main thread in single-threaded case, in general decreasing
892 // with the number of internal threads doing the work.
893 BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->UseRealTime();
894
895 // Measure the total CPU consumption, use it to decide for how long to
896 // run the benchmark loop. This will always measure to no less than the
897 // time spent by the main thread in single-threaded case.
898 BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->MeasureProcessCPUTime();
899
900 // A mixture of the last two. Measure the total CPU consumption, but use the
901 // wall clock to decide for how long to run the benchmark loop.
902 BENCHMARK(BM_OpenMP)->Range(8, 8<<10)->MeasureProcessCPUTime()->UseRealTime();
903 ```
904
905 ### Controlling Timers
906
907 Normally, the entire duration of the work loop (`for (auto _ : state) {}`)
908 is measured. But sometimes, it is necessary to do some work inside of
909 that loop, every iteration, but without counting that time to the benchmark time.
910 That is possible, although it is not recommended, since it has high overhead.
911
912 <!-- {% raw %} -->
913 ```c++
914 static void BM_SetInsert_With_Timer_Control(benchmark::State& state) {
915 std::set<int> data;
916 for (auto _ : state) {
917 state.PauseTiming(); // Stop timers. They will not count until they are resumed.
918 data = ConstructRandomSet(state.range(0)); // Do something that should not be measured
919 state.ResumeTiming(); // And resume timers. They are now counting again.
920 // The rest will be measured.
921 for (int j = 0; j < state.range(1); ++j)
922 data.insert(RandomNumber());
923 }
924 }
925 BENCHMARK(BM_SetInsert_With_Timer_Control)->Ranges({{1<<10, 8<<10}, {128, 512}});
926 ```
927 <!-- {% endraw %} -->
928
929 <a name="manual-timing" />
930
931 ## Manual Timing
932
933 For benchmarking something for which neither CPU time nor real-time are
934 correct or accurate enough, completely manual timing is supported using
935 the `UseManualTime` function.
936
937 When `UseManualTime` is used, the benchmarked code must call
938 `SetIterationTime` once per iteration of the benchmark loop to
939 report the manually measured time.
940
941 An example use case for this is benchmarking GPU execution (e.g. OpenCL
942 or CUDA kernels, OpenGL or Vulkan or Direct3D draw calls), which cannot
943 be accurately measured using CPU time or real-time. Instead, they can be
944 measured accurately using a dedicated API, and these measurement results
945 can be reported back with `SetIterationTime`.
946
947 ```c++
948 static void BM_ManualTiming(benchmark::State& state) {
949 int microseconds = state.range(0);
950 std::chrono::duration<double, std::micro> sleep_duration {
951 static_cast<double>(microseconds)
952 };
953
954 for (auto _ : state) {
955 auto start = std::chrono::high_resolution_clock::now();
956 // Simulate some useful workload with a sleep
957 std::this_thread::sleep_for(sleep_duration);
958 auto end = std::chrono::high_resolution_clock::now();
959
960 auto elapsed_seconds =
961 std::chrono::duration_cast<std::chrono::duration<double>>(
962 end - start);
963
964 state.SetIterationTime(elapsed_seconds.count());
965 }
966 }
967 BENCHMARK(BM_ManualTiming)->Range(1, 1<<17)->UseManualTime();
968 ```
969
970 <a name="setting-the-time-unit" />
971
972 ## Setting the Time Unit
973
974 If a benchmark runs a few milliseconds it may be hard to visually compare the
975 measured times, since the output data is given in nanoseconds per default. In
976 order to manually set the time unit, you can specify it manually:
977
978 ```c++
979 BENCHMARK(BM_test)->Unit(benchmark::kMillisecond);
980 ```
981
982 Additionally the default time unit can be set globally with the
983 `--benchmark_time_unit={ns|us|ms|s}` command line argument. The argument only
984 affects benchmarks where the time unit is not set explicitly.
985
986 <a name="preventing-optimization" />
987
988 ## Preventing Optimization
989
990 To prevent a value or expression from being optimized away by the compiler
991 the `benchmark::DoNotOptimize(...)` and `benchmark::ClobberMemory()`
992 functions can be used.
993
994 ```c++
995 static void BM_test(benchmark::State& state) {
996 for (auto _ : state) {
997 int x = 0;
998 for (int i=0; i < 64; ++i) {
999 benchmark::DoNotOptimize(x += i);
1000 }
1001 }
1002 }
1003 ```
1004
1005 `DoNotOptimize(<expr>)` forces the *result* of `<expr>` to be stored in either
1006 memory or a register. For GNU based compilers it acts as read/write barrier
1007 for global memory. More specifically it forces the compiler to flush pending
1008 writes to memory and reload any other values as necessary.
1009
1010 Note that `DoNotOptimize(<expr>)` does not prevent optimizations on `<expr>`
1011 in any way. `<expr>` may even be removed entirely when the result is already
1012 known. For example:
1013
1014 ```c++
1015 /* Example 1: `<expr>` is removed entirely. */
1016 int foo(int x) { return x + 42; }
1017 while (...) DoNotOptimize(foo(0)); // Optimized to DoNotOptimize(42);
1018
1019 /* Example 2: Result of '<expr>' is only reused */
1020 int bar(int) __attribute__((const));
1021 while (...) DoNotOptimize(bar(0)); // Optimized to:
1022 // int __result__ = bar(0);
1023 // while (...) DoNotOptimize(__result__);
1024 ```
1025
1026 The second tool for preventing optimizations is `ClobberMemory()`. In essence
1027 `ClobberMemory()` forces the compiler to perform all pending writes to global
1028 memory. Memory managed by block scope objects must be "escaped" using
1029 `DoNotOptimize(...)` before it can be clobbered. In the below example
1030 `ClobberMemory()` prevents the call to `v.push_back(42)` from being optimized
1031 away.
1032
1033 ```c++
1034 static void BM_vector_push_back(benchmark::State& state) {
1035 for (auto _ : state) {
1036 std::vector<int> v;
1037 v.reserve(1);
1038 auto data = v.data(); // Allow v.data() to be clobbered. Pass as non-const
1039 benchmark::DoNotOptimize(data); // lvalue to avoid undesired compiler optimizations
1040 v.push_back(42);
1041 benchmark::ClobberMemory(); // Force 42 to be written to memory.
1042 }
1043 }
1044 ```
1045
1046 Note that `ClobberMemory()` is only available for GNU or MSVC based compilers.
1047
1048 <a name="reporting-statistics" />
1049
1050 ## Statistics: Reporting the Mean, Median and Standard Deviation / Coefficient of variation of Repeated Benchmarks
1051
1052 By default each benchmark is run once and that single result is reported.
1053 However benchmarks are often noisy and a single result may not be representative
1054 of the overall behavior. For this reason it's possible to repeatedly rerun the
1055 benchmark.
1056
1057 The number of runs of each benchmark is specified globally by the
1058 `--benchmark_repetitions` flag or on a per benchmark basis by calling
1059 `Repetitions` on the registered benchmark object. When a benchmark is run more
1060 than once the mean, median, standard deviation and coefficient of variation
1061 of the runs will be reported.
1062
1063 Additionally the `--benchmark_report_aggregates_only={true|false}`,
1064 `--benchmark_display_aggregates_only={true|false}` flags or
1065 `ReportAggregatesOnly(bool)`, `DisplayAggregatesOnly(bool)` functions can be
1066 used to change how repeated tests are reported. By default the result of each
1067 repeated run is reported. When `report aggregates only` option is `true`,
1068 only the aggregates (i.e. mean, median, standard deviation and coefficient
1069 of variation, maybe complexity measurements if they were requested) of the runs
1070 is reported, to both the reporters - standard output (console), and the file.
1071 However when only the `display aggregates only` option is `true`,
1072 only the aggregates are displayed in the standard output, while the file
1073 output still contains everything.
1074 Calling `ReportAggregatesOnly(bool)` / `DisplayAggregatesOnly(bool)` on a
1075 registered benchmark object overrides the value of the appropriate flag for that
1076 benchmark.
1077
1078 <a name="custom-statistics" />
1079
1080 ## Custom Statistics
1081
1082 While having these aggregates is nice, this may not be enough for everyone.
1083 For example you may want to know what the largest observation is, e.g. because
1084 you have some real-time constraints. This is easy. The following code will
1085 specify a custom statistic to be calculated, defined by a lambda function.
1086
1087 ```c++
1088 void BM_spin_empty(benchmark::State& state) {
1089 for (auto _ : state) {
1090 for (int x = 0; x < state.range(0); ++x) {
1091 benchmark::DoNotOptimize(x);
1092 }
1093 }
1094 }
1095
1096 BENCHMARK(BM_spin_empty)
1097 ->ComputeStatistics("max", [](const std::vector<double>& v) -> double {
1098 return *(std::max_element(std::begin(v), std::end(v)));
1099 })
1100 ->Arg(512);
1101 ```
1102
1103 While usually the statistics produce values in time units,
1104 you can also produce percentages:
1105
1106 ```c++
1107 void BM_spin_empty(benchmark::State& state) {
1108 for (auto _ : state) {
1109 for (int x = 0; x < state.range(0); ++x) {
1110 benchmark::DoNotOptimize(x);
1111 }
1112 }
1113 }
1114
1115 BENCHMARK(BM_spin_empty)
1116 ->ComputeStatistics("ratio", [](const std::vector<double>& v) -> double {
1117 return std::begin(v) / std::end(v);
1118 }, benchmark::StatisticUnit::kPercentage)
1119 ->Arg(512);
1120 ```
1121
1122 <a name="memory-usage" />
1123
1124 ## Memory Usage
1125
1126 It's often useful to also track memory usage for benchmarks, alongside CPU
1127 performance. For this reason, benchmark offers the `RegisterMemoryManager`
1128 method that allows a custom `MemoryManager` to be injected.
1129
1130 If set, the `MemoryManager::Start` and `MemoryManager::Stop` methods will be
1131 called at the start and end of benchmark runs to allow user code to fill out
1132 a report on the number of allocations, bytes used, etc.
1133
1134 This data will then be reported alongside other performance data, currently
1135 only when using JSON output.
1136
1137 <a name="using-register-benchmark" />
1138
1139 ## Using RegisterBenchmark(name, fn, args...)
1140
1141 The `RegisterBenchmark(name, func, args...)` function provides an alternative
1142 way to create and register benchmarks.
1143 `RegisterBenchmark(name, func, args...)` creates, registers, and returns a
1144 pointer to a new benchmark with the specified `name` that invokes
1145 `func(st, args...)` where `st` is a `benchmark::State` object.
1146
1147 Unlike the `BENCHMARK` registration macros, which can only be used at the global
1148 scope, the `RegisterBenchmark` can be called anywhere. This allows for
1149 benchmark tests to be registered programmatically.
1150
1151 Additionally `RegisterBenchmark` allows any callable object to be registered
1152 as a benchmark. Including capturing lambdas and function objects.
1153
1154 For Example:
1155 ```c++
1156 auto BM_test = [](benchmark::State& st, auto Inputs) { /* ... */ };
1157
1158 int main(int argc, char** argv) {
1159 for (auto& test_input : { /* ... */ })
1160 benchmark::RegisterBenchmark(test_input.name(), BM_test, test_input);
1161 benchmark::Initialize(&argc, argv);
1162 benchmark::RunSpecifiedBenchmarks();
1163 benchmark::Shutdown();
1164 }
1165 ```
1166
1167 <a name="exiting-with-an-error" />
1168
1169 ## Exiting with an Error
1170
1171 When errors caused by external influences, such as file I/O and network
1172 communication, occur within a benchmark the
1173 `State::SkipWithError(const std::string& msg)` function can be used to skip that run
1174 of benchmark and report the error. Note that only future iterations of the
1175 `KeepRunning()` are skipped. For the ranged-for version of the benchmark loop
1176 Users must explicitly exit the loop, otherwise all iterations will be performed.
1177 Users may explicitly return to exit the benchmark immediately.
1178
1179 The `SkipWithError(...)` function may be used at any point within the benchmark,
1180 including before and after the benchmark loop. Moreover, if `SkipWithError(...)`
1181 has been used, it is not required to reach the benchmark loop and one may return
1182 from the benchmark function early.
1183
1184 For example:
1185
1186 ```c++
1187 static void BM_test(benchmark::State& state) {
1188 auto resource = GetResource();
1189 if (!resource.good()) {
1190 state.SkipWithError("Resource is not good!");
1191 // KeepRunning() loop will not be entered.
1192 }
1193 while (state.KeepRunning()) {
1194 auto data = resource.read_data();
1195 if (!resource.good()) {
1196 state.SkipWithError("Failed to read data!");
1197 break; // Needed to skip the rest of the iteration.
1198 }
1199 do_stuff(data);
1200 }
1201 }
1202
1203 static void BM_test_ranged_fo(benchmark::State & state) {
1204 auto resource = GetResource();
1205 if (!resource.good()) {
1206 state.SkipWithError("Resource is not good!");
1207 return; // Early return is allowed when SkipWithError() has been used.
1208 }
1209 for (auto _ : state) {
1210 auto data = resource.read_data();
1211 if (!resource.good()) {
1212 state.SkipWithError("Failed to read data!");
1213 break; // REQUIRED to prevent all further iterations.
1214 }
1215 do_stuff(data);
1216 }
1217 }
1218 ```
1219 <a name="a-faster-keep-running-loop" />
1220
1221 ## A Faster KeepRunning Loop
1222
1223 In C++11 mode, a ranged-based for loop should be used in preference to
1224 the `KeepRunning` loop for running the benchmarks. For example:
1225
1226 ```c++
1227 static void BM_Fast(benchmark::State &state) {
1228 for (auto _ : state) {
1229 FastOperation();
1230 }
1231 }
1232 BENCHMARK(BM_Fast);
1233 ```
1234
1235 The reason the ranged-for loop is faster than using `KeepRunning`, is
1236 because `KeepRunning` requires a memory load and store of the iteration count
1237 ever iteration, whereas the ranged-for variant is able to keep the iteration count
1238 in a register.
1239
1240 For example, an empty inner loop of using the ranged-based for method looks like:
1241
1242 ```asm
1243 # Loop Init
1244 mov rbx, qword ptr [r14 + 104]
1245 call benchmark::State::StartKeepRunning()
1246 test rbx, rbx
1247 je .LoopEnd
1248 .LoopHeader: # =>This Inner Loop Header: Depth=1
1249 add rbx, -1
1250 jne .LoopHeader
1251 .LoopEnd:
1252 ```
1253
1254 Compared to an empty `KeepRunning` loop, which looks like:
1255
1256 ```asm
1257 .LoopHeader: # in Loop: Header=BB0_3 Depth=1
1258 cmp byte ptr [rbx], 1
1259 jne .LoopInit
1260 .LoopBody: # =>This Inner Loop Header: Depth=1
1261 mov rax, qword ptr [rbx + 8]
1262 lea rcx, [rax + 1]
1263 mov qword ptr [rbx + 8], rcx
1264 cmp rax, qword ptr [rbx + 104]
1265 jb .LoopHeader
1266 jmp .LoopEnd
1267 .LoopInit:
1268 mov rdi, rbx
1269 call benchmark::State::StartKeepRunning()
1270 jmp .LoopBody
1271 .LoopEnd:
1272 ```
1273
1274 Unless C++03 compatibility is required, the ranged-for variant of writing
1275 the benchmark loop should be preferred.
1276
1277 <a name="disabling-cpu-frequency-scaling" />
1278
1279 ## Disabling CPU Frequency Scaling
1280
1281 If you see this error:
1282
1283 ```
1284 ***WARNING*** CPU scaling is enabled, the benchmark real time measurements may
1285 be noisy and will incur extra overhead.
1286 ```
1287
1288 you might want to disable the CPU frequency scaling while running the
1289 benchmark, as well as consider other ways to stabilize the performance of
1290 your system while benchmarking.
1291
1292 See [Reducing Variance](reducing_variance.md) for more information.