| 1 |
# Reducing Variance |
| 2 |
|
| 3 |
<a name="disabling-cpu-frequency-scaling" /> |
| 4 |
|
| 5 |
## Disabling CPU Frequency Scaling |
| 6 |
|
| 7 |
If you see this error: |
| 8 |
|
| 9 |
``` |
| 10 |
***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead. |
| 11 |
``` |
| 12 |
|
| 13 |
you might want to disable the CPU frequency scaling while running the |
| 14 |
benchmark, as well as consider other ways to stabilize the performance of |
| 15 |
your system while benchmarking. |
| 16 |
|
| 17 |
Exactly how to do this depends on the Linux distribution, |
| 18 |
desktop environment, and installed programs. Specific details are a moving |
| 19 |
target, so we will not attempt to exhaustively document them here. |
| 20 |
|
| 21 |
One simple option is to use the `cpupower` program to change the |
| 22 |
performance governor to "performance". This tool is maintained along with |
| 23 |
the Linux kernel and provided by your distribution. |
| 24 |
|
| 25 |
It must be run as root, like this: |
| 26 |
|
| 27 |
```bash |
| 28 |
sudo cpupower frequency-set --governor performance |
| 29 |
``` |
| 30 |
|
| 31 |
After this you can verify that all CPUs are using the performance governor |
| 32 |
by running this command: |
| 33 |
|
| 34 |
```bash |
| 35 |
cpupower frequency-info -o proc |
| 36 |
``` |
| 37 |
|
| 38 |
The benchmarks you subsequently run will have less variance. |
| 39 |
|
| 40 |
<a name="reducing-variance" /> |
| 41 |
|
| 42 |
## Reducing Variance in Benchmarks |
| 43 |
|
| 44 |
The Linux CPU frequency governor [discussed |
| 45 |
above](user_guide#disabling-cpu-frequency-scaling) is not the only source |
| 46 |
of noise in benchmarks. Some, but not all, of the sources of variance |
| 47 |
include: |
| 48 |
|
| 49 |
1. On multi-core machines not all CPUs/CPU cores/CPU threads run the same |
| 50 |
speed, so running a benchmark one time and then again may give a |
| 51 |
different result depending on which CPU it ran on. |
| 52 |
2. CPU scaling features that run on the CPU, like Intel's Turbo Boost and |
| 53 |
AMD Turbo Core and Precision Boost, can temporarily change the CPU |
| 54 |
frequency even when the using the "performance" governor on Linux. |
| 55 |
3. Context switching between CPUs, or scheduling competition on the CPU the |
| 56 |
benchmark is running on. |
| 57 |
4. Intel Hyperthreading or AMD SMT causing the same issue as above. |
| 58 |
5. Cache effects caused by code running on other CPUs. |
| 59 |
6. Non-uniform memory architectures (NUMA). |
| 60 |
|
| 61 |
These can cause variance in benchmarks results within a single run |
| 62 |
(`--benchmark_repetitions=N`) or across multiple runs of the benchmark |
| 63 |
program. |
| 64 |
|
| 65 |
Reducing sources of variance is OS and architecture dependent, which is one |
| 66 |
reason some companies maintain machines dedicated to performance testing. |
| 67 |
|
| 68 |
Some of the easier and effective ways of reducing variance on a typical |
| 69 |
Linux workstation are: |
| 70 |
|
| 71 |
1. Use the performance governor as [discussed |
| 72 |
above](user_guide#disabling-cpu-frequency-scaling). |
| 73 |
1. Disable processor boosting by: |
| 74 |
```sh |
| 75 |
echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost |
| 76 |
``` |
| 77 |
See the Linux kernel's |
| 78 |
[boost.txt](https://www.kernel.org/doc/Documentation/cpu-freq/boost.txt) |
| 79 |
for more information. |
| 80 |
2. Set the benchmark program's task affinity to a fixed cpu. For example: |
| 81 |
```sh |
| 82 |
taskset -c 0 ./mybenchmark |
| 83 |
``` |
| 84 |
3. Disabling Hyperthreading/SMT. This can be done in the Bios or using the |
| 85 |
`/sys` file system (see the LLVM project's [Benchmarking |
| 86 |
tips](https://llvm.org/docs/Benchmarking.html)). |
| 87 |
4. Close other programs that do non-trivial things based on timers, such as |
| 88 |
your web browser, desktop environment, etc. |
| 89 |
5. Reduce the working set of your benchmark to fit within the L1 cache, but |
| 90 |
do be aware that this may lead you to optimize for an unrealistic |
| 91 |
situation. |
| 92 |
|
| 93 |
Further resources on this topic: |
| 94 |
|
| 95 |
1. The LLVM project's [Benchmarking |
| 96 |
tips](https://llvm.org/docs/Benchmarking.html). |
| 97 |
1. The Arch Wiki [Cpu frequency |
| 98 |
scaling](https://wiki.archlinux.org/title/CPU_frequency_scaling) page. |