Debian Codegraph @

testing/performance/perftest-in-a-nutshell/perfdocs/index.rst

raw
1 =================================
2 Performance Testing in a Nutshell
3 =================================
4
5 .. contents::
6 :depth: 3
7
8 .. note::
9 This page is still under construction, and is missing the videos.
10
11 This document provides all the information needed regarding the basics of performance testing our products. A set of videos, and instructions can be found to guide you across the various subjects related to performance testing. If you're new to performance testing, starting from the first section would be the most helpful for you.
12
13 Help! I have a regression. What do I do?
14 ----------------------------------------
15
16 Most people will first be introduced to performance testing at Mozilla through a bug like this:
17
18 .. image:: ./perf_alert_bug.png
19 :alt: Performance Alert Bug
20 :scale: 75%
21 :align: center
22
23
24 The title of the bug will include the tests, and subtests that caused the largest change, the platform(s) the changes were detected on, and the date. It will also be prefixed with a range of the size (in percentage) of the changes. In the bug details, the bug which triggered the alert will also be found in the ``Regressed by`` field, and there will be a needinfo for the patch author.
25
26 **Bug overview and the first comment (aka comment 0):**
27
28 An alert summary table will always be found in the first comment (comment 0) of a performance regression bug, it provides all the information required to understand what regressed.
29
30 .. image:: ./perf_alert_comment_zero.png
31 :alt: Comment Zero from a Performance Alert Bug
32 :scale: 75%
33 :align: center
34
35
36 * In the first sentence of the alert, a link to the actual push/commit that caused it is provided.
37 * Below that, a table that summarizes the regressions, and improvements is shown.
38 * Each row in the table represents a detected regression or improvement.
39
40 * ``Ratio``: Provides the size of the change, and links to a graph for the metric.
41 * ``Test``: The name of the test, and metric that changed. It links to the documentation for the test.
42 * ``Platform``: The name of the platform that the change was detected on.
43 * ``Options``: Specify the conditions in which the test was run (e.g. cold, warm, bytecode-cached).
44 * ``Absolute Values``: The absolute values of the change.
45 * ``Performance Profiles``: Before, and after Gecko profiles of the test (this is not available for all test frameworks yet).
46
47 * Below the table, an ``alert summary`` link can be found which provides a different view of this table, a full in-depth explanation of the alert summary link can be found here `Perfherder Alerts View`_. Some additional debugging information such as ``side-by-side`` videos can be found here. It will also include any additional tests that triggered alerts after this bug was filed.
48 * Then some information on what we expect from the patch author is provided regarding how long they have to respond to the alert before it gets backed out along with links to the guidelines for handling regressions.
49 * Finally, a helpful command to run these tests on try is provided using ``./mach try perf --alert <ALERT-NUM>``. :ref:`See here for more information about mach try perf <Mach Try Perf>`.
50
51 **From the alert summary comment, there are multiple things that could be verified:**
52 * Check the graphs to ensure that the regression/improvements are very visible.
53 * Look at the alert summary to see if there are any available side-by-side videos to visualize.
54 * Check the test description to see what the test is doing, as well as what the metrics that changed are measuring.
55 * Compare the before/after performance profiles to see what might have changed. See the section on `Using the Firefox Profiler`_ for more information on this.
56
57 With all of the information found from those, there are two main things that can be done. The first is investigating the profiles (or using some other tools), and finding an issue that needs to be fixed. During the process of investigating and verifying the fix, it will become necessary to verify the fix by running the test. Proceed to the `Running Performance Tests`_ section for information about this, and then the `Performance Comparisons`_ section for help with doing performance comparisons in CI which is needed for verifying if a fix will resolve the alert.
58
59 The second is requesting a confirmation from the performance sheriff that the patch which caused the alert is definitely the correct one. This can happen when the metric is very noisy, and the change is small (in the area of 2-3%, our threshold of detection). The sheriff will conduct more retriggers on the test, and may ask some clarifying questions about the patch.
60
61 If there are any questions about the alert, or additional help is needed with debugging the alert feel free to needinfo the performance sheriff that reported the bug. The performance sheriff most suitable for adding a needinfo to can be identified on the regression bug via the user who added a ``status-firefox [X]: --- → affected`` comment. In the future, this person `will be identified in comment zero <https://bugzilla.mozilla.org/show_bug.cgi?id=1914174>`_.
62
63 Alert Resolution
64 ----------------
65
66 **There are 3 main resolutions for these alert bugs which depend on what you find in your investigations:**
67 #. A ``WONTFIX`` resolution which implies that a change was detected, but it won't be fixed. It's possible to have this resolution on a bug which produces regressions, but the improvements outweigh those regressions. Harness-related changes are often resolved this way as well since we consider them baseline changes.
68 #. An ``INVALID`` resolution which implies that the detection was invalid, and there wasn't a change to performance metrics. These are generally rare, as performance sheriffs tend to invalidate the alerts before a bug is produced, and tend to be related to infrastructure changes or very noisy tests where a culprit can't be determined accurately.
69 #. A ``FIXED`` resolution which implies that a change was detected, and a fix was made to resolve it.
70
71 If it's unclear when an alert may be resolved, it's recommended to file a follow-up bug, and close the alert as ``INCOMPLETE``. If this cannot be done, then it's strongly recommended to reach out to the performance sheriff.
72
73 Alert Monitoring, and the Regression Policy
74 -------------------------------------------
75
76 There is a bugbot rule that monitors the activity of performance alerts. After 1 week of inactivity in the bug, a needinfo will be made for the regressor author to provide an update on any progress. This also notifies performance sheriffs.
77
78 If an alert is flagged in one of these notifications, and performance sheriffs find that there has been absolutely no activity on the alert since it was filed, then **it will become a candidate for a backout** in accordance with our `regression policy <https://www.mozilla.org/en-US/about/governance/policies/regressions/>`_. Otherwise, if there has been some activity, we will simply request an update on any progress that has been made.
79
80 The full process for performance sheriffs who handle these notifications proceeds as follows (`more detailed information can be found here <perf-sheriffing.html#how-to-handle-inactive-alerts>`_):
81 #. A daily email of alert bugs with no activity is obtained.
82 #. We check if the developer (regressor author) has previously responded to the bug.
83 #. **If they have responded in the past**, we reach out to ask them to provide an update on the progress that has been made.
84 #. **If they have not responded in the past**, we reach out to ask them to provide an update and mention that the regressor patch has been **added as a candidate for backout** due to lack of activity.
85 #. If the developer is not responding in either of these cases, after 24 hours, we reach out to their manager with similar messages.
86 #. If the patch is a candidate for backout, and neither the regressor author or their manager responds after 24 hours, we will request a backout of the regressor patch.
87
88 Closing the bug with an `Alert Resolution`_ would be the ideal way to get the Bugbot to stop setting needinfos. Otherwise, it expects some activity there on a weekly basis if it's being actively investigated. If the alert should be investigated some day, it could be closed as ``INCOMPLETE`` with a follow-up bug. There's also a keyword you can add if you absolutely want to keep it open though. The keyword is ``backlog-deferred`` but please only use it sparingly otherwise these kinds of bugs often end up sitting around for years and then get closed when they can no longer be reproduced in our CI (due to machine, test, platform changes).
89
90 Perfherder Alerts View
91 ----------------------
92 When you click on the "Alerts Summary" hyperlink it will take you to an alert summary table on Perfherder which looks like the following screenshot:
93
94 .. image:: ./perfherder_alertsview.png
95 :alt: Sample Perfherder Alert Summary
96 :scale: 75%
97 :align: center
98
99 * The table has 1 performance metric per row that has either improved or regressed a metric.
100 * From left to right, the columns and icons you need to be concerned about as a developer are:
101
102 * ``Graph icon``: Takes you to a graph that shows the history of the metric.
103 * ``Test``: A hyperlink to all the test settings, test owner, and their contact information. As well as the name of the subtest (in our case SpeedIndex, and loadtime).
104 * ``Platform``: Platform of metric which regressed.
105 * ``Debug Tools``: Tools available to help visualize and debug regressions.
106 * ``Information``: Historical data distribution (multimodal data, ok, or n/a if not enough information is available).
107 * ``Tags & Options``: Specify the conditions in which the test was run (e.g. cold, warm, bytecode-cached).
108 * ``Magnitude of Change``: How much the metric improved or regressed (green colour indicates an improvement and red indicates a regression).
109 * ``Confidence``: Confidence value of metric (number is not out of 100) higher number means higher confidence.
110
111 Running Performance Tests
112 -------------------------
113
114 Performance tests can either be run locally, or in CI using try runs. In general, it's recommended to use try runs to verify the performance changes your patch produces (if any). This is because the hardware that we run tests on may not have the same characteristics as local machines so local testing may not always produce the same performance differences. Using try runs also allows you to use our performance comparison tooling such as `Compare View <https://treeherder.mozilla.org/perfherder/comparechooser>`_ and `PerfCompare <https://perf.compare/>`_. See the `Performance Comparisons`_ section for more information on that.
115
116 It's still possible that a local test can reproduce a change found in CI though, but it's not guaranteed. To run a test locally, refer to the harness documentation test lists, such as this one for `Raptor tests <raptor.html#raptor-tests>`_. There are four main ways that you'll find to run these tests:
117
118 * ``./mach raptor`` for :ref:`Raptor`
119 * ``./mach talos-test`` for :ref:`Talos`
120 * ``./mach perftest`` for :ref:`MozPerftest`
121 * ``./mach awsy`` for :ref:`AWSY`
122
123 It's also possible to run all the alerting tests using ``./mach perftest``. To do this, find the alert summary ID/number, then use it in the following command::
124
125 ./mach perftest <ALERT-NUMBER>
126
127 To run the exact same commands as what is run in CI, add the ``--alert-exact`` option. The test(s) to run can also be specified by using the ``--alert-tests`` option.
128
129 Performance Comparisons
130 -----------------------
131
132 Comparing performance metrics across multiple try runs is an important step in the performance testing process. It's used to ensure that changes don't regress our metrics, to determine if a performance improvement is produced from a patch, and among other things, used to verify that a fix resolves a performance alert.
133
134 We currently use PerfCompare for comparing performance numbers. Landing on PerfCompare, two search comparison workflows are available: Compare with a base or Compare over time. Compare with a base allows up to three new revisions to compare against a base revision. Although talos is set at the default, any other testing framework or harness can also be selected before clicking the Compare button. :ref:`You can find more information about using PerfCompare here <PerfCompare>`.
135
136 .. image:: ./perfcomparehomescreen.png
137 :alt: PerfCompare Selection Interface for Revisions/Pushes to Compare
138 :scale: 50%
139 :align: center
140
141 Our old tool for comparing performance numbers, `Compare View <https://treeherder.mozilla.org/perfherder/comparechooser>`_, will be replaced by PerfCompare early next year. The first interface that's seen in that process is the following which is used to select two pushes (based on the revisions) to compare.
142
143 .. image:: ./compare_view_selection.png
144 :alt: Selection Interface for Revisions/Pushes to Compare
145 :scale: 50%
146 :align: center
147
148 At the same time, the framework to compare will need to be selected. By default, the Talos framework is selected, but this can be changed after the Compare button is pressed.
149
150 After the compare button is pressed, a visualization of the comparisons is shown. More information on what the various columns in the comparison mean can be found in `this documentation <standard-workflow.html#compareview>`_.
151
152
153 Using the Firefox Profiler
154 --------------------------
155
156 The Firefox Profiler can be used to help with debugging performance issues in your code. `See here for documentation <https://profiler.firefox.com/docs/#/>`_ on how it can be used to better understand where the regressing code is, and what might be causing the regression. Profiles are provided on most alert summary bugs from before, and after the regression (see first section above).
157
158 If those are not provided in the alert summary, they can always be generated for a test by clicking on the graphs link (the percent-change ratio in an alert summary), selecting a dot in the graph from before or after a change, and clicking the job link. Then, once the job panel opens up in Treeherder, select ``Generate performance profile`` to start a new task that produces a performance profile. See the following graphic illustrating this process:
159
160 .. image:: ./perf_alert_profile_from_graph.png
161 :alt: Getting a Profile from an Alerting Test
162 :scale: 75%
163 :align: center
164
165 Additionally, you can also use the overflow menu and generate a profile:
166
167 .. image:: ./create_profile_triple_dot.png
168 :alt: Creating a profile through the overflow menu
169 :scale: 50%
170 :align: center
171
172 Most Raptor/Browsertime tests produce a performance profile by default at the end of their test run, but Talos, MozPerftest, and AWSY tests do not. As previously mentioned, for regression/improvement alerts, you can find a before and after link of these profiles in Comment 0:
173
174 .. image:: ./perf_alert_comment_zero_before-after.png
175 :alt: View before/after profiles from alerts
176 :scale: 50%
177 :align: center
178
179 You can also find the profiles in the artifacts tab of the Raptor test:
180
181 .. image:: ./raptor_extra_profiler_run.png
182 :alt: Find extra profiler run profiles in treeherder task
183 :scale: 50%
184 :align: center
185
186 To generate the profiles locally, you can pass the flags ``--extra-profiler-run`` or ``--gecko-profile`` which repeat the test for an extra iteration with the profiler enabled, or run the test from the beginning with the profiler enabled for three iterations, respectively. It's also possible to specify more configuration such as the profiled threads, the sampling interval or the profiler features being enabled. The parameters used in a profiling run can be copied directly from the about:profiling page in any Nightly build: click the button at the top of the page, then pick the option "Copy parameters for performance tests".
187
188 Side-by-Side
189 ------------
190
191 Side-by-Side is a job that compares the visual metrics provided by Browsertime for two consecutive pushes. This job is only applicable to pageload-type jobs, and both revisions must run on the same platform with identical configuration.
192
193 Generate a Side-by-Side Job
194 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
195
196 To generate a side-by-side job, follow these steps:
197
198 * Select the desired job.
199 * Navigate to the "Performance" tab and press the "Generate side-by-side" button.
200
201 .. image:: ./generate-side-by-side.png
202 :alt: Generate side-by-side task
203 :scale: 50%
204 :align: center
205
206 Once the process is initiated, a new job will appear in the format `side-by-side-*job-name*`.
207
208 Viewing Results
209 ^^^^^^^^^^^^^^^
210 Once the newly generated job is complete, you can view the result by selecting the job. This type of job provides two types of video (`cold and warm <raptor-metrics.html#warm-pageload>`_). For each set, there are 2 viewing modes (normal and slow-motion).
211
212 Each video includes an annotated description containing the visual metrics provided by Browsertime.
213
214 Adding Performance Tests
215 ------------------------
216
217 This section is under construction.
218
219
220 Additional Help
221 ---------------
222
223 Reach out to the Performance Testing, and Tooling team in the `#perftest channel on Matrix <https://matrix.to/#/#perftest:mozilla.org>`_, or the #perf-help channel on Slack.