Debian Codegraph @

Doc/library/statistics.rst

raw
1 :mod:`!statistics` --- Mathematical statistics functions
2 ========================================================
3
4 .. module:: statistics
5 :synopsis: Mathematical statistics functions
6
7 .. moduleauthor:: Steven D'Aprano <[email protected]>
8 .. sectionauthor:: Steven D'Aprano <[email protected]>
9
10 .. versionadded:: 3.4
11
12 **Source code:** :source:`Lib/statistics.py`
13
14 .. testsetup:: *
15
16 from statistics import *
17 import math
18 __name__ = '<doctest>'
19
20 --------------
21
22 This module provides functions for calculating mathematical statistics of
23 numeric (:class:`~numbers.Real`-valued) data.
24
25 The module is not intended to be a competitor to third-party libraries such
26 as `NumPy <https://numpy.org>`_, `SciPy <https://scipy.org/>`_, or
27 proprietary full-featured statistics packages aimed at professional
28 statisticians such as Minitab, SAS and Matlab. It is aimed at the level of
29 graphing and scientific calculators.
30
31 Unless explicitly noted, these functions support :class:`int`,
32 :class:`float`, :class:`~decimal.Decimal` and :class:`~fractions.Fraction`.
33 Behaviour with other types (whether in the numeric tower or not) is
34 currently unsupported. Collections with a mix of types are also undefined
35 and implementation-dependent. If your input data consists of mixed types,
36 you may be able to use :func:`map` to ensure a consistent result, for
37 example: ``map(float, input_data)``.
38
39 Some datasets use ``NaN`` (not a number) values to represent missing data.
40 Since NaNs have unusual comparison semantics, they cause surprising or
41 undefined behaviors in the statistics functions that sort data or that count
42 occurrences. The functions affected are ``median()``, ``median_low()``,
43 ``median_high()``, ``median_grouped()``, ``mode()``, ``multimode()``, and
44 ``quantiles()``. The ``NaN`` values should be stripped before calling these
45 functions::
46
47 >>> from statistics import median
48 >>> from math import isnan
49 >>> from itertools import filterfalse
50
51 >>> data = [20.7, float('NaN'),19.2, 18.3, float('NaN'), 14.4]
52 >>> sorted(data) # This has surprising behavior
53 [20.7, nan, 14.4, 18.3, 19.2, nan]
54 >>> median(data) # This result is unexpected
55 16.35
56
57 >>> sum(map(isnan, data)) # Number of missing values
58 2
59 >>> clean = list(filterfalse(isnan, data)) # Strip NaN values
60 >>> clean
61 [20.7, 19.2, 18.3, 14.4]
62 >>> sorted(clean) # Sorting now works as expected
63 [14.4, 18.3, 19.2, 20.7]
64 >>> median(clean) # This result is now well defined
65 18.75
66
67
68 Averages and measures of central location
69 -----------------------------------------
70
71 These functions calculate an average or typical value from a population
72 or sample.
73
74 ======================= ===============================================================
75 :func:`mean` Arithmetic mean ("average") of data.
76 :func:`fmean` Fast, floating-point arithmetic mean, with optional weighting.
77 :func:`geometric_mean` Geometric mean of data.
78 :func:`harmonic_mean` Harmonic mean of data.
79 :func:`kde` Estimate the probability density distribution of the data.
80 :func:`kde_random` Random sampling from the PDF generated by kde().
81 :func:`median` Median (middle value) of data.
82 :func:`median_low` Low median of data.
83 :func:`median_high` High median of data.
84 :func:`median_grouped` Median (50th percentile) of grouped data.
85 :func:`mode` Single mode (most common value) of discrete or nominal data.
86 :func:`multimode` List of modes (most common values) of discrete or nominal data.
87 :func:`quantiles` Divide data into intervals with equal probability.
88 ======================= ===============================================================
89
90 Measures of spread
91 ------------------
92
93 These functions calculate a measure of how much the population or sample
94 tends to deviate from the typical or average values.
95
96 ======================= =============================================
97 :func:`pstdev` Population standard deviation of data.
98 :func:`pvariance` Population variance of data.
99 :func:`stdev` Sample standard deviation of data.
100 :func:`variance` Sample variance of data.
101 ======================= =============================================
102
103 Statistics for relations between two inputs
104 -------------------------------------------
105
106 These functions calculate statistics regarding relations between two inputs.
107
108 ========================= =====================================================
109 :func:`covariance` Sample covariance for two variables.
110 :func:`correlation` Pearson and Spearman's correlation coefficients.
111 :func:`linear_regression` Slope and intercept for simple linear regression.
112 ========================= =====================================================
113
114
115 Function details
116 ----------------
117
118 Note: The functions do not require the data given to them to be sorted.
119 However, for reading convenience, most of the examples show sorted sequences.
120
121 .. function:: mean(data)
122
123 Return the sample arithmetic mean of *data* which can be a sequence or iterable.
124
125 The arithmetic mean is the sum of the data divided by the number of data
126 points. It is commonly called "the average", although it is only one of many
127 different mathematical averages. It is a measure of the central location of
128 the data.
129
130 If *data* is empty, :exc:`StatisticsError` will be raised.
131
132 Some examples of use:
133
134 .. doctest::
135
136 >>> mean([1, 2, 3, 4, 4])
137 2.8
138 >>> mean([-1.0, 2.5, 3.25, 5.75])
139 2.625
140
141 >>> from fractions import Fraction as F
142 >>> mean([F(3, 7), F(1, 21), F(5, 3), F(1, 3)])
143 Fraction(13, 21)
144
145 >>> from decimal import Decimal as D
146 >>> mean([D("0.5"), D("0.75"), D("0.625"), D("0.375")])
147 Decimal('0.5625')
148
149 .. note::
150
151 The mean is strongly affected by `outliers
152 <https://en.wikipedia.org/wiki/Outlier>`_ and is not necessarily a
153 typical example of the data points. For a more robust, although less
154 efficient, measure of `central tendency
155 <https://en.wikipedia.org/wiki/Central_tendency>`_, see :func:`median`.
156
157 The sample mean gives an unbiased estimate of the true population mean,
158 so that when taken on average over all the possible samples,
159 ``mean(sample)`` converges on the true mean of the entire population. If
160 *data* represents the entire population rather than a sample, then
161 ``mean(data)`` is equivalent to calculating the true population mean μ.
162
163
164 .. function:: fmean(data, weights=None)
165
166 Convert *data* to floats and compute the arithmetic mean.
167
168 This runs faster than the :func:`mean` function and it always returns a
169 :class:`float`. The *data* may be a sequence or iterable. If the input
170 dataset is empty, raises a :exc:`StatisticsError`.
171
172 .. doctest::
173
174 >>> fmean([3.5, 4.0, 5.25])
175 4.25
176
177 Optional weighting is supported. For example, a professor assigns a
178 grade for a course by weighting quizzes at 20%, homework at 20%, a
179 midterm exam at 30%, and a final exam at 30%:
180
181 .. doctest::
182
183 >>> grades = [85, 92, 83, 91]
184 >>> weights = [0.20, 0.20, 0.30, 0.30]
185 >>> fmean(grades, weights)
186 87.6
187
188 If *weights* is supplied, it must be the same length as the *data* or
189 a :exc:`ValueError` will be raised.
190
191 .. versionadded:: 3.8
192
193 .. versionchanged:: 3.11
194 Added support for *weights*.
195
196
197 .. function:: geometric_mean(data)
198
199 Convert *data* to floats and compute the geometric mean.
200
201 The geometric mean indicates the central tendency or typical value of the
202 *data* using the product of the values (as opposed to the arithmetic mean
203 which uses their sum).
204
205 Raises a :exc:`StatisticsError` if the input dataset is empty,
206 if it contains a zero, or if it contains a negative value.
207 The *data* may be a sequence or iterable.
208
209 No special efforts are made to achieve exact results.
210 (However, this may change in the future.)
211
212 .. doctest::
213
214 >>> round(geometric_mean([54, 24, 36]), 1)
215 36.0
216
217 .. versionadded:: 3.8
218
219
220 .. function:: harmonic_mean(data, weights=None)
221
222 Return the harmonic mean of *data*, a sequence or iterable of
223 real-valued numbers. If *weights* is omitted or ``None``, then
224 equal weighting is assumed.
225
226 The harmonic mean is the reciprocal of the arithmetic :func:`mean` of the
227 reciprocals of the data. For example, the harmonic mean of three values *a*,
228 *b* and *c* will be equivalent to ``3/(1/a + 1/b + 1/c)``. If one of the
229 values is zero, the result will be zero.
230
231 The harmonic mean is a type of average, a measure of the central
232 location of the data. It is often appropriate when averaging
233 ratios or rates, for example speeds.
234
235 Suppose a car travels 10 km at 40 km/hr, then another 10 km at 60 km/hr.
236 What is the average speed?
237
238 .. doctest::
239
240 >>> harmonic_mean([40, 60])
241 48.0
242
243 Suppose a car travels 40 km/hr for 5 km, and when traffic clears,
244 speeds-up to 60 km/hr for the remaining 30 km of the journey. What
245 is the average speed?
246
247 .. doctest::
248
249 >>> harmonic_mean([40, 60], weights=[5, 30])
250 56.0
251
252 :exc:`StatisticsError` is raised if *data* is empty, any element
253 is less than zero, or if the weighted sum isn't positive.
254
255 The current algorithm has an early-out when it encounters a zero
256 in the input. This means that the subsequent inputs are not tested
257 for validity. (This behavior may change in the future.)
258
259 .. versionadded:: 3.6
260
261 .. versionchanged:: 3.10
262 Added support for *weights*.
263
264
265 .. function:: kde(data, h, kernel='normal', *, cumulative=False)
266
267 `Kernel Density Estimation (KDE)
268 <https://www.itm-conferences.org/articles/itmconf/pdf/2018/08/itmconf_sam2018_00037.pdf>`_:
269 Create a continuous probability density function or cumulative
270 distribution function from discrete samples.
271
272 The basic idea is to smooth the data using `a kernel function
273 <https://en.wikipedia.org/wiki/Kernel_(statistics)>`_.
274 to help draw inferences about a population from a sample.
275
276 The degree of smoothing is controlled by the scaling parameter *h*
277 which is called the bandwidth. Smaller values emphasize local
278 features while larger values give smoother results.
279
280 The *kernel* determines the relative weights of the sample data
281 points. Generally, the choice of kernel shape does not matter
282 as much as the more influential bandwidth smoothing parameter.
283
284 Kernels that give some weight to every sample point include
285 *normal* (*gauss*), *logistic*, and *sigmoid*.
286
287 Kernels that only give weight to sample points within the bandwidth
288 include *rectangular* (*uniform*), *triangular*, *parabolic*
289 (*epanechnikov*), *quartic* (*biweight*), *triweight*, and *cosine*.
290
291 If *cumulative* is true, will return a cumulative distribution function.
292
293 A :exc:`StatisticsError` will be raised if the *data* sequence is empty.
294
295 `Wikipedia has an example
296 <https://en.wikipedia.org/wiki/Kernel_density_estimation#Example>`_
297 where we can use :func:`kde` to generate and plot a probability
298 density function estimated from a small sample:
299
300 .. doctest::
301
302 >>> sample = [-2.1, -1.3, -0.4, 1.9, 5.1, 6.2]
303 >>> f_hat = kde(sample, h=1.5)
304 >>> xarr = [i/100 for i in range(-750, 1100)]
305 >>> yarr = [f_hat(x) for x in xarr]
306
307 The points in ``xarr`` and ``yarr`` can be used to make a PDF plot:
308
309 .. image:: kde_example.png
310 :alt: Scatter plot of the estimated probability density function.
311
312 .. versionadded:: 3.13
313
314
315 .. function:: kde_random(data, h, kernel='normal', *, seed=None)
316
317 Return a function that makes a random selection from the estimated
318 probability density function produced by ``kde(data, h, kernel)``.
319
320 Providing a *seed* allows reproducible selections. In the future, the
321 values may change slightly as more accurate kernel inverse CDF estimates
322 are implemented. The seed may be an integer, float, str, or bytes.
323
324 A :exc:`StatisticsError` will be raised if the *data* sequence is empty.
325
326 Continuing the example for :func:`kde`, we can use
327 :func:`kde_random` to generate new random selections from an
328 estimated probability density function:
329
330 >>> data = [-2.1, -1.3, -0.4, 1.9, 5.1, 6.2]
331 >>> rand = kde_random(data, h=1.5, seed=8675309)
332 >>> new_selections = [rand() for i in range(10)]
333 >>> [round(x, 1) for x in new_selections]
334 [0.7, 6.2, 1.2, 6.9, 7.0, 1.8, 2.5, -0.5, -1.8, 5.6]
335
336 .. versionadded:: 3.13
337
338
339 .. function:: median(data)
340
341 Return the median (middle value) of numeric data, using the common "mean of
342 middle two" method. If *data* is empty, :exc:`StatisticsError` is raised.
343 *data* can be a sequence or iterable.
344
345 The median is a robust measure of central location and is less affected by
346 the presence of outliers. When the number of data points is odd, the
347 middle data point is returned:
348
349 .. doctest::
350
351 >>> median([1, 3, 5])
352 3
353
354 When the number of data points is even, the median is interpolated by taking
355 the average of the two middle values:
356
357 .. doctest::
358
359 >>> median([1, 3, 5, 7])
360 4.0
361
362 This is suited for when your data is discrete, and you don't mind that the
363 median may not be an actual data point.
364
365 If the data is ordinal (supports order operations) but not numeric (doesn't
366 support addition), consider using :func:`median_low` or :func:`median_high`
367 instead.
368
369 .. function:: median_low(data)
370
371 Return the low median of numeric data. If *data* is empty,
372 :exc:`StatisticsError` is raised. *data* can be a sequence or iterable.
373
374 The low median is always a member of the data set. When the number of data
375 points is odd, the middle value is returned. When it is even, the smaller of
376 the two middle values is returned.
377
378 .. doctest::
379
380 >>> median_low([1, 3, 5])
381 3
382 >>> median_low([1, 3, 5, 7])
383 3
384
385 Use the low median when your data are discrete and you prefer the median to
386 be an actual data point rather than interpolated.
387
388
389 .. function:: median_high(data)
390
391 Return the high median of data. If *data* is empty, :exc:`StatisticsError`
392 is raised. *data* can be a sequence or iterable.
393
394 The high median is always a member of the data set. When the number of data
395 points is odd, the middle value is returned. When it is even, the larger of
396 the two middle values is returned.
397
398 .. doctest::
399
400 >>> median_high([1, 3, 5])
401 3
402 >>> median_high([1, 3, 5, 7])
403 5
404
405 Use the high median when your data are discrete and you prefer the median to
406 be an actual data point rather than interpolated.
407
408
409 .. function:: median_grouped(data, interval=1.0)
410
411 Estimates the median for numeric data that has been `grouped or binned
412 <https://en.wikipedia.org/wiki/Data_binning>`_ around the midpoints
413 of consecutive, fixed-width intervals.
414
415 The *data* can be any iterable of numeric data with each value being
416 exactly the midpoint of a bin. At least one value must be present.
417
418 The *interval* is the width of each bin.
419
420 For example, demographic information may have been summarized into
421 consecutive ten-year age groups with each group being represented
422 by the 5-year midpoints of the intervals:
423
424 .. doctest::
425
426 >>> from collections import Counter
427 >>> demographics = Counter({
428 ... 25: 172, # 20 to 30 years old
429 ... 35: 484, # 30 to 40 years old
430 ... 45: 387, # 40 to 50 years old
431 ... 55: 22, # 50 to 60 years old
432 ... 65: 6, # 60 to 70 years old
433 ... })
434 ...
435
436 The 50th percentile (median) is the 536th person out of the 1071
437 member cohort. That person is in the 30 to 40 year old age group.
438
439 The regular :func:`median` function would assume that everyone in the
440 tricenarian age group was exactly 35 years old. A more tenable
441 assumption is that the 484 members of that age group are evenly
442 distributed between 30 and 40. For that, we use
443 :func:`median_grouped`:
444
445 .. doctest::
446
447 >>> data = list(demographics.elements())
448 >>> median(data)
449 35
450 >>> round(median_grouped(data, interval=10), 1)
451 37.5
452
453 The caller is responsible for making sure the data points are separated
454 by exact multiples of *interval*. This is essential for getting a
455 correct result. The function does not check this precondition.
456
457 Inputs may be any numeric type that can be coerced to a float during
458 the interpolation step.
459
460
461 .. function:: mode(data)
462
463 Return the single most common data point from discrete or nominal *data*.
464 The mode (when it exists) is the most typical value and serves as a
465 measure of central location.
466
467 If there are multiple modes with the same frequency, returns the first one
468 encountered in the *data*. If the smallest or largest of those is
469 desired instead, use ``min(multimode(data))`` or ``max(multimode(data))``.
470 If the input *data* is empty, :exc:`StatisticsError` is raised.
471
472 ``mode`` assumes discrete data and returns a single value. This is the
473 standard treatment of the mode as commonly taught in schools:
474
475 .. doctest::
476
477 >>> mode([1, 1, 2, 3, 3, 3, 3, 4])
478 3
479
480 The mode is unique in that it is the only statistic in this package that
481 also applies to nominal (non-numeric) data:
482
483 .. doctest::
484
485 >>> mode(["red", "blue", "blue", "red", "green", "red", "red"])
486 'red'
487
488 Only hashable inputs are supported. To handle type :class:`set`,
489 consider casting to :class:`frozenset`. To handle type :class:`list`,
490 consider casting to :class:`tuple`. For mixed or nested inputs, consider
491 using this slower quadratic algorithm that only depends on equality tests:
492 ``max(data, key=data.count)``.
493
494 .. versionchanged:: 3.8
495 Now handles multimodal datasets by returning the first mode encountered.
496 Formerly, it raised :exc:`StatisticsError` when more than one mode was
497 found.
498
499
500 .. function:: multimode(data)
501
502 Return a list of the most frequently occurring values in the order they
503 were first encountered in the *data*. Will return more than one result if
504 there are multiple modes or an empty list if the *data* is empty:
505
506 .. doctest::
507
508 >>> multimode('aabbbbccddddeeffffgg')
509 ['b', 'd', 'f']
510 >>> multimode('')
511 []
512
513 .. versionadded:: 3.8
514
515
516 .. function:: pstdev(data, mu=None)
517
518 Return the population standard deviation (the square root of the population
519 variance). See :func:`pvariance` for arguments and other details.
520
521 .. doctest::
522
523 >>> pstdev([1.5, 2.5, 2.5, 2.75, 3.25, 4.75])
524 0.986893273527251
525
526
527 .. function:: pvariance(data, mu=None)
528
529 Return the population variance of *data*, a non-empty sequence or iterable
530 of real-valued numbers. Variance, or second moment about the mean, is a
531 measure of the variability (spread or dispersion) of data. A large
532 variance indicates that the data is spread out; a small variance indicates
533 it is clustered closely around the mean.
534
535 If the optional second argument *mu* is given, it should be the *population*
536 mean of the *data*. It can also be used to compute the second moment around
537 a point that is not the mean. If it is missing or ``None`` (the default),
538 the arithmetic mean is automatically calculated.
539
540 Use this function to calculate the variance from the entire population. To
541 estimate the variance from a sample, the :func:`variance` function is usually
542 a better choice.
543
544 Raises :exc:`StatisticsError` if *data* is empty.
545
546 Examples:
547
548 .. doctest::
549
550 >>> data = [0.0, 0.25, 0.25, 1.25, 1.5, 1.75, 2.75, 3.25]
551 >>> pvariance(data)
552 1.25
553
554 If you have already calculated the mean of your data, you can pass it as the
555 optional second argument *mu* to avoid recalculation:
556
557 .. doctest::
558
559 >>> mu = mean(data)
560 >>> pvariance(data, mu)
561 1.25
562
563 Decimals and Fractions are supported:
564
565 .. doctest::
566
567 >>> from decimal import Decimal as D
568 >>> pvariance([D("27.5"), D("30.25"), D("30.25"), D("34.5"), D("41.75")])
569 Decimal('24.815')
570
571 >>> from fractions import Fraction as F
572 >>> pvariance([F(1, 4), F(5, 4), F(1, 2)])
573 Fraction(13, 72)
574
575 .. note::
576
577 When called with the entire population, this gives the population variance
578 σ². When called on a sample instead, this is the biased sample variance
579 s², also known as variance with N degrees of freedom.
580
581 If you somehow know the true population mean μ, you may use this
582 function to calculate the variance of a sample, giving the known
583 population mean as the second argument. Provided the data points are a
584 random sample of the population, the result will be an unbiased estimate
585 of the population variance.
586
587
588 .. function:: stdev(data, xbar=None)
589
590 Return the sample standard deviation (the square root of the sample
591 variance). See :func:`variance` for arguments and other details.
592
593 .. doctest::
594
595 >>> stdev([1.5, 2.5, 2.5, 2.75, 3.25, 4.75])
596 1.0810874155219827
597
598
599 .. function:: variance(data, xbar=None)
600
601 Return the sample variance of *data*, an iterable of at least two real-valued
602 numbers. Variance, or second moment about the mean, is a measure of the
603 variability (spread or dispersion) of data. A large variance indicates that
604 the data is spread out; a small variance indicates it is clustered closely
605 around the mean.
606
607 If the optional second argument *xbar* is given, it should be the *sample*
608 mean of *data*. If it is missing or ``None`` (the default), the mean is
609 automatically calculated.
610
611 Use this function when your data is a sample from a population. To calculate
612 the variance from the entire population, see :func:`pvariance`.
613
614 Raises :exc:`StatisticsError` if *data* has fewer than two values.
615
616 Examples:
617
618 .. doctest::
619
620 >>> data = [2.75, 1.75, 1.25, 0.25, 0.5, 1.25, 3.5]
621 >>> variance(data)
622 1.3720238095238095
623
624 If you have already calculated the sample mean of your data, you can pass it
625 as the optional second argument *xbar* to avoid recalculation:
626
627 .. doctest::
628
629 >>> m = mean(data)
630 >>> variance(data, m)
631 1.3720238095238095
632
633 This function does not attempt to verify that you have passed the actual mean
634 as *xbar*. Using arbitrary values for *xbar* can lead to invalid or
635 impossible results.
636
637 Decimal and Fraction values are supported:
638
639 .. doctest::
640
641 >>> from decimal import Decimal as D
642 >>> variance([D("27.5"), D("30.25"), D("30.25"), D("34.5"), D("41.75")])
643 Decimal('31.01875')
644
645 >>> from fractions import Fraction as F
646 >>> variance([F(1, 6), F(1, 2), F(5, 3)])
647 Fraction(67, 108)
648
649 .. note::
650
651 This is the sample variance s² with Bessel's correction, also known as
652 variance with N-1 degrees of freedom. Provided that the data points are
653 representative (e.g. independent and identically distributed), the result
654 should be an unbiased estimate of the true population variance.
655
656 If you somehow know the actual population mean μ you should pass it to the
657 :func:`pvariance` function as the *mu* parameter to get the variance of a
658 sample.
659
660 .. function:: quantiles(data, *, n=4, method='exclusive')
661
662 Divide *data* into *n* continuous intervals with equal probability.
663 Returns a list of ``n - 1`` cut points separating the intervals.
664
665 Set *n* to 4 for quartiles (the default). Set *n* to 10 for deciles. Set
666 *n* to 100 for percentiles which gives the 99 cuts points that separate
667 *data* into 100 equal sized groups. Raises :exc:`StatisticsError` if *n*
668 is not least 1.
669
670 The *data* can be any iterable containing sample data. For meaningful
671 results, the number of data points in *data* should be larger than *n*.
672 Raises :exc:`StatisticsError` if there is not at least one data point.
673
674 The cut points are linearly interpolated from the
675 two nearest data points. For example, if a cut point falls one-third
676 of the distance between two sample values, ``100`` and ``112``, the
677 cut-point will evaluate to ``104``.
678
679 The *method* for computing quantiles can be varied depending on
680 whether the *data* includes or excludes the lowest and
681 highest possible values from the population.
682
683 The default *method* is "exclusive" and is used for data sampled from
684 a population that can have more extreme values than found in the
685 samples. The portion of the population falling below the *i-th* of
686 *m* sorted data points is computed as ``i / (m + 1)``. Given nine
687 sample values, the method sorts them and assigns the following
688 percentiles: 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%.
689
690 Setting the *method* to "inclusive" is used for describing population
691 data or for samples that are known to include the most extreme values
692 from the population. The minimum value in *data* is treated as the 0th
693 percentile and the maximum value is treated as the 100th percentile.
694 The portion of the population falling below the *i-th* of *m* sorted
695 data points is computed as ``(i - 1) / (m - 1)``. Given 11 sample
696 values, the method sorts them and assigns the following percentiles:
697 0%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, 100%.
698
699 .. doctest::
700
701 # Decile cut points for empirically sampled data
702 >>> data = [105, 129, 87, 86, 111, 111, 89, 81, 108, 92, 110,
703 ... 100, 75, 105, 103, 109, 76, 119, 99, 91, 103, 129,
704 ... 106, 101, 84, 111, 74, 87, 86, 103, 103, 106, 86,
705 ... 111, 75, 87, 102, 121, 111, 88, 89, 101, 106, 95,
706 ... 103, 107, 101, 81, 109, 104]
707 >>> [round(q, 1) for q in quantiles(data, n=10)]
708 [81.0, 86.2, 89.0, 99.4, 102.5, 103.6, 106.0, 109.8, 111.0]
709
710 .. versionadded:: 3.8
711
712 .. versionchanged:: 3.13
713 No longer raises an exception for an input with only a single data point.
714 This allows quantile estimates to be built up one sample point
715 at a time becoming gradually more refined with each new data point.
716
717 .. function:: covariance(x, y, /)
718
719 Return the sample covariance of two inputs *x* and *y*. Covariance
720 is a measure of the joint variability of two inputs.
721
722 Both inputs must be of the same length (no less than two), otherwise
723 :exc:`StatisticsError` is raised.
724
725 Examples:
726
727 .. doctest::
728
729 >>> x = [1, 2, 3, 4, 5, 6, 7, 8, 9]
730 >>> y = [1, 2, 3, 1, 2, 3, 1, 2, 3]
731 >>> covariance(x, y)
732 0.75
733 >>> z = [9, 8, 7, 6, 5, 4, 3, 2, 1]
734 >>> covariance(x, z)
735 -7.5
736 >>> covariance(z, x)
737 -7.5
738
739 .. versionadded:: 3.10
740
741 .. function:: correlation(x, y, /, *, method='linear')
742
743 Return the `Pearson's correlation coefficient
744 <https://en.wikipedia.org/wiki/Pearson_correlation_coefficient>`_
745 for two inputs. Pearson's correlation coefficient *r* takes values
746 between -1 and +1. It measures the strength and direction of a linear
747 relationship.
748
749 If *method* is "ranked", computes `Spearman's rank correlation coefficient
750 <https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient>`_
751 for two inputs. The data is replaced by ranks. Ties are averaged so that
752 equal values receive the same rank. The resulting coefficient measures the
753 strength of a monotonic relationship.
754
755 Spearman's correlation coefficient is appropriate for ordinal data or for
756 continuous data that doesn't meet the linear proportion requirement for
757 Pearson's correlation coefficient.
758
759 Both inputs must be of the same length (no less than two), and need
760 not to be constant, otherwise :exc:`StatisticsError` is raised.
761
762 Example with `Kepler's laws of planetary motion
763 <https://en.wikipedia.org/wiki/Kepler's_laws_of_planetary_motion>`_:
764
765 .. doctest::
766
767 >>> # Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, and Neptune
768 >>> orbital_period = [88, 225, 365, 687, 4331, 10_756, 30_687, 60_190] # days
769 >>> dist_from_sun = [58, 108, 150, 228, 778, 1_400, 2_900, 4_500] # million km
770
771 >>> # Show that a perfect monotonic relationship exists
772 >>> correlation(orbital_period, dist_from_sun, method='ranked')
773 1.0
774
775 >>> # Observe that a linear relationship is imperfect
776 >>> round(correlation(orbital_period, dist_from_sun), 4)
777 0.9882
778
779 >>> # Demonstrate Kepler's third law: There is a linear correlation
780 >>> # between the square of the orbital period and the cube of the
781 >>> # distance from the sun.
782 >>> period_squared = [p * p for p in orbital_period]
783 >>> dist_cubed = [d * d * d for d in dist_from_sun]
784 >>> round(correlation(period_squared, dist_cubed), 4)
785 1.0
786
787 .. versionadded:: 3.10
788
789 .. versionchanged:: 3.12
790 Added support for Spearman's rank correlation coefficient.
791
792 .. function:: linear_regression(x, y, /, *, proportional=False)
793
794 Return the slope and intercept of `simple linear regression
795 <https://en.wikipedia.org/wiki/Simple_linear_regression>`_
796 parameters estimated using ordinary least squares. Simple linear
797 regression describes the relationship between an independent variable *x* and
798 a dependent variable *y* in terms of this linear function:
799
800 *y = slope \* x + intercept + noise*
801
802 where ``slope`` and ``intercept`` are the regression parameters that are
803 estimated, and ``noise`` represents the
804 variability of the data that was not explained by the linear regression
805 (it is equal to the difference between predicted and actual values
806 of the dependent variable).
807
808 Both inputs must be of the same length (no less than two), and
809 the independent variable *x* cannot be constant;
810 otherwise a :exc:`StatisticsError` is raised.
811
812 For example, we can use the `release dates of the Monty
813 Python films <https://en.wikipedia.org/wiki/Monty_Python#Films>`_
814 to predict the cumulative number of Monty Python films
815 that would have been produced by 2019
816 assuming that they had kept the pace.
817
818 .. doctest::
819
820 >>> year = [1971, 1975, 1979, 1982, 1983]
821 >>> films_total = [1, 2, 3, 4, 5]
822 >>> slope, intercept = linear_regression(year, films_total)
823 >>> round(slope * 2019 + intercept)
824 16
825
826 If *proportional* is true, the independent variable *x* and the
827 dependent variable *y* are assumed to be directly proportional.
828 The data is fit to a line passing through the origin.
829 Since the *intercept* will always be 0.0, the underlying linear
830 function simplifies to:
831
832 *y = slope \* x + noise*
833
834 Continuing the example from :func:`correlation`, we look to see
835 how well a model based on major planets can predict the orbital
836 distances for dwarf planets:
837
838 .. doctest::
839
840 >>> model = linear_regression(period_squared, dist_cubed, proportional=True)
841 >>> slope = model.slope
842
843 >>> # Dwarf planets: Pluto, Eris, Makemake, Haumea, Ceres
844 >>> orbital_periods = [90_560, 204_199, 111_845, 103_410, 1_680] # days
845 >>> predicted_dist = [math.cbrt(slope * (p * p)) for p in orbital_periods]
846 >>> list(map(round, predicted_dist))
847 [5912, 10166, 6806, 6459, 414]
848
849 >>> [5_906, 10_152, 6_796, 6_450, 414] # actual distance in million km
850 [5906, 10152, 6796, 6450, 414]
851
852 .. versionadded:: 3.10
853
854 .. versionchanged:: 3.11
855 Added support for *proportional*.
856
857 Exceptions
858 ----------
859
860 A single exception is defined:
861
862 .. exception:: StatisticsError
863
864 Subclass of :exc:`ValueError` for statistics-related exceptions.
865
866
867 :class:`NormalDist` objects
868 ---------------------------
869
870 :class:`NormalDist` is a tool for creating and manipulating normal
871 distributions of a `random variable
872 <http://www.stat.yale.edu/Courses/1997-98/101/ranvar.htm>`_. It is a
873 class that treats the mean and standard deviation of data
874 measurements as a single entity.
875
876 Normal distributions arise from the `Central Limit Theorem
877 <https://en.wikipedia.org/wiki/Central_limit_theorem>`_ and have a wide range
878 of applications in statistics.
879
880 .. class:: NormalDist(mu=0.0, sigma=1.0)
881
882 Returns a new *NormalDist* object where *mu* represents the `arithmetic
883 mean <https://en.wikipedia.org/wiki/Arithmetic_mean>`_ and *sigma*
884 represents the `standard deviation
885 <https://en.wikipedia.org/wiki/Standard_deviation>`_.
886
887 If *sigma* is negative, raises :exc:`StatisticsError`.
888
889 .. attribute:: mean
890
891 A read-only property for the `arithmetic mean
892 <https://en.wikipedia.org/wiki/Arithmetic_mean>`_ of a normal
893 distribution.
894
895 .. attribute:: median
896
897 A read-only property for the `median
898 <https://en.wikipedia.org/wiki/Median>`_ of a normal
899 distribution.
900
901 .. attribute:: mode
902
903 A read-only property for the `mode
904 <https://en.wikipedia.org/wiki/Mode_(statistics)>`_ of a normal
905 distribution.
906
907 .. attribute:: stdev
908
909 A read-only property for the `standard deviation
910 <https://en.wikipedia.org/wiki/Standard_deviation>`_ of a normal
911 distribution.
912
913 .. attribute:: variance
914
915 A read-only property for the `variance
916 <https://en.wikipedia.org/wiki/Variance>`_ of a normal
917 distribution. Equal to the square of the standard deviation.
918
919 .. classmethod:: NormalDist.from_samples(data)
920
921 Makes a normal distribution instance with *mu* and *sigma* parameters
922 estimated from the *data* using :func:`fmean` and :func:`stdev`.
923
924 The *data* can be any :term:`iterable` and should consist of values
925 that can be converted to type :class:`float`. If *data* does not
926 contain at least two elements, raises :exc:`StatisticsError` because it
927 takes at least one point to estimate a central value and at least two
928 points to estimate dispersion.
929
930 .. method:: NormalDist.samples(n, *, seed=None)
931
932 Generates *n* random samples for a given mean and standard deviation.
933 Returns a :class:`list` of :class:`float` values.
934
935 If *seed* is given, creates a new instance of the underlying random
936 number generator. This is useful for creating reproducible results,
937 even in a multi-threading context.
938
939 .. versionchanged:: 3.13
940
941 Switched to a faster algorithm. To reproduce samples from previous
942 versions, use :func:`random.seed` and :func:`random.gauss`.
943
944 .. method:: NormalDist.pdf(x)
945
946 Using a `probability density function (pdf)
947 <https://en.wikipedia.org/wiki/Probability_density_function>`_, compute
948 the relative likelihood that a random variable *X* will be near the
949 given value *x*. Mathematically, it is the limit of the ratio ``P(x <=
950 X < x+dx) / dx`` as *dx* approaches zero.
951
952 The relative likelihood is computed as the probability of a sample
953 occurring in a narrow range divided by the width of the range (hence
954 the word "density"). Since the likelihood is relative to other points,
955 its value can be greater than ``1.0``.
956
957 .. method:: NormalDist.cdf(x)
958
959 Using a `cumulative distribution function (cdf)
960 <https://en.wikipedia.org/wiki/Cumulative_distribution_function>`_,
961 compute the probability that a random variable *X* will be less than or
962 equal to *x*. Mathematically, it is written ``P(X <= x)``.
963
964 .. method:: NormalDist.inv_cdf(p)
965
966 Compute the inverse cumulative distribution function, also known as the
967 `quantile function <https://en.wikipedia.org/wiki/Quantile_function>`_
968 or the `percent-point
969 <https://web.archive.org/web/20190203145224/https://www.statisticshowto.datasciencecentral.com/inverse-distribution-function/>`_
970 function. Mathematically, it is written ``x : P(X <= x) = p``.
971
972 Finds the value *x* of the random variable *X* such that the
973 probability of the variable being less than or equal to that value
974 equals the given probability *p*.
975
976 .. method:: NormalDist.overlap(other)
977
978 Measures the agreement between two normal probability distributions.
979 Returns a value between 0.0 and 1.0 giving `the overlapping area for
980 the two probability density functions
981 <https://www.rasch.org/rmt/rmt101r.htm>`_.
982
983 .. method:: NormalDist.quantiles(n=4)
984
985 Divide the normal distribution into *n* continuous intervals with
986 equal probability. Returns a list of (n - 1) cut points separating
987 the intervals.
988
989 Set *n* to 4 for quartiles (the default). Set *n* to 10 for deciles.
990 Set *n* to 100 for percentiles which gives the 99 cuts points that
991 separate the normal distribution into 100 equal sized groups.
992
993 .. method:: NormalDist.zscore(x)
994
995 Compute the
996 `Standard Score <https://www.statisticshowto.com/probability-and-statistics/z-score/>`_
997 describing *x* in terms of the number of standard deviations
998 above or below the mean of the normal distribution:
999 ``(x - mean) / stdev``.
1000
1001 .. versionadded:: 3.9
1002
1003 Instances of :class:`NormalDist` support addition, subtraction,
1004 multiplication and division by a constant. These operations
1005 are used for translation and scaling. For example:
1006
1007 .. doctest::
1008
1009 >>> temperature_february = NormalDist(5, 2.5) # Celsius
1010 >>> temperature_february * (9/5) + 32 # Fahrenheit
1011 NormalDist(mu=41.0, sigma=4.5)
1012
1013 Dividing a constant by an instance of :class:`NormalDist` is not supported
1014 because the result wouldn't be normally distributed.
1015
1016 Since normal distributions arise from additive effects of independent
1017 variables, it is possible to `add and subtract two independent normally
1018 distributed random variables
1019 <https://en.wikipedia.org/wiki/Sum_of_normally_distributed_random_variables>`_
1020 represented as instances of :class:`NormalDist`. For example:
1021
1022 .. doctest::
1023
1024 >>> birth_weights = NormalDist.from_samples([2.5, 3.1, 2.1, 2.4, 2.7, 3.5])
1025 >>> drug_effects = NormalDist(0.4, 0.15)
1026 >>> combined = birth_weights + drug_effects
1027 >>> round(combined.mean, 1)
1028 3.1
1029 >>> round(combined.stdev, 1)
1030 0.5
1031
1032 .. versionadded:: 3.8
1033
1034
1035 Examples and Recipes
1036 --------------------
1037
1038
1039 Classic probability problems
1040 ****************************
1041
1042 :class:`NormalDist` readily solves classic probability problems.
1043
1044 For example, given `historical data for SAT exams
1045 <https://nces.ed.gov/programs/digest/d17/tables/dt17_226.40.asp>`_ showing
1046 that scores are normally distributed with a mean of 1060 and a standard
1047 deviation of 195, determine the percentage of students with test scores
1048 between 1100 and 1200, after rounding to the nearest whole number:
1049
1050 .. doctest::
1051
1052 >>> sat = NormalDist(1060, 195)
1053 >>> fraction = sat.cdf(1200 + 0.5) - sat.cdf(1100 - 0.5)
1054 >>> round(fraction * 100.0, 1)
1055 18.4
1056
1057 Find the `quartiles <https://en.wikipedia.org/wiki/Quartile>`_ and `deciles
1058 <https://en.wikipedia.org/wiki/Decile>`_ for the SAT scores:
1059
1060 .. doctest::
1061
1062 >>> list(map(round, sat.quantiles()))
1063 [928, 1060, 1192]
1064 >>> list(map(round, sat.quantiles(n=10)))
1065 [810, 896, 958, 1011, 1060, 1109, 1162, 1224, 1310]
1066
1067
1068 Monte Carlo inputs for simulations
1069 **********************************
1070
1071 To estimate the distribution for a model that isn't easy to solve
1072 analytically, :class:`NormalDist` can generate input samples for a `Monte
1073 Carlo simulation <https://en.wikipedia.org/wiki/Monte_Carlo_method>`_:
1074
1075 .. doctest::
1076
1077 >>> def model(x, y, z):
1078 ... return (3*x + 7*x*y - 5*y) / (11 * z)
1079 ...
1080 >>> n = 100_000
1081 >>> X = NormalDist(10, 2.5).samples(n, seed=3652260728)
1082 >>> Y = NormalDist(15, 1.75).samples(n, seed=4582495471)
1083 >>> Z = NormalDist(50, 1.25).samples(n, seed=6582483453)
1084 >>> quantiles(map(model, X, Y, Z)) # doctest: +SKIP
1085 [1.4591308524824727, 1.8035946855390597, 2.175091447274739]
1086
1087 Approximating binomial distributions
1088 ************************************
1089
1090 Normal distributions can be used to approximate `Binomial
1091 distributions <https://mathworld.wolfram.com/BinomialDistribution.html>`_
1092 when the sample size is large and when the probability of a successful
1093 trial is near 50%.
1094
1095 For example, an open source conference has 750 attendees and two rooms with a
1096 500 person capacity. There is a talk about Python and another about Ruby.
1097 In previous conferences, 65% of the attendees preferred to listen to Python
1098 talks. Assuming the population preferences haven't changed, what is the
1099 probability that the Python room will stay within its capacity limits?
1100
1101 .. doctest::
1102
1103 >>> n = 750 # Sample size
1104 >>> p = 0.65 # Preference for Python
1105 >>> q = 1.0 - p # Preference for Ruby
1106 >>> k = 500 # Room capacity
1107
1108 >>> # Approximation using the cumulative normal distribution
1109 >>> from math import sqrt
1110 >>> round(NormalDist(mu=n*p, sigma=sqrt(n*p*q)).cdf(k + 0.5), 4)
1111 0.8402
1112
1113 >>> # Exact solution using the cumulative binomial distribution
1114 >>> from math import comb, fsum
1115 >>> round(fsum(comb(n, r) * p**r * q**(n-r) for r in range(k+1)), 4)
1116 0.8402
1117
1118 >>> # Approximation using a simulation
1119 >>> from random import seed, binomialvariate
1120 >>> seed(8675309)
1121 >>> mean(binomialvariate(n, p) <= k for i in range(10_000))
1122 0.8406
1123
1124
1125 Naive bayesian classifier
1126 *************************
1127
1128 Normal distributions commonly arise in machine learning problems.
1129
1130 Wikipedia has a `nice example of a Naive Bayesian Classifier
1131 <https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Person_classification>`_.
1132 The challenge is to predict a person's gender from measurements of normally
1133 distributed features including height, weight, and foot size.
1134
1135 We're given a training dataset with measurements for eight people. The
1136 measurements are assumed to be normally distributed, so we summarize the data
1137 with :class:`NormalDist`:
1138
1139 .. doctest::
1140
1141 >>> height_male = NormalDist.from_samples([6, 5.92, 5.58, 5.92])
1142 >>> height_female = NormalDist.from_samples([5, 5.5, 5.42, 5.75])
1143 >>> weight_male = NormalDist.from_samples([180, 190, 170, 165])
1144 >>> weight_female = NormalDist.from_samples([100, 150, 130, 150])
1145 >>> foot_size_male = NormalDist.from_samples([12, 11, 12, 10])
1146 >>> foot_size_female = NormalDist.from_samples([6, 8, 7, 9])
1147
1148 Next, we encounter a new person whose feature measurements are known but whose
1149 gender is unknown:
1150
1151 .. doctest::
1152
1153 >>> ht = 6.0 # height
1154 >>> wt = 130 # weight
1155 >>> fs = 8 # foot size
1156
1157 Starting with a 50% `prior probability
1158 <https://en.wikipedia.org/wiki/Prior_probability>`_ of being male or female,
1159 we compute the posterior as the prior times the product of likelihoods for the
1160 feature measurements given the gender:
1161
1162 .. doctest::
1163
1164 >>> prior_male = 0.5
1165 >>> prior_female = 0.5
1166 >>> posterior_male = (prior_male * height_male.pdf(ht) *
1167 ... weight_male.pdf(wt) * foot_size_male.pdf(fs))
1168
1169 >>> posterior_female = (prior_female * height_female.pdf(ht) *
1170 ... weight_female.pdf(wt) * foot_size_female.pdf(fs))
1171
1172 The final prediction goes to the largest posterior. This is known as the
1173 `maximum a posteriori
1174 <https://en.wikipedia.org/wiki/Maximum_a_posteriori_estimation>`_ or MAP:
1175
1176 .. doctest::
1177
1178 >>> 'male' if posterior_male > posterior_female else 'female'
1179 'female'
1180
1181
1182 ..
1183 # This modelines must appear within the last ten lines of the file.
1184 kate: indent-width 3; remove-trailing-space on; replace-tabs on; encoding utf-8;