Debian Codegraph @

statistics/TODO

raw
1 # -*- org -*-
2 #+CATEGORY: statistics
3
4 * From: James Theiler <[email protected]>
5 To: John Lamb <[email protected]>
6 Cc: [email protected]
7 Subject: Re: Collecting statistics for time dependent data?
8 Date: Thu, 9 Dec 2004 14:18:36 -0700 (MST)
9
10 On Thu, 9 Dec 2004, John Lamb wrote:
11
12 ] Raimondo Giammanco wrote:
13 ] > Hello,
14 ] >
15 ] > I was wondering if there is a way to compute "running" statistics with
16 ] > gsl.
17 ] >
18 ] Yes you can do it, but there's nothing in GSL that does it and its eay
19 ] enough that you don't need GSL. Something like (untested)
20 ]
21 ] double update_mean( double* mean, int* n, double x ){
22 ] if( *n == 1 )
23 ] *mean = x;
24 ] else
25 ] *mean = (1 - (double)1 / *n ) * *mean + x / n;
26 ] }
27 ]
28 ] will work and you can derive a similar method for updating the variance
29 ] using the usual textbook formula.
30 ]
31 ] var[x] = (1/n) sum x^2_i - mean(x)^2
32 ]
33 ] I don't know if there is a method that avoids the rounding errors. I
34 ] don't know why so many textbooks repeat this formula without the
35 ] slightest warning that it can go so badly wrong.
36 ]
37 ]
38
39 Stably updating mean and variance is remarkably nontrivial. There was
40 a series of papers in Comm ACM that discussed the issue; the final one
41 (that I know of) refers back to the earlier ones, and it can be found
42 in D.H.D. West, Updating mean and variance estimates: an improved
43 method, Comm ACM 22:9, 532 (1979) [* I see Luke Stras just sent this
44 reference! *]. I'll just copy out the pseudocode since the paper is
45 old enough that it might not be easy to find. This, by the way, is
46 generalized for weighted data, so it assumes that you get a weight and
47 a data value (W_i and X_i) that you use to update the estimates XBAR
48 and S2:
49
50 SUMW = W_1
51 M = X_1
52 T = 0
53 For i=2,3,...,n
54 {
55 Q = X_i - M
56 TEMP = SUM + W_i // typo: He meant SUMW
57 R = Q*W_i/TEMP
58 M = M + R
59 T = T + R*SUMW*Q
60 SUMW = TEMP
61 }
62 XBAR = M
63 S2 = T*n/((n-1)*SUMW)
64
65
66
67 jt
68
69 --
70 James Theiler Space and Remote Sensing Sciences
71 MS-B244, ISR-2, LANL Los Alamos National Laboratory
72 Los Alamos, NM 87545 http://nis-www.lanl.gov/~jt
73
74
75 * Look at STARPAC ftp://ftp.ucar.edu/starpac/ and Statlib
76 http://lib.stat.cmu.edu/ for more ideas
77
78 * Try using the Kahan summation formula to improve accuracy for the
79 NIST tests (see Brian for details, below is a sketch of the algorithm).
80
81 sum = x(1)
82 c = 0
83
84 DO i = 2, 1000000, 1
85 y = x(i) - c
86 t = sum + y
87 c = (t - sum) - y
88 sum = t
89 ENDDO
90
91 * Prevent incorrect use of unsorted data for quartile calculations
92 using a typedef for sorted data (?)
93
94 * Rejection of outliers
95
96 * Time series. Auto correlation, cross-correlation, smoothing (moving
97 average), detrending, various econometric things. Integrated
98 quantities (area under the curve). Interpolation of noisy data/fitting
99 -- maybe add that to the existing interpolation stuff.What about
100 missing data and gaps?
101
102 There is a new GNU package called gretl which does econometrics
103
104 * Statistical tests (equal means, equal variance, etc).
105