| 1 |
# -*- org -*- |
| 2 |
#+CATEGORY: statistics |
| 3 |
|
| 4 |
* From: James Theiler <[email protected]> |
| 5 |
To: John Lamb <[email protected]> |
| 6 |
Cc: [email protected] |
| 7 |
Subject: Re: Collecting statistics for time dependent data? |
| 8 |
Date: Thu, 9 Dec 2004 14:18:36 -0700 (MST) |
| 9 |
|
| 10 |
On Thu, 9 Dec 2004, John Lamb wrote: |
| 11 |
|
| 12 |
] Raimondo Giammanco wrote: |
| 13 |
] > Hello, |
| 14 |
] > |
| 15 |
] > I was wondering if there is a way to compute "running" statistics with |
| 16 |
] > gsl. |
| 17 |
] > |
| 18 |
] Yes you can do it, but there's nothing in GSL that does it and its eay |
| 19 |
] enough that you don't need GSL. Something like (untested) |
| 20 |
] |
| 21 |
] double update_mean( double* mean, int* n, double x ){ |
| 22 |
] if( *n == 1 ) |
| 23 |
] *mean = x; |
| 24 |
] else |
| 25 |
] *mean = (1 - (double)1 / *n ) * *mean + x / n; |
| 26 |
] } |
| 27 |
] |
| 28 |
] will work and you can derive a similar method for updating the variance |
| 29 |
] using the usual textbook formula. |
| 30 |
] |
| 31 |
] var[x] = (1/n) sum x^2_i - mean(x)^2 |
| 32 |
] |
| 33 |
] I don't know if there is a method that avoids the rounding errors. I |
| 34 |
] don't know why so many textbooks repeat this formula without the |
| 35 |
] slightest warning that it can go so badly wrong. |
| 36 |
] |
| 37 |
] |
| 38 |
|
| 39 |
Stably updating mean and variance is remarkably nontrivial. There was |
| 40 |
a series of papers in Comm ACM that discussed the issue; the final one |
| 41 |
(that I know of) refers back to the earlier ones, and it can be found |
| 42 |
in D.H.D. West, Updating mean and variance estimates: an improved |
| 43 |
method, Comm ACM 22:9, 532 (1979) [* I see Luke Stras just sent this |
| 44 |
reference! *]. I'll just copy out the pseudocode since the paper is |
| 45 |
old enough that it might not be easy to find. This, by the way, is |
| 46 |
generalized for weighted data, so it assumes that you get a weight and |
| 47 |
a data value (W_i and X_i) that you use to update the estimates XBAR |
| 48 |
and S2: |
| 49 |
|
| 50 |
SUMW = W_1 |
| 51 |
M = X_1 |
| 52 |
T = 0 |
| 53 |
For i=2,3,...,n |
| 54 |
{ |
| 55 |
Q = X_i - M |
| 56 |
TEMP = SUM + W_i // typo: He meant SUMW |
| 57 |
R = Q*W_i/TEMP |
| 58 |
M = M + R |
| 59 |
T = T + R*SUMW*Q |
| 60 |
SUMW = TEMP |
| 61 |
} |
| 62 |
XBAR = M |
| 63 |
S2 = T*n/((n-1)*SUMW) |
| 64 |
|
| 65 |
|
| 66 |
|
| 67 |
jt |
| 68 |
|
| 69 |
-- |
| 70 |
James Theiler Space and Remote Sensing Sciences |
| 71 |
MS-B244, ISR-2, LANL Los Alamos National Laboratory |
| 72 |
Los Alamos, NM 87545 http://nis-www.lanl.gov/~jt |
| 73 |
|
| 74 |
|
| 75 |
* Look at STARPAC ftp://ftp.ucar.edu/starpac/ and Statlib |
| 76 |
http://lib.stat.cmu.edu/ for more ideas |
| 77 |
|
| 78 |
* Try using the Kahan summation formula to improve accuracy for the |
| 79 |
NIST tests (see Brian for details, below is a sketch of the algorithm). |
| 80 |
|
| 81 |
sum = x(1) |
| 82 |
c = 0 |
| 83 |
|
| 84 |
DO i = 2, 1000000, 1 |
| 85 |
y = x(i) - c |
| 86 |
t = sum + y |
| 87 |
c = (t - sum) - y |
| 88 |
sum = t |
| 89 |
ENDDO |
| 90 |
|
| 91 |
* Prevent incorrect use of unsorted data for quartile calculations |
| 92 |
using a typedef for sorted data (?) |
| 93 |
|
| 94 |
* Rejection of outliers |
| 95 |
|
| 96 |
* Time series. Auto correlation, cross-correlation, smoothing (moving |
| 97 |
average), detrending, various econometric things. Integrated |
| 98 |
quantities (area under the curve). Interpolation of noisy data/fitting |
| 99 |
-- maybe add that to the existing interpolation stuff.What about |
| 100 |
missing data and gaps? |
| 101 |
|
| 102 |
There is a new GNU package called gretl which does econometrics |
| 103 |
|
| 104 |
* Statistical tests (equal means, equal variance, etc). |
| 105 |
|