Miller's errorbars (was: Re: Spacecraft earth-flyby data reveals

Surfer wrote:
On Thu, 20 Aug 2009 20:33:16 +0100 (BST), Tom Roberts
<tjroberts137@xxxxxxxxxxxxx> wrote:
As I discussed in http://arxiv.org/abs/physics/0608238,
....

I found your analysis quite convincing from a technical point of view,
but while examining Miller's data reduction algorithm, I saw something
that the analysis failed to take into account.

Right. I did not "take into account" your misconceptions and mistakes.

Simplifying somewhat, Miller's algorithm is equivalent to performing
two distinct operations:
1) Average the raw data
2) Filter the averaged data so as to remove noise.

He applied no "filter", in the sense we use that word today in this
context. What he did is subtract what he thought was a linear
background; he did this after averaging the full turns, but before
averaging the two 1/2-turn averages. The latter averaging is not nearly
as important to the errorbars as the former, so in keeping with your
"simplifying" I'll ignore it for this discussion (keep in mind that my
paper dealt with his complete algorithm).

Since averaging allows one to calculate the standard error in the
mean, that would seem the natural place to start error analysis, which
is what your paper does.

However, examination of the algorithm shows that it is possible to
perform the above operations in reverse order ie:
1) Filter the raw data so as to remove noise
2) Average the filtered data.
(and still get the exact same output from the algorithm !)

Yes, when by "filter" you mean subtracting the linear drift between
markers 1 and 17. These are commutative arithmetic operations for each
marker [#].

[#] Using a true filter would not be commutative, but
since he didn't use one it's OK.

When the algorithm is reordered in this way, the standard errors in
the means can now be calculated from the filtered data. But since the
filtered data contains much less noise than the raw data, the error
bars are now very much smaller.

[[Mod. note -- That's only true if the errors in different data points
were statistically independent. (There are a other requirements, too,
but independence is the most-often-violated one.) In contrast, if
(to take an extreme example) the errors were perfectly correlated,
i.e., every point were in error by the same amount, then the filtering
would have no effect on that error.
-- jt]]

Yes to the Moderator, no to Surfer.

The problem is that by subtracting the linear background turn-by-turn
you introduce an enormous CORRELATION between the background being
subtracted and the data. You did not take this into account. There is no
such correlation when subtracting the average background from the
average data (or rather, the correlation that remains is the one desired
to be removed).

What this amounts to is that your analysis failed to account for a
quite powerful noise reduction capability inherent in Miller's
algorithm.

No. Any "noise reduction capability" of Miller's algorithm cannot
possibly be affected by interchanging commutative arithmetic operations.
There is a reason that your algorithm gives the same "signal" as his,
and that same reason applies to the errorbars as well, but one does need
to compute them correctly.

What this amounts to is you not computing the errorbars correctly -- you
ignored an enormous CORRELATION.

Here's an example, contrived to be simple, clear, and to show my point,
not to mimic Miller's data: Consider a system with 5 markers per turn
and 3 turns; as in Miller's data sheets, each column is a marker
(=orientation), each row is a turn (imagine an adjustment between turns
of -10 -- Miller actually did do that sometimes, so it's not inappropriate):

10.0 10.1 10.0 9.9 10.0
0.0 0.1 0.0 -0.1 0.0
-10.0 -9.9 -10.0 -10.1 -10.0

Miller's algorithm [M] gives an obvious "signal" of amplitude 0.1.
Surfer's algorithm [S] gives an identical "signal". The errorbars from
[M] are enormously larger than that. If one neglects the correlation
introduced, the "errorbars" from [S] are zero. Does anyone seriously
think that this "signal" is truly measured PERFECTLY here??? -- there is
a clear and obvious variation a hundred times larger!

[M] average each column, then subtract the line from
average(column 1) to average(column 5).

[S] for each turn subtract the line from column 1 to
column 5, then average the result columnwise.

Two rather obvious points:
A) Since interchanging commutative arithmetic operations does not
affect the information contained in the data, errorbars
computed by ignoring the correlations of [S] are just plain
wrong.
B) The VALIDITY of the "signal" found this way is inherently
dependent on the uncertainty in the background subtraction
being SMALLER than the "signal" -- if the TRUE background
varied by 0.1 or more during a turn, the "signal" can clearly
be COMPLETELY BOGUS.
For Miller's results this is especially true, as his algorithm
FORCES the noise to look just like a "signal" (see section
III of my paper). So you cannot distinguish bogus from true
just by looking at it.

Now look at figure 3 of my paper to see how NON-linear Miller's
background actually is. It varies from linearity by a factor of >10 more
than the "signal" that Miller found. In the above example, a variation
in the TRUE background of ~1 would completely negate any claims of a
"signal".

Yes, you can see a background of 10, 0, and -10 in
the three turns; but you don't KNOW that this is true.
This example gives you no means to evaluate how large
the true variations in the background are; fortunately
Miller's data do -- see my Fig. 3 -- they invalidate
his "signal".

Note that there is indeed a way to improve the errorbars. This cannot
possibly be done by interchanging commutative arithmetic operations, but
it CAN be done by applying physical modeling. See section IV of my
paper: by modeling the data as the sum of an orientation-dependent
signal and a time-dependent drift, the errorbars are greatly reduced,
and the "signal" disappears. This depends inherently on the ability to
separate the signal (orientation dependence) from the drift, which in
turn depends on the physical situation of his measurements -- read the
paper to see how that can be done.

This is a general rule: you want to model the backgrounds as accurately
as possible, in order to subtract them. One must take into account the
inherent uncertainties in the subtraction and include them in the
errorbars. For Miller's algorithm [M], the uncertainties in the
background subtraction are negligible compared to the scatter of the
data points themselves; for Surfer's algorithm [S] it is not obvious at
all how to proceed -- no matter, because interchanging commutative
arithmetic operations cannot possibly affect the errorbars, and we do
know what they are for [M]. For the analysis of my section IV, the
uncertainties in the background subtraction are the only contribution to
the errorbars (read the paper to see why).

Bottom line: if you want to accept Miller's results, you must accept
WHAT HE DID. Your claims about an alternative are both wrong and
irrelevant. WHAT HE DID has enormous errorbars, even though he did not
compute and display them (consistent with common practice of his day,
but NOT of ours).

This is one of the few ancient experiments for which the
original data sheets are available, so we KNOW how large
his errorbars actually are -- they COMPLETELY negate his
"results".

Tom Roberts

.

Quantcast