Re: information theory problem

Michael Olea wrote:

J.A. Legris wrote:

mark-t2@xxxxxxxxx wrote:
Let's posit an event E, which you assume will occur
with probability 0.3

Then someone tells you it will actually occur with probability 0.5
How much information, in bits, does that message contain?

At this point the question is ill-posed.

Information, in the Shannon sense, is always a "reduction of uncertainty".
So you have to have some uncertainty to reduce. "Uncertainty" is the
entropy of a probability distribution. You need an uncertainty over the
probability of event E - a probability distribution over the probability
of event E (a "prior" - e.g. the probability that the probability of event
E is between 0.29 and 0.31 is 0.6 ...).

To make it concrete lets say event E is "coin comes up heads". You want to
estimate the probability of E. You need an initial distribution over the
interval [0, 1], which will change as evidence comes in. Because this is a
distribution over a continuous, real-valued variable (the "bias" of the
coin) it must be a probability *density* function - the probability of any
specific value (like 0.3) is zero (infinitesimal). You can only assign a
finite probability to an interval (or the union of intervals).

So you have some pdf (probability density function), which is zero outside
[0, 1], nonnegative in [0, 1]. And whose integral from 0 to 1 = 1. To get
the probability that the bias lies in a particular subinterval [a, b] you
integrate the pdf over [a, b].

To get the entropy (in bits) of the pdf you integrate -pdf(x)log2(pdf(x))
with x ranging from 0 to 1.

Now some evidence comes in, a report that the probability of event E is
0.5. Still ill-formed - you need a means of updating the pdf given the
report. You get a new pdf after the report, compute its entropy, subtract
that entropy from the entropy of the old pdf, and that is the information
in bits the report provides.

So lets say your original pdf was the uniform distribution: pdf(x) = 1, x
in
[0, 1]. This has an entropy of 0 bits, which may seem strange, but for
continuous distributions only differences in entropy have meaning.

Now lets say that after the report pdf(x) = 32 for x in [31/64, 33/64],
and zero elsewhere. This distribution has an entropy of -5 bits (which
should now not seem strange); old entropy - new entropy = 0 - -5bits = 5
bits. The report has provided 5 bits of information (it has reduced your
uncertainty by 5 bits). Intuitively, it has shrunk the interval of
uncertainty by a factor of 32.

My textbook does not address this question. I have an idea
regarding the solution, but seek other opinions/analysis...

Caveat: the naive formula, as found in any communications
book, does not apply... i.e. log(1/p) = log 2 = 1 bit, is not it.

It's not the formula that's naive, but its application.

That would be the answer, if E was assumed P = 0.5, and
then you received a message that it had occurred. ...

Right.

... However,
in this case, the message predicts E with P = 0.5, which
you had a priori assigned 0.3

That's a little muddled. You need an estimate of P(E) before the report
and an estimate of P(E) after the report, where "estimate" is a pdf.

Further, you discount the accuracy of his message, assigning
it a 80% chance of reliability. (hence 20% chance that your
original 0.3 estimate is correct) Now how much information
does his message convey?

Still muddled. Lets go back to flipping coins. Say you start with the
uniform distribution. You flip the coin 10 times and it comes up heads 3
times. Calculate the pdf and its entropy. Now you get a report "I flipped
that same coin 40 times, and I got 20 heads". Pool the data, calulate the
new pdf and its entropy, subtract new entropy from old. Exercise left to
reader.


Superman! Help!

[Look! up in the sky! ...]

Hehe.

But, I'm no expert in information theory. Just know the basics.

-- Michael
Thanks for stating relatively clearly what my experienced intuition told me.
--
JosephKK
Gegen dummheit kampfen die Gotter Selbst, vergebens.  
--Schiller
.

Relevant Pages

  • Re: behavior as mapping
    ... estimating a probability distribution, the distribution ... sequence with equal probability - since you have microsecond temporal ... reduction of the entropy Pto the entropy P ... If there were 4 genes we would need 2 bits of binding site info. ...
    (comp.ai.philosophy)
  • Re: behavior as mapping
    ... estimating a probability distribution, the distribution ... sequence with equal probability - since you have microsecond temporal ... reduction of the entropy Pto the entropy P ... If there were 4 genes we would need 2 bits of binding site info. ...
    (comp.ai.philosophy)
  • Re: An Inflationary Account - #4 - Why THAT definition - Part B
    ... where p is a probability and lg represents the logarithm to base 2. ... twice in that formula for entropy. ... it could be said that 'information theory' itself is simply ...
    (talk.origins)
  • Re: "boundary condition" definition
    ... Some people say that "entropy" as used in information theory is the ... is driven by a dissipation of energy. ... By "process" I mean the existence of empirical constraints ... necessarily alters the probability distribution. ...
    (sci.philosophy.meta)
  • An Inflationary Account - #4 - Why THAT definition - Part B
    ... where p is a probability and lg represents the logarithm to base 2. ... twice in that formula for entropy. ... I guess the most surprising thing is that a central formula for ... it could be said that 'information theory' itself is simply ...
    (talk.origins)