Relative Entropy

From: Josh (Josh_at_NOMAIL.NOM)
Date: 02/25/05 

Date: Fri, 25 Feb 2005 12:55:03 +0100

 From Thomas-Cover

The relative entropy between two probability
mass functions p(x) and q(x) is defined as

D(p|q)=\sum_x p(x)log(p(x)/q(x))

In the above definition, we use the convention
(based on continuity arguments) that
0*log(0/q) = 0, p*log(p/0)=inf

How I have to interpret this equalities?

\forall q!=0 0*log(0/q)=0
and
\forall p!=0 p*log(p/0)=inf ?

What about the case where q(x)=p(x)=0?
Thanks, Josh.

Relevant Pages

  • Relative Entropy pq
    ... The relative entropy between two probability ... mass functions pand qis defined as ... In the above definition, we use the convention ...
    (sci.math)
  • Re: Factoring and rationals
    ... and _compute_ the empirical probability that the ... See the word "convention" in my original msg. ... there are several equally plausable enumerations to choose ... over rationals "chosen at random", and it looks messier to me too. ...
    (sci.crypt)
  • Re: Distance Between 2 Randomly-chosen Points on a Sphere
    ... Does this "convention" extend to all sets? ... >> are randomly selected on a circle, the uniform distribution of points ... >> on the circle does not satisfy scale and translation invariance. ... You must be aware that probability theory has some application to real ...
    (sci.math)
  • Re: " Wouldnt we agree with a mathematical writing ?
    ... There is a convention in probability theory that capital letters stand ... for random variables, lower-case letters for values of those variables. ...
    (sci.math)
  • Re: Relative Entropy
    ... >>mass functions pand qis defined as ... >>In the above definition, we use the convention ... >>Thanks, Josh. ... Is there some convention based on continuity arguments in this ...
    (sci.stat.math)