Re: Relative Entropy

From: Hiu Chung Law (antispam_at_antispam.org)
Date: 02/25/05

Date: 25 Feb 2005 14:49:25 GMT

Josh <Josh@nomail.nom> wrote:
> From Thomas-Cover

> The relative entropy between two probability
> mass functions p(x) and q(x) is defined as

> D(p|q)=\sum_x p(x)log(p(x)/q(x))

> In the above definition, we use the convention
> (based on continuity arguments) that
> 0*log(0/q) = 0, p*log(p/0)=inf

> How I have to interpret this equalities?

> \forall q!=0 0*log(0/q)=0
> and
> \forall p!=0 p*log(p/0)=inf ?

> What about the case where q(x)=p(x)=0?
> Thanks, Josh.

There may be better answer, but you can view the above using
the following limit:

lim x->0+ x log x = 0

x should approach 0 from the positive side, in order
to avoid complex arithmetic...