Re: Why kullback-leibler distance?

kalyan wrote:
Hi,

I am working with calculating the kullback-leibler distance between
two probability distribution. When this distance measure is not
symmetric, I was wondering why is the distance measure almost a
standard to calculate the distances of PDF ( probability
distribution
functions).


One of the reasons for importance is its link to the likelihood
function and to maximum likelihood theory, although little of this is
used when treating it as a distance measure I think. In its
max-likelihood background, the non-symmetry is natural.


Or are there any distance measures which overcome this problem of
symmetry. I am calculating distances between multivariate gaussian
distributions.


There are other non-symmetric distances which overcome the problem of
what to do if you having non-matching zero probabilities. Onec of
these is the "continuous ranked probabilty score", although you may
need to move this from a sample-version to a population-version of the
measure.

There are symmetric distance measures which are usually framed as
distances between CDFs rather than PDFs, but should still be good for
what you are doing. For univariate cases, these are the basis of the
Kolmogorov-Smirnov, Cramer-von Mises and Anderson Darling tests ...
you may need to search to find how to convert these ideas to a
multivariate situation. The advantage of CDF-based measures is that
they tend not to be overly inflenced by locations where one or other
density is zero.

David Jones


.

Relevant Pages

  • Re: Planar Linkage Problem
    ... uniform distribution on [0,2pi) ... then of course the distance between endpoints is just 1. ... because there's zero probability that distance ...
    (sci.math)
  • Re: Howard Hersheys Challenge of Sean Pitmans Assumptions
    ... Poisson distribution describes the probability of exactly k events ... likely to be found within a fixed distance." ...
    (talk.origins)
  • Re: I will rebuild physics from the ground up, as promised.
    ... 10 meters, and science predicts that it will require 1 second to traverse ... measurements show that the 10 meters are traversed in just 0,9 seconds. ... distance of 10 meters is measured at a distance using standard surveying ... Let the probability that length exists at any ...
    (sci.physics)
  • Time As the Only Physical Geometric Dimension
    ... time line they will maintain constant distance. ... derived from time and probability (especially Probable ... The first three rows are a perfect spacetime representation. ... Granting a solitary distance between two particles requires: ...
    (sci.physics)
  • Re: I will rebuild physics from the ground up, as promised.
    ... I could probably write a damn book with all the wacky ideas I've been ... measurements show that the 10 meters are traversed in just 0,9 seconds. ... and the distance of 10 meters is not disputed. ... the probability that length exists any any point from one endpoint to the ...
    (sci.physics)