Re: Explanation of Maximum Entropy

From: Ron Hardin (rhhardin_at_mindspring.com)
Date: 08/24/04 

Date: Tue, 24 Aug 2004 15:20:52 GMT

John Uebersax wrote:
>
> Could anyone kindly suggest online resources that give a simple,
> clear, basic explanation of Maximum Entropy as it applies statistical
> estimation?
>
> I understand that maxent may have advantages in certain applications,
> like image processing. What is not clear to me is whether it has
> practical implications for more routine applications. (In other
> words, part of my question is whether this is a fundamental, broadly
> applicable innovation--or revolution--within statistics).
>
> If it helps, we can focus the question by posing the specific example
> of coinflipping. If one observes 6 heads out of 10 flips of a
> potentially biased coin, then one can use standard Bayesian methods to
> infer the posterior probability distribution of Pr(heads)--i.e., the
> probability density for each value of Pr(heads)in the range (0, 1).
> If the prior distribution is uniform, then the posterior distribution
> would have a specific beta distribution form with a mode of .6.
>
> How would this be affected according to a Maximum Entropy approach?
>
> p.s. I understand that the standard Bayesian approach might use a
> better prior than a uniform distribution.
>
> --
> John Uebersax

There's a long and enlightening essay by Edwin T. Jaynes
``Where do we stand on Maximum Entropy?'' in _The Maximum Entropy
Formalism_ Levine and Tribus, eds., MIT Press 1978

He derives dice geometry imperfections from R. Wolf's ``random experiments''
data from dice tossing from 1850-1890, comparing its power with
chi-squared. It's very well written, though I haven't read it for
years. It predicts a bright future for Maximum Entropy.

You get chi-squared by dropping some terms from Maximum Entropy, as I recall. 

-- 
Ron Hardin
rhhardin@mindspring.com
On the internet, nobody knows you're a jerk.

Relevant Pages

  • Re: Explanation of Maximum Entropy
    ... John Uebersax wrote: ... > If the prior distribution is uniform, ... > How would this be affected according to a Maximum Entropy approach? ... define the class intervals such that each has kl/j observations; ...
    (sci.stat.math)
  • Re: skewed normal distribution
    ... (see also Maximum entropy probability distribution in wikipedia) ... So the maxent distributions give best fit pdfs for any number of moments. ...
    (sci.stat.math)
  • Re: Explanation of Maximum Entropy
    ... They standard maximum entropy approach is to find the distribution ... that the expectations have these values. ... Dept. of Statistics and Dept. of Computer Science radford@utstat.utoronto.ca ...
    (sci.stat.math)
  • Re: Dynamical Systems and Expansion-Contraction
    ... maximum entropy corresponds to the ... distribution is the "first cousin" of the uniform distribution, ... normal/Gaussian distributions for the continuous case aren't "ranked" ...
    (sci.physics)
  • Explanation of Maximum Entropy
    ... I understand that maxent may have advantages in certain applications, ... applicable innovation--or revolution--within statistics). ... If the prior distribution is uniform, ... How would this be affected according to a Maximum Entropy approach? ...
    (sci.stat.math)