### Maximum Entropy Distributions

#### Posted by David Corfield

While we eagerly await Tom’s second post on Entropy, Diversity and Cardinality, here’s a small puzzle for you to ponder. If all of the members of that family of entropies he told us about are so interesting, why is it that so many of our best loved distributions are maximum entropy distributions (under various constraints) for Shannon entropy?

For example, the normal distribution, $N(\mu, \sigma^2)$, is the maximum Shannon entropy distribution for distributions over the reals with mean $\mu$ and variance $\sigma^2$. And there are others, including exponential and uniform (here) and Poisson and Binomial (here). So why no famous distributions maximising Tsallis or Renyi entropy? (People do look at maximising these, e.g., here.)

Follow up question: has this property of the normal distribution anything to do with the central limit theorem? This is relevant.

## Re: Maximum Entropy Distributions

As someone who until recently had no clue about entropy, let alone maximizing it, I’d like to share this. It’s a pleasant account of how, if you wanted to model the behaviour of an expert translator of French (e.g. if you wanted to train a machine to do it?), you’d naturally run into the concept of maximum entropy.

As it says, the maximum entropy method is this: