The n-Category Café

Thanks for the reference!

As you say, that’s about Fisher information rather than the Fisher metric. But the two things are closely related, and a priori, I’d expect a deformation or generalization of Fisher information to go hand in hand with a deformation or generalization of the Fisher metric. I’ve only just downloaded the paper, so at the moment I have no idea what could be going on. If you figure it out, please let me know!

I suppose my post was an implicit invitation for people to tell me that the Fisher metric can be deformed! Let’s see if I can begin to figure out what’s happening in the paper that James linked to.

If I’m understanding correctly, they do something like the following. For a real parameter $q$, they define the $q$-Fisher information of a family $(f(-; \theta))_\theta$ of probability distributions to be

$$\int f(x; \theta)^{q - 2} \biggl( \frac{\partial f}{\partial \theta} \biggr)^2 d x.$$

When $q = 1$, this gives the standard Fisher info. They prove some kind of generalized Cramér-Rao inequality for it.

They don’t discuss the Fisher metric, but given the definition above, the obvious thing to do is to put

$$\langle t, u \rangle = \sum p_i^{q - 2} t_i u_i.$$

Here $t, u \in \mathbb{R}^n$ are tangent vectors at a probability distribution $p$ on $\{1, \ldots, n\}$, that is, points in $\mathbb{R}^n$ such that $\sum t_i = 0 = \sum u_i$. Again, this recovers the standard Fisher metric when $q = 1$.

You can do this, but I haven’t yet understood why you’d want to.

Good. Part 7 of that series has an account of the basic fact at the heart of my post: that the Riemannian metric induced by the square root of ordinary relative entropy is the Fisher metric.

One could come at it from another angle:

1. Relative entropy is a canonical concept, because it’s uniquely characterized by a short list of properties that you and I both know well.

2. Since the Fisher metric on the simplex is derived from relative entropy by a canonical process, that means that the Fisher metric is also a canonical concept.

3. And transporting the Fisher metric from the simplex to the positive orthant of the sphere along the diffeomorphism you implicitly mention (take square roots in each coordinate), which is also a natural enough move, gives the round metric on the orthant.

So, we should also accept the round metric on the orthant as a canonical concept.

That’s not a symmetry argument, except to the extent that symmetry (permuation of coordinates) is involved in the axiomatization of relative entropy. But it is an argument — if one was needed! — that the geodesic metric on an orthant of the sphere is a natural thing.

What if I instead said the following?

The flat metric makes the simplex of probability distributions on an $n$-element set isometric to a subset of the unit sphere in $\mathbb{R}^{n+1}$.

The usual flat metric on $\mathbb{R}^{n+1}$ is highly canonical because its symmetry group—technically, its isometry group—has the maximum possible dimension of any isometry group of any Riemannian manifold of that dimension.

But it’s not at all clear to me that the flat metric is very canonical in the context of the simplex of probability distributions, as such.

Since I’m a radical at heart, I’m inclined to a solution like this: since the Fisher metric reveals the space of probability distributions to be a piece of the unit sphere, but the symmetries only become evident when we consider the whole sphere, not just this piece, let’s figure out how to do probability theory using the whole sphere.

So, instead of working with probabilities $p_i \in [0,1]$ with

$$\sum_{i = 1}^n p_i = 1$$

we should work with some quantities $\psi_i \in [-1,1]$ with

$$\sum_{i = 1}^n \psi_i^2 = 1$$

These new quantities should be related to the probabilities by

$$p_i = \psi_i^2$$

I believe the map $\psi \mapsto p$ maps the positive orthant of the unit sphere isometrically to the simplex with its Fisher metric. But if this is true, it seems every orthant of the unit sphere should be mapped isometrically to the simplex with its Fisher metric! After all, there’s nothing special about the positive orthant if what we’re doing to $\psi \in S^{n-1}$ is squaring each of its components.

(It’s a bit hard to visualize how the sphere gets folded down to the simplex by this map, but I can do it when $n = 2$.)

Of course, physicists will notice that $\psi \in S^{n-1}$ looks like a quantum state, and the relation between $\psi$ and $p$ looks like the usual relation between amplitudes and probabilities… except that we’re doing real quantum theory instead of the more common complex quantum theory.

I’ve written about real quantum theory and it’s a perfectly fine thing, though not quite as nice as the complex version:

• Quantum theory and division algebras.

Just as the symmetries of an $n$-dimensional complex Hilbert space form the group $U(n)$, those of a real Hilbert space form the group $O(n)$. Those are the symmetries I was after.

Mathematicians may flinch at this intrusion of “physics” in what had been a discussion of probability theory, but I think that would be unfair. Quantum theory is really just an alternative version of probability theory, different from “classical” probability theory but just as consistent. The only reason it’s part of “physics” is that our universe happens to use this alternative version, and the first people to discover this alternative version were people who carefully studied atoms and stuff.

There are plenty of other reasons to get worried about my radical attempt to “explain” why the Fisher metric makes the simplex round. In fact there are so many that I don’t think I’ll try to list them right now, or try to deal with them. I’ll just mention that physicists typically pass from quantum theory back to classical probability theory by choosing an orthonormal basis $e_i$ of an $n$-dimensional Hilbert space. This lets them define components

$$\psi_i = \langle e_i, \psi \rangle$$

for any vector in the Hilbert space, and probabilities $p_i = |\psi_i|^2$ for any unit vector $\psi$. But when we have a real Hilbert space, this choice determines a “positive orthant” in the unit sphere, and breaks the symmetries from $O(n)$ down to $S_n$. So, the extra symmetries of the sphere are lost when we pick this basis… but the roundness of the Fisher metric on the simplex may be a kind of ghostly hint that these symmetries had been there.

I realize all this sounds rather weird.

Mathematicians may flinch at this intrusion of “physics” in what had been a discussion of probability theory, but I think that would be unfair.

John, I have a feeling you may very reasonably have had me in mind when you wrote that. I completely accept what you write in that paragraph.

Personally speaking, I have nothing against physics. I loved physics at school, though I was dismayed that when I was about 17 or 18, I suddenly dropped from being top of the class to not-so-top of the class. Apparently my ability to do it began to fall off a cliff, the way we often see happening with our mathematics students.

My only problem with physics is an extremely simple one: I don’t know much of it. So when I read explanations of mathematics in physical terms, I very often can’t understand them. That’s all.

If I learn some piece of mathematics, and it also happens to be a part of physics, then great: I’ve accidentally learned some physics (though usually without really understanding the physical background). For instance, I accidentally learned a bit about thermodynamics from working with entropy.

Mind you, I suppose there’s something else going on in the psychological background. Suppose someone were to tell me a story about points on the unit sphere, but insisted on calling them “quantum states” rather than “points on the sphere”. (You didn’t do that — it’s just an example.) Then I’d feel I was missing out on something important, because I know that “quantum state” means a great deal to some people and not much to me. The words we use matter. I’d prefer the other person to either stick to “points on the sphere”, so that the story had to be told and motivated purely in mathematical terms, or take the time to explain what a quantum state is physically, so that I could share that understanding.

Anyway, John, your comment makes perfect sense even to me :-)

Thanks.

I can only guess at how this might work for probability distributions on infinite sets. Information geometers certainly consider this scenario, but I don’t know much about it. Maybe John can say something.

All I can say for sure is that relative entropy (unlike ordinary Shannon entropy) generalizes gracefully from finite sets to arbitrary measurable spaces. Given probability measures $\nu$ and $\mu$ on a measurable space $\Omega$, one defines the entropy of $\nu$ relative to $\mu$ to be

$$H(\nu \| \mu) = \int_\Omega \log \biggl( \frac{d\nu}{d\mu} \biggr) d\nu \in [0, \infty],$$

where that’s a Radon–Nikodym derivative on the right-hand side. For instance, when $\Omega = \mathbb{N}$, this reduces to the same formula as for finite sets but with the sum now countably infinite. But I haven’t attempted to go deeper.

James wrote:

Do you know if something similar works for the space of probability distributions on infinite sets, especially countable ones?

It must. I describe part of the story here:

• Information geometry (part 7).

Like most people in this game, I skip right past countably infinite sets and work more generally with measure spaces, or more precisely measurable spaces: spaces equipped with a $\sigma$-algebra of subsets. (For a countable set you can just use all subsets.)

Given two probability measures $\mu,\nu$ on a measurable space $\Omega$, their relative entropy is defined by

$$S(\mu, \nu) = \int_\Omega \; \log(\frac{d\mu}{d\nu}) \; d\mu$$

where $\frac{d\mu}{d\nu}$ is the Radon–Nikodym derivative. So, the relative entropy is, at least prima facie, only well-defined when $\mu$ is absolutely continuous with respect to $\nu$.

To avoid worrying about this all the time, and get a bunch of probability measures whose relative entropies are all well-defined, I fix a $\sigma$-finite measure $\omega$ on my measurable space $\Omega$, and then consider probability measures of the form $p \omega$ where $p \in L^1(\Omega,\omega)$ is any nonnegative function that integrates to 1: that is, any probability distribution.

With these “boring technicalities” taken care of in an appendix, I then let myself derive the Fisher metric from relative entropy in a fairly free-wheeling way. Anyone who wants to dig deeper into the technicalities should read this:

• Raymond Streater, Introduction to non-parametric estimation, in Algebraic and Geometric Methods in Statistics, eds. Paolo Gibilisco, Eva Riccomagno, Maria Piera Rogantin and Henry P. Wynn, Cambridge U. Press, Cambridge, 2009.

The technicalities are not actually boring if you like analysis.

One thing neither Streater or I do is to develop the picture of probability distributions with their Fisher metric as forming an ‘orthant’ of an infinite-dimensional sphere. This should just be

$$\{ \psi \in L^2(\Omega,\omega) : \psi \ge 0 \; \text{ and } \; \int_\Omega \psi^2 \, d\omega = 1 \}$$

and I’m hoping its interior is a Hilbert manifold. Here $\psi$ is just another way of talking about our probability distribution $p$: we have $p = \psi^2$.

Streater shoots off in another direction and makes probability distributions into a Banach manifold in a different way, but I fear I’m exhausting everyone’s patience with infinite-dimensional manifolds.

Good questions.

There are various characterization theorems for the Rényi and $q$-logarithmic entropies, both ordinary and relative. These are good theoretical reasons for being interested in those entropies.

However, I don’t know of any similar characterization theorems for quantities of the form $H_q^\lambda$. It wouldn’t surprise me if such theorems were known to somebody, though, because this kind of thing has been investigated pretty thoroughly over the years.

For reasons to study $\ell_p$ and $L_p$ spaces, try checking out this MathOverflow question and this n-Cafe post.

I had it in mind too :-)