The n-Category Café

Dusko’s paper uses FCA ( = Formal Concept Analysis). There was a Dagstuhl meeting a few years ago at which various people working in FCA met people working in Domain theory and Chu spaces. (For instance one of the papers there was Zhang, G.Q.: Chu spaces, concept lattices, and domains. In Brookes, S., Panan-
gaden, P., eds.: Electronic Notes in Theoretical Computer Science. Volume 83.,
Elsevier (2004).) Given the link between Chu, domains and logic, is there a `logic of bio-diversity’?

>>>Can you quantify their diversity?

>>Yes, he can, and in many different ways!

>Not to say: in diverse ways. ;-)

Actually, can we take that idea seriously? If we pick a probability distribution, its Renyi extropy gives a family of quantities measuring “diversity” depending on a continuous parameter alpha, which (if we fix a distribution from which to draw alpha) we can interpret as another probability distribution on the set of possible diversities. Could we iterate this process, producing not just effective numbers, but effective numbers of effective numbers, and so on?

For example, if a population is very evenly divided into n equal populations, then all of its Renyi extropies will be approximately n, and so the diversity of extropies would be only slightly more than 1. On the other hand, if the Shannon extropy of an arbitrary distribution is equal to n, we would expect the Renyi extropies to be more varied, and the diversity of extropies would be larger.

Perhaps one could even recover, in some special circumstances, the original distribution from knowledge, say, the Shannon extropy, the Shannon extropy of the Renyi extropies, the Shannon extropy of the Renyi extropies of the Renyi extropies, etc.?

Urs wrote…

But I am imagining that the mathematics of diversity can somehow give more universal answers?

I hope not. I view it as part of the general idea of decategorification.

Can you give me a rough idea of what “mathematics of diversity” can accomplish? What would be a typical theorem in diversity-theory? What do we learn from diversity theory? How is diversity theory different from just statistics?

These are excellent questions that deserve long answers. I’ll try to restrain myself.

The first thing is that there are deep and difficult questions that have nothing to do with statistics. In fact, my own involvement in the mathemtics of diversity is entirely non-statistical. There are very serious statistical challenges in measuring diversity: for example, how can you possibly guess the number of species too rare to show up in your sample? But personally, my interests lie elsewhere.

Earlier this year, I gave a talk about diversity measurement at the (British) National Centre for Statistical Ecology. (I had to come clean and begin by admitting that I knew almost no statistics and almost no ecology.) At some point in the talk I showed a slide saying something like “we assume that our community is fully censused”, causing laughter from the audience. But the point is that even if you know about every last beetle in your community, producing meaningful invariants is still non-trivial.

So, if it’s not statistical, what is it?

Maybe I’ll say more another time, but for now I’ll just point out the information-theoretic connections. Urs, you’ve probably seen enough posts about this to know that there’s a close connection with entropy. An ecological community of $n$ species can be crudely modelled as a probability distribution

$$p = (p_1, \ldots, p_n),$$

where $p_i$ represents the relative abundance (or proportion) of the $i$th species. Ecologists often measure the diversity of the community as the Shannon entropy of the distribution, namely

$$H(p) = -\sum p_i \log(p_i),$$

or (better) as its exponential,

$$e^{H(p)} = p_1^{-p_1} p_2^{-p_2} \cdots p_n^{-p_n}.$$

Then there are various related entropies, such as the Rényi and Tsallis entropies, appearing in the physics, information theory and statistics literature. Many of them have appeared in the ecology literature too.

Are these quantities really relevant ecologically? Well, that’s been the subject of lots of debate. Ultimately what we want is some theorem saying: “any diversity measure satisfying conditions X, Y and Z must be one of the following”. There’s a long tradition of such theorems in information theory, tapping into the theory of functional equations. Some of the conditions involved are clearly well-motivated ecologically, and some are more tenuous.

In my opinion, the most incisive work on ecological diversity measurement rises high above ecology. It becomes about something far more general, something mathematically universal. That’s really why I’m interested, attractive as the ecological applications may be.

Here’s a recent example. Take an ecological community partitioned into $m$ geographical areas or “subcommunities”. Ecologists have long asked: how much of the whole community’s diversity can be attributed to the diversity within the individual subcommunities, and how much to the variation between the subcommunities? You can imagine this might affect decisions on how to allocate resources for conservation.

The average diversity within each subcommunities is traditionally called the $\alpha$-diversity, the diversity between the subcommunities is called the $\beta$-diversity. Those are loose descriptions only. To turn them into precise quantities is no mean feat, and in fact people did it wrongly for several decades.

It was shown in 2007 (by sometime Café contributor Lou Jost) that, in fact, if you want $\alpha$- and $\beta$-diversity to be independent in an intuitively obvious sense, then there’s only one possible way to define them. It’s essentially a theorem about functional equations, but as far as I know it’s not one in the functional equation literature. And it’s a definitive answer; it’s the canonical way of partitioning diversity.

(I should mention that a similar result had been obtained in the late 1970s, by the Canadian statistician Rick Routledge, unknown to Jost at the time. But either Routledge didn’t realize the significance of his own work, or he was rather too quiet about it.)

I want to say much more, especially about how this links in to the theory of magnitude/Euler characteristic of enriched categories and lax colimits. But I think this comment is long enough already.

Thanks for the reply, Tom!

I wrote:

But I am imagining that the mathematics of diversity can somehow give more universal answers?

Your first reaction to this is to say…

I hope not.

…but I gather there is some misunderstanding between us at this point about which hopes are being discussed, because right afterwards you do allude to precisely such more universal answers, when you write, explicitly:

the most incisive work on ecological diversity measurement rises high above ecology. It becomes about something far more general, something mathematically universal.

and before that

Ultimately what we want is some theorem saying: “any diversity measure satisfying conditions X, Y and Z must be one of the following”.

Concerning this last point: how would you describe to an information-theorist the difference between information theory and diversity theory?

(That’s probably the question I should have asked instead of “How is diversity theory different from just statistics?”)