Shannon Entropy from Category Theory
Posted by John Baez
I’m giving a talk at Categorical Semantics of Entropy on Wednesday May 11th, 2022. You can watch it live on Zoom if you register, or recorded later. Here’s the idea:
Shannon entropy is a powerful concept. But what properties single out Shannon entropy as special? Instead of focusing on the entropy of a probability measure on a finite set, it can help to focus on the “information loss”, or change in entropy, associated with a measure-preserving function. Shannon entropy then gives the only concept of information loss that is functorial, convex-linear and continuous. This is joint work with Tom Leinster and Tobias Fritz.
You can see the slides now, here.
It’s fun to talk about work that I did with Tobias Fritz and Tom Leinster here on the -Café — I’ve never given a talk where I went into as much detail as I will now. In fact I will talk a bit about all these:
• John Baez, Tobias Fritz and Tom Leinster, A characterization of entropy in terms of information loss, 2011.
• Tom Leinster, An operadic introduction to entropy, 2011.
• John Baez and Tobias Fritz, A Bayesian characterization of relative entropy, 2014.
• Tom Leinster, A short characterization of relative entropy, 2017.
• Nicolas Gagné and Prakash Panangaden, A categorical characterization of relative entropy on standard Borel spaces, 2017.
• Tom Leinster, Entropy and Diversity: the Axiomatic Approach, 2020.
• Arthur Parzygnat, A functorial characterization of von Neumann entropy, 2020.
• Arthur Parzygnat, Towards a functorial description of quantum relative entropy, 2021.
• Tai-Danae Bradley, Entropy as a topological operad derivation, 2021.
Matrices
It seems a little weird to be working with actual functions between finite sets, rather than matrices where each row is a probability measure. Is this what’s happening when you talk about “convex combinations of functions”? I couldn’t quite parse that slide.