The n-Category Café

Interesting observartions.

A tiny comment:

I think everybody will agree that the general pattern of statistical mechanics is indeed about more about inference than about nature per se. But at some point you want to apply all this to a particular case. Usually this amounts to specifying a Hamiltonian function.

And the precise details of that function is what encodes information about nature.

So there is a bit of information about nature - encoded in a Hamiltonian - and then there are means to extract certain parts of that information (entropy maximization, etc.).

Interestingly, while quantum mechanics is in a way nothing but statistical mechanics analytically continued to the complex plane, we usually tend to regard not just the Hamiltonian in quantum mechanics as encoding information about nature, but also the rest of the formalism.

Whether that “rest of the formalism” is really just a manifestation of our thinking or a genuine aspect of nature is hotly debated in all those discussions concerning the “interpretation of quantum mechanics”.

In connection with Fuchs’ viewpoint, see this discussion by Ray Streater, entitled “Locality in the EPR experiment”. It starts this way:

I. The von Neumann collapse postulate

In this section, we show that the postulate of von Neumann, that on measurement the wave function collapses to an eigenstate of the observable being measured, follows from Bayes’s rule for conditioning probabilities in classical probability.

See the following, which cites Caticha’s paper:

• Thomas Marlow, Relationalism vs. Bayesianism

I love this discussion. It always brings me joy to see someone mention Jaynes as well :)

I’m now thinking about this stuff from a completely different perspective, i.e. finance and economics, but I think similar arguments carry over (as unobvious as that may sound). Jaynes similarly applied his ideas to economics. I think he was a visionary and will someday be recognized as such by a larger audience. I only wish I had stumbled onto his work earlier so that I might have met him.

Best regards,
Eric

In connection with the application of constructions of mathematical physics to machine learning, as well as Bayesianism, I thought some recent work of Shalizi and Crutchfield should be mentioned:

Pattern Discovery and Computational Mechanics

Computational mechanics is a method for discovering, describing and quantifying patterns, using tools from statistical physics. It constructs optimal, minimal models of stochastic processes and their underlying causal structures. These models tell us about the intrinsic computation embedded within a process—how it stores and transforms information. Here we summarize the mathematics of computational mechanics, especially recent optimality and uniqueness results. We also expound the principles and motivations underlying computational mechanics, emphasizing its connections to the minimum description length principle, PAC theory, and other aspects of machine learning.

Shalizi’s views on Bayesian approaches (to use a crude cover term) are interesting; I think he can be fairly described as a skeptic, as indicated here, here, and here (Pet peeves: Physicists who do not distinguish between a random variable (“X = the roll of a die”) and the value it takes (“x=5”). People who report numbers without error-bars or confidence-intervals. Bayesians.)

A discussion of complexity by Murray Gell-Mann, summarizing material in his book The Quark and the Jaguar, echoes the connections made in this post. Note in particular the emphasis on the relevance of a particular notion of regularities in this excerpt:

A measure that corresponds much better to what is usually meant by complexity in ordinary conversation, as well as in scientific discourse, refers not to the length of the most concise description of an entity (which is roughly what AIC is), but to the length of a concise description of a set of the entity’s regularities. Thus something almost entirely random, with practically no regularities, would have effective complexity near zero. So would something completely regular, such as a bit string consisting entirely of zeroes. Effective complexity can be high only a region intermediate between total order and complete disorder.

There can exist no procedure for finding the set of all regularities of an entity. But classes of regularities can be identified. Finding regularities typically refers to taking the available data about the entity, processing it in some manner into, say, a bit string, and then dividing that string into parts in a particular way and looking for mutual AIC among the parts. If a string is divided into two parts, for example, the mutual AIC can be taken to be the sum of the AIC’s of the parts minus the AIC of the whole. An amount of mutual algorithmic information content above a certain threshold can be considered diagnostic of a regularity. Given the identified regularities, the corresponding effective complexity is the AIC of a description of those regularities.

More precisely, any particular regularities may be regarded as embedding the entity in question in a set of entities sharing the regularities and differing only in other respects. In general, the regularities associate a probability with each entity in the set. (The probabilities are in many cases all equal but they may differ from one member of the set to another.) The effective complexity of the regularities can then be defined as the AIC of the description of the set of entities and their probabilities. (Specifying a given entity, such as the original one, requires additional information.)

I can’t resist quoting from another of Gell-Mann’s essays posted at SFI, entitled “Nature Conformable To Herself”, which is highly relevant to the topic of David’s post:

To answer those questions, we need to deal first with the widespread notion that all scientific theory is nothing but a set of constructs with which the human mind attempts to grasp reality, a notion associated with the German philosopher Immanuel Kant. Although I had heard of that belief many times, I first came into collision with it thirty-six years ago in Paris.

At that time, I was a visiting professor at the Collège de France, founded by Francis I more than four hundred years earlier. (As far as I know, I was the first visiting professor in the history of that venerable institution.) My office was in the laboratory of experimental physics established by Francis Perrin, a well-known scientist who was a permanent professor at the Collège. On visits to the offices of the junior experimentalists down the hall, I noticed that they spent a certain amount of time drawing little pictures in their notebooks, which I assumed at first must be diagrams of experimental apparatus. Many of the drawings turned out, however, to be sketches of a gallows for hanging the vice-director of the lab, whose rigid ideas drove them crazy.

I soon got to know the sous-directeur, and we conversed on various subjects, one of which was Project Ozma, an early attempt to detect possible signals from other technical civilizations on planets orbiting nearby stars. The corresponding project nowadays is called the Search for Extraterrestrial Intelligence. We discussed how communication might take place if alien intelligences broadcasting signals were close enough to the solar system, assuming that both interlocutors would have the patience to wait years for the signals to be transmitted back and forth. I suggested that we might try beep, beep-beep, beep-beep-beep, etc. to indicate the numbers 1, 2, 3, and so forth, and then perhaps 1, 2, 3,…42, 44…..60, 62…….92, for the atomic numbers of the 90 chemical elements that are stable—1 to 92 except for 43 and 61. “Wait,” said the sous-directeur, “that is absurd. Those numbers up to 92 would mean nothing to such aliens…. Why, if they have 90 stable chemical elements as we do, then they must also have the Eiffel Tower and Brigitte Bardot.”

That is how I became acquainted with the fact that French schools taught a kind of neo-Kantian philosophy, according to which the laws of nature are nothing but Kantian “categories” used by the human mind to describe reality.