A Brief Introduction to: Information Theory, Excess Entropy and Computational Mechanics

David Feldman, April 1998

This article contains more information than I was interested in as my current interest is in complexity. However, his axiomatic definition of information entropy was interesting and useful to complexity. He list three axioms:

1. Entropy reaches a maximum when the distribution is uniform.
2. Entropy is a continuous function of the probability function.
3. Entropy is the same for every set of probabilities in the probability function

These are my words not his. It’s my interpretation of his mathematical relationships.

The first axiom states that if the probability of any state for a system is the same as any other state then entropy of the system is at its maximum. The second axiom states that any arbitrary small change in the system should lead to a small change in the entropy. And, the third axiom states that any sample from the system should return the same entropy as any other sample. This describes an unstructured complex system.

For a structured complex system applying these axioms yields some interesting insights. A complex system never has a uniform distribution of probabilities, so the entropy of a complex system is never maximized. A structured complex system is often partially or totally nonlinear, and sensitive to initial conditions. Cause and effect are not necessarily relatable. A small change in the system can have a large impact of the system’s entropy. A structured complex system is sensitive to both space and time history. So, almost by definition, a sample cannot be representative of the system’s entropy.

This last point has been concerning me for some time with respect to market research, marketing research and polling. Most of the systems we seek to gain sight about are by definition now structured complex systems. As a result, sampling will yield unreliable results.

A Brief Introduction to: Information Theory, Excess Entropy and Computational Mechanics
David Feldman, April 1998