Showing posts with label entropy. Show all posts
Showing posts with label entropy. Show all posts

Thursday, September 5, 2013

Change, Entropy and Complexity

A video clip from Al Gore's speech on his book The Future. He talks about change, entropy and complexity. While I applaud his courage of introducing the concept of complexity in the public speech to a general audience and I love his metaphors, he doesn't quite get right. But it's a great attempt.

http://www.c-spanvideo.org/clip/4463895

Friday, January 7, 2011

Chaos, Complexity and Entropy

A physics talk for non-physicists by Michael Baranger

“The twenty-first century is starting with a huge bang. For the person in the street, the bang is about a technical revolution that may eventually dwarf the industrial revolution of the 18th and 19th centuries, having already produced a drastic change in the rules of economics. For the scientifically minded, one aspect of this bang is the complexity revolution, which is changing the focus of research in all scientific disciplines, for instance human biology and medicine. What role does physics, the oldest and simplest science, have to play in this? Being a theoretical physicist to the core, I want to focus on theoretical physics. Is it going to change also?

Twentieth-century theoretical physics came out of the relativistic revolution and the quantum mechanical revolution. It was all about simplicity and continuity (in spite of quantum jumps). Its principal tool was calculus. Its final expression was field theory.

Twenty-first-century theoretical physics is coming out of the chaos revolution. It will be about complexity and its principal tool will be the computer. Its final expression remains to be found. Thermodynamics, as a vital part of theoretical physics, will partake in the transformation.”

The author describes calculus as being limited to smooth functions that can be approximated by a series of straight lines. “For at least 200 years, theoretical science fed on this calculus idea. The mathematicians invented concepts like continuity and analyticity to describe smoothness more precisely. And the discovery of Calculus led to an explosion of further discoveries. The branch of mathematics so constituted, known as Analysis, is not only the richest of all the branches, but also by far the most useful for applications to quantitative science, from physics to engineering, from astronomy to hydrodynamics, from materials science to oceanography. Theoretical scientists became applied mathematicians, and applied mathematicians are people for whom analysis is second nature. Integrals, differential equations, series expansions, integral representations of special functions, etc . . . . , these are the tools that calculus has provided and that are capable of solving an amazing variety of problems in all areas of quantitative knowledge.”

Scientists and engineers accepted this assumption so long that they forgot completely about it. “Yes, the enormous success of calculus is in large part responsible for the decidedly reductionist attitude of most twentieth century science, the belief in absolute control arising from detailed knowledge. Yes, the mathematicians were telling us all along that smooth curves were the exception, not the rule: we did not listen!”

Chaos is the exception that finally broke through. “Chaos is the rediscovery that calculus does not have infinite power. In its widest possible meaning, chaos is the collection of those mathematical truths that have nothing to do with calculus. And this is why it is distasteful to twentieth century physicists.”

Chaos can exist in both time and space. Chaos is space is called a fractal. “There are many possible definitions of the word fractal. A very loose and general definition is this: a fractal is a geometric figure that does not become simpler when you analyze it into smaller and smaller parts. Which implies, of course, that it is not smooth.”

Fractals exist in mathematics, geometry and almost everywhere in nature.

Chaos in time is the result of dynamical systems, a system that is capable of changing its configuration over time. “The signature of time-chaos is something called “sensitivity to initial conditions”.”

“Sensitivity to initial conditions is the death of reductionism. It says that any small uncertainty that may exist in the initial conditions will grow exponentially with time, and eventually (very soon, in most cases) it will become so large that we will lose all useful knowledge of the state of the system. Even if we know the state of the system very precisely now, we cannot predict the future trajectory forever. We can do it for a little while, but the error grows exponentially and we have to give up at some point.”

Time and space chaos are closely related. “Every chaotic dynamical system is a fractal-manufacturing machine. Conversely, every fractal can be seen as the possible result of the prolonged action of time-chaos.”

Chaos can exist for very simple systems and is always nonlinear.

“At the present time, the notion of complex system is not precisely delineated yet. This is normal. As people work on complex systems more and more, they will gain better understanding of their defining properties. Now, however, the idea is somewhat fuzzy and it differs from author to author. But there is fairly complete agreement that the “ideal” complex systems, those which we would like most to understand, are the biological ones, and especially the systems having to do with people: our bodies, our groupings, our society, our culture. Lacking a precise definition, we can try to convey the meaning of complexity by enumerating what seem to be the most typical properties. Some of these properties are shared by many non-biological systems as well.”

  • Complex systems contain many constituents interacting nonlinearly.
  • The constituents of a complex system are interdependent.
  • A complex system possesses a structure spanning several scales. At every scale we find a structure.
  • A complex system is capable of emerging behavior. “Emergence happens when you switch the focus of attention from one scale to the coarser scale above it. A certain behavior, observed at a certain scale, is said to be emergent if it cannot be understood when you study, separately and one by one, every constituent of this scale, each of which may also be a complex system made up of finer scales. Thus the emerging behavior is a new phenomenon special to the scale considered, and it results from global interactions between the scale’s constituents.”
  • A complex system is capable of self organization. “The combination of structure and emergence leads to self-organization, which is what happens when an emerging behavior has the effect of changing the structure or creating a new structure.”
  • A complex adaptive system can exhibit self reproduction. “There is a special category of complex systems which was created especially to accommodate living beings. They are the complex adaptive systems. As their name indicates, they are capable of changing themselves to adapt to a changing environment. They can also change the environment to suit themselves. Among these, an even narrower category are self-reproducing: they know birth, growth, and death.”
  • Complexity involves an interplay between chaos and non-chaos.
  • Complexity involves an interplay between cooperation and competition. “Once again this is an interplay between scales. The usual situation is that competition on scale n is nourished by cooperation on the finer scale below it (scale n+1).”

I find the author’s discussion on entropy unsatisfactory and unconvincing. After several pages of proof and discussion, he writes, “The conclusion is that our dimensionless entropy, which measures our lack of knowledge, is a purely subjective quantity. It has nothing to do with the fundamental laws of particles and their interactions. It has to do with the fact that chaos messes up things; that situations that were initially simple and easy to know in detail, will become eventually so complicated, thanks to chaos, that we are forced to give up trying to know them.”

Having learned about entropy from thermodynamics and being able to derive the concept of entropy from simple considerations of the Carnot Cycle, it’s a real property to me. One implication of thermodynamic entropy is the impossibility of perpetual motion. Energy gets dissipated in the form of heat. Perhaps I’m not knowledgeable enough to see the difference between thermodynamic entropy and information entropy. Perhaps I still have more to learn, or unlearn…

Chaos, Complexity and Entropy: A physics talk for non-physicists, Michael Baranger, MIT and NECSI, MIT-CTP-3112

Thursday, November 18, 2010

A Brief Introduction to: Information Theory, Excess Entropy and Computational Mechanics

David Feldman, April 1998

This article contains more information than I was interested in as my current interest is in complexity. However, his axiomatic definition of information entropy was interesting and useful to complexity. He list three axioms:

1. Entropy reaches a maximum when the distribution is uniform.
2. Entropy is a continuous function of the probability function.
3. Entropy is the same for every set of probabilities in the probability function

These are my words not his. It’s my interpretation of his mathematical relationships.

The first axiom states that if the probability of any state for a system is the same as any other state then entropy of the system is at its maximum. The second axiom states that any arbitrary small change in the system should lead to a small change in the entropy. And, the third axiom states that any sample from the system should return the same entropy as any other sample. This describes an unstructured complex system.

For a structured complex system applying these axioms yields some interesting insights. A complex system never has a uniform distribution of probabilities, so the entropy of a complex system is never maximized. A structured complex system is often partially or totally nonlinear, and sensitive to initial conditions. Cause and effect are not necessarily relatable. A small change in the system can have a large impact of the system’s entropy. A structured complex system is sensitive to both space and time history. So, almost by definition, a sample cannot be representative of the system’s entropy.

This last point has been concerning me for some time with respect to market research, marketing research and polling. Most of the systems we seek to gain sight about are by definition now structured complex systems. As a result, sampling will yield unreliable results.

A Brief Introduction to: Information Theory, Excess Entropy and Computational Mechanics
David Feldman, April 1998

Tuesday, September 28, 2010

Arcadia: A Play on Complexity

Arcadia is a play by Tom Stoppard that weaves time, social mores, mathematics and science. It can be interpreted on many levels. Of interest to me is that in many ways, this play is about complexity, and the play is complex (or at least complicated).

Lets start with the title. Arcadia refers to a vision of pastoralism and harmony with nature. The term is derived from the Greek province of the same name which dates to antiquity; the province's mountainous topography and sparse population of pastoralists later caused the word Arcadia to develop into a poetic byword for an idyllic vision of unspoiled wilderness. Arcadia is associated with bountiful natural splendor, harmony, and is often inhabited by shepherds. The concept also figures in Renaissance mythology. Commonly thought of as being in line with Utopian ideals, Arcadia differs from that tradition in that it is more often specifically regarded as unattainable. Furthermore, it is seen as a lost, Edenic form of life, contrasting to the progressive nature of Utopian desires.

The inhabitants were often regarded as having continued to live after the manner of the Golden Age, without the pride and avarice that corrupted other regions. It is also sometimes referred to in English poetry as Arcady. The inhabitants of this region bear an obvious connection to the figure of the Noble savage, both being regarded as living close to nature, uncorrupted by civilization, and virtuous. (From Wikipedia)

The Latin phrase “Et in Arcadia ego”appears on the tomb in a 1647 painting by Nicolus Poussin. It is meant as a cautionary of the impermanence of life: even in Arcadia you will die.

Arcadia has another meaning that is connected to complexity. Arcadia is named after Arcas. In Greek mythology, Arcas was the son of Zeus and Callisto. Callisto was a nymph of the goddess Artemis. Zeus, being a flirtatious god, wanted Callisto for a lover. As she would not be with anyone but Artemis, Zeus cunningly disguised himself as Artemis and seduced Callisto. The child resulting from their union was called Arcas.

Hera (Zeus' wife), became jealous, and in anger, transformed Callisto into a bear. She would have done the same or worse to her son, had Zeus not hidden Arcas in an area of Greece that would come to be called Arcadia, in his honor. There Arcas safely lived until one day, during one of the court feasts held by King Lycaon, Arcas was placed upon the burning altar as a sacrifice to the gods. He then said to Zeus "If you think that you are so clever, make your son whole and unharmed." At this Zeus became enraged. He made Arcas whole and then directed his anger toward Lycaon, turning him into the first werewolf. (Some of the myths have Arcas cut into pieces and served to Zeus.) (See Arcas, Greek Myth Index, and Callisto.)

The significance of this double meaning of the title will become meaningful as I describe the play.

Quoting from SFF Net, “Arcadia is a play that stands up to numerous readings and viewings. The synopsis below barely scratches the surface of its complexity and depth. It also gives away a couple of major plot points best experienced first-hand. Read on at your peril.

The action of Arcadia takes place in single space, a room on the garden front of a very large country house in Derbyshire, but in two times, the present and the early years of the nineteenth century. It opens as Thomasina Coverly, a precocious thirteen-year-old math student, receives a lesson from her tutor, twenty-two-year-old Septimus Hodge. The two are discussing Fermat's theorem, Newton and other matters of mathematics and physics when they are interrupted by Ezra Chater, a third-rate poet. Chater accuses Hodge of having been spied in a "carnal embrace" with Mrs. Chater, a charge Hodge makes little effort to deny. Meanwhile, Thomasina's mother, Lady Croom, is wrangling with her landscape architect, Richard Noakes, who wants to clutter the immaculately kept grounds with a gloomy hermitage and other gothic paraphernalia.

The second scene moves to the twentieth century. Coverly descendants still reside at the estate: young Chloe, mathematician Valentine and mute, mysterious Gus. They are also hosts to best-selling author Hannah Jarvis, there to research a history of the estate's gardens, and to literary scholar Bernard Nightingale, who intends to prove that Lord Byron, the great Romantic poet, visited Sidley Park and killed Ezra Chater in a duel.

The next scene, however, demonstrates that, even though Byron did visit Sidley Park in 1809, it was Hodge whom the cuckolded Chater challenged to a duel.”

The structure of the play is based on the interplay of two time periods in the same room combined with the social mores of each time period. The back and forth nature of the play increases in tempo until the close of the play when both sets of characters are in the same scene.

Thomasina is indeed perceptive, creative and bold. Picking up the dialog in Scene 3:

THOMASINA: You are churlish with me because mama is paying attention to your friend. Well, let them elope, they cannot turn back the advancement of knowledge. I think it is an excellent discovery. Each week I plot your equations dot for dot, xs against ys in all manner of algebraical relation, and every week they draw themselves as commonplace geometry, as if the world of forms were nothing but arcs and angles. God's truth, Septimus, if there is an equation for a curve like a bell, there must be an equation for one like a bluebell, and if a bluebell, why not a rose? Do we believe nature is written in numbers?

SEPTIMUS: We do.

THOMASINA: Then why do your equations only describe the shapes of manufacture?

SEPTIMUS: I do not know.

THOMASINA: Armed thus, God could only make a cabinet.

SEPTIMUS: He has mastery of equations which lead into infinities where we cannot follow.

THOMASINA: What a faint-heart! We must work outward from the middle of the maze. We will start with something simple. (She picks up the apple leaf.) I will plot this leaf and deduce its equation. You will be famous for being my tutor when Lord Byron is dead and forgotten.

With this dialog, the concept of complexity and fractals is introduced.

The conversation quickly turns to another theme:

SEPTIMUS: Back to Cleopatra.

THOMASINA: Is it Cleopatra? I hate Cleopatra!

SEPTIMUS: You hate her? Why?

THOMASINA: Everything is turned to love with her. New love, absent love, lost love – I never knew a heroine that makes such noodles of our sex. It needs only a Roman general to drop anchor outside the window and away goes the empire like a christening mug in a pawn shop. If Queen Elizabeth had been a Ptolemy history would have been quite different – we would be admiring the pyramids of Rome and the great Sphinx of Verona.

SEPTIMUS; God save us.

THOMASINA: But instead, the Egyptian noodle made carnal embrace with the enemy who burned the great library of Alexandria without so much as a fine for all that is overdue. Oh, Septimus! - can you bear it? All the lost plays of the Athenians! Two hundred at least by Aeschylus, Sophocles, Euripides – thousands of poems – Aristotle's own library brought to Egypt by the noodle's ancestors! Can we sleep for grief?

SEPTIMUS: By counting our stock. Seven plays Aeschylus, seven Sophocles, nineteen from Euripides, my lady! You should no more grieve for the rest of them for a buckle lost from your first shoe, or your lesson book which will be lost when you are old. We shed as we pick up, like travelers who must carry everything in their arms, and what we let fall will be picked up by those behind. The procession is very long and life is very short. We die on the march. But there is nothing outside the march so nothing can be lost to it. The missing plays of Sophocles will turn up piece by piece, or be written again in another language. Ancient cures for diseases will reveal themselves once more. Mathematical discoveries glimpsed and lost to view will have their time again. You do not suppose my lady, that if all of Archimedes had been hidden in the great library of Alexandria, we would still be at loss for a corkscrew? I have no doubt that the improved steam-driven heat-engine which puts Mr. Noaks into an ecstasy that he and it and the modern age should all coincide, was well established on papyrus.

This theme has to do with evolution. For Teilhard de Chardin, the noosphere emerges through and is constituted by the interaction of human minds. The noosphere has grown in step with the organization of the human mass in relation to itself as it populates the earth. As mankind organizes itself in more complex social networks, the higher the noosphere will grow in awareness. This is an extension of Teilhard's Law of Complexity/Consciousness, the law describing the nature of evolution in the universe. (See Wikipedia)

It also describes human history. Complexification is the driving force of history, and individuals only retard or advance that natural inevitable change.

In scene 4, in modern time, Hannah and Valentine are talking. In the course of research into the history of Sidley Park, they have discovered Thomasina's notes and mathematics lesson book. Hannah reads from the book:

HANNAH: I, Thomasina Coverly, have found a truly wonderful method whereby all the forms of nature must give up their numerical secrets and draw themselves through number alone. The margin being too mean for my purpose, the reader must look elsewhere for the New Geometry of Irregular Forms discovered by Thomasina Coverly.

In other words, fractals.

Valentine explains what he thinks Thomasina is writing about, and Hannah asks:

HANNAH: Is it difficult?

VALENTINE: The maths isn't difficult. It's what you did at school. You have some x-and-y equation. Any value for x gives you a value for y. So you put a dot where it's right for both x and y. Then you take the next value for x which gives you another value for y, and when you've done that a few times you join up the dots and that's your graph of whatever the equation is.

HANNAH: And is that what she's doing?

VALENTINE: No. Not exactly. Not at all. What she's doing is, every time she works out a value for y, she's using that as her next value for x. And so on. Like a feedback. She's feeding the solution back into the equation, and then solving it again. Iteration, you see.

HANNAH: And that's surprising, is it?

VALENTINE: Well, it is a bit. It's the technique I'm using on my grouse numbers, and it hasn't been around for much longer than, well, call it twenty years.

Valentine describes the work he is doing trying to understand population trends of grouse in the Park. He has data of all the grouse shot since 1870, and is trying to understand the mathematical relationship. He is attempting to formulate the logistic equation for population growth. In discrete form, this equation is known as the logistic map, a very simple equation that exhibits chaotic properties in well defined regions.

Valentine comments, “It's about the behavior of numbers. This thing works for any phenomenon which eats its own numbers – measles epidemics, rainfall averages, cotton prices, it's a natural phenomenon in itself. Spooky.”

“When your Thomasina was doing maths it had been the same maths for a couple of thousand years. Classical. And for a century after Thomasina. Then maths left the real world behind, just like modern art, really. Nature was classical, maths was suddenly Picassos. But now nature is having the last laugh. The freaky stuff is turning out to be the mathematics of the natural world.”
After being pressed by Hannah, Valentine continues, “If you knew the algorithm and fed it back say ten thousand times, each time there'd be a dot somewhere on the screen. You'd never know where to expect the next dot. But gradually you'd start to see this shape, because every dot will be inside the shape of this leaf. It wouldn't be the leaf, it would be a mathematical object. But, yes. The unpredictable and the predetermined unfold together to make everything the way it is. It's how nature creates itself, on every scale, the snowflake and the snowstorm.”

Then Valentine makes the following comment about the significance of complexity, “It makes me so happy. To be at the beginning again, knowing almost nothing. People were talking about the end of physics. Relativity and quantum looked as if they were going to clean out the whole problem between them. A theory of everything. But they only explained the very big and the very small. The universe, the elementary particles. The ordinary-sized stuff which is our lives, the things people write poetry about - clouds - daffodils - waterfalls - and what happens in a cup of coffee when the cream goes in - these things are full of mystery, as mysterious to us as the heavens were to the Greeks. We're better at predicting events at the edge of the galaxy or inside the nucleus of an atom than whether it'll rain on auntie's garden party three Sundays from now. Because the problem turns out to be different. We can't even predict the next drip from a dripping tap when it gets irregular. Each drip sets up the conditions for the next, the smallest variation blows prediction apart, and the weather is unpredictable the same way, will always be unpredictable. When you push the numbers through the computer you can see it on the screen. The future is disorder. A door like this has cracked open five or six times since we got up on our hind legs. It's the best possible time to be alive, when almost everything you thought you knew is wrong.”

Parenthetically, this is the excitement I feel as a physicist (education only) about complexity science.

One of the other themes that run through this play (there are many) is thermodynamics and entropy.

Hannah and Valentine are talking in modern time:

VALENTINE: Listen - you know your tea's getting cold.

HANNAH: I like it cold.

VALENTINE: (Ignoring that) I'm telling you something. Your tea gets cold by itself, it doesn't get hot by itself. Do you think that's odd?

HANNAH: No.

VALENTINE: Well, it is odd. Heat goes to cold. It's a one-way street. Your tea will end up at room temperature. What's happening to your tea is happening to everything everywhere. The sun and the stars. It'll take a while but we're all going to end up at room temperature. When your hermit set up shop nobody understood this. But let's say you're right, in 18-whatever nobody knew more about heat than this scribbling nutter living in a hovel in Derbyshire.

Towards the end of the play when the four characters are on stage at the same time:

SEPTIMUS: So, we are all doomed!

THOMASINA: (Cheerfully) Yes.

VALENTINE: Like a steam engine, you see. She didn't have the maths, not remotely. She saw what things meant, way ahead, like seeing a picture.

SEPTIMUS: This is not science. This is story-telling.

THOMASINA: Is it a waltz now?

SEPTIMUS: No.

VALENTINE: Like a film.

HANNAH: What did she see?

VALENTINE: That you can't run the film backwards. Heat was the first thing which didn't work that way. Not like Newton. A film of a pendulum, or a ball falling through the air - backwards, it looks the same.

HANNAH: The ball would be going the wrong way.

VALENTINE: You'd have to know that. But with heat - friction - a ball breaking a window

HANNAH: Yes.

VALENTINE: It won't work backwards.

HANNAH: Who thought it did?

VALENTINE: She saw why. You can put back the bits of glass but you can't collect up the heat of the smash. It's gone.

SEPTIMUS: So the Improved Newtonian Universe must cease and grow cold. Dear me.

VALENTINE: The heat goes into the mix.
(He gestures to indicate the air in the room, in the universe.)

THOMASINA: Yes, we must hurry if we are going to dance.

VALENTINE: And everything is mixing the same way, all the time, irreversibly ...

SEPTIMUS: Oh, we have time, I think.

VALENTINE: ... till there's no time left. That's what time means.

SEPTIMUS: When we have found all the mysteries and lost all the meaning, we will be alone, on an empty shore.

THOMASINA: Then we will dance. Is this a waltz?

SEPTIMUS: It will serve.

I believe what Stoppard is trying to indicate here is the possible linkage between complexity and entropy. A subject I will write about later.

A complex system cannot be taken apart and put back together remaining the same as it was before. You can do that with a complicated system, even though you will lose energy in the process.

And, that brings back to the title - Arcadia - the place where in spite of its perfection, everyone still dies. And, Arcas who is burned up or cut into pieces as a test for Zeus, to see if he can reverse the process and unlike Humpty Dumpty, put Arcas back together again.

Arcadia, Tom Stoppard, Faber and Faber, 1993
Arcadia (Dramatized), Tom Stoppard, L. A. Theatre Works, 2010 (CD)