Reinventing Society in the Wake of Big Data

A Conversation with Alex (Sandy) Pentland, The Edge, 8/30/12

"With Big Data we can now begin to actually look at the details of social interaction and how those play out, and are no longer limited to averages like market indices or election results. This is an astounding change. The ability to see the details of the market, of political revolutions, and to be able to predict and control them is definitely a case of Promethean fire—it could be used for good or for ill, and so Big data brings us to interesting times. We're going to end up reinventing what it means to have a human society."

I have more concerns than positive thoughts about this  provocative discussion.

Foremost is his understanding and beliefs related to complexity: "The notion that it is connections between people that is really important is key, because researchers have mostly been trying to understand things like financial bubbles using what is called Complexity Science or Web Science. But these older ways of thinking about Big Data leaves the humans out of the equation. What actually matters is how the people are connected together by the machines and how, as a whole, they create a financial market, a government, a company, and other social structures."

Complexity is all about connections, whether it is molecules (including genes), things, insects, animals and certainly social systems of humans. Melanie Mitchell's book Complexity: A Guided Tour, is a tour of the sciences of complexity, a broad set of efforts that seek to explain how large-scale complex, organized, and adaptive behavior can emerge from simple interactions among myriad individuals. And, in One, Two, a Few and Many, I summarize this book and many other sources of information on complexity.

The problem is that if you have a truly complex systems composed of intelligent agents, more data will lead you nowhere. Remember, if we're talking about humans here, you have a complex system of complex systems. If it's truly complex, no matter how deep you dive into the data, there will always be uncertainty. (Think of a fractal.) And, if the systems is in disequilibrium, cause and effect are unrelated.

Pentland states, "With Big Data you can easily get false correlations, for instance, "On Mondays, people who drive to work are more likely to get the flu." If you look at the data using traditional methods, that may actually be true, but the problem is why is it true? Is it causal? Is it just an accident? You don't know. Normal analysis methods won't suffice to answer those questions. What we have to come up with is new ways to test the causality of connections in the real world far more than we have ever had to do before. We no can no longer rely on laboratory experiments; we need to actually do the experiments in the real world." Which won't help. That's like dropping a grain of sand on a pile of sand (think hourglass) and observing what happens. Then drawing a conclusion from your experiment when nothing happens. However, the next grain of sand you drop may cause a whole avalanche to occur. The best you can ever do is to gather historical data on the system and talk about probabilities. And, it's not normal statistics. Now imagine that each grain of sand has intelligence and you have the croquet game described by Lewis Carroll's surrealistic version of the game in the popular children's novel Alice's Adventures in Wonderland; a hedgehog was used as the ball, a flamingo the mallet, and playing cards as the hoops.

"Alice thought she had never seen such a curious croquet-ground in her life; it was all ridges and furrows; the balls were live hedgehogs, the mallets live flamingos, and the soldiers had to double themselves up and to stand on their hands and feet, to make the arches.

The chief difficulty Alice found at first was in managing her flamingo: she succeeded in getting its body tucked away, comfortably enough, under her arm, with its legs hanging down, but generally, just as she had got its neck nicely straightened out, and was going to give the hedgehog a blow with its head, it would twist itself round and look up in her face, with such a puzzled expression that she could not help bursting out laughing: and when she had got its head down, and was going to begin again, it was very provoking to find that the hedgehog had unrolled itself, and was in the act of crawling away: besides all this, there was generally a ridge or furrow in the way wherever she wanted to send the hedgehog to, and, as the doubled-up soldiers were always getting up and walking off to other parts of the ground, Alice soon came to the conclusion that it was a very difficult game indeed.

The players all played at once without waiting for turns, quarrelling all the while, and fighting for the hedgehogs; and in a very short time the Queen was in a furious passion, and went stamping about, and shouting 'Off with his head!' or 'Off with her head!' about once in a minute."

Another problem I have with Pentland's vision of the future is that the opportunities for abuse is enormous and will certainly happen. He recognizes this by writing, "Another important issue with Big Data is that since this data is mostly about people, there are enormous issues about privacy, data ownership, and data control. You can imagine using Big Data to make a world that is incredibly invasive, incredibly 'Big Brother'… George Orwell was not nearly creative enough when he wrote 1984."

There is some inherent limit to our ability to understand and predict the behavior of complex systems akin to the Heisenberg Uncertainty Principle in quantum physics. Heisenberg stated the principle in 1927 as "The more precisely the position is determined, the less precisely the momentum is known in this instant, and vice versa." What occurs in complex systems is not the same as what happens in quantum physics, and we've not stated a similar law for complex systems, yet.

However, my biggest fear is that we will build the systems Pentland envisions.

"This is the first time in human history that we have the ability to see enough about ourselves that we can hope to actually build social systems that work qualitatively better than the systems we've always had. That's a remarkable change. It's like the phase transition that happened when writing was developed or when education became ubiquitous, or perhaps when people began being tied together via the Internet.

The fact that we can now begin to actually look at the dynamics of social interactions and how they play out, and are not just limited to reasoning about averages like market indices is for me simply astonishing. To be able to see the details of variations in the market and the beginnings of political revolutions, to predict them, and even control them, is definitely a case of Promethean fire. Big Data can be used for good or bad, but either way it brings us to interesting times. We're going to reinvent what it means to have a human society."

William Calvin wrote in his book, The Cerebral Symphony: Seashore Reflections on the Structure of Consciousness, “Inconsistency is part of flexibility, of nature's strategy of keeping options open. Animals that cannot adapt to new environments will not survive the incessant fluctuations of climate. Judicial systems that cannot grow and change with our society's evolving problems will become rigid anachronisms that promote social earthquakes. Consistency and rationality are human virtues in dealing with certain potentially orderly situations; we make excellent use of them in engineering and legal systems, but we shouldn't expect living systems to have made them centerpiece of their operation in a changing, unpredictable world.”

In the book, A Simpler Way, Margaret Wheatley and  Myron Kellner-Rogers write, "The universe is a living, creative, experimenting experience of discovering what's possible at all levels of scale from microbe to cosmos. Life's natural tendency is to organize. Life organizes into greater levels of complexity to support more diversity and greater sustainability. Life organizes around a self. Organizing is always an act of creating an identity. Life self-organizes. Networks, patterns, and structures emerge without external imposition or direction. Organization wants to happen. People are intelligent, creative, adaptive, self-organizing, and meaning seeking. Organizations are living systems. They too are intelligent, creative, adaptive, self-organizing, meaning-seeking."

Life needs messes. Let's not attempt to get rid of them all and in so doing destroy life.