Reinventing Society in the Wake of Big Data

A Conversation with Alex (Sandy) Pentland, The Edge, 8/30/12

"With Big Data we can now begin to actually look at the details of social interaction and how those play out, and are no longer limited to averages like market indices or election results. This is an astounding change. The ability to see the details of the market, of political revolutions, and to be able to predict and control them is definitely a case of Promethean fire—it could be used for good or for ill, and so Big data brings us to interesting times. We're going to end up reinventing what it means to have a human society."

I have more concerns than positive thoughts about this  provocative discussion.

Foremost is his understanding and beliefs related to complexity: "The notion that it is connections between people that is really important is key, because researchers have mostly been trying to understand things like financial bubbles using what is called Complexity Science or Web Science. But these older ways of thinking about Big Data leaves the humans out of the equation. What actually matters is how the people are connected together by the machines and how, as a whole, they create a financial market, a government, a company, and other social structures."

Complexity is all about connections, whether it is molecules (including genes), things, insects, animals and certainly social systems of humans. Melanie Mitchell's book Complexity: A Guided Tour, is a tour of the sciences of complexity, a broad set of efforts that seek to explain how large-scale complex, organized, and adaptive behavior can emerge from simple interactions among myriad individuals. And, in One, Two, a Few and Many, I summarize this book and many other sources of information on complexity.

The problem is that if you have a truly complex systems composed of intelligent agents, more data will lead you nowhere. Remember, if we're talking about humans here, you have a complex system of complex systems. If it's truly complex, no matter how deep you dive into the data, there will always be uncertainty. (Think of a fractal.) And, if the systems is in disequilibrium, cause and effect are unrelated.

Pentland states, "With Big Data you can easily get false correlations, for instance, "On Mondays, people who drive to work are more likely to get the flu." If you look at the data using traditional methods, that may actually be true, but the problem is why is it true? Is it causal? Is it just an accident? You don't know. Normal analysis methods won't suffice to answer those questions. What we have to come up with is new ways to test the causality of connections in the real world far more than we have ever had to do before. We no can no longer rely on laboratory experiments; we need to actually do the experiments in the real world." Which won't help. That's like dropping a grain of sand on a pile of sand (think hourglass) and observing what happens. Then drawing a conclusion from your experiment when nothing happens. However, the next grain of sand you drop may cause a whole avalanche to occur. The best you can ever do is to gather historical data on the system and talk about probabilities. And, it's not normal statistics. Now imagine that each grain of sand has intelligence and you have the croquet game described by Lewis Carroll's surrealistic version of the game in the popular children's novel Alice's Adventures in Wonderland; a hedgehog was used as the ball, a flamingo the mallet, and playing cards as the hoops.

"Alice thought she had never seen such a curious croquet-ground in her life; it was all ridges and furrows; the balls were live hedgehogs, the mallets live flamingos, and the soldiers had to double themselves up and to stand on their hands and feet, to make the arches.

The chief difficulty Alice found at first was in managing her flamingo: she succeeded in getting its body tucked away, comfortably enough, under her arm, with its legs hanging down, but generally, just as she had got its neck nicely straightened out, and was going to give the hedgehog a blow with its head, it would twist itself round and look up in her face, with such a puzzled expression that she could not help bursting out laughing: and when she had got its head down, and was going to begin again, it was very provoking to find that the hedgehog had unrolled itself, and was in the act of crawling away: besides all this, there was generally a ridge or furrow in the way wherever she wanted to send the hedgehog to, and, as the doubled-up soldiers were always getting up and walking off to other parts of the ground, Alice soon came to the conclusion that it was a very difficult game indeed.

The players all played at once without waiting for turns, quarrelling all the while, and fighting for the hedgehogs; and in a very short time the Queen was in a furious passion, and went stamping about, and shouting 'Off with his head!' or 'Off with her head!' about once in a minute."

Another problem I have with Pentland's vision of the future is that the opportunities for abuse is enormous and will certainly happen. He recognizes this by writing, "Another important issue with Big Data is that since this data is mostly about people, there are enormous issues about privacy, data ownership, and data control. You can imagine using Big Data to make a world that is incredibly invasive, incredibly 'Big Brother'… George Orwell was not nearly creative enough when he wrote 1984."

There is some inherent limit to our ability to understand and predict the behavior of complex systems akin to the Heisenberg Uncertainty Principle in quantum physics. Heisenberg stated the principle in 1927 as "The more precisely the position is determined, the less precisely the momentum is known in this instant, and vice versa." What occurs in complex systems is not the same as what happens in quantum physics, and we've not stated a similar law for complex systems, yet.

However, my biggest fear is that we will build the systems Pentland envisions.

"This is the first time in human history that we have the ability to see enough about ourselves that we can hope to actually build social systems that work qualitatively better than the systems we've always had. That's a remarkable change. It's like the phase transition that happened when writing was developed or when education became ubiquitous, or perhaps when people began being tied together via the Internet.

The fact that we can now begin to actually look at the dynamics of social interactions and how they play out, and are not just limited to reasoning about averages like market indices is for me simply astonishing. To be able to see the details of variations in the market and the beginnings of political revolutions, to predict them, and even control them, is definitely a case of Promethean fire. Big Data can be used for good or bad, but either way it brings us to interesting times. We're going to reinvent what it means to have a human society."

William Calvin wrote in his book, The Cerebral Symphony: Seashore Reflections on the Structure of Consciousness, “Inconsistency is part of flexibility, of nature's strategy of keeping options open. Animals that cannot adapt to new environments will not survive the incessant fluctuations of climate. Judicial systems that cannot grow and change with our society's evolving problems will become rigid anachronisms that promote social earthquakes. Consistency and rationality are human virtues in dealing with certain potentially orderly situations; we make excellent use of them in engineering and legal systems, but we shouldn't expect living systems to have made them centerpiece of their operation in a changing, unpredictable world.”

In the book, A Simpler Way, Margaret Wheatley and  Myron Kellner-Rogers write, "The universe is a living, creative, experimenting experience of discovering what's possible at all levels of scale from microbe to cosmos. Life's natural tendency is to organize. Life organizes into greater levels of complexity to support more diversity and greater sustainability. Life organizes around a self. Organizing is always an act of creating an identity. Life self-organizes. Networks, patterns, and structures emerge without external imposition or direction. Organization wants to happen. People are intelligent, creative, adaptive, self-organizing, and meaning seeking. Organizations are living systems. They too are intelligent, creative, adaptive, self-organizing, meaning-seeking."

Life needs messes. Let's not attempt to get rid of them all and in so doing destroy life.

Too Fast to Fail

"We're going to need a bigger boat." A paraphrase from police chief Martin Brody (Roy Scheider) in Jaws after seeing the shark for the first time.

 http://www.motherjones.com/politics/2013/02/high-frequency-trading-danger-risk-wall-street

Click here to see the quote in  a film clip.

The Wealth of Networks: How Social Production Transforms Markets and Freedom

I normally write extensive reviews of the books I read. I’m not able to do this with this book. Yochai Benkler’s book is so comprehensive in scope and detail (515pp) that it defies summarization. Suffice it to say that this is the authoritative book on the subjects of networks and social production. It’s a difficult read (the academic language) but rewards the reader with an unsurpassed view of the landscape it covers. In the author’s own words in the acknowledgments, “Another great debt is to David Grais, who spent many hours mentoring me in my first law job, bought me my first copy of Strunk and White, and, for all practical purposes, taught me how to write in English; as he reads these words, he will be mortified, I fear, to be associated with a work of authorship as undisciplined as this, with so many excessively long sentences, replete with dependent clauses and unnecessarily complex formulations of quite simple ideas.” Lawrence Lessig comments, “In this book, Benkler establishes himself as the leading intellectual of the information age.”

What is written here is my attempt to cover the salient points of the introduction.

He begins with, “Information, knowledge, and culture are central to human freedom and human development. How they are produced and exchanged in our society critically affects the way we see the state of the world as it is and might be; who decides these questions; and how we, as societies and polities, come to understand what can and ought to be done. For more than 150 years, modern complex democracies have depended in large measure on an industrial information economy for these basic functions. In the past decade and a half, we have begun to see a radical change in the organization of information production. Enabled by technological change, we are beginning to see a series of economic, social, and cultural adaptations that make possible a radical transformation of how we make the information environment we occupy as autonomous individuals, citizens, and members of cultural and social groups. It seems passe´ today to speak of “the Internet revolution.” In some academic circles, it is positively naıve. But it should not be. The change brought about by the networked information environment is deep. It is structural. It goes to the very foundations of how liberal markets and liberal democracies have coevolved for almost two centuries.

A series of changes in the technologies, economic organization, and social practices of production in this environment has created new opportunities for how we make and exchange information, knowledge, and culture. These changes have increased the role of nonmarket and nonproprietary production, both by individuals alone and by cooperative efforts in a wide range of loosely or tightly woven collaborations. These newly emerging practices have seen remarkable success in areas as diverse as software development and investigative reporting, avant-garde video and multiplayer online games. Together, they hint at the emergence of a new information environment, one in which individuals are free to take a more active role than was possible in the industrial information economy of the twentieth century. This new freedom holds great practical promise: as a dimension of individual freedom; as a platform for better democratic participation; as a medium to foster a more critical and self-reflective culture; and, in an increasingly information dependent global economy, as a mechanism to achieve improvements in human development everywhere.

The rise of greater scope for individual and cooperative nonmarket production of information and culture, however, threatens the incumbents of the industrial information economy. At the beginning of the twenty-first century, we find ourselves in the midst of a battle over the institutional ecology of the digital environment. A wide range of laws and institutions— from broad areas like telecommunications, copyright, or international trade regulation, to minutiae like the rules for registering domain names or whether digital television receivers will be required by law to recognize a particular code—are being tugged and warped in efforts to tilt the playing field toward one way of doing things or the other. How these battles turn out over the next decade or so will likely have a significant effect on how we come to know what is going on in the world we occupy, and to what extent and in what forms we will be able—as autonomous individuals, as citizens, and as participants in cultures and communities—to affect how we and others see the world as it is and as it might be.”

He explains the emergence of the networked information economy, “The most advanced economies in the world today have made two parallel shifts that, paradoxically, make possible a significant attenuation of the limitations that market-based production places on the pursuit of the political values central to liberal societies. The first move, in the making for more than a century, is to an economy centered on information (financial services, accounting, software, science) and cultural (films, music) production, and the manipulation of symbols (from making sneakers to branding them and manufacturing the cultural significance of the Swoosh). The second is the move to a communications environment built on cheap processors with high computation capabilities, interconnected in a pervasive network—the phenomenon we associate with the Internet. It is this second shift that allows for an increasing role for nonmarket production in the information and cultural production sector, organized in a radically more decentralized pattern than was true of this sector in the twentieth century. The first shift means that these new patterns of production—nonmarket and radically decentralized—will emerge, if permitted, at the core, rather than the periphery of the most advanced economies. It promises to enable social production and exchange to play a much larger role, alongside property- and market based production, than they ever have in modern democracies.”

He makes three observations about this emergence:

1. Nonproprietary strategies have always been more important in information production than they were in the production of steel or automobiles, even when the economics of communication weighed in favor of industrial models.
2. We have in fact seen the rise of nonmarket production to much greater importance. Individuals can reach and inform or edify millions around the world. Such a reach was simply unavailable to diversely motivated individuals before, unless they funneled their efforts through either market organizations or philanthropically or state-funded efforts.
3. And, likely most radical, new, and difficult for observers to believe, is the rise of effective, large-scale cooperative efforts—peer production of information, knowledge, and culture.

He states that individuals have enhanced autonomy: “The networked information economy improves the practical capacities of individuals along three dimensions: (1) it improves their capacity to do more for and by themselves; (2) it enhances their capacity to do more in loose commonality with others, without being constrained to organize their relationship through a price system or in traditional hierarchical models of social and economic organization; and (3) it improves the capacity of individuals to do more in formal organizations that operate outside the market sphere. This enhanced autonomy is at the core of all the other improvements I describe. Individuals are using their newly expanded practical freedom to act and cooperate with others in ways that improve the practiced experience of democracy, justice and development, a critical culture, and community.”

The major battle to be fought is between open and proprietary models. “The battle over the relative salience of the proprietary, industrial models of information production and exchange and the emerging networked information economy is being carried out in the domain of the institutional ecology of the digital environment. In a wide range of contexts, a similar set of institutional questions is being contested: To what extent will resources necessary for information production and exchange be governed as a commons, free for all to use and biased in their availability in favor of none? To what extent will these resources be entirely proprietary, and available only to those functioning within the market or within traditional forms of well funded nonmarket action like the state and organized philanthropy? We see this battle played out at all layers of the information environment: the physical devices and network channels necessary to communicate; the existing information and cultural resources out of which new statements must be made; and the logical resources—the software and standards—necessary to translate what human beings want to say to each other into signals that machines can process and transmit. Its central question is whether there will, or will not, be a core common infrastructure that is governed as a commons and therefore available to anyone who wishes to participate in the networked information environment outside of the market-based, proprietary framework.”

He writes about four choices: “There are four methodological choices represented by the thesis that I have outlined up to this point, and therefore in this book as a whole, which require explication and defense. The first is that I assign a very significant role to technology. The second is that I offer an explanation centered on social relations, but operating in the domain of economics, rather than sociology. The third and fourth are more internal to liberal political theory. The third is that I am offering a liberal political theory, but taking a path that has usually been resisted in that literature—considering economic structure and the limits of the market and its supporting institutions from the perspective of freedom, rather than accepting the market as it is, and defending or criticizing adjustments through the lens of distributive justice. Fourth, my approach heavily emphasizes individual action in nonmarket relations. Much of the discussion revolves around the choice between markets and nonmarket social behavior. In much of it, the state plays no role, or is perceived as playing a primarily negative role, in a way that is alien to the progressive branches of liberal political thought. In this, it seems more of a libertarian or an anarchistic thesis than a liberal one. I do not completely discount the state, as I will explain. But I do suggest that what is special about our moment is the rising efficacy of individuals and loose, nonmarket affiliations as agents of political economy. Just like the market, the state will have to adjust to this new emerging modality of human action. Liberal political theory must first recognize and understand it before it can begin to renegotiate its agenda for the liberal state, progressive or otherwise.”

This book is well worth studying and would make a great text for discussion group. If anyone out there has read the book and would like to collaborate on such an endeavor, please let me know.

The Wealth of Networks: How Social Production Transforms Markets and Freedom, Yochai Benkler, Yale University Press, 2006, 515pp

The Wealth of Networks (PDF)

Monkeynomics: A Monkey Economy as Irrational as Ours

Some lessons applicable to complexity and markets.

Laurie Santos looks for the roots of human irrationality by watching the way our primate relatives make decisions. A clever series of experiments in "monkeynomics" shows that some of the silly choices we make, monkeys make too.



Laurie Santos runs the Comparative Cognition Laboratory (CapLab) at Yale, where she and collaborators across departments (from psychology to primatology to neurobiology) explore the evolutionary origins of the human mind by studying lemurs, capuchin monkeys and other primates. The twist: Santos looks not only for positive humanlike traits, like tool-using and altruism, but irrational ones, like biased decisionmaking.

In elegant, carefully constructed experiments, Santos and CapLab have studied how primates understand and categorize objects in the physical world -- for instance, that monkeys understand an object is still whole even when part of it is obscured. Going deeper, their experiments also search for clues that primates possess a theory of mind -- an ability to think about what other people think.

Most recently, the lab has been looking at behaviors that were once the province mainly of novelists: jealousy, frustration, judgment of others' intentions, poor economic choices. In one experiment, Santos and her team taught monkeys to use a form of money, tradeable for food. When certain foods became cheaper, monkeys would, like humans, overbuy. As we humans search for clues to our own irrational behaviors, Santos' research suggests that the source of our genius for bad decisions might be our monkey brains.

Markets as an Innovation Commons

A market is a type of commons as well, and in some cases may even be considered an "innovation commons".

I have read about mythology, anthropology and history, but I am by no means expert. Over the years I've developed a sense of the development of markets that I want to share. Much of this has been developed through conversations with people, especially my partner, Donna Prestwood. It is not rigorous research, but I do want to share the story I've developed. I've put this Italics, because it is a story, not a history.

During times when our ancient ancestors were hunter/gatherers and lived in tribes, the concept of territory was developed (See I told you it was a story. I started with "once upon a time".) Knowledge of what existed outside the territory was limited. The "other" who lived outside your territory was either enemies or strangers. Both were feared. An incursion into your territory was almost surely to provoke an attack, even if the incursion was not an attack itself.

As the boundaries of the territory became established and known or even marked, the "other" avoided incursion unless intent upon an attack. Paths of travel began to be developed along the boundaries of the territories. The first paths bypassed the territories of the tribes.

Sometime during this development people got the idea of bringing gifts to the boundary of their own territory, to the edge of the known. The gifts were left a s a peace offering. Among some tribes this led to the development of concept of potlatch. The tribes brought gifts to the boundary in a type of asynchronous exchange. Potlatch cultures developed when winning meant giving the better gift. Either as peace offerings or potlatch, the practice did reduce the amount of physical conflicts.

Since strangers passed along these boundaries in transit around the territories, it was only natural that somehow they began to enter into the mix. The resulting development led to the creation of markets along these boundary/paths. And, eventually this led to the concentration of villages around the markets, especially at crossroads.

Early in our development, humans developed the sense of having to bring gifts to the edge of the known. In some cultures this ended up as sacrifices. In others, it was food or precious goods. We carry that tradition on now in the form of gifts brought to the altar of our religion. The altar representing symbolically the boundary between what is known and the unknowable.

As a result, the original concept of a market was that it was held on "sacred" ground; it was safe. The boundary was not a place for war. It was a place of peace. Markets today should be "sacred" in that sense. This may be why we are so incensed when someone violates the trust of the market and cheats, lies, or steals.

In some cultures the concept of exchanging gifts at the borders, (Still celebrated by the way every Friday evening in the fall at Texas high school football games with the students of the two schools exchanging gifts at the 50 yard line before the start of the game. Apparently the gifts don't work to bring peace and hostilities have to be undertaken however ritual the hostilities are.) led to the development of a bartering system. From that came the concept of the development of value for different types of goods. Note that adding strangers into the mix brought different types of goods into the market not indigenous to the tribes.

In what sense is a market an innovation commons? Goods produced or manufactured are the embodiment of knowledge. In the ancient market, an individual skilled in making axes could teach the "other" how to make the ax. But, it's more efficient and rewarding to exchange the ax for other things needed or money. Even if the goods are grown, harvested, hunted or extracted, their presence at the market embodies the application of knowledge. Information was also exchanged at a market as well, in a loose bartering sense.

One of the things that makes a market work is the mechanism of valuing goods and services. Another is the trust and safety. Others? I welcome comments.

Are we now at a point in the development of civilization where an innovation commons would work? If so, what are conditions or principles that would assure its success? What can we learn from successful markets?

What can we learn from e-Bay? I don't know enough about it to comment, so I would welcome other comments. One element of e-Bay I do know about is the rating system. Buyers and sellers rate each other after the transaction. A bad reputation prevents you from further participation in the market.

Hyper-Capitalism: What Ever Happened to Free Markets?

by John Renesch

Market fundamentalists are fond of citing Adam Smith’s “invisible hand” theories as expressed in his book The Wealth on Nations, considered by many to be the sacred text of capitalism. Of course Smith could hardly have predicted the degree to which the market would be so manipulated as it has over the century and a half since he penned his 1776 book. Political lobbying and campaign contributions in the U.S. with the resultant government subsidies, tax advantages , changes to corporate law and other tilting of the free-market scale in favor of some to the detriment of others has created a market far from the unfettered model Smith had envisioned.

But that doesn’t stop the market fundamentalists from shouting “let the market resolve things”, “keep government out of the marketplace” and labeling any attempt to bring the scales back into balance as “socialism.” What market fundamentalists do not admit or wish to have known is the market is far from free. Special interests have sought advantages over generations so that those with the most to spend can effectively purchase preferred treatment by government decree and thus the rich get richer at the expense of the not-so-rich.

In effect what they are saying when they insist on keeping hands off the market is “don’t mess with the system that has been manipulated to our tastes” or, in essence, “we have the system customized to our advantage and we don’t want anyone messing with it.” Late comers might be saying “I have learned how to play with the present rules and I am doing quite well at working the system; please don’t mess it up because then I’ll have to learn a different way.” Either way, it is one-ups-man-ship, with a relatively small percentage of the population wanting to maintain its advantage over the rest of us. This tramples on the founding principles of equal opportunity, liberty and justice for all.

More

And click on Newsletter