Clay Shirky looks at "cognitive surplus" -- the shared, online work we do with our spare brain cycles. While we're busy editing Wikipedia, posting to Ushahidi (and yes, making LOLcats), we're building a better, more cooperative world.
Clay Shirky believes that new technologies enabling loose collaboration — and taking advantage of “spare” brainpower — will change the way society works
Thursday, July 29, 2010
Fractals and the art of roughness
At TED2010, mathematics legend Benoit Mandelbrot develops a theme he first discussed at TED in 1984 -- the extreme complexity of roughness, and the way that fractal math can find order within patterns that seem unknowably complicated.
Studying complex dynamics in the 1970s, Benoit Mandelbrot had a key insight about a particular set of mathematical objects: that these self-similar structures with infinitely repeating complexities were not just curiosities, as they'd been considered since the turn of the century, but were in fact a key to explaining non-smooth objects and complex data sets -- which make up, let's face it, quite a lot of the world. Mandelbrot coined the term "fractal" to describe these objects, and set about sharing his insight with the world.
The Mandelbrot set (expressed as z² + c) was named in Mandelbrot's honor by Adrien Douady and John H. Hubbard. Its boundary can be magnified infinitely and yet remain magnificently complicated, and its elegant shape made it a poster child for the popular understanding of fractals. Led by Mandelbrot's enthusiastic work, fractal math has brought new insight to the study of pretty much everything, from the behavior of stocks to the distribution of stars in the universe.
Studying complex dynamics in the 1970s, Benoit Mandelbrot had a key insight about a particular set of mathematical objects: that these self-similar structures with infinitely repeating complexities were not just curiosities, as they'd been considered since the turn of the century, but were in fact a key to explaining non-smooth objects and complex data sets -- which make up, let's face it, quite a lot of the world. Mandelbrot coined the term "fractal" to describe these objects, and set about sharing his insight with the world.
The Mandelbrot set (expressed as z² + c) was named in Mandelbrot's honor by Adrien Douady and John H. Hubbard. Its boundary can be magnified infinitely and yet remain magnificently complicated, and its elegant shape made it a poster child for the popular understanding of fractals. Led by Mandelbrot's enthusiastic work, fractal math has brought new insight to the study of pretty much everything, from the behavior of stocks to the distribution of stars in the universe.
In Defense of Difference
Seed, 7/29/10 by Montenegro & Glavin
This past January, at the St. Innocent Russian Orthodox Cathedral in Anchorage, Alaska, friends and relatives gathered to bid their last farewell to Marie Smith Jones, a beloved matriarch of her community. At 89 years old, she was the last fluent speaker of the Eyak language. In May 2007 a cavalry of the Janjaweed — the notorious Sudanese militia responsible for the ongoing genocide of the indigenous people of Darfur — made its way across the border into neighboring Chad. They were hunting for 1.5 tons of confiscated ivory, worth nearly $1.5 million, locked in a storeroom in Zakouma National Park. Around the same time, a wave of mysterious frog disappearances that had been confounding herpetologists worldwide spread to the US Pacific Northwest. It was soon discovered that Batrachochytrium dendrobatidis, a deadly fungus native to southern Africa, had found its way via such routes as the overseas trade in frog’s legs to Central America, South America, Australia, and now the United States. One year later, food riots broke out across the island nation of Haiti, leaving at least five people dead; as food prices soared, similar violence erupted in Mexico, Bangladesh, Egypt, Cameroon, Ivory Coast, Senegal and Ethiopia.
All these seemingly disconnected events are the symptoms, you could say, of a global epidemic of sameness. It has no precise parameters, but wherever its shadow falls, it leaves the landscape monochromatic, monocultural, and homogeneous. Even before we’ve been able to take stock of the enormous diversity that today exists — from undescribed microbes to undocumented tongues — this epidemic carries away an entire human language every two weeks, destroys a domesticated food-crop variety every six hours, and kills off an entire species every few minutes. The fallout isn’t merely an assault to our aesthetic or even ethical values: As cultures and languages vanish, along with them go vast and ancient storehouses of accumulated knowledge. And as species disappear, along with them go not just valuable genetic resources, but critical links in complex ecological webs.
More
This past January, at the St. Innocent Russian Orthodox Cathedral in Anchorage, Alaska, friends and relatives gathered to bid their last farewell to Marie Smith Jones, a beloved matriarch of her community. At 89 years old, she was the last fluent speaker of the Eyak language. In May 2007 a cavalry of the Janjaweed — the notorious Sudanese militia responsible for the ongoing genocide of the indigenous people of Darfur — made its way across the border into neighboring Chad. They were hunting for 1.5 tons of confiscated ivory, worth nearly $1.5 million, locked in a storeroom in Zakouma National Park. Around the same time, a wave of mysterious frog disappearances that had been confounding herpetologists worldwide spread to the US Pacific Northwest. It was soon discovered that Batrachochytrium dendrobatidis, a deadly fungus native to southern Africa, had found its way via such routes as the overseas trade in frog’s legs to Central America, South America, Australia, and now the United States. One year later, food riots broke out across the island nation of Haiti, leaving at least five people dead; as food prices soared, similar violence erupted in Mexico, Bangladesh, Egypt, Cameroon, Ivory Coast, Senegal and Ethiopia.
All these seemingly disconnected events are the symptoms, you could say, of a global epidemic of sameness. It has no precise parameters, but wherever its shadow falls, it leaves the landscape monochromatic, monocultural, and homogeneous. Even before we’ve been able to take stock of the enormous diversity that today exists — from undescribed microbes to undocumented tongues — this epidemic carries away an entire human language every two weeks, destroys a domesticated food-crop variety every six hours, and kills off an entire species every few minutes. The fallout isn’t merely an assault to our aesthetic or even ethical values: As cultures and languages vanish, along with them go vast and ancient storehouses of accumulated knowledge. And as species disappear, along with them go not just valuable genetic resources, but critical links in complex ecological webs.
More
How We Make Choices
This is a very interesting talk by Sheena Iyengar based on studies of how people in different cultures view and make choices. It's important on two levels:
We all think we're good at making choices; many of us even enjoy making them. Sheena Iyengar looks deeply at choosing and has discovered many surprising things about it. For instance, her famous "jam study," done while she was a grad student, quantified a counterintuitive truth about decisio nmaking -- that when we're presented with too many choices, like 24 varieties of jam, we tend not to choose anything at all. (This and subsequent, equally ingenious experiments have provided rich material for Malcolm Gladwell and other pop chroniclers of business and the human psyche.)
Iyengar's research has been informing business and consumer-goods marketing since the 1990s. But she and her team at the Columbia Business School throw a much broader net. Her analysis touches, for example, on the medical decision making that might lead up to choosing physician-assisted suicide, on the drawbacks of providing too many choices and options in social-welfare programs, and on the cultural and geographical underpinning of choice. She's just published her first book, The Art of Choosing, which shares her research in an accessible and charming story that draws examples from her own life.
- Our consumer based economy is driven by choice. The relationship between choice and economic growth is not covered in this talk. But, it gives you a lot of insight into how our equating freedom of choice with a good life is trapping us.
- It's important to collaboration, community efforts and the understanding of an innovation commons. For these types of efforts to work, choice is not a freedom but a responsibility toward the group. This is most often seen in how we look at ecology and the future.
We all think we're good at making choices; many of us even enjoy making them. Sheena Iyengar looks deeply at choosing and has discovered many surprising things about it. For instance, her famous "jam study," done while she was a grad student, quantified a counterintuitive truth about decisio nmaking -- that when we're presented with too many choices, like 24 varieties of jam, we tend not to choose anything at all. (This and subsequent, equally ingenious experiments have provided rich material for Malcolm Gladwell and other pop chroniclers of business and the human psyche.)
Iyengar's research has been informing business and consumer-goods marketing since the 1990s. But she and her team at the Columbia Business School throw a much broader net. Her analysis touches, for example, on the medical decision making that might lead up to choosing physician-assisted suicide, on the drawbacks of providing too many choices and options in social-welfare programs, and on the cultural and geographical underpinning of choice. She's just published her first book, The Art of Choosing, which shares her research in an accessible and charming story that draws examples from her own life.
Some Thoughts on Complexity
Preface
Complexity includes at least three states of being where many of the rules we take for granted no longer apply. Concepts like causation, predictability, repeatability, control, analysis, determinism, linearity, and even centralization don’t work in complex systems. When dealing with a complex system it’s as though you’ve gone through a looking glass and everything you thought you knew is no longer valid. Furthermore, many complex systems exist at a human scale. We interact with many of these complex systems in daily life, natural and man made, and as a result become part of some of these complex systems.
It requires a paradigm shift to begin to comprehend complex systems. I shudder to use the word because it’s been so over used and trivialized. But I can think of no other concept to describe the shift. Complexity requires new perceptions, new beliefs, new ways of thinking and new rules for problem solving
The first barrier to be overcome is perception. Our belief systems are so strong, our scotomas so large, and our fears so great that resistance to accept the new reality is enormous. That’s understandable. After all, our world is built on the concepts listed above that aren’t valid in complex systems.
There is a fable about Columbus arriving in the new world and the natives unable to “see” the three ships. They had no mental construct, or paradigm, that would allow them to perceive the ships. A medicine man sat for days looking out at the water. By studying the pattern of the ripples caused by the ships, he was able to finally see them.
In essence that’s about the extent of our perception of complex systems. We can see the effects, and in some cases measure the effects, but we don’t understand how they work, based on our previous paradigm.
The three types of complex systems are critical state, chaotic and emergent. Earthquakes result from a critical state, and markets appear to be in a critical state. Weather is chaotic. And, massively parallel systems of intelligent agents exhibit emergence, like termites.
Lewis Carroll described a complex system with intelligent agents in Alice’s Adventures in Wonderland:
Alice thought she had never seen such a curious croquet ¬ground in her life: it was all ridges and furrows: the croquet balls were live hedgehogs, and the mallets live flamingoes, and the soldiers had to double themselves up and stand on their hands and feet, to make the arches.
The chief difficulty Alice found at first was in managing her flamingo: she succeeded in getting its body tucked away, com¬fortably enough, under her arm, with its legs hanging down, but generally, just as she had got its neck nicely straightened out, and was going to give the hedgehog a blow with its head, it would twist itself round and look up in her face, with such a puz¬zled expression that she could not help bursting out laughing; and, when she had got its head down, and was going to begin again, it was very provoking to find that the hedgehog had unrolled itself, and was in the act of crawling away: besides all this, there was generally a ridge or a furrow in the way wherever she wanted to send the hedgehog to, and, as the doubled-up soldiers were always getting up and walking off to other parts of the ground, Alice soon came to the conclusion that it was a very difficult game indeed.
The players all played at once, without waiting for turns, quarreling all the while, and fighting for the hedgehogs; and in a very short time the Queen was in a furious passion, and went stamping about, and shouting "Off with his head!" or "Off with her head!" about once in a minute. Alice began to feel very uneasy: to be sure, she had not as yet had any dispute with the Queen, but she knew that it might happen any minute, "and then," thought she, "what would become of me? They're dreadfully fond of beheading people here: the great wonder is, that there's any one left alive!"
Complexity is transdisciplinary, i.e. a discipline that crosses many older disciplines. There are examples of complexity in math, chemistry, physics, geology, weather and the environment, economics, markets, biology, medicine, etc.
One last closing thought. I’m not writing about complicated systems. Complicated systems in general do not exhibit the properties of complex systems.
Skills Required to Deal with Complexity
The skill set required to deal with complexity hasn’t been developed yet as you can appreciate as it is a new paradigm that crosses almost all older disciplines. However, there are some things I can say about the concepts that have to be learned.
The common concepts across all complex systems that must be taught are:
The concept of intellective skills was developed by Shoshana Zuboff in In the Age of the Smart Machine: The Future of Work and Power:
…The thinking this operator refers to is of a different quality form the thinking that attended the display of action-centered skills. It combines abstraction, explicit inference, and procedural reasoning. Taken together, these elements make possible a new set of competencies that I call Intellective skills. As long as the new technology signals only deskilling – the diminished importance of action-centered skills – there will be little probability of developing critical judgment at the data interface. To rekindle such judgment, though on a new more abstract footing, a re-skilling process is required. Mastery in a computer-mediated environment depends upon developing intellective skills.
Interpersonal skills will have to include collaboration, decentralization, leadership, creativity, innovativeness, flexibility, openness, self development, tolerance for ambiguity and uncertainty and others I’m sure.
Management of Complexity
First, if it’s true complexity and not just an extremely complicated system, there is no known way to manage it. The best we can do now is to manage the effects of a complex system. For example, I mentioned earthquakes as being the result of a complex system in a critical state. We can’t manage the earthquake or even predict when an earthquake will occur or its magnitude. But, we do know from the history of an earthquake prone fault system that there is a probability of a certain magnitude of earthquake occurring. We then design buildings to survive that magnitude of earthquake.
No analysis of the BP and Toyota examples has been done to determine if the systems are complex or just extremely complicated. Or, what type of complex system they might be.
Tad Patzek, chairman of the Petroleum and Geosystems Engineering Department at the University of Texas, testified in Congress last month about the Gulf of Mexico oil spill. His comments included observations on broad, long-term trends in industry, government and academia. He reported, "Horrible things happen when complex technologies and procedures overtake humans, who service the technologies falsely assuming complete control.” Patzek asserts that it is a complex system with emergent properties. I doubt that this is true. I think it most likely a complex system in a critical state. I suspect its behavior is more like an earthquake (criticality) than a termite mound (emergence). It probably has some characteristics of both as the human and software subsystems (intelligent agents) in the system could exhibit some characteristics of emergence. However, his conclusions are valid: large events like this will occur, and we can't predict when they will occur or prevent them. Understandably, he never states this as directly as I did. How can you tell that to politicians or business people? (Remember the Queen in the croquet game?)
The software developers have tamed the complicatedness of large software programs such as operating systems by turning a hoard of software developers, who are not in the same organization, loose on the project and then by facilitating the development of a complex system of intelligent agents (individual programmers) with positive emergent properties. This requires a degree of openness, decentralization, flexibility, cooperation and loss of control not presently possible in most modern corporations. Nor, do we know how to apply these principles in systems other than software.
For a copy of this article, click here.
Complexity includes at least three states of being where many of the rules we take for granted no longer apply. Concepts like causation, predictability, repeatability, control, analysis, determinism, linearity, and even centralization don’t work in complex systems. When dealing with a complex system it’s as though you’ve gone through a looking glass and everything you thought you knew is no longer valid. Furthermore, many complex systems exist at a human scale. We interact with many of these complex systems in daily life, natural and man made, and as a result become part of some of these complex systems.
It requires a paradigm shift to begin to comprehend complex systems. I shudder to use the word because it’s been so over used and trivialized. But I can think of no other concept to describe the shift. Complexity requires new perceptions, new beliefs, new ways of thinking and new rules for problem solving
The first barrier to be overcome is perception. Our belief systems are so strong, our scotomas so large, and our fears so great that resistance to accept the new reality is enormous. That’s understandable. After all, our world is built on the concepts listed above that aren’t valid in complex systems.
There is a fable about Columbus arriving in the new world and the natives unable to “see” the three ships. They had no mental construct, or paradigm, that would allow them to perceive the ships. A medicine man sat for days looking out at the water. By studying the pattern of the ripples caused by the ships, he was able to finally see them.
In essence that’s about the extent of our perception of complex systems. We can see the effects, and in some cases measure the effects, but we don’t understand how they work, based on our previous paradigm.
The three types of complex systems are critical state, chaotic and emergent. Earthquakes result from a critical state, and markets appear to be in a critical state. Weather is chaotic. And, massively parallel systems of intelligent agents exhibit emergence, like termites.
Lewis Carroll described a complex system with intelligent agents in Alice’s Adventures in Wonderland:
Alice thought she had never seen such a curious croquet ¬ground in her life: it was all ridges and furrows: the croquet balls were live hedgehogs, and the mallets live flamingoes, and the soldiers had to double themselves up and stand on their hands and feet, to make the arches.
The chief difficulty Alice found at first was in managing her flamingo: she succeeded in getting its body tucked away, com¬fortably enough, under her arm, with its legs hanging down, but generally, just as she had got its neck nicely straightened out, and was going to give the hedgehog a blow with its head, it would twist itself round and look up in her face, with such a puz¬zled expression that she could not help bursting out laughing; and, when she had got its head down, and was going to begin again, it was very provoking to find that the hedgehog had unrolled itself, and was in the act of crawling away: besides all this, there was generally a ridge or a furrow in the way wherever she wanted to send the hedgehog to, and, as the doubled-up soldiers were always getting up and walking off to other parts of the ground, Alice soon came to the conclusion that it was a very difficult game indeed.
The players all played at once, without waiting for turns, quarreling all the while, and fighting for the hedgehogs; and in a very short time the Queen was in a furious passion, and went stamping about, and shouting "Off with his head!" or "Off with her head!" about once in a minute. Alice began to feel very uneasy: to be sure, she had not as yet had any dispute with the Queen, but she knew that it might happen any minute, "and then," thought she, "what would become of me? They're dreadfully fond of beheading people here: the great wonder is, that there's any one left alive!"
Complexity is transdisciplinary, i.e. a discipline that crosses many older disciplines. There are examples of complexity in math, chemistry, physics, geology, weather and the environment, economics, markets, biology, medicine, etc.
One last closing thought. I’m not writing about complicated systems. Complicated systems in general do not exhibit the properties of complex systems.
Skills Required to Deal with Complexity
The skill set required to deal with complexity hasn’t been developed yet as you can appreciate as it is a new paradigm that crosses almost all older disciplines. However, there are some things I can say about the concepts that have to be learned.
The common concepts across all complex systems that must be taught are:
- Systems : We need to teach - What a system is, The types of systems, How one determines the type of a system, When you have included enough of the system, How a system responds to its environment and History of systems
- Reductionism and holism: Both concepts need to be taught, along with an understanding of when each one should be applied.
- Statistics: Statistics is a branch of mathematics concerned with collecting and interpreting data. Depending on the system, different types of statistics need to be used. People need to know how to collect and interpret data taken from the history of all types of complex systems. And, we need to teach the difference between probability and risk.
- Pattern recognition: In Emergence, Steven Johnson comments, “As the futurist Ray Kurzweil writes, 'Humans are far more skilled at recognizing patterns than in thinking through logical combinations, so we rely on this aptitude for almost all of our mental processes. Indeed, pattern recognition comprises the bulk of our neural circuitry. These faculties make up for the extremely slow speed of human neurons.' ...the brain is a massively parallel system, with 100 billion neurons all working away at the same time.” We need to educate people in how to use this capability, at a conscious level, to see patterns in systems or to use tools to assist in that recognition process.
- New ways of looking at the future and the development of a long term perspective.
- Techniology: The study of how technology is developed and how society influences what technologies get developed and how technology influences society (think sociology or ecology)
- There are several important concepts that must be taught: Chaos, Criticality and Emergence
- Decentralization: “As we enter the Era of Decentralization, there is an important educational challenge: How can we help people become intellectually engaged with the new types of systems and new ways of thinking that characterize this new era? To date, schools and other educational institutions have done little, if anything, to engage students with the idea of decentralization. Instead, they often perpetuate centralized explanations and approaches” comments Mitchel Resnick, Turtles, Termites, and Traffic Jams. “There is an apparent paradox in people's reactions to decentralized systems.” Resnick explains. “On one hand is the allure of decentralization. They are fascinated by systems that are organized without an organizer, coordinated without a coordinator.” On the other hand, “When people see patterns in the world, they intuitively assume that the patterns are created by lead or by seed.” That's the educational challenge. Sanders and McCabe write, “Complex adaptive systems, and models thereof, are characterized by distributed organizations or networks, whose parts all influence each other, either directly or through feedback loops, which continually evolve and adapt to accomplish overarching goals. This is in fundamental contrast to the top-down, hierarchical management structures found in most government organizations and in much of corporate America, where local experimentation, innovation and adaption are discouraged in favor of rigid bureaucratic rules and planning procedures. Simple cause and effect relationships do not characterize complex adaptive systems, and hence most of the conventional policy-planning tools currently used by decision-makers, both government and corporate, are inappropriate and ineffective. Complexity science offers new ways of understanding, thinking about and designing organizational systems that are capable of responding to and influencing complex nonlinear relationships. Understanding the local dynamics in a complex system can provide great insight into the behavior of the overall system and help identify key leverage points of change and transformation.”
- Trans-disciplinary approaches: “Complexity science is truly an interdisciplinary science” write Irene Sanders and Judith McCabe. “Adopting a systems view of the world meant that the questions were too big for any one discipline alone to answer. As scientists began looking for connections among the different types of complex systems, the boundaries between disciplines began to open. As a result, we are witnessing the integration of knowledge across the disciplines – the physical sciences, social sciences and the humanities. Insights about complex systems are emerging across a broad spectrum of fields - from physics, mathematics and computer science, to biology, oceanology, neuroscience, art and architecture. From health care to city planning, complexity science is creating a fundamental shift in the way we view the dynamics and interactions of complex systems.” This the real message of complexity science (in the sense of Marshall McLuhan's “The medium is the message.”) Complexity science is the “medium” through which multiple disciplines can together transcend their parochial views, and facilitate the emergence of a new world view.
The concept of intellective skills was developed by Shoshana Zuboff in In the Age of the Smart Machine: The Future of Work and Power:
…The thinking this operator refers to is of a different quality form the thinking that attended the display of action-centered skills. It combines abstraction, explicit inference, and procedural reasoning. Taken together, these elements make possible a new set of competencies that I call Intellective skills. As long as the new technology signals only deskilling – the diminished importance of action-centered skills – there will be little probability of developing critical judgment at the data interface. To rekindle such judgment, though on a new more abstract footing, a re-skilling process is required. Mastery in a computer-mediated environment depends upon developing intellective skills.
Interpersonal skills will have to include collaboration, decentralization, leadership, creativity, innovativeness, flexibility, openness, self development, tolerance for ambiguity and uncertainty and others I’m sure.
Management of Complexity
First, if it’s true complexity and not just an extremely complicated system, there is no known way to manage it. The best we can do now is to manage the effects of a complex system. For example, I mentioned earthquakes as being the result of a complex system in a critical state. We can’t manage the earthquake or even predict when an earthquake will occur or its magnitude. But, we do know from the history of an earthquake prone fault system that there is a probability of a certain magnitude of earthquake occurring. We then design buildings to survive that magnitude of earthquake.
No analysis of the BP and Toyota examples has been done to determine if the systems are complex or just extremely complicated. Or, what type of complex system they might be.
Tad Patzek, chairman of the Petroleum and Geosystems Engineering Department at the University of Texas, testified in Congress last month about the Gulf of Mexico oil spill. His comments included observations on broad, long-term trends in industry, government and academia. He reported, "Horrible things happen when complex technologies and procedures overtake humans, who service the technologies falsely assuming complete control.” Patzek asserts that it is a complex system with emergent properties. I doubt that this is true. I think it most likely a complex system in a critical state. I suspect its behavior is more like an earthquake (criticality) than a termite mound (emergence). It probably has some characteristics of both as the human and software subsystems (intelligent agents) in the system could exhibit some characteristics of emergence. However, his conclusions are valid: large events like this will occur, and we can't predict when they will occur or prevent them. Understandably, he never states this as directly as I did. How can you tell that to politicians or business people? (Remember the Queen in the croquet game?)
The software developers have tamed the complicatedness of large software programs such as operating systems by turning a hoard of software developers, who are not in the same organization, loose on the project and then by facilitating the development of a complex system of intelligent agents (individual programmers) with positive emergent properties. This requires a degree of openness, decentralization, flexibility, cooperation and loss of control not presently possible in most modern corporations. Nor, do we know how to apply these principles in systems other than software.
For a copy of this article, click here.
Wednesday, July 28, 2010
Complexity and Oil Spills from Platforms
Tad Patzek, chairman of the Petroleum and Geosystems Engineering Department at the University of Texas, briefed the Energy and Environment Subcommittee of the Energy and Commerce Committee on June 6, 2010. He reported, "Horrible things happen when complex technologies and procedures overtake humans, who service the technologies falsely assuming complete control. In this briefing, I attempt to explain the blowout of the BP exploratory well in terms of complexity, technology and science. I argue that organizational structures and human behavior have not kept pace with the complex technologies we — the engineers and scientists — have created.” I discussed my initial reaction to the report in an earlier blog, Complexity and the Gulf of Mexico oil Spill. He thought that perhaps the system of a platform oil well was complex – one constructed of intelligent agents with the property of emergent behavior. I agreed with him that I thought it was a complex system but speculated that it was best represented by a critical state system. I commented that analysis of historical data was required to determine what type of system it might be.
I did find some historical data on platform rigs. Because of Federal law regulating wells, the Bureau of Ocean Energy Management, U.S. Department of the Interior, keeps records of oil spills for platforms in the outer continental shelf (OCS). There were 251 spills reported of amounts greater than 50 barrels (bbl) from 1964 through 2009. A total of 261,052 bbl (10.96 million gallons) were reported spilled over this time period (if the spill was greater than 50 bbl). Beginning in 1970, reports were required for spills between 10 and 50 bbl. This second data base contains information on an additional 449 spills with a total 8,972 bbl. These two data bases can be found at:
www.mms.gov/incidents/Excel/Spills50bbl1964-2009.xls
www.mms.gov/incidents/Excel/SpillsbblCY1970to2009.xls
The data for 251 large oil spills on platforms in the OCS are shown in condensed form in the video here (http://www.archive.org/details/PlatformOilSpillHistory). It displays the water depth of the well on the vertical axis and the total amount spilled on the horizontal axis. The year is shown running below the horizontal axis. Both scales are logarithmic. This is the raw data that people in the field would see as events occurred. To make this video I used a Google Gadget, Motion Graph, and Google Docs.
In order to determine if the platform system is a critical state system, it’s instructive to look at the distribution of the magnitude of the spills. If it is a complex system in a critical state, that distribution should be an inverse power law.
The distribution is shown below for the large spill data set. Note that I eliminated all the data with exactly 50 bbl, the lower limited of the required reporting, as suspicious. And, to make the data easier to display on a graph and subsequent analysis, I eliminated data over 8,000 bbl (9 data points). As you will see later in this report, this makes little difference to the analysis.
First, you will note that this curve does look like an inverse power law (1/XN). Additionally, it has a very long tail, especially if you remember that there are 9 data points beyond the upper limit extending to 160,000 bbl. This indicates that, based on historical data, even though the majority of the spills are relatively small, there is a small but finite probability that a very large spill can occur.
Analysis of these data indicates that the distribution is indeed an inverse power law with a power of 1.56.
Performing the same analysis on the small spill (less than 50bbl) data set, over a different time period and different range of data, showed that the results were nearly identical. The power was -1.53 with a correlation coefficient of 0.94. This is a strong indication that something similar to a complex system in a critical state exists for this system as this suggests scale invariance, i.e. it looks the same at any scale.
However, that is hidden chaotic behavior if you look at the details and not just averages.
In this graph, all the data is shown as logarithms. On a log – log plot, a power curve shows up as a straight line. Note that the left edge of the envelope is almost exactly an inverse square relationship, how earthquakes present themselves. However, for spills larger than 500 bbl, the behavior appears chaotic. Unfortunately, the data indicates that for spills over 1,000 bbl, it’s just as likely that the spill will be 1,000 or 160,000 bbl.
There are several reasons why these data could indicate this apparent chaotic behavior:
You can download a copy of this article here.
For more information about complexity, read my article “1, 2, a Few, Many”. Or, clcik on the term complexity in the left sidebar of this blog.
I did find some historical data on platform rigs. Because of Federal law regulating wells, the Bureau of Ocean Energy Management, U.S. Department of the Interior, keeps records of oil spills for platforms in the outer continental shelf (OCS). There were 251 spills reported of amounts greater than 50 barrels (bbl) from 1964 through 2009. A total of 261,052 bbl (10.96 million gallons) were reported spilled over this time period (if the spill was greater than 50 bbl). Beginning in 1970, reports were required for spills between 10 and 50 bbl. This second data base contains information on an additional 449 spills with a total 8,972 bbl. These two data bases can be found at:
www.mms.gov/incidents/Excel/Spills50bbl1964-2009.xls
www.mms.gov/incidents/Excel/SpillsbblCY1970to2009.xls
The data for 251 large oil spills on platforms in the OCS are shown in condensed form in the video here (http://www.archive.org/details/PlatformOilSpillHistory). It displays the water depth of the well on the vertical axis and the total amount spilled on the horizontal axis. The year is shown running below the horizontal axis. Both scales are logarithmic. This is the raw data that people in the field would see as events occurred. To make this video I used a Google Gadget, Motion Graph, and Google Docs.
In order to determine if the platform system is a critical state system, it’s instructive to look at the distribution of the magnitude of the spills. If it is a complex system in a critical state, that distribution should be an inverse power law.
The distribution is shown below for the large spill data set. Note that I eliminated all the data with exactly 50 bbl, the lower limited of the required reporting, as suspicious. And, to make the data easier to display on a graph and subsequent analysis, I eliminated data over 8,000 bbl (9 data points). As you will see later in this report, this makes little difference to the analysis.
First, you will note that this curve does look like an inverse power law (1/XN). Additionally, it has a very long tail, especially if you remember that there are 9 data points beyond the upper limit extending to 160,000 bbl. This indicates that, based on historical data, even though the majority of the spills are relatively small, there is a small but finite probability that a very large spill can occur.
Analysis of these data indicates that the distribution is indeed an inverse power law with a power of 1.56.
Performing the same analysis on the small spill (less than 50bbl) data set, over a different time period and different range of data, showed that the results were nearly identical. The power was -1.53 with a correlation coefficient of 0.94. This is a strong indication that something similar to a complex system in a critical state exists for this system as this suggests scale invariance, i.e. it looks the same at any scale.
However, that is hidden chaotic behavior if you look at the details and not just averages.
In this graph, all the data is shown as logarithms. On a log – log plot, a power curve shows up as a straight line. Note that the left edge of the envelope is almost exactly an inverse square relationship, how earthquakes present themselves. However, for spills larger than 500 bbl, the behavior appears chaotic. Unfortunately, the data indicates that for spills over 1,000 bbl, it’s just as likely that the spill will be 1,000 or 160,000 bbl.
There are several reasons why these data could indicate this apparent chaotic behavior:
- We don’t have a large enough data set. Given the extremely long tail of this distribution, the apparent chaotic behavior could be the result of sampling error. In which case the last statement about probabilities would be invalid.
- There could be a problem in estimating spills over 1,000 bbl.
- There are variables in the response to the spill that could turn a 1,000 bbl spill to 160,000 bbl.
- The specific conditions of the large spills are significantly different.
- The data set includes the performance of many platforms over a long time span. Behavior of the different systems above 1,000 bbl could be different. Behavior of systems could change over time.
- The system becomes chaotic under the conditions of a large spill. The above curve is reminiscent of the logistic map that demonstrates regions of chaotic behavior.
- Cause and effect are disconnected. As Buchanan wrote, “...that the greatest of events have no special or exceptional causes.”
- A power law can only be generated by some process that is steeped in history. “...the future emerges out of a string of accidents, each leaving its indelible trace on the course of events,” Buchanan commented.
- The complex system in a critical state exhibits the property of self similarity or scale invariance. It looks the same at all scales. Again, referring to Buchanan, “...the Gutenberg-Richter power law says that the process behind earthquakes is scale invariant, and the unavoidable implication is that the great quakes are no more special or unusual than the tiny shudders constantly rippling beneath our feet.”
- Systems that exhibit self organizing critically are ubiquitous.
- A new type of statistics has to be applied to complex systems in a critical state. They do not follow normal, i.e. Gaussian, statistics. Large events are a lot more likely to occur than a normal distribution would predict. Nassim Nicholas Taleb wrote in The Black Swan, “A Black Swan is a highly improbable event with three characteristics: It is unpredictable; it carries a massive impact; and after the fact, we concoct an explanation that makes it appear less random, and more predictable than it was.”
- The results of attempts to “improve” the behavior of a complex system in a critical state are unpredictable and can actual make the performance worse. Since the 1890's the forest service had a policy of stamping out a forest fire as quickly as possible. The logic behind that policy was that it was better to catch a fire small before it became big. In 1998 the Yellowstone fire consumed almost 800,000 acres, 36% of the park, and the number and intensity of forest fires was increasing everywhere. Also in 1998, the geologists Bruce Malamud, Gleb Morein, and Donald Turcotte of Cornell University gathered extensive data on forest fires. They demonstrated that the number of acres consumed in a forest fire followed an inverse power law of 2.48. A forest is a complex system in a critical state, at least for forest fires. The actions of the forest service were inadvertently making the critical state more dangerous by not allowing the small naturally occurring fires to keep the system in its natural state.
You can download a copy of this article here.
For more information about complexity, read my article “1, 2, a Few, Many”. Or, clcik on the term complexity in the left sidebar of this blog.
Wednesday, July 21, 2010
An Economy of Grinds
In his column for the New York Times (7/12/10), David Brooks introduces the concepts of princes and grinds in an economy. Princes are, according to Brooks, someone you’d like to sit next to at a luncheon. “If you go to business conferences, you know that at lunch it is definitely better to be seated next to a prince than a grind. Princes, who can be male or female, are senior executives at major corporations. They are almost always charming, smart and impressive. They’ve read interesting books. They’ve got well-rehearsed takes on the global situation. They can drop impressive names as they tell you about their visits to the White House, Moscow or Beijing. If you’re having lunch or dinner with a prince, you’re going to have a good time.” Grinds are people who are boring. “Grinds, on the other hand, tend to have started their own company or their own hedge fund. They’re often too awkward to work in a large organization and too intense to work for anybody but themselves. Over lunch, they can be socially inert. You try to draw them out by probing for one or two subjects of interest to them. But as often as not, you find yourself playing conversational ping-pong with a master of the monosyllabic response. Every once in a while you’ll run into one who can’t help but let you know how much smarter he is than you or anybody else in the room. Sitting at this lunch is about as pleasant for him as watching a cockroach crawl up his arm. He’d much rather be back working in front of his computer screen.”
But, he goes on to say, Princes often turn out to be untrustworthy and grinds turn out to build the economy through small business startups and entrepreneurial efforts that create jobs. He correctly deduces that unfortunately, the princes (who caused the recession) are doing fine now, but the grinds are not. The grinds are suffering. “The big companies are posting excellent earnings. They’re sitting on mountains of cash. The aspiring grinds, meanwhile, are dead in the water. Small businesses are not growing. They are not hiring. They are struggling to stay alive.”
Brooks spends the rest of the essay describing the differences between the princes and the grinds.
I agree with almost all of what he writes. My two exceptions are:
1. Including hedge fund creators in the Grinds. They’re still just manipulating money, not creating anything that serves a public good or creates anything of extrinsic value.
2. Not mentioning innovation. It’s innovation that creates real economic growth and opportunities for jobs and capital investment. It’s innovation that both of his types of people have forgotten.
But, he goes on to say, Princes often turn out to be untrustworthy and grinds turn out to build the economy through small business startups and entrepreneurial efforts that create jobs. He correctly deduces that unfortunately, the princes (who caused the recession) are doing fine now, but the grinds are not. The grinds are suffering. “The big companies are posting excellent earnings. They’re sitting on mountains of cash. The aspiring grinds, meanwhile, are dead in the water. Small businesses are not growing. They are not hiring. They are struggling to stay alive.”
Brooks spends the rest of the essay describing the differences between the princes and the grinds.
I agree with almost all of what he writes. My two exceptions are:
1. Including hedge fund creators in the Grinds. They’re still just manipulating money, not creating anything that serves a public good or creates anything of extrinsic value.
2. Not mentioning innovation. It’s innovation that creates real economic growth and opportunities for jobs and capital investment. It’s innovation that both of his types of people have forgotten.
Tuesday, July 6, 2010
Blogs, Wikis, Podcasts and Other Powerful Web Tools for Classrooms
"Tim Berners-Lee had a grand vision for the Internet when he began development of the World Wide Web in 1989. "The original thing I wanted to do,' Berners- Lee said, 'was make it a collaborative medium, a place where we [could] all meet and read and write'." This the opening two sentences for this really valuable book. The book is dated somewhat by the constant development and diffusion of new social technologies. However, for a teacher, or anyone for that matter, that's interested in learning about the fundamental tools, that sometimes are called collectively, Web 2.0, this is a great book. He not only describes the tools and their applications, he specifically tells the reader how to get the tool and use in simple step by step language.
So why get all excited about these new tools? Aren't they just incremental changes to the existing ways of doing thinks? Not at all. These tools collectively are disruptive to our ways of teaching, learning, working and living.
"No matter how you look at it, we are creating what author Douglas Rushkoff calls a "Society of Authorship' where every teacher, every student, every person will have the ability to contribute ideas and experiences to the larger body of knowledge that is the Internet. And, in doing so, Rushkoff says, we will be writing the human story, in real time, together, a vision that asks each of us to participate."
McLuhan warned us 40 years ago, that the new information technologies were going to alter our perception and ways of thinking. " ... William D. Winn, Director of the learning Center at the University of Washington, believes that years of computer use creates children that 'think differently from us. They develop hypertext minds. They leap around. It's as though their cognitive structures were parallel, not sequential.' In other words, today's students may not be well-suited to the more linear progression of learning that most educational systems employ."
Unlike their students, most teachers have not been brought up with these altered ways of thinking. Many try, but are like foreigners who speak with an accent. And, the educational institutions are even slower to change.
The toolbox that this book is based on is:
1. Weblogs (or blogs)
2. Wikis
3. Rich Site Summary (RSS)
4. Agregators
5. Social Bookmarking
6. Online Photo Galleries
7. Audio/video-casting
"In large measure, it is blogs that have opened up the Read/Write frontier for content creation to the web, and millions of people have been quick to take advantage of the opportunity. Remember, a new blog is being created every second, and that shows no sign of slowing down." Blogging in its truest from engages people in a "process of thinking in words, not simply accounting of the days events or feelings. '
" ... Fernette and Brock Eide's research shows that blogging in its true form has a great deal of potential positive impact on students. They found that blogs can
• Promote critical and analytical thinking
• Be a powerful promoter of creative, intuitive, and associational thinking
• Promote analogical thinking
• Be a powerful medium for increasing access and exposure to quality information
• Combine the best of solitary reflection and social interaction."
Richardson takes time to teach how teachers can help students identify the quality of the information that get from the Internet. He correctly points out that this skill is part of the new literacy,
Wikis provide easy collaboration for all. Perhaps the most successful wiki to date is Wikipedia, an encyclopedia based on the wiki platform. "the first wiki was created by Ward Cunningham in 1995, who was looking to create an easy authoring tool that might spur people to publish. And the key word here is easy, because plainly put, a wiki is a website where anyone can edit anything at any time they want."
RSS, rich text summary or really simple syndication, is a technology that allows you to syndicate your work just like newspaper columnists do, so that is published around the world. It also allows you to draw information from the web around the world to you on topics that interest you, automatically, and almost instantaneously.
Aggregators are tools used with RSS and other forms of syndication to aggregate information for you.
Social bookmarking is a way the web “learns” from us. When we bookmark a web site or web page, and categorize it, we are telling others that this is valuable information. It’s the Dewey Decimal System of the Internet.
Online photo-galleries are a way to share photographs with others.
Audio/video-casting are not broad casting, but narrow casting, providing the equivalent of radio and TV over the Internet. Although services like You Tube have millions of viewers for a video.
This is a great book for a beginner to learn about and how to use web 2.0 tools in the classroom, and in your professional life.
Blogs, Wikis, Podcasts, and Other Powerful Web Tools for Classrooms
Will Richardson
Corwin Press, 2006, 149 pages
The authors blog is a rich source of material for understanding web 2.0 tools for education
So why get all excited about these new tools? Aren't they just incremental changes to the existing ways of doing thinks? Not at all. These tools collectively are disruptive to our ways of teaching, learning, working and living.
"No matter how you look at it, we are creating what author Douglas Rushkoff calls a "Society of Authorship' where every teacher, every student, every person will have the ability to contribute ideas and experiences to the larger body of knowledge that is the Internet. And, in doing so, Rushkoff says, we will be writing the human story, in real time, together, a vision that asks each of us to participate."
McLuhan warned us 40 years ago, that the new information technologies were going to alter our perception and ways of thinking. " ... William D. Winn, Director of the learning Center at the University of Washington, believes that years of computer use creates children that 'think differently from us. They develop hypertext minds. They leap around. It's as though their cognitive structures were parallel, not sequential.' In other words, today's students may not be well-suited to the more linear progression of learning that most educational systems employ."
Unlike their students, most teachers have not been brought up with these altered ways of thinking. Many try, but are like foreigners who speak with an accent. And, the educational institutions are even slower to change.
The toolbox that this book is based on is:
1. Weblogs (or blogs)
2. Wikis
3. Rich Site Summary (RSS)
4. Agregators
5. Social Bookmarking
6. Online Photo Galleries
7. Audio/video-casting
"In large measure, it is blogs that have opened up the Read/Write frontier for content creation to the web, and millions of people have been quick to take advantage of the opportunity. Remember, a new blog is being created every second, and that shows no sign of slowing down." Blogging in its truest from engages people in a "process of thinking in words, not simply accounting of the days events or feelings. '
" ... Fernette and Brock Eide's research shows that blogging in its true form has a great deal of potential positive impact on students. They found that blogs can
• Promote critical and analytical thinking
• Be a powerful promoter of creative, intuitive, and associational thinking
• Promote analogical thinking
• Be a powerful medium for increasing access and exposure to quality information
• Combine the best of solitary reflection and social interaction."
Richardson takes time to teach how teachers can help students identify the quality of the information that get from the Internet. He correctly points out that this skill is part of the new literacy,
Wikis provide easy collaboration for all. Perhaps the most successful wiki to date is Wikipedia, an encyclopedia based on the wiki platform. "the first wiki was created by Ward Cunningham in 1995, who was looking to create an easy authoring tool that might spur people to publish. And the key word here is easy, because plainly put, a wiki is a website where anyone can edit anything at any time they want."
RSS, rich text summary or really simple syndication, is a technology that allows you to syndicate your work just like newspaper columnists do, so that is published around the world. It also allows you to draw information from the web around the world to you on topics that interest you, automatically, and almost instantaneously.
Aggregators are tools used with RSS and other forms of syndication to aggregate information for you.
Social bookmarking is a way the web “learns” from us. When we bookmark a web site or web page, and categorize it, we are telling others that this is valuable information. It’s the Dewey Decimal System of the Internet.
Online photo-galleries are a way to share photographs with others.
Audio/video-casting are not broad casting, but narrow casting, providing the equivalent of radio and TV over the Internet. Although services like You Tube have millions of viewers for a video.
This is a great book for a beginner to learn about and how to use web 2.0 tools in the classroom, and in your professional life.
Blogs, Wikis, Podcasts, and Other Powerful Web Tools for Classrooms
Will Richardson
Corwin Press, 2006, 149 pages
The authors blog is a rich source of material for understanding web 2.0 tools for education
Complexity and Gulf of Mexico Oil Spill
"Tad Patzek, chairman of the Petroleum and Geosystems Engineering Department at the University of Texas, testified in Congress last month about the Gulf of Mexico oil spill. His comments included observations on broad, long-term trends in industry, government and academia; excerpts of his prepared remarks on those issues are below.
Patzek covered a range of other topics, including recommendations on specific avenues of research for improved drilling and cleanup methods."
Austin American Statesman, 7/4/10
Dr. Patzek briefed the Energy and Environment Subcommittee of the Energy and Commerce Committee on June 6, 2010
He reported, "Horrible things happen when complex technologies and procedures overtake humans, who service the technologies falsely assuming complete control.
In this briefing, I attempt to explain the blowout of the BP exploratory well in terms of complexity, technology and science. I argue that organizational structures and human behavior have not kept pace with the complex technologies we — the engineers and scientists — have created.
Given the structural changes in the industry, academia and government, this tragedy has been at least 20 years in the making. It seems that the human inability to grasp and execute the complex steps of a deepwater drilling procedure led to the tragic outcome."
This briefing is well worth reading as he introduces the concept of complexity to explain accidents in large, complicated systems of man and technology. This is an extremely important area for research and understanding.
I think his assertion that BP exploratory well Mississippi Canyon Block 252-01 is a complex system is valid, although he does not prove it so. And, without good historical data on wells of this type, it is difficult to determine what type of complex system it is.
Patzek asserts that it is a complex system with emergent properties. I doubt that this is true. I think it most likely a complex system in a critical state. I suspect its behavior is more like an earthquake (criticality) than a termite mound (emergence). (It probably has some characteristics of both as the human (intelligent agents) in the system could exhibit some characteristics of emergence.)
However, his conclusions are valid: large events like this will occur, and we can't predict when they will occur or prevent them. Understandably, he never states this as directly as I did. How can you tell that to politicians or business people? (See 1, 2, a Few, Many for more information)
Using earthquakes as an example: How do we deal with them? We build structures that will sustain their integrity through high impact, low probability events. We research the history and determine the probability of an event in different geological zones. And, while we don't have an early warning system for earthquakes, in many cases we do for tsunamis. We monitor the ocean and warn of a tsunamis presence and path.
Patzek is correct in discussing the philosophical impacts of complexity science, and how our technology has informed us, leaving us scotomas in our perception of the systems we are constructing.
It's time to start perceiving, recognizing and understanding the complex systems we live in. They are everywhere once you know what to look for.
Patzek covered a range of other topics, including recommendations on specific avenues of research for improved drilling and cleanup methods."
Austin American Statesman, 7/4/10
Dr. Patzek briefed the Energy and Environment Subcommittee of the Energy and Commerce Committee on June 6, 2010
He reported, "Horrible things happen when complex technologies and procedures overtake humans, who service the technologies falsely assuming complete control.
In this briefing, I attempt to explain the blowout of the BP exploratory well in terms of complexity, technology and science. I argue that organizational structures and human behavior have not kept pace with the complex technologies we — the engineers and scientists — have created.
Given the structural changes in the industry, academia and government, this tragedy has been at least 20 years in the making. It seems that the human inability to grasp and execute the complex steps of a deepwater drilling procedure led to the tragic outcome."
This briefing is well worth reading as he introduces the concept of complexity to explain accidents in large, complicated systems of man and technology. This is an extremely important area for research and understanding.
I think his assertion that BP exploratory well Mississippi Canyon Block 252-01 is a complex system is valid, although he does not prove it so. And, without good historical data on wells of this type, it is difficult to determine what type of complex system it is.
Patzek asserts that it is a complex system with emergent properties. I doubt that this is true. I think it most likely a complex system in a critical state. I suspect its behavior is more like an earthquake (criticality) than a termite mound (emergence). (It probably has some characteristics of both as the human (intelligent agents) in the system could exhibit some characteristics of emergence.)
However, his conclusions are valid: large events like this will occur, and we can't predict when they will occur or prevent them. Understandably, he never states this as directly as I did. How can you tell that to politicians or business people? (See 1, 2, a Few, Many for more information)
Using earthquakes as an example: How do we deal with them? We build structures that will sustain their integrity through high impact, low probability events. We research the history and determine the probability of an event in different geological zones. And, while we don't have an early warning system for earthquakes, in many cases we do for tsunamis. We monitor the ocean and warn of a tsunamis presence and path.
Patzek is correct in discussing the philosophical impacts of complexity science, and how our technology has informed us, leaving us scotomas in our perception of the systems we are constructing.
It's time to start perceiving, recognizing and understanding the complex systems we live in. They are everywhere once you know what to look for.
Subscribe to:
Posts (Atom)