Reading Robert Austin's book Measuring and Managing Performance in Organizations (Dorset House, 1996), was such an eye-opening experience for me that it has merited mention in this blog several times before this, most prominently in In Defense of Misbehavior. Once you understand measurement dysfunction, in which the performance measurement of individuals almost inevitably drives dysfunction into an organization, you will never again see the world in quite the same way.
The gist of it is this: an old axiom of business is "You cannot manage what you do not measure." But in his Ph.D. thesis, Austin, formerly an executive with Ford Motor Company and now on the faculty of the Harvard Business School, applies agency theory, a form of game theory, to show that in all but the most trivial of situations, it is impossible to measure all metrics applicable to an individual's performance. There will always be aspects of performance that cannot be measured. When rewarded (or punished) based on the metrics you do measure, the person being measured will be highly incentivized to game the system by slacking off on the those unmeasurable metrics. A classic example are call center agents hanging up on customers in the middle of calls to meet their quotas.
Austin's work has been cited by Tom DeMarco ("Mad About Measurement") in Why Does Software Cost So Much? (Dorset House, 1995), and by Joel Spolsky ("Measurement") in Joel on Software (Apress, 2004). It compelled me to get a copy of his dissertation (Robert Daniel Austin, Theories of measurement and dysfunction in organizations, Carnegie Mellon University, University Microfilm, #9522945, 1995), and even put a talk together on the topic which I've given several times. Given the recent horrifying cover story in BusinessWeek, "Managing by the Numbers: How IBM improves quality by tracking employees' every move" (September 8, 2008) , this topic is bound to become popular blog fodder once again.
But that's not what I'm here to talk about. I'm here to talk about why free markets don't work.
A few years ago, a large telecommunications equipment manufacturer decided to save money by eliminating all desktop PC support and having their product developers support their own predominantly-Windows desktop computers. This was done, of course, as a money savings move.
Did they save money? On paper sure, because they eliminated a bunch of IT staff from the bottom line. In reality, who the hell knows? All they really did was move their desktop IT support from where its cost could be measured to where it could not be measured. Most of the product line of this manufacturer was Linux-based, so the developers, who were earning hefty salaries for their Linux and telecommunications expertise, were not Windows experts. Most of their Windows support experience came from buying a home PC from BestBuy.
I suspect those developers were the most expensive IT support staff on the planet. But its all good because those costs weren't measured. Nor was the loss in productivity, the reduction in efficiency and quality in the work environment, and the moral hit to both the developers and the IT staff that came with the assumption that those two skill sets were interchangeable. I mean, really, it's just all computers, right? How hard can it be?
Sure, you could see this as a form of measurement dysfunction: some VP got a bonus by gaming the system. This is how people most often think of measurement dysfunction, as a more general term for incentive distortion. But this can also be seen in another way, not inadvertent dysfunction caused by the measurement of performance of individuals, but in a deliberate application so as to move costs to where they could not be measured. It's measurement dysfunction as management strategy.
Pollution is another form of measurement dysfunction. Companies realize cost savings by not dealing with the environmental costs of manufacturing. That's because, without government regulation, environmental costs are not borne by the polluter. The costs are pushed off onto the downstream neighbors of the manufacturer, or into the surrounding community, even onto future generations. As long as you just think locally and take a short term view, there is a strong incentive to move environmental costs to where they cannot be measured. Offshoring is just another form of this: moving environmental, or even societal, costs to where they are not measured.
In his book, Best Business Crime Writing of the Year (Random House, 2002), James Surowiecki collects magazine and newspaper articles on the spate of corporate crime that lead to the demise of Enron, WorldCom, Tyco, and other best-in-class companies. The actions of the executives described in these articles illustrate another form measurement dysfunction. Federal, state and local laws are a form of measurement. There is a strong (given the huge dollar amounts involved, I'm tempted to say irresistible) incentive to game the system by testing the limits and loopholes of those laws in an effort to derive revenue from areas for which the results of actions are not measured. I suspect that many of these executives didn't even realize they had committed a crime until the men with badges, guns, and handcuffs showed up with warrants, subpoenas, and writs. The Sarbanes-Oxley Act is an attempt to fix this dysfunction by measuring a metric that had not been measured before.
Measurement dysfunction is much more broadly applicable than just as incentive distortion. That's why it's known by other names in other contexts: unintended consequences, perverse incentives, and moral hazard are some I know about. With all due respect to my libertarian friends (and I have a lot of them), I argue that measurement dysfunction is why free markets don't work.
There will always be an economic incentive for participants in a truly free market to dump costs into or derive revenue from any gray area in which there is no measurement. And in an information economy, there are a lot of gray areas. That's because the measurement of all dimensions of market costs is impossible. Adam Smith described an "invisible hand" that guides markets to the most efficient allocation of resources. But for many metrics of market cost, there is no hand at all, invisible or otherwise. I'm not arguing for controlled markets. But I am arguing for regulated markets: that a significant (and perhaps unfortunate) amount of government regulation is necessary to act as a visible hand, forcing market participants to bear the full costs of their transactions.
In the realm of performance measurement of individuals, Austin argues against extrinsic motivation, for example bonuses tied to specific goals, because of the measurement dysfunction this produces. Instead, intrinsic motivation, for example corporate culture, causes individuals to do the right thing independent of incentives. If intrinsic motivators worked at the corporate level, free markets could work as well. But it's been argued that corporations are psychopathic because, devoid of any other legal restraint, their sole purpose is to maximize shareholder value.
I'm not saying that corporations are evil. I'm the founder and CEO (and janitor) of a corporation myself. But I am saying that it would be foolish not to acknowledge the measurement dysfunction that occurs in a truly free market.