Value Is Where You Find It

Here is the story as it was related to me.

More than a decade ago, the Pfizer pharmaceutical company was developing a new drug to treat hypertension. Unfortunately, the phase one clinical trials of sildenafil citrate were so disappointing that the researchers decided to cut their losses and end the trials early. They asked the participants in the trial, who were predominantly older men, to return the unused drug samples.

Many of them refused.

Then someone broke into the clinic from which the trials were being run and made off with some of the remaining sildenafil samples.

The researchers knew something was up, but what? They started interviewing the participants, and that is when they discovered that their experimental drug, which would eventually be marketed under the name Viagra, had an unintended side effect.

I’d like to think that once the researchers realized the implications, more than one of them swung by the local Mercedes dealership on the way home from work.

Sometimes the value in your product isn’t where you think it is.

About the time those clinical trials began, I started working for the research and development arm of a large high-tech equipment manufacturer. During the next decade, I worked on seven different development projects, for as many as six release cycles, all but one of which led to a successful commercial product. All of those projects, even the one that was cancelled (maybe especially the one that was cancelled), produced useful collateral beyond the product itself. This collateral included reusable code, useful design patterns, and best practices, not to mention the experience stored inside the heads of the developers.

Seldom if ever did I see this R&D organization try to leverage this collateral beyond breaking up high-performance teams and reassigning individual members to unrelated projects where for the most part they were encouraged to shut up and get busy. It’s hard to imagine a poorer use of human resources, or a worse job of leveraging the full value of a project.

Sometimes the value is in the people.

When I finally resigned from that organization, the predominant emotion I felt was gratitude. My time there was a tremendous learning experience. I worked on many different unrelated projects, learned to come up to speed quickly, worked with a lot of great people, got along well with a broad range of personalities, exercised a variety of skills in several different roles, developed expertise in several different processes, languages, and tool chains, was exposed to lots of architectures and design patterns, and learned a lot about the various product lines and the business side of the house. I could not have asked for a better training program, and it has served me well in my subsequent work.

When it came time for me to literally walk out the door of that organization, I had to hunt down an administrative assistant to take my badge and walk me out of the building. No exit interview, no meeting with anyone from HR, no sign of any manager, not even, as had happened in the past, security guards to escort me out. If I didn’t have a sturdy ego, I would have had some self-doubts.

I’ve given a lot of thought about why this organization might have failed to leverage potentially valuable collateral, whether it was in products or people.

The use of silos in the organization may have made it difficult to leverage collateral from a project in one silo to one in another silo. Silos made it much harder to transfer staff and other resources, temporarily or permanently. It also made it difficult to leverage the experience of engineers who had been successfully transferred out of a silo back to the legacy systems on which they had once worked when customer problems occurred. I remember being specifically forbidden by one department head to consult on a complex legacy product for which I was one of the few remaining developers. Good for his silo, bad for the other, and definitely bad for the customer.

A forced ranking system may have rewarded the hoarding of information and technology. It introduced a natural selection towards producing results efficiently in the short term, without regard for any long term consequences. It punished the reporting of negative results, ignoring the fact that negative results are still results. Although learning what doesn’t work is what we call “experience”, admitting that you tried something that didn’t work made it a little more likely you would be in the next wave of layoffs.

For sure the trend of treating engineers as a fungible commodity didn’t help.

Exploiting collateral, whether it’s people, technology, or unintended consequences, may not be cheap. Merely identifying it can be expensive. Committing to using it carries risk. Exploits like code reuse frequently fail for legitimate reasons. The exploitation may only pay off in the long run, and so requires a long term view.

Engineers frequently prefer to reinvent rather than reuse. It takes some experience to realize that there is no glory in solving a problem that has already been effectively solved by someone else. Some programming languages make reuse high effort. Some software designs make it difficult.

Sildenafil citrate was an unusual failure. Most negative results do not give rise to a whole new, immensely profitable, industry. But I wish we did a better job extracting the full value of our work, whether it’s in the technology, the people, the processes, or the products themselves.

People Are Smart In Different Ways

I learned that there were different ways to be smart long before I met Mark Uris. Mark had an M.B.A. and a background as a buyer for a major retail chain, when for some reason he decided to follow a different career path and become a software developer. He was not the best software developer I'd ever met. In fact, he pretty much sucked at it. I'm sure he knew it, too.

But the great thing about information technology is that it exploits a broad range of skill sets like nobody's business. Mark was no exception. When he went into management and ended up with responsibility for purchasing everything from individual disk drives to entire server farms, he really came into his own. There was no one as good as he was at dealing with vendors, negotiating prices, and somehow keeping everyone on both sides of the table happy. Well, for sure, our upper management was happier than our vendors, but even the vendors didn't have too much to complain about. Mark understood the principle of Win-Win. He also managed his group of system administrators with skill, good goofy humor, and humanity.

Mark may have had some notoriety because of his father, Leon Uris, author of Exodus and other epic novels. But to me, Mark was just the guy down the hall that knew where the spare parts were, who knew who would fix my workstation (and more importantly, when), the guy who saved the organization a ton of money while getting us engineers some really cool hardware, and who could be counted on for the latest bad joke.

Mark Jay Uris died of cancer on March 29th at age 56. He was survived by his wife, two daughters, and four siblings. He was a reliable friend and a valued colleague.

When you're young, you live with this misconception that not only are you immortal, but so are your friends. It comes as a shock when you discover for the first time that this isn't the case. It doesn't get any easier the older you get.

I am already missing him.

Implications of Memory Models (or Lack of Them) for Java, C, and C++ Developers

I'll be giving a talk on Implications of Memory Models (or Lack of Them) for Java, C, and C++ Developers at Avaya Inc. at 120th and Pecos in Westminster, Colorado, on Thursday, April 5, from 1500 to 1600, in the auditorium.

(Chip Overclock) will discuss why the Double Checked Locking Pattern, which he used to implement lazy singetons in several commercial products, doesn’t work reliably on modern processors. He will describe how he constructed an object in C++ on a single-core hyperthreaded processor only to find upon using that object that its virtual table apparently hadn’t been initialized. Modern microprocessors implement memory models such that what you may think is happening maybe isn’t happening when you think it is. Wackiness ensues. More restrictive approaches to multi-threaded design are necessary, even on single-core processors.

It's not open to the public, but the slides (which I updated yesterday, for those of you that can attend) are available on the Digital Aggregates web site here.