A Swiss Cheese of Errors

 In 2021, an F-35B fighter jet rolled off the front of the aircraft carrier HMS Queen Elizabeth during a failed take off. "Rolled" is the probably the right term, as British carriers do not use a catapult like U.S. carriers. The pilot ejected and landed on the flight deck with only minor injuries.

It was discovered later that a protective cover - part of the "red gear" because of its color - over the left engine intake had mistakenly been left in place. It was sucked into the compressor inlet of the single center-mounted jet engine, reducing power to where it was insufficient for take off.

The U.S. and its allies recovered the carcass of the F-35B. Which is good, because if they hadn't, somebody else would have.

As you would expect after totaling a bleeding edge US$80M aircraft, part of the enormously expensive and troubled U.S. F-35 program, there was a lengthy post mortem report written. I read a short (about forty pages) summary and analysis of this report this morning by Aerosurrance, a U.K. based aviation consultancy.

There is a concept in the study of organizational and complex systems failures - which is a hobby of mine that I've written about here before - called the Swiss cheese model. This is where "holes" in redundant layers of safety systems and checks (because no such system is perfect) just happen to line up at exactly the wrong time to produce a catastrophic failure.

This report was like that: maintenance crews were overworked, fatigued, and under staffed; procedures were insufficient or not followed; poor design of the red gear; no sharing of similar failures, four of which had occurred before in the U.S.; etc. (And after having previously read several long ProPublica articles about failures in the U.S. Navy that were in part due to similar issues with overwork and fatigue, I'm seeing a pattern.)

While reading this article, it occurred to me that we already have a term for Swiss cheese types of failures, and have had for eons, we just tend not to use it when it the result is tragic.

We call them "a comedy of errors".

Boom Town

In 1969, a forty-kiloton nuclear bomb was detonated underground in Colorado, near the town of Parachute, between Glenwood Springs and Grand Junction, west of Denver and a little south of what is now Interstate 70. The test shaft was over 8,400 feet deep. It was a test to see if small nuclear devices could free up natural gas deep underground, part of Project Plowshare. Edward Teller, one of the developers of the H-bomb, was there during the test.

In 1973, three thirty-kiloton nuclear bombs were detonated underground in Colorado, in Rio Blanco County, in the northwestern part of the state. It was another Project Plowshare effort to find peaceful uses for nuclear bombs.

These efforts to extract natural gas were a partial success. Partial because the gas was so radioactive it could not be sold, so was instead burned off.

As a result, Colorado passed an amendment to its state constitution such that approval for the detonation of a nuclear device in the state has to pass a state-wide popular vote. It is the only state in the U.S. with such a requirement.

This all happened before the Spousal Unit and I moved to the Denver area in 1989. But that's not the only adventure with radioactivity you may find in Colorado.

Given the depth of the H-bomb detonation near Parachute, the site is probably not nearly as radioactive as the site of the old Rocky Flats Nuclear Weapons Plant, between Denver and Boulder Colorado, just a few minutes drive from my home, and which I drove right past when I commuted to Boulder every day to work in the early 1990s. Rocky Flats - somewhat euphemistically - manufactured triggers for nuclear bombs; the trigger for a nuclear (fusion) bomb is an atomic (fission) bomb. Rocky Flats was an EPA superfund site for a long long time after it closed in 1992, following an FBI raid.

Radioactivity is a natural phenomena here in Colorado, thanks to decaying uranium ore underground that creates radioactive radon gas. Like a lot of folks in our neighborhood, we have radon mitigation in our home: a fan that runs 24x7 that pulls air (and presumably radon) out of our crawl space and exhausts it above the house.

Huge piles of mine tailings around Colorado mountain towns contain a lot of uranium ore, which was just a waste product when it was originally dug out during the gold and silver mining era. No one at the time had any idea they were creating an environmental hazard that would last, for all practical purposes, forever.

For reasons unrelated to any of this, my tiny little company owns a couple of geiger counters. One day, the Spousal Unit asked me if my watch with a tritium dial gave off enough radiation to be detectable. Good question. Tritium is a radioactive isotope of hydrogen. Its radioactive decay produces helium, an electron (a.k.a. beta particle), and an electron anti-neutrino. Tritium watch dials have hour and hand markers that are tiny vials containing tritium gas and a phosphorescent compound. The beta particles from the decaying tritium excite the phosphor, making it glow. No external light source is needed, although in time all the tritium will have decayed and the vial will stop glowing. Gun sights for use at night may use this mechanism too.

I swept one of my geiger counters close over the watch, and got a reading of normal background radiation. Beta particles from tritium decay are so feeble energetically that they typically can't penetrate more than about a quarter inch of air, much less human skin. Most probably can't make it past the wall of the vial.

Then, I happened to sweep the geiger counter across one of my other watches, where it went completely nuts. It turns out an old mechanical French Army surplus watch from the 1960s that I own has a radium dial. I had no idea; that wasn't in the original description when I purchased it. Radium is a decay product of naturally occurring uranium, and radon gas is in turn a decay product of radium. As it decays, radium produces ionizing radiation in the form of helium nuclei (a.k.a. alpha particles) and gamma radiation.

That watch dial was hot. The watch now resides in a lead-lined envelope on a shelf in the basement, along with some samples of uranium ore.

In these parts, it's no accident that we all have a healthy glow.

References

Westword, "The Boom Years", 2023-07-30

Wikipedia, "Rocky Flats Plant", 2023-06-19

Model Collapse

A few decades ago I was working at the National Center for Atmospheric Research, a national lab in Boulder Colorado sponsored by the National Science Foundation. Although our missions were completely different, we had a lot in common operationally with the Department of Energy labs, like Los Alamos and Lawrence Livermore, regarding supercomputers and large data storage systems, so we did a lot of collaboration with them.

I had a boss at NCAR that once remarked that the hidden agenda behind the DoE labs was that they were a work program for physicists. Sometimes, often without much warning, you need a bunch of physicists for a Manhattan Project kind of activity. And you can't just turn out experienced Ph.D. physicists at the drop of a hat; it takes years or even decades. So for reasons of national security and defense policy you have to maintain a pipeline of physicist production, and a means to keep them employed and busy so that they can get the experience they need. Then you've got them when you need them.

This always seemed very forward thinking to me. The kind of forward thinking you hope someone in the U.S. Government is doing.

It came to me today that this is the same issue in the screen writers' and actors' strike.

Machine learning (ML) algorithms, of which Large Language Models (LMMs) are just one example, need almost unbelievably large amounts of data to train their huge neural networks. There is a temptation to use the output of ML models to train other ML models because it's relatively cheap and easy to create more input data, where as expensive humans can take a long time to do it. But training an ML model with the output of another ML model leads to an effect called "model collapse".

I mentioned an article on VentureBeat (which cites an academic paper) on this topic in a prior blog article. The VentureBeat article by Carl Franzen provides the following metaphor:

If you trained an ML model to recognize cats, you could feed it billions of "natural" real-life examples of data about blue cats and yellow cats. Then if you asked it questions about cats, you would get answers containing blue cats, and yellow cats, and maybe even occasionally green cats.

But suppose yellow cats were relatively rarely represented in your data, whether they were rare in the real world or not. Mostly then you would get answers about blue cats, almost never yellow cats, and rarely if ever green cats.

Then you started training your new improved ML model on the output of the the prior ML model. The new "synthetic" data set would dilute out all the examples of yellow cats. Eventually you would have model that didn't even recognize yellow cats at all.

This is one example of model collapse: the ML model no longer represents the real-world, and cannot be relied upon to deliver accurate results.

This is what will happen if you eliminate the human elements from your screenwriting or acting (or software development), using AI algorithms to write and to synthesize and portray characters (or write software). If you don't have a full pipeline constantly producing people who have training and experience at writing or acting (or writing software, or whatever it is you need), you no longer have a way to generate the huge human-created and human-curated datasets you need to train your AIs. The models collapse, and eventually the writing or portrayal of characters (or the software) in no way represents the real world.

That valley isn't even uncanny; it's just wrong.

But you can't just gen up more competent, trained, experienced writers or actors (or software developers) on the spur of the moment. It takes years to do that. By the time you realize you're in trouble, it's too late.

This is the precipice some folks want us to move towards today.