Human Nature Resilient Processes

The tsunami induced nuclear crisis in Japan is rightfully scaring the bejeezus out of everyone who is paying attention... How on earth could a system with such potential for devastation have relied on such a fragile process for disaster recovery?

Japan's nuclear power plants relied on external power in order to shut down safely. No external power, no safe shutdown.  I'm sure that someone who analyzed this process "computed the odds" and determined that the likelihood of external power loss occurring at the same time that a plant shutdown was required was incredibly low - virtually impossible - nothing to worry about.

Well... They must have missed a decimal point or something, because the unlikely happened.

As the renowned Murphy's experiences would remind us, unlikely things happen all the time... or more precisely... things that we judge to be unlikely happen all the time.

Are these process failures really unlikely, or are they an expression of the true nature of things?

Processes must take into consideration the environment in which they operate... and that environment always has at least two natures that must be addressed:

• Physical Reality - aka nature with a capital 'N'
• Social Reality - aka Human Nature

Japan's current nuclear problems might seem to be a clear case of not properly analyzing the physical reality of power plant near shoreline near ocean fault, but time after time we find that Human Nature screws up processes that should have worked. Somewhere in Japan's equation a human's judgment was influenced to overlook what's now obvious.

Human's are motivated by the darnedest things...  and one of those things is economics.  I would have said greed, but that would be unfair.  Humans, and by extension their companies, are motivated by profit and loss.  Potential gain versus potential pain is a major driving force in all of our endeavors.

Years ago I worked for a company that built safety systems for power plants that use coal, gas or oil to generate steam.  The process by which these plants operate is conceptually simple - burn fuel to generate heat, and use that heat to boil water, and use the steam from the boiling water to spin a turbine that generates electricity.  The not-so-simple part of this process is to burn the fuel as quickly as possible so that you can generate a lot of electricity.  You have to maintain just the right ratio of fuel and air at all times, so you're constantly pumping both fuel and air into your furnace.  It's actually a controlled explosion.

If something goes wrong and your flame goes out, you need to shut down the fuel and air flow as quickly as you can.  If there's a lot of hot unburned fuel and air in your furnace and you get a spark you are very likely to hear a very loud boom (the least of your worries when your power plant explodes).

To avoid explosions, companies like the one that I worked for built systems that monitor the flames and shut down the fuel and air flow when a problem is detected.  Interestingly enough, the impetus for installing these systems usually came from Insurance Companies rather than from the plant operators.  Most plant operators trusted their workers to manage the burners... most insurance companies didn't.

Let me add here that it's very expensive to shut down and restart a power plant.
You can't just flick a switch... and it can take hours to restore service.  Hours of lost revenue and additional expense.  Consequently, if a "safety" system shuts down your plant in error, the plant operators get very upset.

One of my coworkers, Ben, told me a story... I have no evidence to support this story, but it's reasonable to believe that it's true...

One day Ben happened to be sitting in the control room of a power plant when our safety system detected a problem.  The alarms went off, and Ben could hear the mechanical relays that would close the valves start "clacking".  The safety system was doing everything that it could to shut down the plant, but nothing happened.

Meanwhile, the control room operators took steps to resolve the problem.  I don't know what they did, but the plant kept running and nothing went boom.

Perplexed, Ben started poking around, and lo and behold it turned out that the safety system was not connected to the valves that it was supposed to close.  Perhaps it had never been connected, perhaps it had been disconnected... the end result was the same.  No safety system.

Of course the control room operators were pretty smug about this... They'd fixed the problem, there was no boom, and most importantly there had been no loss of revenue.

Human nature broke this safety process.  The motivation to "keep it running" was far more compelling than the motivation to "stay safe".

Humans will always take risks.  If your process is not resilient enough to survive these risks, then it's going to break.

I believe that nuclear power is key to our future... and new designs that rely on Physics can produce much safer plants... but if we don't also address Human Nature it's just a matter of time before we hear that very loud boom again.