Symptom First Archives - understand-culture

Eyeglasses for Hubble

The Hubble Space Telescope was launched on April 24, 1990. On June 27, 1990, NASA announced that it was flawed. Its main mirror had an error “10 times larger than the specified tolerance,” caused by a lens spacing error in a test device.

In December 1993 astronauts installed COSTAR, a device that “acted like a pair of eyeglasses” for the telescope’s instruments. Later instruments were built with their own corrective optics. COSTAR was removed in 2009 and is now in the National Air and Space Museum.

The cause was found. The fix was still applied in front of the mirror, not to it.

A Band-Aid Solution

American English has a full set of words for fixing the visible part of a problem.

a Band-Aid solution — “serving as a temporary or expedient remedy or solution.” The brand name has been used this way since 1968.

kick the can down the road — to put off work on an issue for a later date. Lawmakers began using it in the mid-1980s.

whack-a-mole — “a situation in which any attempt to deal with a problem is met with another problem or a reoccurrence of the original problem.”

putting out fires — dealing with urgent problems one at a time as they come up.

a quick fix — a solution that works now.

Nobody Ever Gets Credit

In 2001 the MIT researchers Nelson Repenning and John Sterman published a study of why process improvement programs so often fail. Its title quotes an engineer at an auto company: “Nobody ever gets credit for fixing problems that never happened.”

Managers who “rely on working harder and shortcuts to meet immediate throughput needs” fall into what the authors call “the Capability Trap.” Working harder shows results at once, a pattern the authors call “better-before-worse.” Fixing the process pays off only later.

Treat the Greatest Threat First

In 1976 an orthopedic surgeon, James Styner, crashed his small plane in Nebraska. His wife was killed and three of his children were critically injured. The crash led to Advanced Trauma Life Support, a course for doctors who treat injured patients. The first course was held in 1978.

The course, now run by the American College of Surgeons, has been taught to “over 1 million doctors in more than 80 countries.” Its concepts include: “Treat the greatest threat to life first.” “A lack of a definitive diagnosis should not delay the application of indicated treatment.” “A detailed history is not essential to begin the evaluation of a patient with acute injuries.”

Continuing Resolutions

When Congress has not passed its regular spending bills by the start of the fiscal year, it passes a continuing resolution. The Congressional Research Service describes its purpose: “to provide temporary funding to continue certain programs and activities” until the regular bills are done.

From fiscal year 1977 to 2023, all regular appropriations were enacted on time only four times: in 1977, 1989, 1995 and 1997. Fiscal 1997 was the last year without any continuing resolution. Fifteen times, a continuing resolution ran for the full year.

The Y2K Windowing Fix

The full fix for the Year 2000 problem was to store years with four digits. A quicker fix, called windowing, kept two digits and simply treated years from 00 to 20 as 2000s. According to New Scientist, an estimated 80 percent of computers fixed in 1999 used it.

During the Y2K work, Gartner researcher Jim Duggan called it “a Band-Aid.” Retired programmer Larry McNish later said the pivot year “just postponed their ‘Y2K’ problem a few decades.” In January 2020 the video game WWE 2K20 “stopped working at midnight on 1 January 2020,” and about 14,000 New York City parking meters stopped accepting credit cards.

It has not been shown that these two failures came from windowing. New Scientist listed them as date bugs of 2020.

In-Family

Foam from the Space Shuttle’s external fuel tank had struck the orbiter on earlier flights. The Columbia Accident Investigation Board found that foam shedding, “which had initially raised serious safety concerns, evolved into “in-family” or “no safety-of-flight” events.” It also found: “NASA does not fully understand the mechanisms that cause foam loss on almost all flights.”

After a large piece of foam came off on flight STS-112, a task was opened to “identify the cause and corrective action.” It was not due until after the next launch. It then slipped past STS-107, the flight on which Columbia was lost. Responsibility moved between the orbiter office and the external tank office. The board wrote: “Integration did not occur.”

Implementation Lapsed

After a cold snap in 1989, power generators in Texas received recommendations for protecting their plants in winter. A federal report on the next freeze, in 2011, found: “These recommendations were not mandatory, and over the course of time implementation lapsed.” It added: “Many of the generators that experienced outages in 1989 failed again in 2011.”

In February 2021 it happened a third time. An official of ERCOT, the Texas grid operator, said winter guidelines were still voluntary: “There aren’t regulatory penalties at the current time.” The final federal report called the event “the largest manually controlled load shedding event in U.S. history.” Freezing problems caused 44.2 percent of the unplanned outages, reductions and failures to start at power plants.

Stop the Bleeding

Google’s book Site Reliability Engineering tells engineers what to do in a major outage: “Your first response in a major outage may be to start troubleshooting and try to find a root cause as quickly as possible. Ignore that instinct!” Instead, “your course of action should be to make the system work as well as it can under the circumstances.” The chapter adds: “Stopping the bleeding should be your first priority.”

The book takes its model from aviation: “Novice pilots are taught that their first responsibility in an emergency is to fly the airplane.” The search for the cause comes afterward, and engineers are told to keep evidence such as logs for it.

Technical Debt

In 1992 the American programmer Ward Cunningham introduced a metaphor at a software conference: “Shipping first-time code is like going into debt.” A little debt speeds development, he wrote, as long as it is paid back promptly. “The danger occurs when the debt is not repaid.”

Technical debt is now a standard term in software teams. Often the shortcut is taken on purpose, and the cost of fixing it later is counted as interest. In Cunningham’s words: “Every minute spent on not-quite-right code counts as interest on that debt.”