A huge jolt convulsed an oil rig in the Gulf of Mexico. The pipe down to the well on the ocean floor, more than a mile below, snapped in two. Workers battled a toxic spill.

That was 2003—seven years before last month’s Deepwater Horizon disaster, which killed 11 people and sent crude spewing into the sea. And in 2004, managers of BP PLC, the oil giant involved in both incidents, warned in a trade journal that the company wasn’t prepared for the long-term, round-the-clock task of dealing with a deep-sea spill.

It still isn’t, as Deepwater Horizon demonstrates and as BP’s chief executive, Tony Hayward said recently. It’s “probably true” that BP didn’t do enough planning in advance of the disaster, Mr. Hayward said. There are some capabilities, he said, “that we could have available to deploy instantly, rather than creating as we go.”

It’s a problem that spans the industry, whose major players include Chevron Corp, Royal Dutch Shell and Petróleo Brasileiro SA. Without adequately planning for trouble, the oil business has focused on developing experimental equipment and techniques to drill in ever deeper waters, according to a Wall Street Journal examination of previous deepwater accidents. As drillers pushed the boundaries, regulators didn’t always mandate preparation for disaster recovery or perform independent monitoring.

The brief, roughly two-decade history of deepwater drilling has seen serious problems: fires, equipment failures, wells that collapsed, platforms that nearly sank. Since last July, one brand-new deepwater rig—among the 40 or so operating in at least 1,000 feet of water in the Gulf—was swept by fire. Another lost power and started to drift, threatening to detach from the wellhead. Poor maintenance at a third deepwater well led to a serious gas leak, according to regulatory records.

By some measures, offshore drilling has become safer in recent years. Industry backers argue that major accidents are rare. The rate of serious injuries in U.S. waters fell 71% between 1998 and 2008, and the number of serious oil spills has also been falling once hurricanes are taken into account. Moreover, deepwater drilling is by some measures safer than drilling in shallower waters, where rigs are often older and operated by smaller companies.

Still, drilling for oil at depths no human could survive presents special risks when something does go wrong. The water pressure is crushing, the seabed temperature is almost freezing, the underground conditions explosive. The rapid push into deeper water means that some projects rely on technology that hasn’t been used before.

“It’s like outer space, in terms of the complexity of the operating environment,” said Robin West, who helped oversee offshore-drilling policy under President Ronald Reagan and is now chairman of PFC Energy, a consulting firm.

In 2008, Chevron was plagued with accidents while using the Discoverer Deep Seas rig in more than 7,000 feet of water in the Gulf. There was a fire, then a leak deep under the sea. Finally the cement and steel casing inside the well collapsed, allowing drilling fluid to flow out of control. Workers stopped the flow only by permanently plugging the well.

Chevron says the well was “safely and permanently” abandoned after the problems. “One of Chevron’s core values is the safety of our employees, contractors and neighbors,” Chevron spokesman Kurt Glaubitz said. “It is fundamental to how we operate.”

BP has led the charge into the deepest, most challenging environments. Last week Mr. Hayward, the CEO, said, “It’s clear that we will find things we can do differently.”

As companies have moved farther offshore, drilling has gotten increasingly expensive. BP was paying nearly $500,000 a day to lease the Deepwater Horizon from Transocean Ltd. and paid roughly that much again for other equipment and services.

One of the most serious safety hazards on rigs are “blowouts,” the uncontrolled flows of oil and natural gas like the one that brought down the Deepwater Horizon. They remain relatively rare, but no more so than in the 1960s, when equipment was much more primitive.

That’s in part because, even as the gear used to fight blowouts has improved, the industry has steadily pushed into deeper waters.

“While drilling as a whole may be advancing to keep up with these environments, some parts lag behind,” Texas A&M professors Samuel Noynaert and Jerome Schubert wrote in a 2005 paper published in an industry journal. “An area that has seen this stagnation and resulting call for change has been blowout control in deep and ultra-deep waters.”

The professors declined to comment for this article.

Serious accidents like the Deepwater Horizon have been rare, but not unheard of. In 2001, an oil-and-gas-production platform off Brazil’s coast exploded and ultimately sank, killing 11 people.

Offshore drilling is almost as old as the oil industry itself. In the 1890s, companies began prospecting for oil from piers extending off the beach near Santa Barbara, Calif. In 1947, Kerr McGee Corp. (which was later acquired by Anandarko Petroleum Corp.) drilled the first well out of sight of land, in the Gulf of Mexico.

In the past decade or so, what had been a steady march into deeper water turned into a sprint, as easier-to-find oil fields dwindled. In 1996, Royal Dutch Shell broke new ground with its Mars platform, which floated in 3,000 feet of water. A decade later, wells in 5,000 feet of water—almost a mile deep—were so common as to be considered relatively routine. Several rigs working today can drill in water as much as 12,000 feet deep, more than two miles above the ocean floor.

Associated Press

Petrobras’ P-36 platform offshore Brazil in the process of sinking after explosions in 2001

Shell says it has operated in the Gulf for five decades without “a significant offshore well incident or platform spill in the deep water Gulf of Mexico.”

Drilling in deeper water doesn’t change the fundamental process, but it makes virtually everything harder. Rigs must be bigger so they can hold more drilling pipe to stretch vast distances. The pipes themselves must be stronger to withstand ocean currents. Equipment on the sea floor must be sturdier to face extreme pressures at depth.

Drill bits must be tougher so they don’t melt in the 400-degree temperatures they encounter deep in the earth. And it is harder for drillers to exert just the right amount of pressure down the well bore, enough to keep oil and gas from spurting upwards—a blowout—but not so much that they crack open the rocks beneath the surface, which could also lead to a blowout.

The use of untested techniques has raised alarm bells among some engineers. In a paper published in a trade journal last year, three industry engineers in Denmark noted that many deepwater projects are “dependent on prototype and novel technologies.” They said, “there is significant uncertainty related to the performance of these systems,” because they haven’t been tested in real-world settings.

They couldn’t be reached for comment.

BP discovered that in 1999 at its Thunder Horse offshore oil field in the Gulf of Mexico, where managers say hundreds of pieces of equipment had to be created from scratch.

Not all the brand-new systems worked. Thunder Horse had a near disaster in 2005, when a faulty control system opened valves and allowed water to flood into the hull of a drilling platform there. The multibillion-dollar platform almost sank. BP spent months fixing equipment damaged in the flood.

And in 2006, as Thunder Horse was getting close to completion, workers discovered a leak in one of the huge sets of valves on the seafloor that control the flow of oil and gas from the wells. An investigation found minute cracks in a protective coating on some of the pipes, allowing corrosion that could, ultimately, have led to breakage of the pipes. BP had to pull the equipment back to the surface for repairs, delaying the project for months and raising the costs.

Equipment failure was also to blame in the case of the Discoverer Enterprise, the rig that ran into trouble in 2003 when the “riser,” the pipe down to the seafloor, snapped in two. That left the Enterprise floating free, with no immediate way to control the well sitting on the sea floor more than a mile below. That well, investigators later concluded, had the potential to spew more oil in one week than was spilled in 1989 by the Exxon Valdez, which ranks as one of the worst U.S. oil spills to date.

In a 2004 article in a trade journal, two BP managers evaluated the company’s response to the Discoverer Enterprise incident. Their conclusion: Although the company’s initial reaction was strong, it had “less focus” on the longer term and wasn’t prepared for the nearly two weeks of round-the-clock response even the fairly small spill required.

A BP spokesman said it follows a “tried-and-tested approach to incident management.”

Catastrophe was averted in the Discoverer Enterprise case because, unlike at the Deepwater Horizon, the well’s “dead-man switch” was triggered when the riser broke. A powerful contraption known as a blowout preventer sheared off the pipe and sealed off the well. Some 2,450 barrels of drilling fluid inside the riser spilled into the Gulf, but the well itself was secure.

Today, the Discoverer Enterprise is located at the site where the Deepwater Horizon sank, sucking up oil from the still-leaking well through a special tube.

Drilling companies have pushed the limits of technology in blowout preventers, also known as BOPs. Multiple technical papers have called into question whether the shears are powerful enough to cut through the tough steel used in modern drilling pipe at the deepest wells. A 2004 study commissioned by federal regulators found that only three of 14 newly built rigs had shears powerful enough to cut through pipe at the equipment’s maximum water depth.

“This grim snapshot illustrates the lack of preparedness in the industry to shear and seal a well with the last line of defense against a blowout,” the study said.

Andy Radford, a policy adviser for the American Petroleum Institute, said the group recommends that all blowout preventers be equipped with shears powerful enough to cut through the pipe being used.

Some subsequent studies, including a 2007 paper co-authored by a BP engineer, have echoed those concerns. “The use of higher strength, higher toughness drill pipe … has in some cases exceeded the capacity of some BOP shear rams to successfully and or reliably shear drill pipe,” the 2007 paper said.

When things go wrong in deep water, the problems are harder to solve. “If we can touch the wellhead, we have a really super high chance of making the flow stop,” said Daniel Eby, vice president of Cudd Well Control, a contractor that helps oil companies stop out-of-control wells. “The problem comes when you can’t touch it. And when you put that wellhead in 5,000 feet of water, we can’t touch it.”

The current crisis is widely expected to send insurance costs higher for deepwater drilling. Lloyd & Partners Ltd., a London broker, recently said it would cut back the amount of pollution insurance it offers to oil companies by a third. In general, rates have risen for all drilling rigs in recent years due to hurricane damage and other issues, but haven’t been consistently higher for deepwater rigs than for those in shallower water.

Government regulators have long known that the deepwater presents special challenges. After the 2003 accident on the Discoverer Enterprise, researchers from the National Oceanic and Atmospheric Administration conducted a study looking at how to tell where oil from an undersea spill would reach the surface, and how to better coordinate with workers respondomg to a spill.

The Minerals Management Service, the government agency that oversees offshore drilling, in recent years moved away from requiring specific safety measures in offshore drilling and instead set broad performance goals that it was up to the industry to meet.

MMS declined to make an official available for an interview for this article. In a statement, the agency said it’s reviewing its oversight in light of the disaster.

In joint MMS-Coast Guard hearings into the Deepwater Horizon accident, Michael Saucier, an MMS official, testified that the agency “highly encouraged,” but didn’t require, companies to have back-up systems to trigger blowout preventers in case of an emergency.

“Highly encourage? How does that translate to enforcement?” Coast Guard Capt. Hung Nguyen, who is co-chairing the investigation, asked at the hearings.

“There is no enforcement,” Mr. Saucier replied.

Coast Guard Lt. Cmdr. Michael Odom, who oversees Coast Guard inspections (the Coast Guard inspects oil-company vessels above the water, while the MMS oversees drilling) testified that current regulations for offshore drilling may be out-of-date. He said many regulations were written years ago, and focused on near-shore drilling operations.

“The pace of technology has definitely outrun the regulations,” Mr. Odom said at the hearing.