Major Failures At The Tevatron

Tevatron main ring from the air. This morning's 5.2 earthquake in the Midwest, with dozens of aftershocks, rattled Chicago skyscrapers and homes in Cincinnati. It must have affected the Tevatron. Fermilab Today, published daily M-F on the web, says nothing about the earthquake, as of 11:30 AM Central, 7 hours after the first shock. Point is any complex and sensitive installation, whether a chemical plant or the Tevatron, would likely crashdown under an onslaught of earthquakes. In 1981, a small local quake at Arzew, Algeria crashed down LNG I and LNG II. The plants vented steam you could see for miles in the fiery glow of the natural gas flares during the night shift. No one was hurt and there was no serious damage, but it took about a week to get the plants up to speed. This is the real safety test for any plant, unplanned instant total shutdown, not ever done deliberately because of the many risks. How would the LHC fare during an earthquake, when even a thunderstorm can rock the Tevatron?

We do know what can go wrong at the Tevatron on a nice day. Not easy to find out. It's a buttoned down organization. Apart from the tritium leak made public, even major failures requiring extensive shutdowns for repairs and re-engineering, are not press released by Fermilab, as though the Tevatron has never suffered any downtime accidents. Even Scientific American has no stories published on Tevatron failures nor could I find any other media stories. The only window on operations at the Tevatron, is the Accelerator Update, a complex technical summary, through a link on Fermilab Today. Digging around with Google you can get info on the December 5, 2003 beam loss accident from a paper by Fermilab's N.V. Mokhov et al, Beam-Induced Damage To the Tevatron Components And What Has Been Done About It at http://lss.fnal.gov/archive/2006/conf/fermilab-conf-06-415-ad.pdf The only list of serious component failures in Run II, was from a talk in 2006 at an annual accelerator conference, EPAC at Edinburg, by Valeri Lebedev from Fermilab. Here's a partial list from the Lebedev pdf, Tevatron Failures section.

01-Mar-03 Downtime 12 Days Safety lead ground fault
05-Dec-03 Downtime 12 Days Catastrophic beam loss
20-Dec-03 Downtime 10 Days Cryostat vacuum leak
15-Mar-04 Downtime 12 Days Helium leak
21-Nov-05 Downtime 3 Wks Helium leak
14-Jan-06 Downtime 2 Wks Cryostat vacuum leak
22-Feb-06 Downtime 1 Wk+ Helium leak

(Subsequently, I found the way into a searchable database, a new website for Fermilab's Library, Technical Publications. No link to it either, oddly enough, from other Fermilab sites. Go to http://bss.fnal.gov/techpubs/fermilab_spires.html Another trick is to use Title Search for any search term like beam loss. Then you get 33 citations on beam loss from SPIRES.)

Besides the short presentation notes on the pdf by Lebedev, here then is Lebedev's detailed paper on the same subject, Tevatron Operational Status And Possible Lessons For The LHC. I couldn't find any transcripts of the candid talk he gave at the EPAC-2006 Conference, which would have been to a big LHC audience in Edinburg. But the full paper does clarify a lot of other problems like the faulty pressure relief (Kautzky) valves at the Tevatron, responsable for 2 quenches. "Considering this a systematic failure, we replaced the failed part in all of about 1200 valves during the 2006 shutdown." Then a discussion of the second catastrophic beam loss. "One of worst failures happened on December 20, 2003 when a hardware failure caused a beam loss with consecutive major multi-house (helium cooling/compression plants) quench with 2/3 of the beam lost in the ring before the beam was aborted. Magnets and beam collimators had to be repaired." Imagine 1200 valve repairs. Did they call every plumber in Chicago?

Mokhov's paper, cited above, is a condensed yet updated version of an earlier paper I found through the new tech docs database with a different name that goes Beam-Induced Damage to the Tevatron Collimators: Analysis and Dynamic Modeling of Beam Loss, Energy Deposition and Ablation. This is the fullest discussion on the 05-Dec-03 Catastrophic Beam Loss without going into repairs and redesign of certain Tevatron components. The shorter later paper was presented by Mokhov at another particle physics conference, the ICFA-HB2006 in Tsukuba, Japan. Since it's the best look at a big accident, here's a summary for nonphysicists.

This beam loss wasn't initially caused by the usual leaks or superconducting magnet quenches. These occurred within a split second of the failure of a Roman pot in the huge CDF detector, one of the 2 main experiments in the Tevatron ring. A Roman pot is a CERNRome group invention from the 1970's, a sampler inserted near a collision event to pickup particles deflected from the collision point of the 2 opposing beams.

This time this Roman pot didn't obey an operator command to withdraw from the beam collisions. It did momentarily, but reinserted itself well into the beams. This spewed particles hitting the Roman pot, upstream into the helium coolant and superconducting magnets. Two-thirds of them quenched, about 640, and of the 24 refrigeration houses used to cool the helium to cool the magnets, 16 failed. A release of helium wasn't discussed in Mokhov's paper, though some was vaporized causing damage.

Trillions of protons and antiprotons unravelled from the beam pipe turn after turn, a "major fraction of the 1.5 MJ beam lost in the ring", according to Lebedev. Mokhov notes, "The entire beam is lost during about 400 turns (8.4 msec). . ." in the blink of an eye.

Fortunately no one is allowed in the ring systems when the accelerators are operating because of the radiation and possible leaks and accidents, so no one was hurt below ground. Above ground effects to personnel were not observed. A lot of glazed eyes in front of monitors at least.

Mokhov focuses on the obvious damage and what was done to the Tevatron to correct a cascade of failures. Two collimators, a primary tungsten and another of stainless steel, took most of the beam energy direct hits. Ordinarily, collimators very near the beams, are used to reduce the size of a beam's halo, an action called cleaning or scaping the beam. Here they minimized the damage to the Tevatron. The tungsten abraded and the stainless steel was grooved as though by a 3mm beam, though the actual beam might have been less than a hair's thickness. A large cryogenic spool containing focusing elements was also badly damaged, but by helium evaporation and pressure rise during the quench. Lebedev cites Mokhov, are we in Moscow yet? Lebedev infers that Mokhov's short paper also covers the December 20 accident, which explains why there's some drilling by a beam through more steel somewhere.

Anyway, what actual loss of tungsten and stainless steel that might have been transformed into some other particles by the proton-antiproton beams is not discussed. Some of these unknown particles could still be in the Tevatron or some were released. Fermilab experiments with beams shot at various targets might show what they are, experiments either before or after the accident, but there is no discussion of this in Mokhov's paper. Neither is there information on what happens to helium or magnets or pipe hit by these very powerful beams. Recall my earlier post, Daily Battles At The Tevatron, that since there is a lithium lens used in the Tevatron it might be the source of tritium contamination at Fermilab. Would someone at Fermilab please check?

After a lot of study, Fermilab did modify the Tevatron to prevent a similar set of events. Some modifications needed were so obvious that it is startling they weren't implemented sooner, like for the AC power distribution systems. The emergency kicker magnets and the CAMAC Abort controls now have their own separate feed from the utility company's sub-station. "It now requires a failure of the substation." writes Mokhov obtusely, that is for the emergency beam dump to fail. Uninterruptable power supplies were also added to the CAMAC Abort rack. Other major modifications were made to the Tevatron, discussed at length in Mokhov's paper.

No more "misbehaved proton beam". There are other Russians at Fermilab with colorful English. I can imagine what you might hear at the Tevatron on a bad day. Beam is falling or is technical malfunction from data sensor or maybe crippled switch? Can we check? Not in a few milliseconds, Nicky.

Even though the Tevatron is closing down by 2010, Fermilab is fighting for another collider installation at Batavia, the enormous ILC or International Linear Collider, still on the Fermilab drawing board. So far it doesn't look good.