LHC Quench Stops CERN: Re-start Delayed Again

After last Friday's massive quench at the LHC, CERN has announced that re-starting the collider will have to wait even beyond 2 months for repairs and downtime. With a planned winter shutdown in late November, even if the collider were ready, there would not be enough time to continue tests. Re-start of the LHC should be in April 2009.

Day 7 into the major accident damaging part of the LHC helium cooling and superconducting magnet systems, indicates little progress as yet. Scientists at CERN are still investigating the failure in Sector 3-4, an eighth of the 17 mile ring of magnets that are normally cooled to near absolute zero temperature. The extent of the damage is greater than was reported by CERN initially. Dismissive at first of the collider's "teething problems", CERN suggested that the LHC could be operational soon. A sketchy press release was followed by another sketchy press release.

Oddly after the BBC discovered the massive quench of about 100 magnets in a log entry on a CERN website, "the entry has since been removed" according to TimesOnline the day after.

It was also TimesOnline that said on September 21st that a connection between 2 giant magnets melted, with a release of one tonne of helium, adding the heat also melted 2 giant magnets, causing an explosive release of liquid helium which blasted helium gas through the ring tunnel. No one was hurt as no personnel were in the collider ring said CERN, normally only off limits when proton beams are operating. CERN said that at the time of the accident last Friday morning, there were no beams and only admitted to "a large helium leak" and "a faulty electrical connection between two magnets, which probably melted".

The only piece of real news by CERN's chief spokesman, Dr James Gillies, who does the standard tour of the major media outlets to reassure journalists, outside of his department's press releases, was about the damaged magnets in question that he identified as giant quadrupoles, in the Telegram UK, September 20th. Later the probably melted connection turned out to be a busbar, a type of reinforced splice of magnet ribbon cable, one of many such connections between magnets.

It wasn't exactly a routine failure. Commissioning was still going on in Point 4 and on either side into the later damaged zone of Sector 3-4. and in 4-5. The high energy test was of the RF system for anticipated higher loads when it would be used to power up beams to 5 TeV, that were scheduled for later this year.

As of today only specialized safety teams and technicians have been allowed in to check for hazards and equipment damage. Following the accident Sector 3-4 showed some continuing warming of magnets and helium, some of the warmest then cooling, but now the Sector has been partially re-cooled. Most magnets are at 30 K, with no further helium leaks reported.

Eventually the damaged Sector 3-4 will be warmed to room temperature for repairs to be effected, but that would mean that remaining helium would have to be drained from the system first, and that hasn't started yet. Since temperatures have been brought down, it seems that the strategy is to cool the helium enough so that it can be safely drained as a liquid and stored as it's very expensive to replace. At least two damaged quadruoles will have to be cut out as they are welded together and then new ones welded in, a difficult and expensive proposition.

Coming on the heels of two other major failures, the hacking of part of the CMS computer system, during the big September 10th media launch, and the destruction of a 30 tonne surface transformer during a thunderstorm shortly afterwards, powering 2 sectors of the LHC cooling system, the need for a more careful re-launch of the collider is apparent to CERN, hence the further delay to spring.

CERN publicly has put on a brave if vague face, with its first very short press release of the 20th September, "Incident in LHC sector 3-4", though it was abundantly clear the day of the accident September 19th that it was more than an incident, as reported by the BBC that day. Even on the day of the accident, according to Scientific American, "CERN said on Friday that "The LHC is on course for [its] first collisions in a matter of weeks", just a day later it announced the minimum two-month repair job." Lately Dr Robert Aymar, Director General of CERN referred to the accident as "undoubtedly a psychological blow."

But there are other factors leading up to the accident that haven't been addressed. Certainly timing the First Beam to suit a media bash opening, when the CMS was being hacked, and still going ahead with it, suggests the need to perform for the media above ordinary safety considerations. Several other factors also indicate a rush to perform for the media and a huge worldwide audience.

Reuters reported September 10th, that there were "Small electrical issues before CERN machine start-up". "Project leader Lyn Evans gave no details . . ."

Operating the collider during a thunderstorm is certainly a known risk at the Tevatron, and 36 hours after the September 10th opening, a 30 tonne surface transformer that powers part of the helium cooling system failed during a lightening strike. Cause of failure still not confirmed by CERN.

According to the LHC Commissioning with Beam page, "The winter shut-down will then be used to commissioning and train the magnets up to full current, such that the 2009 run will start at the full 14 TeV design energy." Lyn Evans reiterated the same point in a talk on "The LHC Machine" at a CERN colloquium, Strings 2008, August 18th. Some installed magnets from another supplier had to be retrained. Earlier this year there were plans to jump to 7 TeV. Then 5 TeV was announced as rather a surprise, the new goal per beam. Where the damaged magnets from this lot of not fully trained magnets?

What if CERN has run out of luck? They have been lucky, not running beams they say during the transformer failure and the huge magnet quench, but what if they had been running beams? Beams could have been lost causing more damage. They ran beams during the hacking of CMS, but luckily there were no collisions.

There's a long road ahead. As Fermilab/SLAC's sponsored magazine, Symmetry wrapped up the transformer accident at the LHC, "These kinds of hiccoughs in starting up a large collider are not surprising as the LHC has millions of critical components."

What? Only millions?


References

Gillis, Alan. "Accident Cripples LHC", The Science of Conundrums", September 19, 2008, http://bigsciencenews.blogspot.com/2008/09/accident-cripples-lhc.html

Higgins, Alexander G. "Small accidents mean big trouble for supercollider", AP/PhysOrg, September 22, 2008, http://www.physorg.com/news141278719.html

CERN. "Incident in sector 3-4", Press Release, September 20th, 2008, http://press.web.cern.ch/Press/PressReleases/Releases2008/PR09.08E.html

CERN. "LHC re-start scheduled for 2009", Press Release, September 23, 2009, http://press.web.cern.ch/Press/PressReleases/Releases2008/PR10.08E.html

Reuters. "Hadron Collider halted for months", Reuters News Video, September 21, 2008, http://www.reuters.com/news/video?videoId=91036

Chalmers, Matthew and Henderson, Mark. "CERN delays atom-smashing over magnet fault", TimesOnline, September 20, 2008, http://www.timesonline.co.uk/tol/news/uk/science/article4789673.ece

Leake, Jonathan. "Oh blast, that's the wrong kind of big bang", TimesOnline, September 21, 2008, http://www.timesonline.co.uk/tol/news/uk/article4794825.ece

Highfield, Roger. "Large Hadron Collider to be turned off for two months following damage", Telegraph UK, September 20, 2008, http://www.telegraph.co.uk/earth/main.jhtml?view=DETAILS&grid=&xml=/earth/2008/09/20/scilhc120.xml

Gillis, Alan. "LHC Not So Safe". The Science of Conundrums, September 12, 2008, http://bigsciencenews.blogspot.com/2008/09/lhc-not-so-safe.html

Gillis, Alan. "LHC Fails Thunderstorm Test", The Science of Conundrums, September 17, 2008, http://bigsciencenews.blogspot.com/2008/09/lhc-fails-thunderstorm-test.html

Lite, Jordan. "Hobbled LHC shuttered for repairs; 'No big deal say scientists'", SciAm, September 22, 2008, http://www.sciam.com/blog/60-second-science/post.cfm?id=hobbled-lhc-shuttered-for-repairs-n-2008-09-22

Reuters. "Small electrical issues before CERN machine start-up", Reuters News, September 10, 2008, http://www.reuters.com/article/latestCrisis/idUSLA57119

CERN. "LHC Commissioning with Beam", LHC Commissioning, page still current, http://lhc-commissioning.web.cern.ch/lhc-commissioning/

Gillis, Alan. "LHC Start-up To Shutdown 2008", The Science of Conundrums, August 22, 2008, http://bigsciencenews.blogspot.com/2008/08/lhc-start-up-to-shut-down-2008.html

CERN. "LHC: countdown to beam begins", CERN Courier, August 18, 2008, http://cerncourier.com/cws/article/cern/35431

Harris, David. "LHC glitch means two month delay" symmetrybreaking, September 20, 2008, http://www.symmetrymagazine.org/breaking/2008/09/20/lhc-glitch-means-two-month-delay/

Accident Cripples LHC

No collisions, no beams either next week at the LHC. The BBC reports an alarming quench of about 100 superconducting magnets today, that heated up as much as 100 C. A tonne of liquid helium spilled into the tunnel and the CERN fire brigade went in. Cause of the quenching has not been announced, nor have any injuries been reported. Liquid helium leaks vaporize back to a gas almost instantly and would freeze or choke personnel present.

Ordinarily magnet quenches occur when proton beams are lost or scatter into magnets, causing helium coolant and magnets to heat and lose their superconductivity and their power to keep proton beams within the collider. CERN doesn't say whether beams were running today during the accident. They would have been at their lowest power 0.45 TeV per beam, though beams 11 times more powerful at 5 TeV were scheduled before October 12th. Vacuum conditions were also lost said the BBC, which suggests beam damage, though CERN hasn't commented.

This is on top of another major failure to the helium cooling system. Last Friday the 12th's thunderstorm burned out a giant 30 tonne 12 Million Volt Amperes surface transformer that powers some of the helium cryogenics system. Although CERN admitted the problem yesterday, without mentioning the thunderstorm, and said the transformer was replaced last weekend, there was no explanation for the long delay in informing the public. The failure of the transformer caused an initial warming in the helium coolant in 2 of the colliders 8 sectors, some as late as September 17th at near 7 K

Whether an ongoing transformer problem contributed to quenching has not been announced. The TimesOnline initial report on damage said, "One of the beams had been captured by Friday, but work was then interrupted by the loss of electrical transformers that power the cryogenic cooling system . . ." A beam apparently running during the thunderstorm, and more than one transformer lost. CERNS's lastest progress report yesterday was sketchy.

The damaged Sector3-4, an eighth of the 17 mile collider has still not been stabilized since this morning's accident. About half of it or a mile length was well above normal 1.9 K design temperature, about a quarter of the sector and its magnets this afternoon at 15:46 PM were about 82 K to 110 K. Currently all magnets have warmed. The warmest show a slight recovery from 110 K down to 99 K. and Sector 3-4 is still in crisis as of the 20th September 0:418 AM, with no accurate readout for about half the magnets, and the rest showing more warming, with a few outside the spike zone climbing abruptly in temperature since this afternoon.

Given the gravity of the accident, repairs would take a week or longer.


References

BBC. "Hadron Collider forced to halt", Sept 19, 2008, BBC News, http://news.bbc.co.uk/1/hi/sci/tech/7626256.stm

CERN. "LHC progress report, week 1", Sept 18, 2008, LHC First Beam, http://lhc-first-beam.web.cern.ch/lhc-first-beam/News/lhc_080918.html

Gillis, Alan. "LHC Fails Thunderstorm Test", Sept 17,2008, The Science of Conundrums, http://bigsciencenews.blogspot.com/2008/09/lhc-fails-thunderstorm-test.html

Henderson, Mark. "'Big Bang Machine' back on collision course after its glitches are fixed", Sept 18, 2008, TimesOnline, http://www.timesonline.co.uk/tol/news/uk/science/article4774817.ece

LHC Fails Thunderstorm Test

Nature had her own ideas about testing LHC safety. A thunderstorm last Friday knocked out some transformers at the LHC near Geneva that are part of the helium cooling system, that cools the magnets that keep the proton beams travelling (near light speed) on a circular path through the collider. Technicians have been scrambling to fix the problems, but not before some magnets warmed well above standard operating temperatures, some reaching almost 7K from the usual ultra cold 1.9K .

Electromagnets at the LHC need to be this cold to be superconducting, or at peak efficiency, in order to deliver extremely high magnetic fields in the 27 km ring of 1200 giant magnets and thousands of smaller ones, at 8.33 Teslas or about 200,000 times the earth's magnetic field strength.

Everything about this atom smasher is boggling, including the numbers, like its cost at about $10 billion and CERN's yearly operating budget of $1 billion, with over 2500 physicists working on site.

Reports from CERN today state that repairs were successful and the collider will be poised for proton collisions next week at a base energy of 0.90 Trillion Electron Volts. CERN is aiming for much higher energies never before attempted, 5 TeV per beam before October 12th.

CERN didn't say whether the collider was operating with beams or not during the emergency. If it had beams shooting through the ring, protons could have scattered, further warming and quenching some warm magnets, that might have exploded if the automatic heaters had also been affected by the power failures. Several sectors did warm partially at the LHC. If they all had, in the worst case scenario, beams might have unravelled and crashed down the LHC, causing a catastrophic failure. With the warming of magnets and helium coolant, up to 40 refrigeration plants above and below ground could have failed, usually with a loss of helium. In December 2003, the Tevatron had a catastrophic beam loss and a major quench, not related to thunderstorms.

Some hassles during storms have hampered operations at Fermilab's Tevatron, currently the most powerful collider until the LHC goes full blast. Even though both colliders are largely underground, lightning travels well though moist earth or wet clay, and part of the LHC in the vicinity of the second largest experiment, the CMS, has been excavated from clay, so unstable that it had to be artificially frozen during excavation of the giant CMS cavern. Hence its nickname at CERN, see-a-mess.

Like any machine the LHC is vulnerable to its environment and its own weaknesses, not forgetting the colossal energies and particle collisions it will produce.


News Stories On LHC Thunderstorm

Henderson, Mark. "'Big Bang Machine' is back on collision course after its glitches are fixed", Sept 18, 2008, TimesOnline,
http://www.timesonline.co.uk/tol/news/uk/science/article4774817.ece
Brouet, Anne-Marie. "Panne de faisceau dans le LHC", Sept 17, 2008, Tribune de Genève, http://www.tdg.ch/geneve/actu/2008/09/16/systeme-froid-gele-faisceau-lhc
Highfield, Roger. "The Large Hadron Collider: First subatomic particle collision to happen next week", Sept 16, 2008, Telegraph UK, http://www.telegraph.co.uk/earth/main.jhtml?view=DETAILS&grid=&xml=/earth/2008/09/16/scilhc116.xml

LHC Costs And Benefits

O'Neill, Martin. "Politics of proton smashing", Sept 17, 2008, New Statesman,
http://www.newstatesman.com/international-politics/2008/09/physics-lhc-cern-scientific

Tevatron Thunderstorms at 100,000 electron volts

Mosher, Dave. "Lightening strikes, Tevatron blinks", Oct/Nov 2006, Symmetry Magazine,
http://www.symmetrymagazine.org/cms/?pid=1000391

Tevatron Quenches and Failures

Gillis, Alan. "Major Failures At The Tevatron", Apr 18, 2008, The Science of Conundrums,
http://bigsciencenews.blogspot.com/2008/04/major-failures-at-tevatron.html

Major Failures At The Tevatron

Tevatron main ring from the air. This morning's 5.2 earthquake in the Midwest, with dozens of aftershocks, rattled Chicago skyscrapers and homes in Cincinnati. It must have affected the Tevatron. Fermilab Today, published daily M-F on the web, says nothing about the earthquake, as of 11:30 AM Central, 7 hours after the first shock. Point is any complex and sensitive installation, whether a chemical plant or the Tevatron, would likely crashdown under an onslaught of earthquakes. In 1981, a small local quake at Arzew, Algeria crashed down LNG I and LNG II. The plants vented steam you could see for miles in the fiery glow of the natural gas flares during the night shift. No one was hurt and there was no serious damage, but it took about a week to get the plants up to speed. This is the real safety test for any plant, unplanned instant total shutdown, not ever done deliberately because of the many risks. How would the LHC fare during an earthquake, when even a thunderstorm can rock the Tevatron?

We do know what can go wrong at the Tevatron on a nice day. Not easy to find out. It's a buttoned down organization. Apart from the tritium leak made public, even major failures requiring extensive shutdowns for repairs and re-engineering, are not press released by Fermilab, as though the Tevatron has never suffered any downtime accidents. Even Scientific American has no stories published on Tevatron failures nor could I find any other media stories. The only window on operations at the Tevatron, is the Accelerator Update, a complex technical summary, through a link on Fermilab Today. Digging around with Google you can get info on the December 5, 2003 beam loss accident from a paper by Fermilab's N.V. Mokhov et al, Beam-Induced Damage To the Tevatron Components And What Has Been Done About It at http://lss.fnal.gov/archive/2006/conf/fermilab-conf-06-415-ad.pdf The only list of serious component failures in Run II, was from a talk in 2006 at an annual accelerator conference, EPAC at Edinburg, by Valeri Lebedev from Fermilab. Here's a partial list from the Lebedev pdf, Tevatron Failures section.

01-Mar-03 Downtime 12 Days Safety lead ground fault
05-Dec-03 Downtime 12 Days Catastrophic beam loss
20-Dec-03 Downtime 10 Days Cryostat vacuum leak
15-Mar-04 Downtime 12 Days Helium leak
21-Nov-05 Downtime 3 Wks Helium leak
14-Jan-06 Downtime 2 Wks Cryostat vacuum leak
22-Feb-06 Downtime 1 Wk+ Helium leak

(Subsequently, I found the way into a searchable database, a new website for Fermilab's Library, Technical Publications. No link to it either, oddly enough, from other Fermilab sites. Go to http://bss.fnal.gov/techpubs/fermilab_spires.html Another trick is to use Title Search for any search term like beam loss. Then you get 33 citations on beam loss from SPIRES.)

Besides the short presentation notes on the pdf by Lebedev, here then is Lebedev's detailed paper on the same subject, Tevatron Operational Status And Possible Lessons For The LHC. I couldn't find any transcripts of the candid talk he gave at the EPAC-2006 Conference, which would have been to a big LHC audience in Edinburg. But the full paper does clarify a lot of other problems like the faulty pressure relief (Kautzky) valves at the Tevatron, responsable for 2 quenches. "Considering this a systematic failure, we replaced the failed part in all of about 1200 valves during the 2006 shutdown." Then a discussion of the second catastrophic beam loss. "One of worst failures happened on December 20, 2003 when a hardware failure caused a beam loss with consecutive major multi-house (helium cooling/compression plants) quench with 2/3 of the beam lost in the ring before the beam was aborted. Magnets and beam collimators had to be repaired." Imagine 1200 valve repairs. Did they call every plumber in Chicago?

Mokhov's paper, cited above, is a condensed yet updated version of an earlier paper I found through the new tech docs database with a different name that goes Beam-Induced Damage to the Tevatron Collimators: Analysis and Dynamic Modeling of Beam Loss, Energy Deposition and Ablation. This is the fullest discussion on the 05-Dec-03 Catastrophic Beam Loss without going into repairs and redesign of certain Tevatron components. The shorter later paper was presented by Mokhov at another particle physics conference, the ICFA-HB2006 in Tsukuba, Japan. Since it's the best look at a big accident, here's a summary for nonphysicists.

This beam loss wasn't initially caused by the usual leaks or superconducting magnet quenches. These occurred within a split second of the failure of a Roman pot in the huge CDF detector, one of the 2 main experiments in the Tevatron ring. A Roman pot is a CERNRome group invention from the 1970's, a sampler inserted near a collision event to pickup particles deflected from the collision point of the 2 opposing beams.

This time this Roman pot didn't obey an operator command to withdraw from the beam collisions. It did momentarily, but reinserted itself well into the beams. This spewed particles hitting the Roman pot, upstream into the helium coolant and superconducting magnets. Two-thirds of them quenched, about 640, and of the 24 refrigeration houses used to cool the helium to cool the magnets, 16 failed. A release of helium wasn't discussed in Mokhov's paper, though some was vaporized causing damage.

Trillions of protons and antiprotons unravelled from the beam pipe turn after turn, a "major fraction of the 1.5 MJ beam lost in the ring", according to Lebedev. Mokhov notes, "The entire beam is lost during about 400 turns (8.4 msec). . ." in the blink of an eye.

Fortunately no one is allowed in the ring systems when the accelerators are operating because of the radiation and possible leaks and accidents, so no one was hurt below ground. Above ground effects to personnel were not observed. A lot of glazed eyes in front of monitors at least.

Mokhov focuses on the obvious damage and what was done to the Tevatron to correct a cascade of failures. Two collimators, a primary tungsten and another of stainless steel, took most of the beam energy direct hits. Ordinarily, collimators very near the beams, are used to reduce the size of a beam's halo, an action called cleaning or scaping the beam. Here they minimized the damage to the Tevatron. The tungsten abraded and the stainless steel was grooved as though by a 3mm beam, though the actual beam might have been less than a hair's thickness. A large cryogenic spool containing focusing elements was also badly damaged, but by helium evaporation and pressure rise during the quench. Lebedev cites Mokhov, are we in Moscow yet? Lebedev infers that Mokhov's short paper also covers the December 20 accident, which explains why there's some drilling by a beam through more steel somewhere.

Anyway, what actual loss of tungsten and stainless steel that might have been transformed into some other particles by the proton-antiproton beams is not discussed. Some of these unknown particles could still be in the Tevatron or some were released. Fermilab experiments with beams shot at various targets might show what they are, experiments either before or after the accident, but there is no discussion of this in Mokhov's paper. Neither is there information on what happens to helium or magnets or pipe hit by these very powerful beams. Recall my earlier post, Daily Battles At The Tevatron, that since there is a lithium lens used in the Tevatron it might be the source of tritium contamination at Fermilab. Would someone at Fermilab please check?

After a lot of study, Fermilab did modify the Tevatron to prevent a similar set of events. Some modifications needed were so obvious that it is startling they weren't implemented sooner, like for the AC power distribution systems. The emergency kicker magnets and the CAMAC Abort controls now have their own separate feed from the utility company's sub-station. "It now requires a failure of the substation." writes Mokhov obtusely, that is for the emergency beam dump to fail. Uninterruptable power supplies were also added to the CAMAC Abort rack. Other major modifications were made to the Tevatron, discussed at length in Mokhov's paper.

No more "misbehaved proton beam". There are other Russians at Fermilab with colorful English. I can imagine what you might hear at the Tevatron on a bad day. Beam is falling or is technical malfunction from data sensor or maybe crippled switch? Can we check? Not in a few milliseconds, Nicky.

Even though the Tevatron is closing down by 2010, Fermilab is fighting for another collider installation at Batavia, the enormous ILC or International Linear Collider, still on the Fermilab drawing board. So far it doesn't look good.

Daily Battles At The Tevatron

A quiet moment at the Tevatron Main Control Room. Murphy's Law usually runs the Tevatron. Anything that can go wrong, will go wrong. Highlights from April 7 to 9, 2008: Booster kicker magnet (MP02) fails. 10E10 of antiprotons lost. Lightening strikes cause store to abort and TeV to quench. Here's the full 5 page report for April 9 to 11, 2008, from the Fermilab Accelerator Update. "The day shift began with the Tevatron (TeV) in quench recovery. . ."

Well, its a big place with most of the aging equipment dating from 1982. "The Tevatron tunnel is slowly sagging." V. Lebedev writes in his paper on Tevetron Operation Status And Possible Lessons For The LHC. As a result, "During 2006 shutdown about 50 quadrupoles we unrolled." And ". . . because of compression of thermo-insulating spacers . . . It took 3 years and 3 major shutdowns to finish shimming for all 772 Tevatron SC dipoles this year. (2006)"

A lot of experiments and projects make for a hectic schedule, over 50, with some 20 centered at the Tevatron. A thousand superconducting magnets, 15 miles of helium pipe, 24 cryogenic refrigerator houses, but utterly dwarfed by the power and size of the monster LHC. A lot of small annoyances, like from the magnets, though supercooled why should they smell? Other strange smells cropping up, odd vapors, but no fires or radiation hot spots detected by biohazard teams. Mysterious and spooky in the old bowels of the accelerators, grimy cement floors and poor light, an aging industrial twilight city, zones contaminated with radiation and off limits to personnel.

The main problem is a plumber's nightmare. Stuck valves, helium leaks that mushroom 700 times in volume from the cryogenic state, nitrogen leaks as the secondary coolant, vacuum leaks from the beam lines, heat leaks usually electrical, and radiation leaks which contaminate the magnets and force nearly all the superconducting magnet quenches. And they quench a lot, an average of 8 quenches per month in 2006, down from 16 per month in 2001.

Even the roof leaks at the Meson Lab. And when there's a prairie thunderstorm it's pandemonium. Power and communications disrupted. Fire alarms triggered all over dozens of buildings. Even without a direct hit the lightening surging through the ground can terminate the Tevatron beam, an emergency situation, an instant beam dump or its 2 beams drilling holes through anything. Wacky? No, it already happened at the Tevatron twice in December, 2003. A catastrophic beam loss. Ron Moore, Tevatron Department Head, Beams Division, referring to one of the beam loss accidents, recently said ". . . the high energy causes damage even within short time periods. Within 16 nanoseconds, one beam burned through about 1.5 meters (about 5 feet) of solid steel."

The Tevatron's Main Injector ring has had an intermittent beam loss problem with a neutrino beam that's fired underground to a target detector in an abandoned mine in northern Minnesota, the Soudan mine, 735 km away. Not science fiction either, but you need Fermilab authorization before you can access the failure data. There's just a note published in Fermilab Today of December 7, 2006. "NuMI (Neutrino Main Injector) suffers intermittent beam loss when MI stashes" occurring December 4-6. The NuMI-MINOS project on neutrino physics is a big one that will probably go on without the old Tevatron, only the new Main Injector required to produce the beam.

The big worry is radiation contamination, like tritium, usually found in nuclear power plant cooling water and then your local river. Fermilab also relies on water cooling for certain elements of the Tevatron, even collecting rainwater for some of their needs. It's an eco friendly gesture. In November 2005, low levels of tritium were found in the waters around Fermilab, in Indian Creek and ponds on the site. First time in 30 years. But don't forget that 2 years before, the Tevatron had at least two beam accidents in 2003, the December 5 hit that quenched two-thirds of its superconducting magnets, and later that month, the December 20 failure about as serious. But how could radioactive tritium have been produced from protons (even though extracted from hydrogen) and antiprotons (from nickel), unless one or the other acted on something else, producing neutrons then acting on boron or lithium to make tritium. It's up to Fermilab to find out how this happened. All they have been doing is taking water samples. How about some basic research? Well almost. They did locate the tritium leak, a pipe from Fermilab connecting two cooling ponds. Fermilab says the problem is fixed with no further tritium detected. But where is the tritium that was discharging into the ponds? Still in the Tevatron system? And how was tritium produced in the first place? According to Lebedev in the same paper cited above on page 5, there is a lithium lens, apparently in the Debuncher. Lebedev adds that the lithium lens was to be upgraded by 2007. So was it damaged then? Transformed into tritium? No analysis provided by Lebedev.

There are many lessons here for the LHC just in Lebedev's 5 page paper. No doubt CERN is fully aware of them and have revised their engineering and operations with Fermilab's help. The problem is CERN is not building a new and better Tevatron, but a much bigger machine with energies surpassing the Tevatron's beam by an enormous factor of 150.

Two reassuring things though. On the main desk at the Main Control Room, a lowtech box of Puffs and a flashlight. Click to enlarge.

You Appear To Be Running A Large LHC

Windows XP at CERN LHC CMS. It's little things like ...please send an error report, that could get us into a lot of trouble. The Compact Muon Solenoid weighs in at 5 jumbo jets with a magnetic field of 4 Teslas at 21,000 amps, the magnet supercooled to -269 degrees C. It's all boggling, the sheer scale and complexity, with 3 other major experiments, each in its own giant cavern and 2 piggyback ones in a monumental 27 km ring tunnel of cryogenics and superconducting magnets for the beam pipes.1700 major interconnections for 1232 dipoles in 15 meter lengths for curving the beams, 400 5-7 meter focusing quadrupoles, another 5,000 corrector magnets. Hundreds of thousands of bolts, 40,000 or 10 km worth of special welds using tungsten gas, 65,000 electrical splices of superconducting cables and a lot of duct tape. For this main ring alone the nominal current is 11,850 amps. Imagine plugging in the CMS toaster for a total draw of 32,850 amps. CERN should be able to do that without knocking out the European Power Grid. But the energy stored in the ring magnets once powered is 10 GJ with 725 MJ in the beam. A small loss is sufficient to quench a superconducting magnet. If somebody turns on their air conditioner in Geneva, the beam could dump 157 kg of TNT equivalent, the ring another 2 metric tonnes.

After sifting through a mountain of info on CERN's friendly websites I'd say the risk of conventional accidents is the first thing to consider. This one failure in the power supply could bring down the entire LHC. Add the other 4 tonnes of TNT equivalent in the CMS. Explosions, electrical fires, rupture of the superfluid helium coolant under considerable pressure in the cryogenic pipe system running throughout the ring and CMS, pumps and compressors failing. The Large Hadron Collider could be a "ten billion dollars or whatever it is" disaster a nanosecond before someone clicks Send Error Report or Don't Send.