LHC Quench Stops CERN: Re-start Delayed Again

After last Friday's massive quench at the LHC, CERN has announced that re-starting the collider will have to wait even beyond 2 months for repairs and downtime. With a planned winter shutdown in late November, even if the collider were ready, there would not be enough time to continue tests. Re-start of the LHC should be in April 2009.

Day 7 into the major accident damaging part of the LHC helium cooling and superconducting magnet systems, indicates little progress as yet. Scientists at CERN are still investigating the failure in Sector 3-4, an eighth of the 17 mile ring of magnets that are normally cooled to near absolute zero temperature. The extent of the damage is greater than was reported by CERN initially. Dismissive at first of the collider's "teething problems", CERN suggested that the LHC could be operational soon. A sketchy press release was followed by another sketchy press release.

Oddly after the BBC discovered the massive quench of about 100 magnets in a log entry on a CERN website, "the entry has since been removed" according to TimesOnline the day after.

It was also TimesOnline that said on September 21st that a connection between 2 giant magnets melted, with a release of one tonne of helium, adding the heat also melted 2 giant magnets, causing an explosive release of liquid helium which blasted helium gas through the ring tunnel. No one was hurt as no personnel were in the collider ring said CERN, normally only off limits when proton beams are operating. CERN said that at the time of the accident last Friday morning, there were no beams and only admitted to "a large helium leak" and "a faulty electrical connection between two magnets, which probably melted".

The only piece of real news by CERN's chief spokesman, Dr James Gillies, who does the standard tour of the major media outlets to reassure journalists, outside of his department's press releases, was about the damaged magnets in question that he identified as giant quadrupoles, in the Telegram UK, September 20th. Later the probably melted connection turned out to be a busbar, a type of reinforced splice of magnet ribbon cable, one of many such connections between magnets.

It wasn't exactly a routine failure. Commissioning was still going on in Point 4 and on either side into the later damaged zone of Sector 3-4. and in 4-5. The high energy test was of the RF system for anticipated higher loads when it would be used to power up beams to 5 TeV, that were scheduled for later this year.

As of today only specialized safety teams and technicians have been allowed in to check for hazards and equipment damage. Following the accident Sector 3-4 showed some continuing warming of magnets and helium, some of the warmest then cooling, but now the Sector has been partially re-cooled. Most magnets are at 30 K, with no further helium leaks reported.

Eventually the damaged Sector 3-4 will be warmed to room temperature for repairs to be effected, but that would mean that remaining helium would have to be drained from the system first, and that hasn't started yet. Since temperatures have been brought down, it seems that the strategy is to cool the helium enough so that it can be safely drained as a liquid and stored as it's very expensive to replace. At least two damaged quadruoles will have to be cut out as they are welded together and then new ones welded in, a difficult and expensive proposition.

Coming on the heels of two other major failures, the hacking of part of the CMS computer system, during the big September 10th media launch, and the destruction of a 30 tonne surface transformer during a thunderstorm shortly afterwards, powering 2 sectors of the LHC cooling system, the need for a more careful re-launch of the collider is apparent to CERN, hence the further delay to spring.

CERN publicly has put on a brave if vague face, with its first very short press release of the 20th September, "Incident in LHC sector 3-4", though it was abundantly clear the day of the accident September 19th that it was more than an incident, as reported by the BBC that day. Even on the day of the accident, according to Scientific American, "CERN said on Friday that "The LHC is on course for [its] first collisions in a matter of weeks", just a day later it announced the minimum two-month repair job." Lately Dr Robert Aymar, Director General of CERN referred to the accident as "undoubtedly a psychological blow."

But there are other factors leading up to the accident that haven't been addressed. Certainly timing the First Beam to suit a media bash opening, when the CMS was being hacked, and still going ahead with it, suggests the need to perform for the media above ordinary safety considerations. Several other factors also indicate a rush to perform for the media and a huge worldwide audience.

Reuters reported September 10th, that there were "Small electrical issues before CERN machine start-up". "Project leader Lyn Evans gave no details . . ."

Operating the collider during a thunderstorm is certainly a known risk at the Tevatron, and 36 hours after the September 10th opening, a 30 tonne surface transformer that powers part of the helium cooling system failed during a lightening strike. Cause of failure still not confirmed by CERN.

According to the LHC Commissioning with Beam page, "The winter shut-down will then be used to commissioning and train the magnets up to full current, such that the 2009 run will start at the full 14 TeV design energy." Lyn Evans reiterated the same point in a talk on "The LHC Machine" at a CERN colloquium, Strings 2008, August 18th. Some installed magnets from another supplier had to be retrained. Earlier this year there were plans to jump to 7 TeV. Then 5 TeV was announced as rather a surprise, the new goal per beam. Where the damaged magnets from this lot of not fully trained magnets?

What if CERN has run out of luck? They have been lucky, not running beams they say during the transformer failure and the huge magnet quench, but what if they had been running beams? Beams could have been lost causing more damage. They ran beams during the hacking of CMS, but luckily there were no collisions.

There's a long road ahead. As Fermilab/SLAC's sponsored magazine, Symmetry wrapped up the transformer accident at the LHC, "These kinds of hiccoughs in starting up a large collider are not surprising as the LHC has millions of critical components."

What? Only millions?


References

Gillis, Alan. "Accident Cripples LHC", The Science of Conundrums", September 19, 2008, http://bigsciencenews.blogspot.com/2008/09/accident-cripples-lhc.html

Higgins, Alexander G. "Small accidents mean big trouble for supercollider", AP/PhysOrg, September 22, 2008, http://www.physorg.com/news141278719.html

CERN. "Incident in sector 3-4", Press Release, September 20th, 2008, http://press.web.cern.ch/Press/PressReleases/Releases2008/PR09.08E.html

CERN. "LHC re-start scheduled for 2009", Press Release, September 23, 2009, http://press.web.cern.ch/Press/PressReleases/Releases2008/PR10.08E.html

Reuters. "Hadron Collider halted for months", Reuters News Video, September 21, 2008, http://www.reuters.com/news/video?videoId=91036

Chalmers, Matthew and Henderson, Mark. "CERN delays atom-smashing over magnet fault", TimesOnline, September 20, 2008, http://www.timesonline.co.uk/tol/news/uk/science/article4789673.ece

Leake, Jonathan. "Oh blast, that's the wrong kind of big bang", TimesOnline, September 21, 2008, http://www.timesonline.co.uk/tol/news/uk/article4794825.ece

Highfield, Roger. "Large Hadron Collider to be turned off for two months following damage", Telegraph UK, September 20, 2008, http://www.telegraph.co.uk/earth/main.jhtml?view=DETAILS&grid=&xml=/earth/2008/09/20/scilhc120.xml

Gillis, Alan. "LHC Not So Safe". The Science of Conundrums, September 12, 2008, http://bigsciencenews.blogspot.com/2008/09/lhc-not-so-safe.html

Gillis, Alan. "LHC Fails Thunderstorm Test", The Science of Conundrums, September 17, 2008, http://bigsciencenews.blogspot.com/2008/09/lhc-fails-thunderstorm-test.html

Lite, Jordan. "Hobbled LHC shuttered for repairs; 'No big deal say scientists'", SciAm, September 22, 2008, http://www.sciam.com/blog/60-second-science/post.cfm?id=hobbled-lhc-shuttered-for-repairs-n-2008-09-22

Reuters. "Small electrical issues before CERN machine start-up", Reuters News, September 10, 2008, http://www.reuters.com/article/latestCrisis/idUSLA57119

CERN. "LHC Commissioning with Beam", LHC Commissioning, page still current, http://lhc-commissioning.web.cern.ch/lhc-commissioning/

Gillis, Alan. "LHC Start-up To Shutdown 2008", The Science of Conundrums, August 22, 2008, http://bigsciencenews.blogspot.com/2008/08/lhc-start-up-to-shut-down-2008.html

CERN. "LHC: countdown to beam begins", CERN Courier, August 18, 2008, http://cerncourier.com/cws/article/cern/35431

Harris, David. "LHC glitch means two month delay" symmetrybreaking, September 20, 2008, http://www.symmetrymagazine.org/breaking/2008/09/20/lhc-glitch-means-two-month-delay/

Accident Cripples LHC

No collisions, no beams either next week at the LHC. The BBC reports an alarming quench of about 100 superconducting magnets today, that heated up as much as 100 C. A tonne of liquid helium spilled into the tunnel and the CERN fire brigade went in. Cause of the quenching has not been announced, nor have any injuries been reported. Liquid helium leaks vaporize back to a gas almost instantly and would freeze or choke personnel present.

Ordinarily magnet quenches occur when proton beams are lost or scatter into magnets, causing helium coolant and magnets to heat and lose their superconductivity and their power to keep proton beams within the collider. CERN doesn't say whether beams were running today during the accident. They would have been at their lowest power 0.45 TeV per beam, though beams 11 times more powerful at 5 TeV were scheduled before October 12th. Vacuum conditions were also lost said the BBC, which suggests beam damage, though CERN hasn't commented.

This is on top of another major failure to the helium cooling system. Last Friday the 12th's thunderstorm burned out a giant 30 tonne 12 Million Volt Amperes surface transformer that powers some of the helium cryogenics system. Although CERN admitted the problem yesterday, without mentioning the thunderstorm, and said the transformer was replaced last weekend, there was no explanation for the long delay in informing the public. The failure of the transformer caused an initial warming in the helium coolant in 2 of the colliders 8 sectors, some as late as September 17th at near 7 K

Whether an ongoing transformer problem contributed to quenching has not been announced. The TimesOnline initial report on damage said, "One of the beams had been captured by Friday, but work was then interrupted by the loss of electrical transformers that power the cryogenic cooling system . . ." A beam apparently running during the thunderstorm, and more than one transformer lost. CERNS's lastest progress report yesterday was sketchy.

The damaged Sector3-4, an eighth of the 17 mile collider has still not been stabilized since this morning's accident. About half of it or a mile length was well above normal 1.9 K design temperature, about a quarter of the sector and its magnets this afternoon at 15:46 PM were about 82 K to 110 K. Currently all magnets have warmed. The warmest show a slight recovery from 110 K down to 99 K. and Sector 3-4 is still in crisis as of the 20th September 0:418 AM, with no accurate readout for about half the magnets, and the rest showing more warming, with a few outside the spike zone climbing abruptly in temperature since this afternoon.

Given the gravity of the accident, repairs would take a week or longer.


References

BBC. "Hadron Collider forced to halt", Sept 19, 2008, BBC News, http://news.bbc.co.uk/1/hi/sci/tech/7626256.stm

CERN. "LHC progress report, week 1", Sept 18, 2008, LHC First Beam, http://lhc-first-beam.web.cern.ch/lhc-first-beam/News/lhc_080918.html

Gillis, Alan. "LHC Fails Thunderstorm Test", Sept 17,2008, The Science of Conundrums, http://bigsciencenews.blogspot.com/2008/09/lhc-fails-thunderstorm-test.html

Henderson, Mark. "'Big Bang Machine' back on collision course after its glitches are fixed", Sept 18, 2008, TimesOnline, http://www.timesonline.co.uk/tol/news/uk/science/article4774817.ece

LHC Fails Thunderstorm Test

Nature had her own ideas about testing LHC safety. A thunderstorm last Friday knocked out some transformers at the LHC near Geneva that are part of the helium cooling system, that cools the magnets that keep the proton beams travelling (near light speed) on a circular path through the collider. Technicians have been scrambling to fix the problems, but not before some magnets warmed well above standard operating temperatures, some reaching almost 7K from the usual ultra cold 1.9K .

Electromagnets at the LHC need to be this cold to be superconducting, or at peak efficiency, in order to deliver extremely high magnetic fields in the 27 km ring of 1200 giant magnets and thousands of smaller ones, at 8.33 Teslas or about 200,000 times the earth's magnetic field strength.

Everything about this atom smasher is boggling, including the numbers, like its cost at about $10 billion and CERN's yearly operating budget of $1 billion, with over 2500 physicists working on site.

Reports from CERN today state that repairs were successful and the collider will be poised for proton collisions next week at a base energy of 0.90 Trillion Electron Volts. CERN is aiming for much higher energies never before attempted, 5 TeV per beam before October 12th.

CERN didn't say whether the collider was operating with beams or not during the emergency. If it had beams shooting through the ring, protons could have scattered, further warming and quenching some warm magnets, that might have exploded if the automatic heaters had also been affected by the power failures. Several sectors did warm partially at the LHC. If they all had, in the worst case scenario, beams might have unravelled and crashed down the LHC, causing a catastrophic failure. With the warming of magnets and helium coolant, up to 40 refrigeration plants above and below ground could have failed, usually with a loss of helium. In December 2003, the Tevatron had a catastrophic beam loss and a major quench, not related to thunderstorms.

Some hassles during storms have hampered operations at Fermilab's Tevatron, currently the most powerful collider until the LHC goes full blast. Even though both colliders are largely underground, lightning travels well though moist earth or wet clay, and part of the LHC in the vicinity of the second largest experiment, the CMS, has been excavated from clay, so unstable that it had to be artificially frozen during excavation of the giant CMS cavern. Hence its nickname at CERN, see-a-mess.

Like any machine the LHC is vulnerable to its environment and its own weaknesses, not forgetting the colossal energies and particle collisions it will produce.


News Stories On LHC Thunderstorm

Henderson, Mark. "'Big Bang Machine' is back on collision course after its glitches are fixed", Sept 18, 2008, TimesOnline,
http://www.timesonline.co.uk/tol/news/uk/science/article4774817.ece
Brouet, Anne-Marie. "Panne de faisceau dans le LHC", Sept 17, 2008, Tribune de Genève, http://www.tdg.ch/geneve/actu/2008/09/16/systeme-froid-gele-faisceau-lhc
Highfield, Roger. "The Large Hadron Collider: First subatomic particle collision to happen next week", Sept 16, 2008, Telegraph UK, http://www.telegraph.co.uk/earth/main.jhtml?view=DETAILS&grid=&xml=/earth/2008/09/16/scilhc116.xml

LHC Costs And Benefits

O'Neill, Martin. "Politics of proton smashing", Sept 17, 2008, New Statesman,
http://www.newstatesman.com/international-politics/2008/09/physics-lhc-cern-scientific

Tevatron Thunderstorms at 100,000 electron volts

Mosher, Dave. "Lightening strikes, Tevatron blinks", Oct/Nov 2006, Symmetry Magazine,
http://www.symmetrymagazine.org/cms/?pid=1000391

Tevatron Quenches and Failures

Gillis, Alan. "Major Failures At The Tevatron", Apr 18, 2008, The Science of Conundrums,
http://bigsciencenews.blogspot.com/2008/04/major-failures-at-tevatron.html

LHC News Update

Well Fuk, over to you.

Right Brian. Over on Earth, 'Red Button Day' is fast approaching. That's when they push the big red button at the Large Hadron Collider.

It's finally a go, right Fuk?

We make it a preliminary feed. We're getting some Russian satellite interference. We're filtering now. Gillies at CERN is on top of it.

Wasn't he on top of it for June? Fuk?

That was full cooldown, Brian.

So we got full cooldown, Fuk?

No Fuking way. They're almost there.

Fuking glitch factory, Fuk? Weren't they gonna hit that Fuking button 3 years ago?

That's affirmative, Brian. But when they do, they're gonna hit a Fuking Home Run.

Hope you're right, Fuk.


The real down to earth story is the Russians have scooped CERN on start-up of the LHC. RIA Novosti World says the collider "will be officially unveiled October 21". The source is a Russian scientist, Alexander Vodopyanov of the Joint Institute for Nuclear Research (Dubna). CERN has been awash with start-up rumors lately, including one in The New York Times, and hasn't commented.

James Gillies, CERN's PR Director, reported in CosmicLog MSNBC, did announce that the timing of the first beam injection will be sometime during the first 2 weeks of September. "The machine is basically cold now." See for yourself. Click on LHC Cooldown Status Live, on this page, from a column of KEY CERN SITES on your right, only 5 of 8 sectors are cold enough, at 1.9 K . But this first injection is for only one beam into one sector of 8, basically a test of one of the Injection Kicker magnets that kicks a proton beam into the main ring from the PS and SPS pre-accelerators. This will be the clockwise beam, the counterclockwise beam test should be a month later. The beam energy isn't specified, but it will probably be 0.45 TeV, which is what the pre-accelerators deliver to the main 17 mile ring, where beams will be further accelerated to 5 TeV, maybe in October as Vodopyanov says, just before the official inaugeration.

CosmicLog also has some useful updates on a rival Fermilab search for the Higgs Boson, some ZZs turning up, yawn. And the Wagner/Sancho suit, to be heard in court on September 2nd. It could be dismissed then or an injunction could be granted to delay LHC start-up. Or proceedings could drag on. Probably the Hawaiian Court will say they don't have jurisdiction.

And all the fuss on the latest LHC Rap Video, not enough juice to power a hearing aid, at 0.0000000000003 GeV. The Cernettes Channel on Youtube can't be beat.

Lately too, you might want to see a dazzling gallery of 28 hi-res images of the LHC by CERN photographers on The Boston Globe on-line edition's Big Picture.

STOP PRESS August 7, 2008

It's official. CERN announced the first attempt to circulate a beam of protons through the LHC ring will be September 10, 2008. The beam energy is what the pre-accelerators can deliver, that is 0.45 TeV. The target is still 5 TeV per beam later this year, with collisions possible before the planned Winter shutdown.

The big event will be on Eurovision, but elsewhere you'll need a paid on-line account with them. There's a big media circus being carefully planned for hundreds of journalists descending from the skies or driving in. The good news is the rest of us will be able to watch it live on up to 4 CERN Webcast Channels for free. Thanks to all at CERN. Don't forget to stock up on your Collider Donuts and don't miss this. If you're worried, it's still too early to be really worried, only a low energy single beam run. Still the LHC has never been tested like this. Expect some sparks and quarks.

The initial test of an Injection Kicker magnet into one sector is way ahead of schedule, set for this weekend. This must be an LHC first. Ahead of schedules and rumors? Amazing!

Make a date to watch the Collider Show here, Wednesday September 10, 2008. It's five hours later in France if you're in New York. There's a CERN LHC LIVE WEBCAMS spot on your right. Click on CERN Live Webcasts. Next one is August 18, 08:00 or 3 AM in NY, lectures on String theory, "Strings 2008".