Thursday, September 25, 2008

LHC Quench Stops CERN: Re-start Delayed Again

After last Friday's massive quench at the LHC, CERN has announced that re-starting the collider will have to wait even beyond 2 months for repairs and downtime. With a planned winter shutdown in late November, even if the collider were ready, there would not be enough time to continue tests. Re-start of the LHC should be in April 2009.

Day 7 into the major accident damaging part of the LHC helium cooling and superconducting magnet systems, indicates little progress as yet. Scientists at CERN are still investigating the failure in Sector 3-4, an eighth of the 17 mile ring of magnets that are normally cooled to near absolute zero temperature. The extent of the damage is greater than was reported by CERN initially. Dismissive at first of the collider's "teething problems", CERN suggested that the LHC could be operational soon. A sketchy press release was followed by another sketchy press release.

Oddly after the BBC discovered the massive quench of about 100 magnets in a log entry on a CERN website, "the entry has since been removed" according to TimesOnline the day after.

It was also TimesOnline that said on September 21st that a connection between 2 giant magnets melted, with a release of one tonne of helium, adding the heat also melted 2 giant magnets, causing an explosive release of liquid helium which blasted helium gas through the ring tunnel. No one was hurt as no personnel were in the collider ring said CERN, normally only off limits when proton beams are operating. CERN said that at the time of the accident last Friday morning, there were no beams and only admitted to "a large helium leak" and "a faulty electrical connection between two magnets, which probably melted".

The only piece of real news by CERN's chief spokesman, Dr James Gillies, who does the standard tour of the major media outlets to reassure journalists, outside of his department's press releases, was about the damaged magnets in question that he identified as giant quadrupoles, in the Telegram UK, September 20th. Later the probably melted connection turned out to be a busbar, a type of reinforced splice of magnet ribbon cable, one of many such connections between magnets.

It wasn't exactly a routine failure. Commissioning was still going on in Point 4 and on either side into the later damaged zone of Sector 3-4. and in 4-5. The high energy test was of the RF system for anticipated higher loads when it would be used to power up beams to 5 TeV, that were scheduled for later this year.

As of today only specialized safety teams and technicians have been allowed in to check for hazards and equipment damage. Following the accident Sector 3-4 showed some continuing warming of magnets and helium, some of the warmest then cooling, but now the Sector has been partially re-cooled. Most magnets are at 30 K, with no further helium leaks reported.

Eventually the damaged Sector 3-4 will be warmed to room temperature for repairs to be effected, but that would mean that remaining helium would have to be drained from the system first, and that hasn't started yet. Since temperatures have been brought down, it seems that the strategy is to cool the helium enough so that it can be safely drained as a liquid and stored as it's very expensive to replace. At least two damaged quadruoles will have to be cut out as they are welded together and then new ones welded in, a difficult and expensive proposition.

Coming on the heels of two other major failures, the hacking of part of the CMS computer system, during the big September 10th media launch, and the destruction of a 30 tonne surface transformer during a thunderstorm shortly afterwards, powering 2 sectors of the LHC cooling system, the need for a more careful re-launch of the collider is apparent to CERN, hence the further delay to spring.

CERN publicly has put on a brave if vague face, with its first very short press release of the 20th September, "Incident in LHC sector 3-4", though it was abundantly clear the day of the accident September 19th that it was more than an incident, as reported by the BBC that day. Even on the day of the accident, according to Scientific American, "CERN said on Friday that "The LHC is on course for [its] first collisions in a matter of weeks", just a day later it announced the minimum two-month repair job." Lately Dr Robert Aymar, Director General of CERN referred to the accident as "undoubtedly a psychological blow."

But there are other factors leading up to the accident that haven't been addressed. Certainly timing the First Beam to suit a media bash opening, when the CMS was being hacked, and still going ahead with it, suggests the need to perform for the media above ordinary safety considerations. Several other factors also indicate a rush to perform for the media and a huge worldwide audience.

Reuters reported September 10th, that there were "Small electrical issues before CERN machine start-up". "Project leader Lyn Evans gave no details . . ."

Operating the collider during a thunderstorm is certainly a known risk at the Tevatron, and 36 hours after the September 10th opening, a 30 tonne surface transformer that powers part of the helium cooling system failed during a lightening strike. Cause of failure still not confirmed by CERN.

According to the LHC Commissioning with Beam page, "The winter shut-down will then be used to commissioning and train the magnets up to full current, such that the 2009 run will start at the full 14 TeV design energy." Lyn Evans reiterated the same point in a talk on "The LHC Machine" at a CERN colloquium, Strings 2008, August 18th. Some installed magnets from another supplier had to be retrained. Earlier this year there were plans to jump to 7 TeV. Then 5 TeV was announced as rather a surprise, the new goal per beam. Where the damaged magnets from this lot of not fully trained magnets?

What if CERN has run out of luck? They have been lucky, not running beams they say during the transformer failure and the huge magnet quench, but what if they had been running beams? Beams could have been lost causing more damage. They ran beams during the hacking of CMS, but luckily there were no collisions.

There's a long road ahead. As Fermilab/SLAC's sponsored magazine, Symmetry wrapped up the transformer accident at the LHC, "These kinds of hiccoughs in starting up a large collider are not surprising as the LHC has millions of critical components."

What? Only millions?


Gillis, Alan. "Accident Cripples LHC", The Science of Conundrums", September 19, 2008,

Higgins, Alexander G. "Small accidents mean big trouble for supercollider", AP/PhysOrg, September 22, 2008,

CERN. "Incident in sector 3-4", Press Release, September 20th, 2008,

CERN. "LHC re-start scheduled for 2009", Press Release, September 23, 2009,

Reuters. "Hadron Collider halted for months", Reuters News Video, September 21, 2008,

Chalmers, Matthew and Henderson, Mark. "CERN delays atom-smashing over magnet fault", TimesOnline, September 20, 2008,

Leake, Jonathan. "Oh blast, that's the wrong kind of big bang", TimesOnline, September 21, 2008,

Highfield, Roger. "Large Hadron Collider to be turned off for two months following damage", Telegraph UK, September 20, 2008,

Gillis, Alan. "LHC Not So Safe". The Science of Conundrums, September 12, 2008,

Gillis, Alan. "LHC Fails Thunderstorm Test", The Science of Conundrums, September 17, 2008,

Lite, Jordan. "Hobbled LHC shuttered for repairs; 'No big deal say scientists'", SciAm, September 22, 2008,

Reuters. "Small electrical issues before CERN machine start-up", Reuters News, September 10, 2008,

CERN. "LHC Commissioning with Beam", LHC Commissioning, page still current,

Gillis, Alan. "LHC Start-up To Shutdown 2008", The Science of Conundrums, August 22, 2008,

CERN. "LHC: countdown to beam begins", CERN Courier, August 18, 2008,

Harris, David. "LHC glitch means two month delay" symmetrybreaking, September 20, 2008,


David said...

I wrote the comment about "millions" of critical components as an estimate. Clearly there are many more components but the number of "critical" ones (ones which might cause the machine to stop working if they failed) is not the whole lot.

kaden said...

I recently came accross your blog and have been reading along. I thought I would leave my first comment. I dont know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.