LHC Quench Stops CERN: Re-start Delayed Again

After last Friday's massive quench at the LHC, CERN has announced that re-starting the collider will have to wait even beyond 2 months for repairs and downtime. With a planned winter shutdown in late November, even if the collider were ready, there would not be enough time to continue tests. Re-start of the LHC should be in April 2009.

Day 7 into the major accident damaging part of the LHC helium cooling and superconducting magnet systems, indicates little progress as yet. Scientists at CERN are still investigating the failure in Sector 3-4, an eighth of the 17 mile ring of magnets that are normally cooled to near absolute zero temperature. The extent of the damage is greater than was reported by CERN initially. Dismissive at first of the collider's "teething problems", CERN suggested that the LHC could be operational soon. A sketchy press release was followed by another sketchy press release.

Oddly after the BBC discovered the massive quench of about 100 magnets in a log entry on a CERN website, "the entry has since been removed" according to TimesOnline the day after.

It was also TimesOnline that said on September 21st that a connection between 2 giant magnets melted, with a release of one tonne of helium, adding the heat also melted 2 giant magnets, causing an explosive release of liquid helium which blasted helium gas through the ring tunnel. No one was hurt as no personnel were in the collider ring said CERN, normally only off limits when proton beams are operating. CERN said that at the time of the accident last Friday morning, there were no beams and only admitted to "a large helium leak" and "a faulty electrical connection between two magnets, which probably melted".

The only piece of real news by CERN's chief spokesman, Dr James Gillies, who does the standard tour of the major media outlets to reassure journalists, outside of his department's press releases, was about the damaged magnets in question that he identified as giant quadrupoles, in the Telegram UK, September 20th. Later the probably melted connection turned out to be a busbar, a type of reinforced splice of magnet ribbon cable, one of many such connections between magnets.

It wasn't exactly a routine failure. Commissioning was still going on in Point 4 and on either side into the later damaged zone of Sector 3-4. and in 4-5. The high energy test was of the RF system for anticipated higher loads when it would be used to power up beams to 5 TeV, that were scheduled for later this year.

As of today only specialized safety teams and technicians have been allowed in to check for hazards and equipment damage. Following the accident Sector 3-4 showed some continuing warming of magnets and helium, some of the warmest then cooling, but now the Sector has been partially re-cooled. Most magnets are at 30 K, with no further helium leaks reported.

Eventually the damaged Sector 3-4 will be warmed to room temperature for repairs to be effected, but that would mean that remaining helium would have to be drained from the system first, and that hasn't started yet. Since temperatures have been brought down, it seems that the strategy is to cool the helium enough so that it can be safely drained as a liquid and stored as it's very expensive to replace. At least two damaged quadruoles will have to be cut out as they are welded together and then new ones welded in, a difficult and expensive proposition.

Coming on the heels of two other major failures, the hacking of part of the CMS computer system, during the big September 10th media launch, and the destruction of a 30 tonne surface transformer during a thunderstorm shortly afterwards, powering 2 sectors of the LHC cooling system, the need for a more careful re-launch of the collider is apparent to CERN, hence the further delay to spring.

CERN publicly has put on a brave if vague face, with its first very short press release of the 20th September, "Incident in LHC sector 3-4", though it was abundantly clear the day of the accident September 19th that it was more than an incident, as reported by the BBC that day. Even on the day of the accident, according to Scientific American, "CERN said on Friday that "The LHC is on course for [its] first collisions in a matter of weeks", just a day later it announced the minimum two-month repair job." Lately Dr Robert Aymar, Director General of CERN referred to the accident as "undoubtedly a psychological blow."

But there are other factors leading up to the accident that haven't been addressed. Certainly timing the First Beam to suit a media bash opening, when the CMS was being hacked, and still going ahead with it, suggests the need to perform for the media above ordinary safety considerations. Several other factors also indicate a rush to perform for the media and a huge worldwide audience.

Reuters reported September 10th, that there were "Small electrical issues before CERN machine start-up". "Project leader Lyn Evans gave no details . . ."

Operating the collider during a thunderstorm is certainly a known risk at the Tevatron, and 36 hours after the September 10th opening, a 30 tonne surface transformer that powers part of the helium cooling system failed during a lightening strike. Cause of failure still not confirmed by CERN.

According to the LHC Commissioning with Beam page, "The winter shut-down will then be used to commissioning and train the magnets up to full current, such that the 2009 run will start at the full 14 TeV design energy." Lyn Evans reiterated the same point in a talk on "The LHC Machine" at a CERN colloquium, Strings 2008, August 18th. Some installed magnets from another supplier had to be retrained. Earlier this year there were plans to jump to 7 TeV. Then 5 TeV was announced as rather a surprise, the new goal per beam. Where the damaged magnets from this lot of not fully trained magnets?

What if CERN has run out of luck? They have been lucky, not running beams they say during the transformer failure and the huge magnet quench, but what if they had been running beams? Beams could have been lost causing more damage. They ran beams during the hacking of CMS, but luckily there were no collisions.

There's a long road ahead. As Fermilab/SLAC's sponsored magazine, Symmetry wrapped up the transformer accident at the LHC, "These kinds of hiccoughs in starting up a large collider are not surprising as the LHC has millions of critical components."

What? Only millions?


References

Gillis, Alan. "Accident Cripples LHC", The Science of Conundrums", September 19, 2008, http://bigsciencenews.blogspot.com/2008/09/accident-cripples-lhc.html

Higgins, Alexander G. "Small accidents mean big trouble for supercollider", AP/PhysOrg, September 22, 2008, http://www.physorg.com/news141278719.html

CERN. "Incident in sector 3-4", Press Release, September 20th, 2008, http://press.web.cern.ch/Press/PressReleases/Releases2008/PR09.08E.html

CERN. "LHC re-start scheduled for 2009", Press Release, September 23, 2009, http://press.web.cern.ch/Press/PressReleases/Releases2008/PR10.08E.html

Reuters. "Hadron Collider halted for months", Reuters News Video, September 21, 2008, http://www.reuters.com/news/video?videoId=91036

Chalmers, Matthew and Henderson, Mark. "CERN delays atom-smashing over magnet fault", TimesOnline, September 20, 2008, http://www.timesonline.co.uk/tol/news/uk/science/article4789673.ece

Leake, Jonathan. "Oh blast, that's the wrong kind of big bang", TimesOnline, September 21, 2008, http://www.timesonline.co.uk/tol/news/uk/article4794825.ece

Highfield, Roger. "Large Hadron Collider to be turned off for two months following damage", Telegraph UK, September 20, 2008, http://www.telegraph.co.uk/earth/main.jhtml?view=DETAILS&grid=&xml=/earth/2008/09/20/scilhc120.xml

Gillis, Alan. "LHC Not So Safe". The Science of Conundrums, September 12, 2008, http://bigsciencenews.blogspot.com/2008/09/lhc-not-so-safe.html

Gillis, Alan. "LHC Fails Thunderstorm Test", The Science of Conundrums, September 17, 2008, http://bigsciencenews.blogspot.com/2008/09/lhc-fails-thunderstorm-test.html

Lite, Jordan. "Hobbled LHC shuttered for repairs; 'No big deal say scientists'", SciAm, September 22, 2008, http://www.sciam.com/blog/60-second-science/post.cfm?id=hobbled-lhc-shuttered-for-repairs-n-2008-09-22

Reuters. "Small electrical issues before CERN machine start-up", Reuters News, September 10, 2008, http://www.reuters.com/article/latestCrisis/idUSLA57119

CERN. "LHC Commissioning with Beam", LHC Commissioning, page still current, http://lhc-commissioning.web.cern.ch/lhc-commissioning/

Gillis, Alan. "LHC Start-up To Shutdown 2008", The Science of Conundrums, August 22, 2008, http://bigsciencenews.blogspot.com/2008/08/lhc-start-up-to-shut-down-2008.html

CERN. "LHC: countdown to beam begins", CERN Courier, August 18, 2008, http://cerncourier.com/cws/article/cern/35431

Harris, David. "LHC glitch means two month delay" symmetrybreaking, September 20, 2008, http://www.symmetrymagazine.org/breaking/2008/09/20/lhc-glitch-means-two-month-delay/

Nostradamus And The LHC

A detail from a watercolor in the Vaticinia Nostradami codex, 1629 AD, at the Central National Library, Rome. The current buzz on the internet is a prophecy by Nostradamus that seems to indicate a colossal disaster for Geneva caused by the LHC. It's so striking, I thought it worth a closer look. While searching for the original French quatrain, number 44 in Century 9, I came across this image from what's being called 'The Lost Book of Nostradamus' from the recent book with this title by Ottavio Cesare Ramotti.

An archer shoots two fish in opposite directions across a gap, within a section of pipe. If you're imagining the LHC proton beams shooting through a detector through a beryllium pipe, and you're from the Renaissance, knowning nothing of physics and little of machinery, how better to illustrate this event? Fish too, in opposite streams, quite remarkable when you recall the quatrain and the 'Raypoz'.

It's not certain that Nostradamus wrote and illustrated this codex of 80 watercolors, something like William Blake's much later books of illuminations. It was attributed to Nostradamus by the title added in about 1689, while Nostradamus lived from 1503-1566. It's possible the codex was produced by Nostradamus' son César, who is known to have been a painter of miniatures and was preparing a booklet as a gift for King Louis XIII of France.

The current codex was presented by a Brother Beroaldus to Cardinal Maffeo Barberini, later Pope Urban VIII who was in office from 1623-1644. The mystery deepens as the codex some how found its way into the Central National Library in Rome, only to be rediscovered by Italian journalists in 1982. After some study, parts of it were found to be derived from an earlier work, the 'Vaticinia de Summis Pontificibus' from the 13-14th century. The 'Marston MS 225' at Yale, is also similar, probably from Bavaria or Bohemia. These earlier works were considered books of prophecies, though whose is in doubt.

If not quite evidence to confirm the Nostradamus LHC prophecy, it set me off on another search through the other 700 or so quatrains where I found another striking one, from Century 4, number 67. Before we look at this one, here is the one that has internet buzzing.

Leave, leave Geneva every last one of you,

Saturn will be converted from gold to iron,

Raypoz will exterminate all who oppose him,(?)

Before the coming the sky will show signs.

Migrés, migrés de Geneue trestous,

Saturne d'or en fer se changera,

Le contre Raypoz exteriminera tous,

Auvant l'aruent le ciel signes fera.

You've got to be careful with translations from Old French. The popular English version is not totally correct. Spellings vary in old texts, words change meanings and some become obscure.

The third line in question is one of a mistake in syntax. The translator was guessing here at the meaning. 'Le contre' clearly means 'the opposite'. 'Raypoz' is not a term used anywhere else in French and has no definite meaning. The opposite Raypos will exteriminate all, is the actual statement. 'Ray' is not French, though evidently it's Nostradamus' abbreviation of 'rayon', meaning 'ray.' 'Poz' is curiously written with a Z, a rare letter like in English, which indicates at least the pronunciation. 'Pos' for 'positive' is current in English as an abbreviation, and 'positif' is 'positive', though neither pos or poz would have been used in the Renaisance. Though the Z makes it clear that it isn't the French 'pos' which if so spelt would be pronounced like 'poe'. So 'poz' definately suggests 'positif'. Note that Nostradamus is consistent, using abbreviations to make up Raypos, as we do today. To call Raypoz the PositiveRay is a sound derivation, though it wouldn't have been understood back then, with the only rays being 'rayons de soleil' or sun rays, sunlight.

What is the Opposite of Raypos? A negative ray. In the case of the LHC, since they're using proton rays, the exact counterpart is antiproton rays, antimatter. So a matter/antimatter explosion destroying Geneva? All of us Trekkies know that. CERN experiments have confirmed it. And the Geneva Airport is a stone's throw away from the giant Atlas Experiment.

Two disturbing bits of information, the detail from the watercolor and the quatrain above. Have a look at number 3, the C4Q67:

The year that Saturn and Mars are equal fiery,

The air is very dry, a long meteor.(?)

From hidden fires a great place burns with heat,

Little rain, a hot wind, wars and raids.

L'an que Saturne & Mars esgaux combuste,

L'air fort sieché longe trajection.

Par feux secrets, d'ardeur grand lieu adust,

Peu pluie, vent chault, guerres, incursions.

You have to think like an astrologer here to make sense of the time clues he left in his works and consider his experience of the world. Nostradamus was himself a famous astrologer, well known for his almanac and the patronage he received at the court of Henry II of France and his Queen Catherine di Medici. But far from being in what we might consider a dubious profession, he was well respected and honest. Having studied with Rabelais at the same school, he was a Doctor of Medicine, perhaps the first of his day to insist on hygiene. Known also as a Mathematician, he was involved in public works projects, like the irrigation of the vast Paine de la Crau, which he also partly financed, near his home at Salon-de-Provence.

Both quatrains have astrological time clues. But Saturn wasn't discovered until after Galileo and the telescope. Well, like the modern method for inferring the presence of a celestial object by its apparent effect on other objects, modern astronomers have made similar guesses. With Nostradamus it was the careful study of Astrology that made Saturn real for him.

In the first quatrain, 'Saturn converted from gold to iron' is a metaphor for a conjuction where Saturn is unfavored, possibly eclipsed. In the other quatrain, 'The year that Saturn and Mars are equally fiery' could mean both are exhaulted. An astrologrer today might be able to put a date on this disaster at the LHC.

'The air is very dry, a long meteor.' is a suspect translation. The literal French is 'The air very dried long distance.' The air is dried by something and there is no meteor. The 'longe trajection' could be 'a long distance' and the drying is clear in the next line, not 'from' but 'by secret (not hidden) fires'. So we have poetically: The air dried for a long way / By secret fires of ardent power, a great place burns.

As a real warning of the burning of the LHC and Geneva, I think that it should be considered seriously. Reconsidering 120 tonnes of helium under 15-20 atmospheres pressure, much of it in an odd superfluid state at critically low 1.9 K temperature, and exposed in the ring to an 8.2 Tesla magnetic field, and the 'Raypoz' and its opposite, what might happen if not a plasma fire, some altered state of helium combusts due to the enormous TeV energies, 5 per beam and a collision force of 10 TeV scheduled this summer. Even worse, some nuclear event, as in an earlier post, The Almost Thermonuclear LHC. If I were in Geneva, I'd pack my bags.

For the Hollywood History Channel version of Nostradamus: The Lost Book, the 5 minute video is the best look at the original watercolors.

For a brief Wikepedia history of the Lost Book and some images.

For more images from the Lost Book.

For an Old French dictionary, geared for Nostradamus.

Finally a big Nostradamus site, with the Centuries in French and English, searchable.

Near An Inevitable LHC Disaster

On the left, a U of Liverpool contribution to the Velo Detector for the LHCb experiment. The sophistication of so many components, most of them custom designed and built for the LHC, means their reliability is unknown. In fact, well before startup, the main control room, the CCC, is flooded with 180,000 alarm events per day.

There are more than 120,000 computer definitions of what constitutes an alarm, over 120,000 things that could go wrong. Of course a lot of the alarm events are false readings or are from faulty sensors. EMS problems can add noise and alter readings. Operator error is another factor. Commonly, computer software and hardware have their own glitches as we all know now. And here at the LHC, everybody is flying blind. You have to have faith in your instruments, because you can't eyeball what's going on. If you noticed a real world fault, it would be an explosion on a security cam down in the ring tunnel.

How do control room operators react? In a typical emergency there is no time to react. Events go forward with nanosecond delays. An idiot light goes on and it's already too late. So emergency systems are automated for the most part. The software decides what to do. Fine if the sensor data can be trusted and the computer program recognizes the emergency and knows what to do in the space of milliseconds. If not, the operators have to think fast and hit the right buttons.

Reminds me of the partial melt-down at the Three Mile Island Nuclear Power Plant in the 1970's. Some main switches for various emergency functions were identical. Which one do you hit if you have to read a tiny label under each one? An investigation showed that some operators glued beer bottle caps on the switch buttons to make their functions clear, like Coors and Bud. If you think that was then and this is now, OK, but virtual switches on a computer screen might be hard to find and click or require some fancy keystrokes, that only so and so is authorized to make, but he's off in the washroom. Lately AEC inspections of a nuclear plant in the U.S. found operators napping, actually asleep on the job. Must be all those late nights with Coors and Bud.

Fortunately CERN's got some experience with control room emergencies from the days of the LEP. The computer systems were painstakingly designed, taking some 300 person years, and have been in use while the older pre-accelerators and systems were operating, like the LEIR, the SPS, the CNGS, and during the current LHC hardware commissioning. Here again, you can't be too careful. CERN has no experience at the TeV level of operations. To start beam commissioning at 5 TeV to save time, because of a backlog of problems that delayed the LHC startup for 3 years, certainly is asking for trouble. The former LEP was producing electron and positron beams of nearly massless particles at only 0.200 TeV. Now it's hadrons, protons and later very heavy lead ions. 5 TeV is only the start. 7 TeV next year and 1,150 TeV for lead ions , the year after.

But from a design and engineering point of view you don't even need these particle beams, to have a major accident. By mid-June this year the entire LHC cryogenic ring is to be cooled down to 1.9 K, near absolute zero. To do just this, you'd be hard pressed to follow a simple description of it's multi-stage complexity, in CERN's giant fridge. 20 fridges actually, 8 plus 8 in the tunnels, but 4 giant main multi-stage refrigeration plants, a thousand inlets and outlets per plant, above ground, using liquid nitrogen for pre-cooling to cool the 120 t of helium, 60 t in the plants and another 60 t in the ring, at 20 atmospheres pressure is a lot of helium (the world production for a year) and a lot of pressure. Simple over pressurising can blow valves or pipe or any fault at design pressure can produce an explosion. Ever blown up a bicycle tire at the gas station? Sure, there are LHC pressure relief valves, but recall the failure-prone Kautzky valves at the Tevatron, operating at a warmer 4.5 K? Above 5 K the superconducting magnets quench, which is one reason why in view of the Tevatron's troubles with recurring quenches, colder helium provides more of a theoretical safety margin. It also turns into a superfluid which has so little viscosity it can leak out through the smallest fracture or pinprick in the cryogenic system. If you thought you had troubles with a leaky bathroom tap, try a leaky cryogenic valve blasting you frozen in a millisecond. Ouch.

To make sure the superconducting magnets don't warm, the helium is pumped through them, so avoiding any hot spots developing which could cause a quench. Elaborate cryogenic Meyer distribution systems are in place within the tunnels themselves, custom built for the purpose. But a fault anywhere down the line at design pressure could explode, even before a relief valve is engaged, calibrated for above normal pressures. But it's all tested, sector by sector. Sure, but tests are short term stresses on equipment, not about failures developing over time. When the whole system is pressurized for the first time, it will be a real test of the cryogenic system, with all the equipment running simultaneously. And don't forget the superconducting magnets will be powered too. That's when engineers cross their fingers.

Over pressurising, in a worst case scenario, means equipment failure, then a pipe or valve failure, an explosion and helium release. Then the superconducting magnets quench, possibly exploding or magnet energy not safely discharged, and refrigeration plants crash.

Under pressurizing could be due to a helium leak, a power failure, or equipment malfunction, then the superconducting magnets quench, with the same dangerous cascade of events.

Normal operating pressure could still be too much for any of thousands of welds and pipe connections, and any valve or line with a hair fracture or defect. Possible leak or explosion and helium release. Then the superconducting magnets quench, and the same dangerous cascade of events ensues.

The best case scenario? The cryogenics and magnets work perfectly. But will they every day, 24 hours a day, until winter shutdown?

I'd say there's a 50-50 chance of a major accident even before complete cool down. The danger is to technicians and engineers who will be down in the tunnels when it happens. A deadly explosion and helium gas asphyxiation, a catastrophic failure at the LHC. No matter how knowledgeable and professional people are, people still make mistakes. It might shock some people into their senses and stop the LHC.

It's All About Energy At The LHC

British CMS Team weighing the Higgs. Even one particle of dust could skew the results. That's why there are about 150 million sensors in the main experiments, 60 million tracker and pixel channels in the CMS alone, to nail the Higgs boson. And one large grocer's weigh scale just in case it's rather heavy.

And there are a lot more sensors and equipment to operate the LHC machine. Thousands of miles of electrical cable and fiber optics feeding four auxiliary control rooms for the experiments and one 5 million Euro main control room or CERN Control Centre (CCC), the Central Computer Room, plus the Tier Zero and Tier One Computer Grid based at CERN. Once operating, the annual LHC electric bill is estimated at $30 million, the power consumption about equivalent to the Canton of Geneva.

Rudiger Schmidt at CERN: "The LHC is a frightening complex accelerator--both from the hardware point of view and from operation."

Frightening and it hasn't been turned on yet. We have an idea of what to expect from the earlier posts on Fermilab's Tevatron. Electrical problems and faults for one, a tedious reality that has surfaced during LHC commissioning, besides some odd surprises. They have to be eliminated. At the LHC scale, even small faults can have potentially disastrous consequences.

One thing to remember is the 27 km LHC pipe when cooled to 1.9 K, near absolute zero, actually shrinks by about 10 meters. Odd problems and failures can crop up. When there's a quench, superconducting magnets affected need to be heated to discharge their energy or they will self-destruct. Last August when Sector 7-8 out of 8 did not expand properly during a warming test, there was a major failure in the beam lines. Some PIM's or plug in modules, like sliding copper fingers, buckled into beam lines, obstructing them in the pipe. Proton beams have yet to be injected, so there was no scare, and the problem was fixed in an ingenious way, some high tech with a lot of low tech. "Ping pong balls" with radio transmitters were blown through the system to locate the 6 defective PIMs.

With the LHC so big, there are other problems peculiar to it. You might laugh, but the Moon influences the size of the ring during tides, producing a variation of a millimeter, raising the Earth's crust 25 cm in the Geneva area. That causes a change in the beam energy of up to two tenths of a thousandth. Adjustments even at this tiny scale have to be made by physicists. N. Mokhov of Fermilab, now working on the LHC machine protection system, says, "The parameters of the LHC beam are so high that microscopic effects can be very destructive to the machine and to the detectors." Not to mention that unstudied anecdotal effect of Full Moons burning out electrical equipment and flooding hospital emergency rooms with accident victims. Can we get Mythbusters in on this one?

The nagging electrical problem affecting the entire LHC, is EMC or Electromagnetic Compatibility. Equipment, sensors, communications and computers, like nice cosy shielded conditions with proper and invariable power supplies, with no interference from high powered up to 8.2 Tesla magnets or odd electron cloud formations in the pre-accelerators or other nastier radiation. The LHC has been an EMC swamp, causing all sorts of equipment failures, noise and false readings that might be genuine that set alarms off all over the place. Even the electric passenger train system above ground emitted stray DC currents influencing the accelerators. That was fixed after a long investigation. Here's a bit more from http://ab-div-po.web.cern.ch/ab-div-po/Infos/Conferences/EMC%20Workshop/Fritz%20EMC_LHC_workshop_041122.pdf:

• Alarm systems - NDE's, false alarms, alarm avalanches, wrong alarms
• Multiple grounding and shielding of equipment
• Entire physics project failures, eg Microvertex detector cancelled
• Large computing effort to remove signal noise in postprocessing
• Fragile designs asking for "quiet environments"
• Teams accusing each other of making "noise"
• Destruction of 600 kW cryogenic compressor
• Beam loss and consequent magnet destruction in TT40
• Fire in experiment NA48
• Fire in BA3 because fire alarm showed a fire elsewhere