Near An Inevitable LHC Disaster

On the left, a U of Liverpool contribution to the Velo Detector for the LHCb experiment. The sophistication of so many components, most of them custom designed and built for the LHC, means their reliability is unknown. In fact, well before startup, the main control room, the CCC, is flooded with 180,000 alarm events per day.

There are more than 120,000 computer definitions of what constitutes an alarm, over 120,000 things that could go wrong. Of course a lot of the alarm events are false readings or are from faulty sensors. EMS problems can add noise and alter readings. Operator error is another factor. Commonly, computer software and hardware have their own glitches as we all know now. And here at the LHC, everybody is flying blind. You have to have faith in your instruments, because you can't eyeball what's going on. If you noticed a real world fault, it would be an explosion on a security cam down in the ring tunnel.

How do control room operators react? In a typical emergency there is no time to react. Events go forward with nanosecond delays. An idiot light goes on and it's already too late. So emergency systems are automated for the most part. The software decides what to do. Fine if the sensor data can be trusted and the computer program recognizes the emergency and knows what to do in the space of milliseconds. If not, the operators have to think fast and hit the right buttons.

Reminds me of the partial melt-down at the Three Mile Island Nuclear Power Plant in the 1970's. Some main switches for various emergency functions were identical. Which one do you hit if you have to read a tiny label under each one? An investigation showed that some operators glued beer bottle caps on the switch buttons to make their functions clear, like Coors and Bud. If you think that was then and this is now, OK, but virtual switches on a computer screen might be hard to find and click or require some fancy keystrokes, that only so and so is authorized to make, but he's off in the washroom. Lately AEC inspections of a nuclear plant in the U.S. found operators napping, actually asleep on the job. Must be all those late nights with Coors and Bud.

Fortunately CERN's got some experience with control room emergencies from the days of the LEP. The computer systems were painstakingly designed, taking some 300 person years, and have been in use while the older pre-accelerators and systems were operating, like the LEIR, the SPS, the CNGS, and during the current LHC hardware commissioning. Here again, you can't be too careful. CERN has no experience at the TeV level of operations. To start beam commissioning at 5 TeV to save time, because of a backlog of problems that delayed the LHC startup for 3 years, certainly is asking for trouble. The former LEP was producing electron and positron beams of nearly massless particles at only 0.200 TeV. Now it's hadrons, protons and later very heavy lead ions. 5 TeV is only the start. 7 TeV next year and 1,150 TeV for lead ions , the year after.

But from a design and engineering point of view you don't even need these particle beams, to have a major accident. By mid-June this year the entire LHC cryogenic ring is to be cooled down to 1.9 K, near absolute zero. To do just this, you'd be hard pressed to follow a simple description of it's multi-stage complexity, in CERN's giant fridge. 20 fridges actually, 8 plus 8 in the tunnels, but 4 giant main multi-stage refrigeration plants, a thousand inlets and outlets per plant, above ground, using liquid nitrogen for pre-cooling to cool the 120 t of helium, 60 t in the plants and another 60 t in the ring, at 20 atmospheres pressure is a lot of helium (the world production for a year) and a lot of pressure. Simple over pressurising can blow valves or pipe or any fault at design pressure can produce an explosion. Ever blown up a bicycle tire at the gas station? Sure, there are LHC pressure relief valves, but recall the failure-prone Kautzky valves at the Tevatron, operating at a warmer 4.5 K? Above 5 K the superconducting magnets quench, which is one reason why in view of the Tevatron's troubles with recurring quenches, colder helium provides more of a theoretical safety margin. It also turns into a superfluid which has so little viscosity it can leak out through the smallest fracture or pinprick in the cryogenic system. If you thought you had troubles with a leaky bathroom tap, try a leaky cryogenic valve blasting you frozen in a millisecond. Ouch.

To make sure the superconducting magnets don't warm, the helium is pumped through them, so avoiding any hot spots developing which could cause a quench. Elaborate cryogenic Meyer distribution systems are in place within the tunnels themselves, custom built for the purpose. But a fault anywhere down the line at design pressure could explode, even before a relief valve is engaged, calibrated for above normal pressures. But it's all tested, sector by sector. Sure, but tests are short term stresses on equipment, not about failures developing over time. When the whole system is pressurized for the first time, it will be a real test of the cryogenic system, with all the equipment running simultaneously. And don't forget the superconducting magnets will be powered too. That's when engineers cross their fingers.

Over pressurising, in a worst case scenario, means equipment failure, then a pipe or valve failure, an explosion and helium release. Then the superconducting magnets quench, possibly exploding or magnet energy not safely discharged, and refrigeration plants crash.

Under pressurizing could be due to a helium leak, a power failure, or equipment malfunction, then the superconducting magnets quench, with the same dangerous cascade of events.

Normal operating pressure could still be too much for any of thousands of welds and pipe connections, and any valve or line with a hair fracture or defect. Possible leak or explosion and helium release. Then the superconducting magnets quench, and the same dangerous cascade of events ensues.

The best case scenario? The cryogenics and magnets work perfectly. But will they every day, 24 hours a day, until winter shutdown?

I'd say there's a 50-50 chance of a major accident even before complete cool down. The danger is to technicians and engineers who will be down in the tunnels when it happens. A deadly explosion and helium gas asphyxiation, a catastrophic failure at the LHC. No matter how knowledgeable and professional people are, people still make mistakes. It might shock some people into their senses and stop the LHC.