This blog is about an accident on Dec 1, 1999, when two of ADITYA’s magnet coils broke apart from their support system and got thrown out. Important lessons can be learnt from this incident.
After joining IPR in Aug 1995, I could make major improvements in ADITYA Tokamak. These have been covered in various blogs given below:
- My Research Journey to Brain : My research journey through life
- 5 Days Miracle in Nuclear Fusion Reactor(ADITYA Tokamak) – This helped in optimizing parameters to get longer discharges with flat top
- Rural technology helps fusion reactor! – This helped in reducing the noise due to improper maintenance of grounding system and improved the number of successful discharges from 20% to 100%
- How a “Photo” brought ADITYA Tokamak(Nuclear Fusion Reactor) back from death! – It helped in solving a major weakness in the insulation system of a critical magnet coil, that was limiting operational space
- Mystery of Error Fields in ADITYA Tokamak(Nuclear Fusion Reactor)! – This helped in understanding of a major design fault that impacted all aspects of machine operation
- Dancing Sparks in ADITYA Tokamak(Nuclear Fusion Reactor) – This solved a major grounding fault and allowed higher operational parameter
After the various weaknesses of machine were solved one by one, we started to operate the machine at higher parameters. ADITYA was designed to operate by changing current in TR1 (Ohmic Coil) from +20 KA to -20 KA. This would have allowed the full Volt-Sec available to achieve the full design parameters. This involved three set of power supplies : (1) Positive Converter, which was to slowly ramp up current up to +20 KA and then quickly reduce it to zero. (2) Auxiliary Converter to take over from +2 KA and change to -2 KA (3) Negative converter to take over from auxiliary converter and drive the current down to -20 KA.
Mostly ADITYA was run from 7.5-8 KA to 0 and thus it was operating with only 20% of 40 KA of current change that it was designed for. With above mentioned improvements, ADITYA started to operate from +12 KA to -2 KA resulting it increase of 110 KA and duration off 130 ms in about 2 years of my joining. Before that it was mostly able to go up to 40-50 kA and not last more than 15-20 ms.
ADITYA control room was in two parts : (1) Main Control room attached to main hall along with Data Acquisition system, CCTV etc. (2) Power Supply Control room. Power supply was also designed to be operated from Main control room but integration was not fully completed and power supply engineers operated from their own control room with a computer managing the power supply in a preprogrammed way with no feedback system coupled.
Many times, I used to go and sit with the power supply engineers in their control room to work on adjusting parameters to achieve higher and better plasma performance. When the power supply was operating there would be some kind of sound that was heard all over. For example, you can hear such a sound in this video:
On Dec 1, 1999, I was in power supply control room trying to work on adjusting parameters. We went through number of discharges (called “shots”) and it was close to the end of the day around 5 PM. Suddenly as we fired a shot 7575, background sound abruptly stopped into a deafening silence! I could feel that something was not right!
There was silence for some time over announcement system too. After silence of one minute, we got information that there seems to be an accident inside. I came out of the power supply control room and got down the stair case and entered ADITYA Main hall. Two heavy magnet coils had broken through their support and flew out of their location. See the video above in slow motion. You would notice after 25 sec, coil showing bit upward motion and then settling back. Again around 40 sec, it starts to rise up uncontrolled and follows with fire and then camera image is disturbed too. After about 2:13 sec, camera starts capturing video again with coils displaced and other things broken as coils get thrown out. We found many things scattered all over the machine and water leakage due to breaking of cooling line etc. We immediately rushed to block the leakage and any other immediate issues needing attention.
I went to inform Director as it was a serious incident to see if he gives any instruction on the matter or forms any committee to investigate. He did not seem much worried about preserving evidences etc and just asked to move on. We did whatever was necessary to end the day and do more investigation about damage next day with fresh mind and went home. I also instructed to keep the area closed as there were lot of debris all over including possible metal pieces etc.
After reaching office next day morning, I rushed to ADITYA hall to see the extent of damage and if machine had died or can be brought back to life. When I entered the hall, it was full of people all over without worrying about things lying all over. A large number of them had never come to see machine as it was going through many folds improvements over last 2 years that I had been around. Some senior scientists had not come even after being requested at times. I found one of them even on top of machine discussing the trajectory of coil as it was flying out and why it fell the way it had fallen rather than the other way. He was not concerned about why it broke apart in the first place.
Do we enjoy seeing damage more than progress?
I had to request all to vacate the hall so that we could clean up the debris and get more details of what damage had been done and if we get any clue about what had happened. There were two magnet coils placed on the top that broke through the support to restrain them and got thrown out. Luckily they took path that did not damage anything significant on the way. I do not recollect the damage to these coils themselves but they were easily reparable/replaceable. They were very much accessible being right on the top.
Every Monday morning, we used to have a ADITYA meeting that was open to whole institute. Next Monday meeting was all packed unlike other meetings. Luckily we had put camera viewing the area where coil broke out of its support and we could see the movements though it was very fast. Many times we used to point camera to some other locations, like in this one:
You can also hear a background sound as the machine was operating. For transparency, we showed videos recorded. We immediately started to get comments that proper maintenance was not being carried out and we can see coils being loose in undergoing movements. We informed them that regular maintenance was being done and has been done as per standard protocol and is recorded in the log book. However murmurs were going around repeating the same trying to blame my team.
Of course my task was to get to the bottom of the issue and see if we can get the machine to work again. As we were moving to higher working parameter space, it was likely that such incidents would take place more often unless we could fix the weaknesses of the machine. After investigations, following facts emerged :
- During shot 7575, due to some fault in the power supply, current in two magnet coils initially were finne but after sometime it started to grow until the power supply safety circuit came into operation and shut down the power supply. I have not been able to locate the documents to give more details. However force applied on magnet coils is typically proportional to the product of two currents. While the power supply safety circuit protected itself by tripping, force applied on coils were much higher than normal.
- When we looked into the design of support system, disturbing fact came out. As seen from the published document about ADITYA, coils support system were designed to take care of forces compressing them towards mid plane and not trying to force it away. Thus the support system was designed with much less strength than needed. Now the question arose as to why this was done and why design missed out forces that would try to expel coils away from the mid plane of the machine?
When I stumbled upon a thick file containing the details of magnet coil design, it became very clear to me as to how this had happened. Probably due to lack of computational facility there was no continuous force vs time plot made for different discharge scenario. Instead two points in the time line were chosen to calculate the forces – (i) At the beginning of the discharge, when current in BV coils were zero and TR coils were positive. Multiplication of these two gave zero force being applied at this time point. (ii) When plasma current was being maintained in flat top phase and Ohmic Power supply was operating through negative converter and current in TR coils were negative (they used to start from +ve and transition to -ve values). In this phase the force was pushing the coils to mid-plane as mentioned in the above paper.
So where did we go wrong? We missed out a scenario in between – when plasma current was in rising phase with finite current flowing in BV coils and TR was having positive currents too as Ohmic Power supply was still operating with positive converter. During this phase force applied would tend to throw the coil apart and we had to provide much stronger support to restrain them. In fact, power supply was rarely operated in negative converter phase and thus most of the time forces being applied would tend to expel them out. This can be seen in this video too. You need to focus your attention in the yellow marked box. A slight movement would be noticed:
One more thing that came out of investigation was that due to change in the dimensions of the coils, support system were adjusted to accommodate the changed dimensions and at least one bolt was not in a position that a tool could reach to tighten it. Only finger was used which would not make it possible to tighten it securely. (Read my blog Mystery of Error Fields in ADITYA Tokamak(Nuclear Fusion Reactor)! ).
Based on this understanding, we repaired the magnet coils as well redesigned the support system to make sure that it can hold larger force trying to expel it as well and also it was easier to maintain. This removed one more weakness of the system and we could start going to even higher parameters.
With this modification, ADITYA became ready for operation at highest parameters. Two limitations still remained:
- Cooling systems for magnet coils were not very proper and with operation at the higher currents, they needed larger intervals to cool them down. We used to put fans too for forced ait cooling specially at few locations.
- Gas purification system was not fully functional and an efficient discharge cleaning system or surface modification system would have helped. A few of these were tried too. I do not have information about their performance after I moved out in 2002.
- Main hindrance continued to be the hesitation in operating ADITYA Pulsed Power Supply(APPS) to full parameters specially the use of negative converter to prolong the discharge. This was due to the concern among power supply group of possible chance of malfunction and damage to the power supply. IPR management did not seem willing to address this issue and kept limiting the performance of the machine.
- APPS was being originally controlled by a PDP-11 computer based software supplied by ABB, Germany. Support to this computer was closed by company and even for small tasks like formatting a drive, one engineer from Germany used to come on a private trip for this purpose only costing lot of money. Eventually I replaced the whole system by a PC based system running Real Time Linux(RTLInux). I was invited in 2001 at the International Conference on RT Linux to give a special talk on this as this was world’s largest power supply being controlled by RTLinux. Subsequently VME based system was developed for this purpose.
- Feedback control remained an additional point of concern. Power supply group was reluctant to use them and many diagnostics system needed for this purpose also needed improvements and calibration.
In research and development, mistakes of wrong designs are common. They are part of learning process and should become part of collective knowledge to move forward. However hiding them or negating them takes away all that learning.
Thanks for posting. This accident and analysis will be valuable to progress on fusion reactor research.