Chapter 1 INTRODUCTION
The Tore Supra data acquisition system (DAS) (Fig. 1) was designed in the early 1980s [1] and has considerably evolved since then. Three generations of data acquisition units still coexist. Totalizing now 50 diagnostics and subsystems, the data acquisition units are operated simultaneously. The first generation was based on Multibus systems with an Intel processor and RMX operating system. Since 1995, most of these systems have been replaced by VME bus systems equipped with Motorola PowerPC processor boards running a Lynx OS real-time OS to obtain a more powerful architecture. The third generation was developed to perform extensive data acquisition for infrared and visible video cameras that produce large amounts of data to handle. It was supported by industrial PC over a Microsoft Windows OS. The first two generations, Multibus and VME bus systems, are diskless with real time OS (RTOS). Diskless systems are desirable for reliability and maintainability as they share common resources like kernel and file system.
The advantages of these architectures are: a common kernel a common file system easy updating and backup reliability (no hard disk) can be installed closest to the detector insensitive to the magnetic field easy to deploy
Page 1
These facilities considerably reduce the maintenance cost. Moreover, open source real-time operating system are now available that may provide free and convenient solutions for DAS. As a result, it was decided to explore an alternative solution based on an open source OS with a diskless system for the fourth generation. Currently, Linux OS is fairly mature to be used on DAS with preemptive and real time features on x86 architecture. In 2010, Linux distributions for PCI bus systems have been evaluated and compared to LynxOS real time OS. The latency and time response to any hardware interrupt has been achieved with x86 and x86 multi-core architecture target processors. The results obtained allowed to choose a version of Linux PCI platform for DAS on Tore Supra. These aspects and the new diagnostic DCEDRE are detailed in the following of the paper.
Page 2
handle. It was supported by industrial PC over a Microsoft Windows OS. The first two generations, Multibus and VME bus systems, are diskless with real time OS (RTOS).
Page 4
Page 5
CPCI bypasses the limit of five PCI slots in PC and offers a wide range of CPU and data acquisition boards. It is possible to use directly PXI cards, PCI and PMC (PCI mezzanine card) cards with adapter in a CPCI crate. The processor unit is not included in the CPCI crate. An industrial PC is connected to CPCI crate by a MXI-4 bridge from National Instruments. MXI-4 bridge provides a remote control of CPCI electronics cards from industrial PC through a fully transparent link connected with fiber optic cabling. The separation of the I/O board from the industrial PC allows having a robust system like VME bus system and a lifespan of more than 10 years.
Page 6
This configuration has the advantage: upgrading of the calculation power without changing all the data acquisition less noisy environment than for industrial PC for very fast data acquisition (2GHz). For a reduced DAS development time, electronic cards providers are selected for hardware availability and drivers quality, so the development time is now down to half between design and delivery. New OS, Linux The first milestone was to find a Linux distribution. The target system for PCI bus system are x86 multi-core processor with industrial PC. X86 processor is widely used in Linux community which has led us not to use commercial distribution. A custom solution has been developed. The main challenge will be to develop a diskless system. A diskless system on Tore Supra (named client) starts with Ethernet networks. Client has only memory and its file system is hosted on a server. The client-server architecture is showed on Figure 3. At startup, a client downloads the kernel on the server. The file system is common to all clients to facilitate the management. The user application development is realized on a compiler.
The target boot is based on PXE (Preboot execution Environment) boot. Some unnecessary services such as energy saving, virtual memory are disabled that slowdown the system. A hard RTOS solution will be evaluated in 2012 with Xenomai real time extension for Linux .
Page 7
Different Linux version 2.6 with different preemption option were chosen to be evaluated: 2.6.28.10 preemptible kernel (low latency desktop) 2.6.26.8-rt16 Complete Preemption (Real- Time) As referred in DAS needs on Tore Supra section a RTOS must meet deadline. Interrupt response time measurement and polling device can evaluate RTOS performance for a given target. Performance tests were also done with Lynx OS on a VME target system to compare Linux to a commercial RTOS. The interrupt response time (Fig. 4) is the interval between the time when a hardware interrupt occur and the time when the user task wake up to run. The interrupt response time is the sum of interrupt latency, interrupt processing, context switching and task wake up.
For response time tests, user task simulates a processing consuming 100% of processing power to measure the impact on Linux to evaluate RTOS closest to reality. A 500s interrupt frequency and 400s processing simulation were used for this test during 1 hour. Linux 2.6.28.10 on x86 dual core target is not far of hard RTOS Lynx OS (Fig. 5). Linux 2.6.28.10 is better in interrupt Information Technology, DYPIET
Page 8
management than Linux 2.6.26.8 RT. Linux 2.6.28.1 interrupt time response is 38s against 50s for Linux 2.6.26.8 RT.
Some VME I/O (input/output) devices used on Lynx OS target do not generate interrupt to indicate that data are available. I/O devices require the application to generate the necessary request in order to interact with them. The application runs in a periodic loop and makes a request every time through the loop, called polling device. Polling is also a way to consider more quickly an event than interrupt. A test was done to find the polling frequency limit on each target system and Linux version
Page 9
Lynx OS takes advantage on Linux. Moreover Lynx OS is completely freeze and do not reply to any request unlike to Linux. Linux 2.6.28.10 have a large advantage on x86 while both versions of Linux are equal on dual core. With upper CPU frequency than PPC, x86 dual core is close to hard RTOS performance.
NEW DIAGNOSTICS WITH LINUX Linux with a system disk has been used successfully on a new diagnostic for the dust detection in 2010. It can count and quantify the dust sucked into a vacuum duct continuously 24/7. In 2011, the Tore Supra DAS custom software was ported on a Linux diskless PCI platform. The new generation was validated on plasma experiment for the so called DCEDRE diagnostic (Fig. 7).
Page 10
This new diagnostic aiming at measuring the full frequency spectrum (0-200MHz) of electrostatic probes signal during high frequency power injection into the plasma gave preliminary results during Ion Cyclotron Range of Frequency (f=57MHz) heating. Two signals are acquired on an Agilent U1066A, 200 MHz digitizer, 12 bit resolution with a wide bandwidth of 100MHz. An external trigger for timing system network starts data acquisition for 100000 samples at 200MHz samples rate. Data are time stamped with NI PXI 6602 card clocked by 1MHz timing system network. Spectrum is obtained by Fourier transform of a set of 105
also in some cases at f/2 (Fig. 8), giving some evidence of the complex interaction of the plasma edge, where the probe is located, with the wave.
Page 11
Figure 8: Blow-up of the power spectrum during ICRF heating experiment (acquisition rate: 200MHz).
Page 12
Chapter 5 IMPROVEMENT OF TORE SUPRA REAL TIME PROCESSING CAPABILITY USING REMOTE PCS.
The Tore Supra tokamak is the largest superconducting magnetic fusion facility in operation. It has been devoted to long-duration high-performance discharge research. The Tore Supra real time measurements and control (RTMC) system has been upgrade in 2001 [1]. It is designed to deal with continuous acquisition during the plasma discharge, fast acquisition triggered on a time window (sampling frequency up to 2 GHz) and Real Time (RT) data processing for plasma control [2]. The simultaneous control of an increasing number of plasma parameters [3] aiming at tokamak operations in a fully steady state regime makes fast acquisitions and RT data processing more and more demanding. The Tore Supra Data Acquisition System (DAS) comprises 44 acquisition units. Most of them are based on VME bus and use Lynx 3.1 as operating system with a Power PC 300MHz CPU board. The main issue is related to fast acquisition operation (typical time window duration is 2s) that makes the data flow rate increasing from about 100 kB/s up to 8 MB/s which could not be handled with the present system. Furthermore, the time delay between two fast acquisition phases must be reduced to be able to catch fast plasma events occurring at almost the same date. As a result of this mode of operation, the amount of real time data has been increased by a factor of 6 during the last 6 years. While the calculation power of the acquisition units is fixed, some of them are not able any more to handle in parallel the data flow rate and the Real Time processing. Since 2001, a work has been initiated to identify the DAS limitation and define solutions to improve its performance in order to satisfy the new requirements from plasma physic studies. This paper deals with the upgrade of the Tore Supra RTMC system. The section 1 describes the present limitation encountered with the VME based acquisition units. The section 2 discuss the architecture of the DAS using remote PCs. The third section is devoted to the performance of the real time data processing using remote PCs.
Page 13
The acquisition parameters are defined by the user. They are uploaded in the CPU before each plasma discharge. Two CPU boards are used in the VME crate. The second processor (CPU#2) Information Technology, DYPIET
Page 14
is used for RT data processing and m actuators via a control loop and dedicated algorithms. It runs a single task during the plasma operation with a response time of a few milliseconds. The application program in the first processor (CPU#1) is made up of several tasks such as the timing chronology of the plasma discharge, the packaging of the raw or processed data in frames to be sent on Ethernet network, the communication with the RT server to transmit the frames and, the management of the events. Intercommunication between the processors is fulfilled by the VME bus through shared memory. The duration of the frame processing in the first CPU requires typically 1.86Cs per octet (maximum flux rate is 525 KB/s). Leaving the VME crate, the frames are sent via a 750 KB/s Ethernet TCP/IP protocol connection to a switch concentrating information from about 15 diagnostics (at Tore Supra, 3 switch are used). Then a 70 MB/s Ethernet link is used to transmit data to the RT server and the data storage. The main limitations (see Fig. 2) are concentrated in the VME crate and the Ethernet link to the Switch.
As an example, we consider a diagnostic with a fast acquisition data flow rate of 6.5 MB/s during 150 ms (1 MB of data). The minimum duration for the data to leave the VME crate (from the ADC to the switch) is 4s; 26 times longer than the acquisition itself. Thus the minimum delay between two triggered fast acquisitions is of the order of 4s. This limits strongly our ability to Information Technology, DYPIET
Page 15
study fast events appearing in the plasma. The main limitation was due to the multiplicity of processes running in the first CPU and the limited power of the processor itself. Recent more powerful processor cannot be used because the operating system Lynx OS 3.1 cannot be easily upgraded.
Page 16
case of a remote connection failure, the data processing is fully handled by the CPU board in the VME crate with the limitations it implicates on the delay between triggered fast acquisitions. While the remote PC is much more powerful than the VME processors in use at Tore Supra, it is able to handle the frame processing of 3 VME acquisition units. A Java-based application has been developed to monitor the fast acquisition processes. This program checks the status of the connection between the VME crate and the remote PC, the status of the VME software (data packet sending) and the status of the frame builder in the remote PC. During the plasma discharge it displays information on VME processes: the number of packets sent to the remote PC and the number of packets waiting for emission on Ethernet. Information about the remote PC is also available: the number of packets received from the VME (cross check) and the number of frame that were built, sent and waiting in the buffer. This application makes part of the tools routinely used during the experiments to monitor the acquisition of diagnostic using fast acquisition.
Page 18
Figure 4: Maximum flow rate along the path of the data from the diagnostic to the storage in the new architecture
The present main limitation is the fast Ethernet connection between the VME crates and the switch. Nevertheless upgrading this connection to 70 MB/s requires the use of new CPUs supporting Giga Ethernet transfer, and Lynx 3.1 as operating system. Such processor boards are already commercially available. Finally, the last limitation we have to face to is the maximum VME bus flow rate which cannot exceed 20 MB/s with DMA mode. A modification Information Technology, DYPIET
Page 19
of such constraint would require the replacement of the VME crate meaning major modification of the Tore Supra DAS. At present 2 diagnostics routinely use the remote PC solution: the Electron Cyclotron Emission (ECE) which gives the electron temperature profile and the Motional Stark Effect (MSE) which gives the current density profile. None of these two diagnostics are able to make fast acquisition without the remote PC. The ECE system has been upgraded to perform 32 points (instead of 16) temperature profiles every 1 ms. In parallel the fast acquisition is performed over the 32 channels on triggered time windows of 120 ms with a data sampling rate of 10Cs. The flux of data that must be handled by the DAS is about 6.5 MB/s and the minimum delay between triggers is 0.7 s. In comparison, using only 16 channels, the minimum delay between two fast acquisitions trigger was of the order of 4s. Another example is the MSE diagnostic. The fast acquisition duration is of the order of 1 s with a data sampling of 4 Cs. Although fewer channels must be recorded the data flux is almost the same than for the ECE diagnostic but it must be sustained over 1 s. Four more diagnostics will benefit from the remote PC power to perform small delay fast acquisition. The architecture described, offers several advantages. While the data acquisition (VME) is now decoupled from the data processing (frame building), the improvement of the RT processing ability can be performed more easily because we benefit from the fast progress on the PC. The remote PC could be used for RT processing of the fast acquisition to get physical parameter as well. By adding a connection to the RT reflective memory network (ScramNet) of Tore Supra these data can be used as inputs to enhance the plasma control efficiency [3]. In particular, a refined localization of plasma instabilities due to MHD could be performed by this means, thus allowing a more accurate response of the actuators
Page 20
Chapter 6 THE DATA ACQUISITION SYSTEM FOR THE INFRARED CAMERAS ON TORE SUPRA
The data acquisition for the infrared measurement system on Tore-Supra is a key element to ensure the supervision of the new actively cooled plasma-facing components of CIEL (Inner Components and Limiter), which is a major upgrade of Tore Supra. It will allow us to follow the thermal evolution of components of Tore-Supra, in particular the Toroidal Pumped Limiter (LPT) (360 15 m long) and the five additional heating launchers. When fully installed, the infrared measurement system will be composed of 12 digital 14-bit infrared cameras. They cover a 1001200 C temperature range, and each picture has a definition of 320 X 240 pixels with a 20 ms time resolution. The objectives of the data acquisition system are real-time recording and analysis of each view element for further postpulse analysis in order to understand the physics phenomenon and ensure the supervision of the plasma facing components, and also to be part of the global feedback control system of Tore Supra.
Overall Description
In order to describe the data acquisition system developed for the infrared measurement system (IR-DAS), we will first define what kind of data (and how many) have to be collected. Secondly, we will describe an experimental session on Tore Supra and the sequential phases of a plasma discharge. The data acquisition system for the infrared measurement system has also a function of real-time processing. It has to use the data in real time in order to insure the protection of the plasma facing components, to prevent them for overheating and at the same time it has to provide some computed data in order to take part in the plasma control and global feedback system of Tore Supra. Finally, we will define key features performed by the data acquisition system and the engineering implementation.
Page 21
Long-term post discharge analysis: many Tore Supra physicists must have access to the
infrared data for several years after the experimental session. The plasma discharge phase is monitored by the Tore Supra supervisor. The data acquisition units (AU) have to follow different triggers and external events delivered by the centralized timing system. Each data acquisition unit has to download new parameters during the preparation of the discharge, start the acquisition and store data back into the Tore Supra database until the end of the discharge.
Page 22
Page 23
Chapter 7 ADVANTAGES
A common kernel A common file system Easy updating and backup Reliability (no hard disk) Can be installed closest to the detector Insensitive to the magnetic field Easy to deploy
Page 24
Chapter 8 CONCLUSION
This study showed that real-time Linux solutions can be used for the data acquisition systems of Tore Supra. Linux on PCI bus systems fulfils Tore Supra needs for soft real time application. Linux 2.6.28.10 preemptible kernel (low latency desktop) on x 86 dual cores is the best solution and obtain equivalent performance to Lynx OS on interrupt time response. PCI platforms will reduce the cost of hardware, development time and operating costs. New Tore Supra DAS architecture is like ITER recommendation for ITER CODAC control system [3]. New DAS will be used easily for testing plant system controllers for ITER CODAC on Tore Supra experiments. In 2011, the new diagnostic DCEDRE has validated the new diskless PCI architecture with Linux OS on experiments. The next steps are: Performance tests with Xenomai in 2012. New diagnostics for plasma control with Linux 2.6 diskless system in 2012. Upgrade data acquisition unit of hybrid heating system on Linux diskless system with real time arc detection algorithm 100s cycle in 2012. For x 86 dual core systems, to dedicate a core to Linux and the other to Xenomai
Page 25
Chapter - 9 REFERENCES
[1] B. Guillerminet et al., Tore supra continuous acquisition system. Fusion Eng. Design, 60, 427-434 (2002). [2] F. Leroux et al., Real time data acquisition system for control and continuous acquisition in Tore Supra, Proceed. 13th IEEE-NPSS Real Time Conf., Montreal (2003). [3] ITER CODAC Guideline for Fast Controlers, I/O Bus Systems and Communication Methods between Interconnected Systems January 2010. [4] F. Leroux, B. Guillerminet, D. Moulin, F. Saint- Laurent, Real time data acquisition system for control and continuous acquisition in Tore Supra, 13th IEEE-NPSS Real Time Conference 2003. [5] B. Guillerminet, Y. Buravand, E. Chatelier, F. Leroux, Experience from Tore Supra Acquisition System and Evolutions, 4th IAEA/TM on Control, Data Acquisition & Remote Participation for Fusion Research (2003) [6] P. Moreau et al., "Towards Control of Steady State Plasma on Tore Supra", Proc. of IEEE Conference on Decision and Control, San Diego, CA, USA, Dec. 2006 [7] D. Moulin, B. Couturier, C. Grisolia, G. Martin, F. Saint-Laurent The real time plasma control system of Tore Supra Fusion Engineering and Design, vol.43 p.371 (1999). [8] D. Guilhem, G. Martin, R. Reichle, H. Roche, M. Jouve, and L. Ducobu et al., Infrared surface temperature measurement for long pulse real time feed-back control in an actively cooled machine, Rev. Sci. Instr., vol. 70, no. 1, pp. 427430. [9] D. Moulin, B. Couturier, L. Ducobu, D. Elbeze, B. Gagey, and B. Guillerminet et al., Real time data acquisition system for control and long pulse operation in Tore Supra, in Proc. RT97 Real Time Conf., Beaune, France, 1997.
Page 26