Proceedings of the 29th Annual International Conference of the IEEE EMBS Cit Internationale, Lyon, France August 23-26,

2007.

ThP2B4.6

Model based and iconic multimodal registration to merge gated cardiac PET, CT and MR sequences
Xavier Bty*, Christine Cavaro-Mnard and Vincent Roullier LISA EA4014 - Universit dAngers 62, av. Notre Dame du Lac 49000 Angers
xavier.baty@univ-angers.fr

Jean-Jacques Le Jeune INSERM U646 10 rue Andr Boquel 49100 Angers, France

Abstract In this paper, we present a multimodal registration method applied to gated positron emission tomography (PET), X-ray Computed Tomography (CT) and Magnetic Resonance (MR) cardiac images. CT images acquired on the same device as the PET ones are used as link to merge anatomical MR images and functional PET images. The registration process is divided in two steps: a 3D structure registration and a grey-levels registration. This approach enables global to local transformations. The structure registration uses a 3D biventricular heart model initialized on CT and MR data to dene a rigid transform. This global registration is then rened with a grey-levels step based on mutual information and free form deformations. To improve endocardium registration, we propose a composite PET-CT image to nd the optimal transformation on MR image. We also take into account the temporal problematic of heart motion by initializing the searched transformation, at a current frame, with the composition of a monomodal transformation (representing the heart motion between the previous and current frames) and a multimodal one (representing the spatial transformation between the two images at the previous frame).

I. I NTRODUCTION Cardiovascular diseases, in particular cardiac insufciency, are exponential growing diseases. Different therapeutic options include chemical drugs, revascularization surgery or heart transplant. Thus non-invasive evaluation of cardiac function is a major point for diagnosis and to follow the ischemic diseases up in order to choose and adapt the better treatment. The complementary natures of magnetic resonance (MR) imaging and positron emission tomography (PET) imaging provide to cardiologists signicant information about myocardial viability evaluation after infarction. In one hand, PET imaging is considered as gold standard for ventricle function evaluation. In the other hand, MR imaging gives detailed anatomical images and is also a reference method to evaluate the left and right ventricle function. X-ray Computed Tomography (CT) imaging is a very high detailed anatomical modality and can also give complementary information about the coronary artery tree. Thus, registration and merging of these images should lead to a more informative image. In cardiac viability studies, a mental registration of information provided by different imaging modalities is routinely performed by clinicians. Automatic registration, based on image processing, is therefore expected to offer better
*Thanks to General Electric Healthcare for funding

accuracy, reproducibility and to save time. Because of the mixed motions of heart and thorax structures, registration of cardiac images is a complex problem. A review of cardiac image registration approaches can be found in [1]. Anyway, these methods used a transmission PET image where we use a more detailed CT image to dene the transformation. Moreover, most methods only merge average PET images to end-diastole MR images because they did not have access to gated PET data. Only work of Shina et al. [2] allows the registration of MR and PET gated images (8 frames per cardiac cycle). They used morphological operators to segment the left ventricle and a surface tting based registration that led to a rigid transformation not well adapted to the cardiac motion. To resolve these points, we have developed a registration process divided in two steps. The rst one is a 3D structure registration that takes into account anatomical information [3]. The rigid transformation thus obtained, initializes the second step that is a grey-level based registration that allows local transformations. In our application, this approach is necessary because images are gated to electrocardiogram (ECG) and thus local deformations can be important. The 3D structure registration uses a 3D biventricular heart model and allows to reduce computing time during grey levels registration by reducing the deformation space. We also take into account the temporal problematic of heart motion by initializing the searched multimodal transformation as the composition of the multimodal transformation at frame t and the monomodal MR temporal transformation between frame t, t + 1. Thus, we reduce the processing time for all the registrations needed. In the following section, we describe the registration method we have developed. The third section shows the results that we comment.

II. C ARDIAC REGISTRATION A. Image acquisition All images used in this study were acquired in Angers Hospital. PET-CT data were scanned in axial slice plane on General Electric Discovery ST. MR data were scanned on 1,5 T General Electric Signa Horizon LX 9.1 in short axis slice plane. The images are gated to ECG (16 images per cardiac

1-4244-0788-5/07/$20.00 2007 IEEE

852

Authorized licensed use limited to: Instituto Politecnico de Lisboa. Downloaded on March 05,2010 at 12:03:40 EST from IEEE Xplore. Restrictions apply.

Fig. 1. Localization of slices - Left: Axial slices of CT and PET (from top to bottom) Right: Short Axis slices of MR, CT and PET (from top to bottom) focused on cardiac area for different frames of the cardiac cycle. CT and PET short axis slices are reformatted from axial slices. TABLE I ACQUISITION PARAMETERS FOR PET, CT AND MR IMAGES . PET 128128 47 3,27 mm circular 50 cm axial CT 512512 335 1,2 mm circular 50 cm axial MR 256256 12 8 mm rectangular 460345 mm2 short axis

Size Number of Slices Slice Thickness Field of View Slice Plane

cycle). Acquisition parameters are given in table I. The PET-CT scan device has the same geometry of acquisition for PET and CT, that allows to merge directly PET and CT images. The "Image Orientation Patient" tag given in DICOM header of short axis MR images allows to access the rotation parameters needed to reformat PET and CT images from axial slice plane to short axis (Fig. 1). B. Medical image registration Bases of medical images registration can be found in [4] and [5]. Image registration between source image (noted S) and target image (noted T) consists in the determination of the optimal transformation that leads S := (S) similar to T [6] : = arg max C(S , T ) (1)

where C is a similarity criterion. The transformation is dened by the vector of parameters over a space depending of chosen transformation. C. Developed scheme The process we have developed, register and merge gated PET and MR heart sequences via CT images. The complex nature of cardiac motion associated to multimodality imaging prevents the use of direct registration and the assumption of functional relation between pixels intensity of PET and MR images. Mkel et al. used transmission PET images as a link between emission PET images and MR ones [7]. Here, we use CT images as link because new acquisition device allows to acquire quasi simultaneously CT and PET images and because CT images have better spatial and temporal resolution than transmission PET images. Then we need to dene the transformation that registers CT and MR images.

Knowing this transformation, we can apply it to PET images (registered to CT during acquisition) in order to have PET images registered to MR images. The global scheme of our registration process is presented on the synoptic diagram on Fig. 2. 1) 3D structure alignment: The 3D structure alignment is the rst step of our process. To perform a global registration between CT and MR volume of rst frame, we choose an afne transformation composed of translation, rotation and scaling. We construct a 3D biventricular heart model based on the work of Mkel and Pham [7]. This model is deformed to t with CT data in one hand and MR data in the other hand. The deformation is performed with an active deformation method based on 3D gradient vector ow (GVF) [8]. Then, we use an Iterative Closest Point (ICP) algorithm [9] to register the two models. Thus, we get a rigid transformation that globally registers all images of PET-CT data to MR ones. But this afne transform does not take into account local deformations proper to each patient. 2) Grey levels based registration: The grey levels registration step assures the local registration. Our work is based on the algorithm proposed by Mattes et al. in [10]. We use a B-Spline kernel based Free Form Deformations to model the transformation. Deformations are obtained by tuning an underlying mesh of control points regularly placed upon the image to transform. The control points displacements are interpolated to obtain a smooth and continuous C 2 transformation. The advantage of B 3 (x) spline basis (2) is its limited support that leads to controlled local deformations and its good trade-off between complexity and smoothness of the obtained deformation. 1 0 |x| 1 6 (4 6x2 3|x|3 ), 3 1 (2 |x|)3 , 1 |x| 2 B (x) = (2) 6 0, 2 |x| In order to take into account all of the structures inside images and to speed up the registration process, we perform a multi-resolution decomposition by computing two pyramids of images by recursive gaussian ltering. At the top of the pyramids, low-scale images can be registered. The obtained transformation is upsampled to initialize the registration at the next level of the pyramids. It permits a global to local approach for the registration process. As similarity criterion, we use the mutual information introduced by Viola et al. [11]. This criterion is well suited for

853
Authorized licensed use limited to: Instituto Politecnico de Lisboa. Downloaded on March 05,2010 at 12:03:40 EST from IEEE Xplore. Restrictions apply.

Fig. 2. Global scheme of the registration process. First CT data are registered to MR data using a 3D heart model. Using the obtained transformation a grey level based registration is performed between composed image of PET-CT and MR images.

(a)

(b)

(c)

(d) Fig. 4. Temporal registration problematic

Fig. 3. Original (a) and composed images with = 0, 3 (b), = 0, 5 (c) and = 0, 7 (d)

multimodal registration. But we can not make the assumption that it exists a functional or statistical relation between the pixels intensities of PET and MR images. Moreover, the intensities of pixels inside cavities and pixels of the myocardial wall are similar on CT image since no contrast agent is used during acquisition (Fig. 3(a)). So we dened a new image (noted Comp) as the composition of PET and CT images (see (3) and Fig. 3(b, c, d)). Comp = CT + (1 ) P ET (3)

This composite image is feasible because images are registered by acquisition. The use of this image should allows to take into account the myocardium contours from PET and CT images and thus improve the registration process by using more suitable data for the similarity criterion. For the optimization, we have to tune a number of parameters depending of the FFD grid size (2 N 2 parameters for a regular square grid of N control points by side). So we use a modied version of BFGS minimization algorithm that allows to specify bounds for the parameters in a large search space and limits calculator memory usage [12]. D. Temporal registration Our application needs the registration of 13 short axis PET images on 13 MR images for 16 frames, that is to say 208 registrations. This prevents a clinical routine application if

the process is not optimized. Basically, for a given slice, the temporal problematic of the cardiac registration is shown on Fig. 4. Ft represents the previous obtained transformation from the multimodal registration of the composed image on the MR one. GM R stands for the obtained transformation from t,t+1 the monomodal registration of two successives MR frames (same for Comp image). As seen on Fig. 4 , it is difcult to analyse the motion extracted from GComp . So knowing Ft , t,t+1 we initialize the transformation Ft+1 as the composition of Ft and GM R . Thus, we obtain an initialization that speeds t,t+1 up the registration process and prevents the optimizer from evolving to local minima. III. R ESULTS AND COMMENTS A. Structure alignment The heart model has been constructed by segmenting heart contours on MR images from a healthy patient. From the contours, a 3D model is constructed (see Fig. 5) using Nuages 1 and GHS3D 2 softwares . The model is initialized, in one hand, on MR data and, in the other hand, on the CT data by indicating the center of the cardiac cavity to the software. Then Gradient Vector Flow forces are computed in order to attract contours of the model to contours of the heart. The 3D model is deformed
1 www-sop.inria.fr/prisme/personnel/geiger/geiger.html 2 www-rocq1.inria.fr/gamma/ghs3D/ghs.html

854
Authorized licensed use limited to: Instituto Politecnico de Lisboa. Downloaded on March 05,2010 at 12:03:40 EST from IEEE Xplore. Restrictions apply.

composite PET-CT images at frame t+1 with a composition of the previous registration at frame t and the temporal registration between two consecutive MR images. The results can be viewed on General Electric Advantage Window workstation using adapted lookup table. The overall process is designed to be as robust and fast as possible.
Fig. 5. Construction of the heart template

R EFERENCES
[1] T. Mkel, P. Clarysse, O. Sipil, N. Pauna, Q. Pham, T. Katila, and I. Magnin, A review of cardiac image registration methods, IEEE Transactions on Medical Imaging, vol. 21, no. 9, pp. 10111021, 2002. [2] S. Sinha, U. Sinha, J. Czernin, G. Porenta, and H. Schelbert, Noninvasive assessment of myocardial perfusion and metabolism: Feasibility of registering gated MR and PET images, American Journal of Roentgenology, vol. 164, no. 2, pp. 301307, 1995. [3] O. Camara, O. Colliot, and I. Bloch, Computational Modeling of Thoracic and Abdominal Anatomy Using Spatial Relationships for Image Segmentation, Real Time Imaging, pp. 263273, 2004. [4] D. Hill, P. Batchelor, M. Holden, and D. Hawkes, Medical image registration, Physics in Medicine and Biology, vol. 46, pp. R1R45, 2001. [5] J. P. W. Pluim, J. B. A. Maintz, and M. A. Viergever, Mutual information based registration of medical images: a survey, IEEE Transactions on Medical Imaging, vol. 22, no. 8, pp. 9861004, 2003. [6] L. G. Brown, A survey of image registration techniques, ACM Computing Surveys, vol. 24, no. 4, pp. 325376, 1992. [7] T. Mkel, Q. Pham, P. Clarysse, J. Nenonen, J. Ltjnen, O. Sipil, H. Hnninen, K. Lauerma, J. Knuuti, T. Katila, and I. Magnin, A 3-D model-based registration approach for the PET, MR and MCG cardiac data fusion, Medical Image Analysis, vol. 7, no. 3, pp. 377389, 2003. [8] C. Xu and J. Prince, Snakes, shapes, and gradient vector ow, IEEE Transactions on Image Processing, vol. 7, pp. 359369, 1998. [9] P. Besl and N. McKay, A method for registration of 3-D shapes, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 14, pp. 239256, 1992. [10] D. Mattes, D. Haynor, H. Vesselle, T. Lewellen, and W. Eubank, PETCT image registration in the chest using free-form deformations, IEEE Transactions on Medical Imaging, vol. 22, no. 1, pp. 120128, 2003. [11] P. Viola and P. Wells, Alignement by maximisation of mutual information, Proceedings of ICCV 1995, Boston MA, pp. 1523, 1995. [12] R. Byrd, C. Zhu, P. Lu, and J. Nocedal, L-BFGS-B fortran routines for large scale bound constrained optimization, ACM Transaction on Mathematical Software, vol. 23, no. 4, pp. 550560, 1997. [13] L. Ibez, W. Schroeder, L. Ng, J. Cates, and the Insight Software Consortium, ITK software guide second edition, www.itk.org, 2005. [14] R. Chandrashekara, R. Mohiaddin, and D. Rueckert, Analysis of 3D myocardial motion in tagged MR images using nonrigid image registration, IEEE Transactions on Medical Imaging, vol. 23, no. 10, pp. 12451250, 2004.

using the Finite Element Method described in [7]. An ICP algorithm is then performed to obtain the global transform that maps CT space to MR space. The structure alignment guided by the constructed heart template permits the scaling and the global alignment of the heart. Fig. 6(a, b) shows the results on PET and MR images after the global registration. Next, the grey level registration step allows local registration. B. Grey levels based registration Algorithms were implemented using the open-source ITK library [13]. The same type of algorithm is used both for CT-MR registration and for MR-MR temporal registration. Furthermore, the monomodal registration algorithm permits the tracking of the myocardium motion as presented in [14]. The parameter for the composed image is set to 0,5 because no noticeable difference was seen on the results for variations between 0,3 to 0,7. The obtained transform is then applied on original PET images. Thus PET and MR data can be merged. Fig. 6 and Fig. 7 shows a registered PET slice superimposed on MRI. Fig. 6 shows the results after afne global registration. Fig. 7 shows the results after local iconic registration; improvements are visible on the bottom of the myocardium. IV. C ONCLUSION We present a method to register multimodal PET, CT and MR cardiac series. The 3D structure registration based on a heart model avoids manual intervention for segmentation and provides a global transformation. The grey levels registration step based on free form deformations and mutual information ensures to take into account local deformations and specicity of images. The temporal registration speeds up the whole process by initializing registration of MR and

(a) in diastole Fig. 6.

(b) in systole

(a) in diastole Fig. 7.

(b) in systole

Rigid registration results.

Local registration results.

855
Authorized licensed use limited to: Instituto Politecnico de Lisboa. Downloaded on March 05,2010 at 12:03:40 EST from IEEE Xplore. Restrictions apply.