Abstract
Magnetic Resonance Imaging (MRI) typically collects data below the Nyquist sampling rate for imaging acceleration. To remove aliasing artifacts, we propose a multi-channel deep generative adversarial network (GAN) model for MRI reconstruction. Because multi-channel GAN matches the parallel data acquisition system architecture on a modern MRI scanner, this model can effectively learn intrinsic data correlation associated with MRI hardware from originally-collected multi-channel complex data. By estimating missing data directly with the trained network, images may be generated from undersampled multi-channel raw data, providing an “end-to-end” approach to parallel MRI reconstruction. By experimentally comparing with other methods, it is demonstrated that multi-channel GAN can perform image reconstruction with an affordable computation cost and an imaging acceleration factor higher than the current clinical standard.
You have full access to this open access chapter, Download conference paper PDF
Similar content being viewed by others
1 Introduction
Magnetic Resonance Imaging (MRI) is a powerful tool for disease diagnosis and treatment. However, the clinical applications of MRI are limited by its intrinsically low imaging speed. On a clinical MRI scanner, data are typically collected below the Nyquist sampling frequency for imaging acceleration. As undersampling may introduce aliasing artifacts, image reconstruction is challenging in MRI.
Parallel imaging is a conventional approach to reconstruct images from undersampled data in MRI. This approach requires a radiofrequency coil array for collecting multi-channel images in parallel [2, 7, 9]. The coil sensitivity differences between channels may be used to remove aliasing artifacts in image reconstruction. Several parallel imaging techniques including GRAPPA [2] and SENSE [9] have been commercialized and are being used as clinical standards. A drawback of these techniques is that coil sensitivity must be calibrated and the calibration procedure may reduce the effective parallel imaging acceleration factor. Recently, compressed sensing has become an active area of research interest in MRI [6]. This approach relies on a sparsity constraint on MRI data and uses an iterative algorithm to reconstruct images. Although a calibration procedure is not needed, the computation is expensive. In addition, most compressed sensing techniques are sensitive to regularization parameters in image reconstruction and their clinical performance is not as robust as parallel imaging.
Deep learning has made many breakthroughs in the field of computer vision and medical imaging. Recent years have seen a number of research works on its applications in MRI reconstruction. For example, a data-driven scheme based on generative adversarial network (GAN) has been used in combination with compressed sensing [8]. This work, like many other deep learning studies, treats MRI reconstruction as a computer vision problem and removes aliasing from DICOM images in grayscale or RGB magnitude generated by a series of pre- and post-processing procedures [5, 8, 10, 14]. However, it should be known that MRI physically collects multi-channel complex data samples in Fourier space (termed k-space in the field of MRI) with both magnitude and phase information. By transforming complex k-space data into a grey-scale DICOM image with pre- and post-processing, a large amount of information is lost, which introduces low data utility in the network. As a result, image reconstruction may not take full advantage of the data-driven power of deep learning.
Our work aims to develop a deep generative model that can process multi-channel MRI raw data. It is expected that image reconstruction may take advantage of all the information in MRI data. The work is inspired by a well-known fact in parallel imaging: multi-channel coil sensitivity introduces k-space data correlation and every data sample may be represented by the convolution of their neighboring samples with a filter. This filter can be well modelled by convolutional neural networks in k-space. By training a GAN with multi-channel k-space raw data, our model may learn and use parallel imaging mechanisms underlying MRI data to reconstruct images from undersampled data. Herein, we present a multi-channel GAN model for MRI reconstruction from undersampled data in this paper. This model has the following features:
-
(1)
The multi-channel GAN receives multi-channel complex undersampled MRI data at the input and generates multi-channel complex fully-sampled MRI data at the output.
-
(2)
The model provides an “end-to-end” approach to MRI reconstruction, i.e., images can be generated directly from raw MRI data. No pre- or post-processing procedures are needed.
-
(3)
A new loss function is introduced to combine adversarial and perceptual loss for improved artifact suppression in image reconstruction.
-
(4)
The model is trained with multiple sets of multi-channel MRI data. In every training step, all the channels of the network are updated simultaneously with the multi-channel raw data.
-
(5)
The trained model is used as a general reconstructor for all new datasets. This is different from conventional MRI methods that use a different reconstructor for different datasets.
2 Method
MRI raw data are collected in complex k-space (Fourier spatial-frequency domain) from a multi-channel (typically 8–32 channels) radiofrequency coil array. If data are fully sampled at the Nyquist frequency, an inverse Fourier transform may generate the real images. To accelerate MRI, k-space is usually undersampled, which introduces aliasing artifacts in image-space. Since multi-channel coil sensitivity may introduce k-space data correlation, a parallel imaging technique (e.g., GRAPPA, SPIRiT) may estimate missing data by the linear convolution of partially collected k-space with a filter. In our work, a GAN-based network is used to model the filter used in parallel imaging for image reconstruction. In GAN pipeline, two models are jointly trained: a generator model G which captures the training data distribution and a discriminator model D which justifies if the generated data come from the distribution of the training data. Through the training process, the generator should be trained to estimate the embedding data manifold and generate samples to fool the discriminator. Then after training the generator alone can be used to generate new samples that are similar to real samples.
Here a multi-channel GAN based model (a multi-input and multi-output system) is trained to estimate complex-valued data in k-space. Given a set of data pairs: fully sampled k-space data y and undersampled zero-filling data \(x = M_Ry\), where \(M_R\) is a undersampling mask with an acceleration factor of R, a generator G is trained to generate fake samples from x and these fake samples should be justified as real samples by the discriminator D. The adversarial loss of the generator is given by:
To minimize the difference between the generated data and ground truth data, a pixelwise MSE loss is also introduced. Here an \(l_1\) cost term is used for more robust performance in noise and blurring suppression:
In addition, a data consistency loss is used to minimize the sum of squared difference between the acquired data samples and their estimates. This loss is formulated as:
The terms above encourage the generator to generate an output that matches the ground truth at the k-space sampling positions. To suppress artifacts that can not be quantified by Eqs. 1, 2 and 3, we introduce a perceptual loss based on high-level features extracted from pre-trained networks and combine the pixelwise loss with perceptual loss for improved visual quality of the reconstructed image. Here let \(\phi _j(\cdot )\) be the j-th layer output of a pre-trained network with shape \(C_j \times W_j \times H_j\). We can use \(\phi _j(\cdot )\) as a feature extractor which captures high-level image characteristics. The perceptual loss is formulated as:
By summing the adversarial loss, pixelwise loss, data consistency loss and perceptual loss together, the overall loss functions for generator and discriminator are formed as:
The parameters \(\alpha \), \(\beta \) and \(\gamma \) are used to balance the adversarial loss, \(l_1\) loss, data consistency loss and perceptual loss. The generator and discriminator are trained with mini-batch stochastic gradient descent and back-propagation algorithms. The two sub-networks are updated alternatively until convergence. The trained generator is then used to reconstruct images from new raw MRI data.
Generator Architecture. We adapt the basic architecture of identity residual network with skip connections [3] for multi-channel MRI reconstruction. During training, multi-channel undersampled and fully sampled complex k-space data are fed into the network. Then 5 residual blocks are stacked sequentially where each block has two convolutional layers and skip connections from block input to output. Each convolutional layer in the residual block consists of 128 feature maps using \(3\times 3\) kernels and is followed by batch normalization and ReLU activation. The network is followed by three convolutional layers with kernel size \(1\times 1\). A VGG-16 network [11] pre-trained on ImageNet is used as feature extractor and the output of relu2_2 layer is used as perceptual feature.
Discriminator Architecture. A discriminator is connected to the generator output. The discriminator is a regular convolutional network which consists of 7 convolutional layers, each of which is followed by batch normalization and ReLU layers. We use 8, 16, 32, 64, 64, 64 feature maps for the first 6 layers, respectively. We use \(3\times 3\) kernels for the first 5 layers and \(1\times 1\) kernels for the last 2 layers. The discriminator output is a scaler between 0 and 1 measuring the estimated score of the “realness” of the generated data.
3 Experiment
3.1 Dataset and Training Details
To validate the proposed method, we perform several experiments with 170 2D multi-channel MRI images. These data are collected from the brain anatomy of different human subjects using an 8 channel coil array with a T1/T2 weighted TSE sequences (axial resolution \(256\times 256\) and FOV \(240\times 240\) mm). The data are randomly grouped into a training (127 images) and a test (43 images) set. To replicate the reconstruction process, we use uniform Cartesian masks with various undersampling factors. A Nvidia Tesla V100 GPU is used for training with batch size 4. The learning rate is initially set as \(1\times 10^{-5}\) and reduced in half every 10,000 iterations. An Adam optimizer is used for optimization. The network is trained for 2,000 epoches, which is about 1,500 min.
3.2 Results and Discussion
The proposed method is used to reconstruct images from undersampled data generated from the 43 test data samples in comparison to GRAPPA [2], SPIRiT [7], Compressive Sensing (CS) [6] and GANCS [8]. GRAPPA, SPIRiT and Compressive Sensing reconstruction uses Berkeley ESPIRiT [12] and BART [13] toolboxes. In GRAPPA and SPIRiT, additional fully-sampled \(18\times 18\) k-space areas in central k-space are used as autocalibration signals (ACS). In GANCS [8], the raw k-space data are transformed to image-space and only the magnitude images are used. Figure 3 shows a few examples with an undersampling factor of 5 in reference to the ground truth images and the Zero-filling (ZF) Fourier reconstruction results, which are used to show the spatial distribution of aliasing artifacts. Quantitative results including Structural SIMilarity (SSIM) and PSNR are given in Table 1. It is found that parallel imaging methods (GRAPPA and SPIRiT) gives higher PSNRs. However, they may generate image blurs with noticeable background noise, indicating outer k-space is not reconstructed well. Compressive Sensing gives better image details with less noise, but at a cost of computation time (Table 1) and considerable artifacts near tissue boundaries. GANCS gives apparently worse performance both qualitatively (Fig. 3) and quantitatively (Table 1). In comparison, the proposed method gives high-quality images with a low computation cost. It should be noted that this new method does not need calibration data, implying the net undersampling factor is higher than that in parallel imaging.
It should be mentioned that the performance of GANCS shown in Fig. 3 is worse than that in the original paper [8]. This should be attributed to the following differences: First, in the previous study, GANCS is used to reconstruct images from radial data. In this study, Cartesian data are used. Compared with radial undersampling, Cartesian undersampling introduces more patterned aliasing artifacts, which cannot be effectively removed with compressed sensing. Second, the previous GANCS study uses a total number of 45,300 training images. This study uses only 127 training data samples. The significant reduction in training data size should be a major factor that affects the reconstruction performance.
In this study, the proposed method is based on the same GAN structure as GANCS. However, a multi-channel architecture is used to process multi-channel complex k-space data. Because more information in MRI raw data is used, better performance can be achieved with less training data. This is an advantage of the proposed method. It should also be pointed out that the proposed method is practically useful in MRI. Most MRI protocols are running with fixed parameters on daily basis in clinical practice. Once the proposed deep learning network is trained with a certain amount of data collected from different patients using a fixed protocol, it can be directly used to reconstruct images for the upcoming new patients scanned with the same protocol. This is more time efficient than conventional parallel imaging, which always requires a calibration procedure with additional data acquisition in every scan.
An investigation is also made on the performance of the trained multi-channel GAN with different acceleration factors. As shown in Figs. 4 and 5, the reconstruction performance is not dramatically degraded until the undersampling factor is higher than 5. In previous studies [15], it has been demonstrated that the standard 8-channel head coil used in this work has a maximal parallel imaging acceleration factor of 4 due to hardware limitation. This indicates that the proposed method can learn not only parallel imaging mechanisms but also useful k-space data features from MRI raw data, making it possible to accelerate MRI beyond the parallel imaging limit.
4 Conclusion
In this paper, we propose a multi-channel GAN model for parallel MRI reconstruction. Compared to other existing deep learning approaches, the proposed method directly uses multi-channel complex-valued k-space data. Instead of learning anatomy structure in image space, we reformulate MRI reconstruction as a data completion problem and learn physical data relationship in k-space with a multi-channel GAN model. The experimental results demonstrate that the proposed method outperforms other state-of-the art MRI reconstruction methods for imaging acceleration.
References
Goodfellow, I., et al.: Generative adversarial nets. In: Advances in Neural Information Processing Systems, pp. 2672–2680 (2014)
Griswold, M.A., et al.: Generalized autocalibrating partially parallel acquisitions (GRAPPA). Magn. Reson. Med. 47(6), 1202–1210 (2002)
He, K., Zhang, X., Ren, S., Sun, J.: Identity map**s in deep residual networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9908, pp. 630–645. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46493-0_38
Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9906, pp. 694–711. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43
Lee, D., Yoo, J., Ye, J. C.: Deep artifact learning for compressed sensing and parallel MRI (2017). ar**v preprint ar**v:1703.01120
Lustig, M., Donoho, D., Pauly, J.M.: Sparse MRI: The application of compressed sensing for rapid MR imaging. Magn. Reson. Med. 58(6), 1182–1195 (2007)
Lustig, M., Pauly, J.M.: SPIRiT: Iterative self-consistent parallel imaging reconstruction from arbitrary k-space. Magn. Reson. Med. 64(2), 457–471 (2010)
Mardani, M., et al.: Deep generative adversarial networks for compressed sensing automates MRI (2017). ar**v preprint ar**v:1706.00051
Pruessmann, K.P., Weiger, M., Scheidegger, M.B., Boesiger, P.: SENSE: sensitivity encoding for fast MRI. Magn. Reson. Med. 42(5), 952–962 (1999)
Schlemper, J., Caballero, J., Hajnal, J.V., Price, A., Rueckert, D.: A deep cascade of convolutional neural networks for MR Image reconstruction. In: Niethammer, M., Styner, M., Aylward, S., Zhu, H., Oguz, I., Yap, P.-T., Shen, D. (eds.) IPMI 2017. LNCS, vol. 10265, pp. 647–658. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-59050-9_51
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition (2014). ar**v preprint ar**v:1409.1556
Uecker, M., et al.: ESPIRiT - an eigenvalue approach to autocalibrating parallel MRI: where SENSE meets GRAPPA. Magn. Reson. Med. 71(3), 990–1001 (2014)
Uecker, M., et al.: Berkeley advanced reconstruction toolbox. In: Proc. Intl. Soc. Mag. Reson. Med., vol. 23, p. 2486 (2015)
Yu, S., et al.: Deep De-Aliasing for fast compressive sensing MRI (2017). ar**v preprint ar**v:1705.07137
Li, Y., Dumoulin, C.: Correlation imaging for multiscan MRI with parallel data acquisition. Magn. Reson. Med. 68(6), 2005–2017 (2012)
Acknowledgement
This research is supported in part by grants from NIH R01 EB022405, NSF ACI 1443054 and IIS 1350885.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2018 Springer Nature Switzerland AG
About this paper
Cite this paper
Zhang, P., Wang, F., Xu, W., Li, Y. (2018). Multi-channel Generative Adversarial Network for Parallel Magnetic Resonance Image Reconstruction in K-space. In: Frangi, A., Schnabel, J., Davatzikos, C., Alberola-López, C., Fichtinger, G. (eds) Medical Image Computing and Computer Assisted Intervention – MICCAI 2018. MICCAI 2018. Lecture Notes in Computer Science(), vol 11070. Springer, Cham. https://doi.org/10.1007/978-3-030-00928-1_21
Download citation
DOI: https://doi.org/10.1007/978-3-030-00928-1_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-00927-4
Online ISBN: 978-3-030-00928-1
eBook Packages: Computer ScienceComputer Science (R0)