1 Introduction

In recent decades, medical imaging has played a critical role in health care areas such as treatment and medical diagnosis [1, 2]. There are several modalities in medical imaging, like computed tomography (CT), single-photon emission computed tomography (SPECT), magnetic resonance imaging (MRI), and positron emission tomography (PET). All those modalities were obtained from different sensors and focused on specific organ features. Typically, one modality cannot give enough information for complete medical diagnosis [1, 2].

The CT images are illustrating the hard tissue like the bone structure. The main advantages of these images are a low time for scan and high image resolution [1, 3]. MRI detects the soft tissue structures or details of a specific organ like the brain, lung, and eyes. It is safer than other scans for babies and pregnant women. It provides a high spatial resolution of any human organ [4]. The drawback for these images is more sensitive to movement, so not better for organs that include movement like mouth tumors [1]. On the other hand, PET provides metabolisms of tissues and functional information for the organ. The PET is obtained high-sensitive images for molecular imaging with low resolution [1, 3, 4]. Finally, SPECT provides molecular information and the blood flow of tissues [3]. It obtained low-resolution images.

Nowadays, image fusion is a technique that plays a vital role in several areas such as object recognition, medical diagnosis, computer vision, and remote sensing [2]. The objective of the image fusion technique is to integrate the significant information from one or more images to produce a fused image contains complementary information without any artifacts, reduce noises and remove the redundancy of the images [4, 5]. This image also must preserve the meaningful information from the source images [6].

The basic three steps for any fusion technique are image decomposition, fusion rule, and image reconstruction [5, 7]. In the image decomposition step, use different decomposition techniques to decompose the two source images into sub-images. The sub-image coefficients have been fused by the fusion rules. The final image became created using one-of-a-kind image reconstruction strategies. The main image fusion procedure is defined in the Fig. 1.

Fig. 1
figure 1

Basic image fusion process [3]

Full size image

Image fusion comprises of three categories pixel-level-based fusion, decision-level-based fusion, and feature-level-based fusion [8, 9]. Pixel-level-based fusion techniques create the informative fused image by merging the source images pixel information. For feature-level-based fusion techniques, some features from the source images are extracted like shape, edges, and directions. These features are combined to create meaningful features for a more descriptive image [3]. Pixel-level-based fusion strategies are classified into spatial domain strategies and transform domain strategies [2, 6].

Spatial domain strategies aim to reconstruct the fused image by detecting image pixels from the source images, where the input image data is retained in the fused images [2]. The core benefit of these strategies is their low computational time. On the other hand, these produce low-contrast images with spatial distortion and color distortion [4, 6]. There are numerous strategies for a spatial domain, just like the principal component analysis (PCA) approach, minimal fusion approach, weighted average approach, average fusion approach, and maximum fusion approach. Transform domain strategies reconstruct the fused image by transforming the source images into the frequency domain. The fusion rule is applied to the low and high-frequency coefficients. The fused image is created by inverting the transformation [3]. The core advantages of the transform domain are avoiding distortion and being more efficient than spatial domain strategies [6].

Multimodal medical image fusion (MMIF) is considered the core type of image fusion to combine or fuse several images from one or more imaging modalities to improve imaging quality while retaining vital information [10]. The detailed information obtained from the combined images to accurately identify the location of the lesion and accurately diagnose diseases [2, 6]. The main combinations of multimodal medical images such as (MRI-CT, MRI-PET, MRI-SPECT, and CT-SPECT) listed in the following Fig. 2.

The multiscale transform (MST) is considered the most common image fusion technique in the transform domain. The MST-based image fusion techniques include non-subsampled contourlet transform (NSCT) [11, 12], Laplacian pyramid (LP) [13], non-subsampled shearlet transform (NSST) [8], and convolutional neural networks (CNN) [14].

Fig. 2
figure 2

Examples of the common combinations of multimodal medical image fusion (MMIF)

Full size image

In this paper, we propose the combination of the NSCT and the PA-PCNN to fuse medical images from different modalities. The proposed method combines the features of the NSCT transform, involving shift-invariance, multi-scale, and multidirectional properties, along with the PA-PCNN to solve the problem of PCNN-free parameters. To verify the effectiveness of the proposed method, we used 100 pairs of multi-modality brain images. Our experiments use five fusion metrics to evaluate the proposed algorithms performance. They are entropy (EN), mutual information (MI), weighted edge information (Q\(^{AB/F}\)), nonlinear correlation information entropy (Q\(_{ncie}\)), and average gradient (AG). All these sections are discussed in detail in Section 4.2. We compared our proposed method with five valuable fusion methods: the NSCT [15], the Non-Subsampled Contourlet Transform-Sparse Representation (NSCT-SR) [16], the Non-Subsampled Contourlet Transform-phase congruency local Laplacian (NSCT-PCLP) [2], the Non-Subsampled Shearlet Transform-Parameter Adaptive Pulse Coupled Neural Network (NSST-PAPCNN) [17], and the Non-Subsampled Contourlet Transform-Pulse Coupled Neural Network (NSCT-PCNN) [18]. The experiments with the proposed algorithm and the comparisons of MIF results are presented in Section 4.4. Experiment results show that the proposed method retains significant and salient features such as high contrast, sharp boundaries, and smooth edges without noise. It also helps doctors diagnose the disease accurately from one image with limited energy and time.

1.1 Motivations

The main motivations for this paper are choosing the most effective method for combining several source images with the following characteristics: high efficiency, high spatial resolution preservation, and low color distortion using the PA-PCNN in the NSCT domain to aid doctors in accurately diagnosing diseases. In addition, creating a new accurate fused image with more detailed information than the input images.

1.2 Contribution

Our proposed medical image fusion method uses the NSCT features, including multi-scale, shift-invariance, and multi-directional properties, along with the PA-PCNN that solve the problem of PCNN-free parameters. The main advantages of PA-PCNN are its high speed and fewer iterations than other PCNN models used for image fusion. The PA-PCNN tool gain high fusion performance and capture the subtle differences and fine details present in the source medical images. The proposed method enhances the output fused image’s high contrast, clarity, and information content.

The main contribution of this paper is to create a high-performance fusion algorithm to detect whole brain regions from different multi-modality medical images with low computational time and preserving image color, image edges and the structure of image without any distortion.

The rest of the paper is as follows. Section 2 focuses on some previous works for fusing medical images. The methods employed in this paper is presented in Section 3. The MMIF proposed algorithm employed in this work is presented in Section 3.3. The experimental results presented in Section 4. We discuss the results berifly in the Section 5. Finally, we conclude and summarize whole the paper in Section 6.

2 Related work

Researchers presented various MMIF approaches. These approaches are examined and accomplished precise results. In this section, we preview and analyze some of this research.

Zhu et al. [2] proposed a new MMIF technique based on the NSCT transform, phase congruency, and local Laplacian energy techniques. The proposed technique is carried out in three major phases. Using the NSCT method, the input images decomposed into the low pass and high pass subbands. Then used the fusion rule based on the phase congruency and the local Laplacian energy to fused the highpass subbands and lowpass subbands. Finally, the final image reconstructed using the inverse NSCT technique. Results evaluated using the nine fusion metrics are the information entropy (IE), the SSIM metric, the CB metric, the Tsallis entropy (TE), the phase congruency (PC) metric, the Q\(_{G}\), the VIFF, and the MI. The proposed technique is tested on CT, MRI, PET, and SPECT images using MATLAB2014a. The results demonstrate that the proposed technique produces fusion images with high-performance and low computational time. The main disadvantage of this technique is that it is not suitable for fusing PET-MRI images.

Yin et al. [17] A presented MMIF approach uses the parameter-adaptive pulse coupled neural network (PAPCNN) approach within the NSST. The NSST decomposed input images into high and low-frequency bands. The low-frequency bands are fused using the weighted sum of eight neighborhood-based modified Laplacian (WSEML) and the PAPCNN method for high-frequency bands. Then the final image is done by using the inverse NSST approach. The experiment used 80 pairs of four image modalities. Results declared that the presented technique has high performance as other compared methods.

**a et al. [19] proposed a MMIF technique based on the PAPCNN method and the convolutional sparse representation (CSR) method in the NSST transform. The NSST decomposed input images into low and high-frequency subbands. Then apply CSR and PAPCNN methods to fuse low and high-frequency subbands. The fused image was presented by the inverse NSST method. The proposed experiment used 70 pairs of input images. Results declared that the proposed technique has high performance in both the objective and visual effects compared with other techniques. The defect of this method is that the quality measures are not suitable for merging MR-T1, MR-T2, and SPECT image modalities.

Vanitha et al. [20] suggested an MMIF method based on spatial frequency PAPCNN (SF-PAPCNN) and the NSST transform. The NSST decomposes the medical images into frequency sub-bands. The maximum rule fuses low-frequency sub-bands, and the SF-PAPCNN fuses the high-frequency sub-bands. The output image is created using the inverse of the NSST transform. The performance of the suggested method is validated using these metrics: weighted edge information (\(Q_{AB/F}\)), standard deviation (SD), mutual information (MI), and entropy (EN). The suggested method fuses images while preserving edge information and structural details. The defect of this method is that the quality measures are not suitable for merging CT, MRI, and SPECT image modalities.

Li et al. [21] presented a MMIF technique based on the Pulse-Coupled Neural Network (PCNN) method in NSCT transform. The presented paper used the NSCT method to decompose the input image and generate more high and low-frequency bands. Then apply the PCNN to fuse the low-frequency bands and the guided image filtering (GIF) and WSEML method as the fusion rule in high-frequency bands. The results declared that the presented technique has high performance in medical image fusion.

Mei et al. [22] proposed the MMIF method using NSCT and adaptive PCNN for combining medical images. This MMIF method uses the NSST to decompose the input images into low- and high-frequency sub-bands. Then use the combination of the Sum of Directional Gradients (SDG) and the PCNN model to fuse the low sub-band frequency coefficients. On the other hand, the high sub-band frequency coefficients are obtained by merging the local energy and the global gradient. The final fused image is generated using the inverse of the NSST transform. The proposed method is compared using the combinations of the NSCT transform. The main advantage of this method is that it achieves better results with low computational time. The defect of this method uses only CT and MRI images.

Vanitha et al. [23] proposed an MMIF method using the PA-PCNN and energy attribute in the NSST domain. The NSST transformed the input images into low- and high-subband coefficients. The energy attribute fuses the low subband coefficients, and the PA-PCNN fuses the high subband coefficients. The fused image is obtained by applying the inverse NSST to the fused coefficients. The proposed technique is carried out on CT, MRI, PET, and SPECT images. The proposed method uses the fused image in the disease treatment. On the other hand, some edges cannot be detected accurately in the MRI-PET and MRI-SPECT modalities.

Sa et al. [18] proposed an MMIF method using the NSCT and PCNN methods. The NSCT decomposes the input images into low- and high-frequency coefficients. The PCNN merges both the high- and low-frequency coefficients. The proposed method used six collections of medical images in different modalities: CT / MRI, MR-T1 / MR-T2, MR-T2 / SPECT, and MR-T1 / PET. The proposed method’s performance is evaluated using these measures: peak signal-to-noise ratio (PSNR), \(Q_{AB/F}\), average gradient (AG), nonlinear correlation information entropy (\(Q_{ncie}\)), SD, MI, and EN. The experimental results show that the proposed method outperforms the compared fusion methods. The benefits of the proposed method achieve promising results that help doctors detect the accurate position of the disease and treat patients.

Table 1 Comparison of various related work
Full size table

These studies are summarised in the following Table 1. In this table, we list the related work in the MIF area. [2] fused CT, MRI, PET, and SPECT images using phase congruency and local Laplacian energy methods in the NSCT domain. It produces high-performance results with low computational time as compared with other fusion methods. However, this work is not suitable for fusing PET and MRI images. [17] used the PAPCNN method in the NSST domain to merge various medical images. This method produces high-performance results but still consumes more time. [19] used the CSR method to fuse low subbands and PAPCNN in high subbands in the NSST domain. This method produces better results except when merging MR-T1, MR-T2, and SPECT images. [20] applied the maximum rule and SF-PAPCNN methods to fuse low and high subbands in the NSST domain to preserve structure and information details but still produce inaccurate results while merging CT, MRI, and SPECT images. [21] fused all types of medical images by using PCNN, GIF, and WSEML methods in the NSCT domain. This method used in clinical diagnosis but highly time-consuming. [22] fused CT and MRI images using adaptive PCNN and SDG in the NSST domain. This method achieves better results and consumes less time, but it cannot be tested on other medical imaging modalities. [23] used the PAPCNN and energy attributes in the NSST domain to treat diseases, but some edges could not be detected accurately. [18] fused all types of medical images using PCNN in the NSCT domain. It also treats patients and detects the accurate position of the disease, but it still consumes time.

3 Methods

This section outlines the methods proposed in this paper. These methods, including the NSCT and the PAPCNN, are declared in detail as follows.

Fig. 3
figure 3

The basic NSCT image decomposition process [18]

Full size image

3.1 Non-subsampled contourlet transform (NSCT)

NSCT is one of the main transformation methods based on the contourlet transform (CNT). CNT is mainly used in image processing and gains high-performance results in this field. The NSCT method solves the CNT shift variant problem that results from up-sampling and down-sampling [2]. NSCT adopts a shift-invariant, multi-scale decomposition, and multi-directional transform. It consists of the Non-Subsampled Pyramid Filter Bank (NSPFB) and the Non-Subsampled Directional Filter Bank (NSDFB) [2, 21]. Figure 3 illustrates the process of image decomposition using the NSCT method.

The NSPFB includes a two-branch filter bank with a lack of down-sampling and up-sampling [2]. It achieves two NSCT features that are shift-invariant and multi-scale decomposition. In NSPFB, source images decomposed into low- and high-pass frequency images. The NSPFB decomposes the low-frequency image iteratively shown in Fig. 3. The NSPFB generates K+1 sub-images, with K high-frequency image while 1 is a low-frequency image [2]. The NSDFB aims to use the high-frequency images produced from the NSPFB decomposition to produce directional sub-images of the same size as the input images. It gets the NSCT multidirectional feature [2].

3.2 Parameter-adaptive pulse-coupled neural network (PA-PCNN)

PCNN is one of the biological artificial neural network methods established by [24] that depends on iterative calculation without training steps. Its widely used in image processing and image fusion as a one-layer network [17]. Setting many free parameters, like various amplitudes and linking strengths, is the main problem in the standard PCNN method. The PAPCNN method is used in image segmentation and image fusion areas. This method aims to solve the problem of setting several free parameters to improve the performance of the fused image. When compared to other traditional PCNN methods, the PA-PCNN method is faster and requires less time. The basic PA-PCNN structure is described in the Fig. 4.

Fig. 4
figure 4

Architecture of the PA-PCNN Method [17]

Full size image

The PA-PCNN method described mathematically using these equations [17, 19, 25]:

$$\begin{aligned} F_{ij}[n]= S_{ij} \end{aligned}$$
(1)
$$\begin{aligned} L_{ij}[n]=V_{L}\sum _{kl}W_{ijkl}Y_{kl}[n-1] \end{aligned}$$
(2)
$$\begin{aligned} U_{ij}[n]=e^{-\alpha _{f}}U_{ij}[n+1]+F_{ij}[n](1+\beta L_{ij}[n]) \end{aligned}$$
(3)
$$\begin{aligned} Y_{ij}[n]={\left\{ \begin{array}{ll} 1, \ U_{ij}[n]>E_{ij}[n-1] \\ 0, \ otherwise \end{array}\right. } \end{aligned}$$
(4)
$$\begin{aligned} E_{ij}[n]=e^{-\alpha _{e}}E_{ij}[n-1]+V_{E}Y_{ij}[n] \end{aligned}$$
(5)
$$\begin{aligned} T_{ij}[n]=T_{ij}[n-1]+Y_{ij}[n] \end{aligned}$$
(6)

Where the feeding input is \(F_{ij}\) and \(L_{ij}\) is a linking input in the i and j position. The total internal activity and the external stimulus are defined by \(U_{ij}\) and \(S_{ij}\), respectively. \(Y_{ij}\) is the neurons output, and the linking strength is \(\beta \). \(E_{ij}\) is the dynamic threshold function. The \(\alpha _{f}\) is an exponential decay coefficient of \(F_{ij}\), and \(\alpha _{e}\) is the time constant for the threshold function. The parameters \(V_{E}\) and \(V_{L}\) are the amplitude of \(E_{ij}\) and \(L_{ij}\), respectively. \(T_{ij}[n]\) is the firing time for each neuron.

n represents the iteration and \(w_{ijkl}\) is a synaptic weight matrix.

\(w_{ijkl}\) = \(\begin{bmatrix} .5 &{} 1 &{} .5\\ 1 &{} 0 &{} 1\\ .5 &{} 1 &{}.5 \end{bmatrix}\)

3.3 Proposed method

In this paper, we propose MMIF technique. The proposed technique includes these steps: NSCT decomposition, the fusion of low-and high-frequency bands, and NSCT reconstruction. The block diagram of the proposed technique is described in Fig. 5.

Fig. 5
figure 5

The schematic diagram of the proposed fusion method

Full size image

The steps of the proposed fusion method for multi-modality medical source images were discussed in Algorithm 1.

Algorithm 1
figure a

the algorithm of the proposed multi-modality medical image fusion method.

Full size image

3.3.1 Image decomposition

In the NSCT decomposition step, we use the NSCT method to decompose multi-modality preregistered medical images A and B into low-and high-frequency coefficients \(L_{A}\), \(H_{A}\), \(L_{B}\), and \(H_{B}\). The \(L_{A}\) is a low-frequency coefficient for image A, and the high-frequency coefficient is \(H_{A}\). The \(L_{B}, H_{B}\) have the same meaning as image A.

3.3.2 Fusion rule

Fusion of low-and high-frequency coefficients using the PA-PCNN method as in (1) to (5) and calculating the firing time as in (6). The fused low-and high-frequency coefficients \(L_{F}\) and \(H_{F}\) are calculated using the following equations:

$$\begin{aligned} L_{F}(i,j)={\left\{ \begin{array}{ll} L_{A}(i,j), \ if \ T_{A,ij}[N]>T_{B,ij}[N] \\ L_{B}(i,j), \ otherwise \end{array}\right. } \end{aligned}$$
(7)
$$\begin{aligned} H_{F}(i,j)={\left\{ \begin{array}{ll} H_{A}(i,j), \ if \ T_{A,ij}[N]>T_{B,ij}[N] \\ H_{B}(i,j), \ otherwise \end{array}\right. } \end{aligned}$$
(8)

where N represents the total number of iterations.

3.3.3 Image reconstruction

In the NSCT reconstruction step, we use the inverse of the NSCT transform to combine the fused low-and high-frequency coefficients \(L_{F}\) and \(H_{F}\) to produce the fused image F.

4 Experiment results

In this section, we discuss the performance and effectiveness of the proposed method. We also simulate the results of the proposed method. All source images used in this experiment are discussed. This section consists of four subsections: datasets, quality measures, performance evaluation, and comparing with other techniques.

4.1 Datasets

To validate the performance of the proposed method, we used 100 pairs of multimodal medical source images, including 25 CT-MRI image pairs, 25 MRI (MR-T1 and MR-T2) image pairs, 25 MRI-PET image pairs, and 25 MRI-SPECT image pairs. These source images were obtained from Harvard Medical School’s Whole Brain Atlas [26]. All image pairs were effectively registered. These source images have the exact spatial resolution of 256 x 256 pixels. The MATLAB R20l8a toolbox was used to run all experiments on a PC running Windows 10 with an Intel Core i7 processor and 8 GB of RAM. Figure 6 shows examples of the source images. Where, data 1 and data 2 represent the CT and MR images, data 3 and data 4 represent the MR-T1 and MR-T2 images, data 5 and data 6 indicate the MRI and PET images, and data 7 and 8 indicate the MRI and SPECT images.

Fig. 6
figure 6

Source images: Data1 (a, b), Data2 (c, d), Data3 (e, f), Data4 (g, h), Data5 (i, j), Data6 (k, l), Data7 (m, n), and Data8 (o, p)

Full size image

4.2 Quality measures

In this subsection, we present some evaluation metrics for medical image fusion. There are various evaluation metrics for image fusion. Our experiments use five fusion metrics to evaluate the proposed algorithm’s performance. They are entropy (EN), Mutual Information (MI), weighted edge information (Q\(^{AB/F}\)), nonlinear correlation information entropy (Q\(_{ncie}\)), and Average Gradient (AG). All of these metrics are discussed as follows.

  • The Entropy (EN) evaluates the information content and the amount of information in the fused image. A high EN value means a high-performance image. It’s defined as:

    $$\begin{aligned} EN=-\sum _{l=0}^{L-1}p_{l}\log _{2}p_{l} \end{aligned}$$
    (9)

    where \(p_{l}\) is the probability of the gray levels of l and L represents a total number of gray levels of an image [18, 27, 28].

  • The Mutual Information (MI) measures the total amount of information contained in the input images and the fused image. A higher MI value means higher image quality [27,28,29]. MI is indicated by:

    $$\begin{aligned} MI=MI_{AF}+MI_{BF} \end{aligned}$$
    (10)

    Where \(MI_{AF}\) represents the mutual information between both the source image A and the fused image F.

    $$\begin{aligned} MI_{AF}=\sum _{m,n}p_{A,F}(m,n)\log _{2}\left[ \frac{p_{A,F}(m,n)}{p_{A}(m)p_{F}(n)}\right] \end{aligned}$$
    (11)

    Where \(p_{A,F}(m,n)\) represents the joint probability of the source and the fused image. (m, n) represents the position of a pixel.

  • The weighted edge information \(({\textbf {Q}}^{AB/F})\) measures edge intensity information and total information moved from the input images to the fused image. Is given by [27, 29]:

    $$\begin{aligned} {{{Q}^{^{AB/F}}{=}\tfrac{{\sum _{m=1}^{M}}{\sum _{n=1}^{N}}\left( Q^{AF}(m,n){W_{A}}(m,n)+Q^{BF}(m,n)W_{B}(m,n)\right) }{\sum _{m=1}^{M}\sum _{n=1}^{N}(W_{A}(m,n)+W_{B}(m,n))}}} \end{aligned}$$
    (12)

    Where \(Q^{AF}\) and \(Q^{BF}\) are the edge information preservation factors, \(W_{A}\) and \(W_{B}\) are the weighted items. The \(Q^{AB/F}\) value ranges between 0 and 1. (m, n) represents the position of a pixel.

  • The Average Gradient (AG) calculates the gradient information and texture details in the fused image [28, 29]. A higher AG value means a high-performance image. It is calculated using an equation.

    $$\begin{aligned} {AG=}\sum _{m=1}^{M}\sum _{n=1}^{N}\sqrt{\tfrac{\left( (F(m,n)-F(m+1,n))^{2}+(F(m,n)-F(m,n+1))^{2}/2\right) }{MN}} \end{aligned}$$
    (13)

    where M and N represent the image sizes and (m, n) represents the position of a pixel.

  • The nonlinear correlation information entropy Q\(_{ncie}\) measures the nonlinear information of the fused image. Q\(_{ncie}\) is denoted by the following formula [5]:

    $$\begin{aligned} Q_{ncie}(X,Y)=2+\sum _{i=1}^{b^2}(\tfrac{n_{i}}{N})\log _{b}(\tfrac{n_{i}}{N}) \end{aligned}$$
    (14)

    Where N refers to the dataset size and \(n_{i}\) refers to the number of samples.

4.3 Performance evaluation

In this subsection, we compared our proposed method with five valuable fusion methods: the NSCT [15], the Non-Subsampled Contourlet Transform-Sparse Representation (NSCT-SR) [16], the Non-Subsampled Contourlet Transform-phase congruency local Laplacian (NSCT-PCLP) [2], the Non-Subsampled Shearlet Transform-Parameter Adaptive Pulse Coupled Neural Network (NSST-PAPCNN) [17], and Non-Subsampled Contourlet Transform-Pulse Coupled Neural Network (NSCT-PCNN) [18]. We use five mainstream medical fusion metrics. The performance of the proposed algorithm is better if all of these metrics have higher values.

Fig. 7
figure 7

Fusion result of “data 1” (a) CT and (b) MR-T2 using (c) NSCT method, (d) NSCT-SR method, (e) NSCT-PCLP method, (f) NSST-PAPCNN method, (g) NSCT-PCNN and the proposed method (h) NSCT-PAPCNN

Full size image
Fig. 8
figure 8

Fusion result of “data 2” (a) CT and (b) MR-T2 using (c) NSCT method, (d) NSCT-SR method, (e) NSCT-PCLP method, (f) NSST-PAPCNN method, (g) NSCT-PCNN and the proposed method (h) NSCT-PAPCNN

Full size image
Fig. 9
figure 9

Fusion result of “data 3” (a) MR-T1 and (b) MR-T2 using (c) NSCT method, (d) NSCT-SR method, (e) NSCT-PCLP method, (f) NSST-PAPCNN method, (g) NSCT-PCNN and the proposed method (h) NSCT-PAPCNN

Full size image
Fig. 10
figure 10

Fusion result of “data 4” (a) MR-T1 and (b) MRR-T2 using (c) NSCT method, (d) NSCT-SR method, (e) NSCT-PCLP method, (f) NSST-PAPCNN method, (g) NSCT-PCNN and the proposed method (h) NSCT-PAPCNN

Full size image
Fig. 11
figure 11

Fusion result of “data 5” (a) MRI and (b) PET using (c) NSCT method, (d) NSCT-SR method, (e) NSCT-PCLP method, (f) NSST-PAPCNN method, (g) NSCT-PCNN and the proposed method (h) NSCT-PAPCNN

Full size image
Fig. 12
figure 12

Fusion result of “data 6” (a) MRI and (b) PET using (c) NSCT method, (d) NSCT-SR method, (e) NSCT-PCLP method, (f) NSST-PAPCNN method, (g) NSCT-PCNN and the proposed method (h) NSCT-PAPCNN

Full size image
Fig. 13
figure 13

Fusion result of “data 7” (a) MRI and (b) SPECT using (c) NSCT method, (d) NSCT-SR method, (e) NSCT-PCLP method, (f) NSST-PAPCNN method, (g) NSCT-PCNN and the proposed method (h) NSCT-PAPCNN

Full size image
Fig. 14
figure 14

Fusion result of “data 8” (a) MRI and (b) SPECT using (c) NSCT method, (d) NSCT-SR method, (e) NSCT-PCLP method, (f) NSST-PAPCNN method, (g) NSCT-PCNN and the proposed method (h) NSCT-PAPCNN

Full size image

4.4 Comparing with other techniques

In this subsection, our proposed method is compared with other methods based on visual and objective quality. We compared image fusion methods, which include all eight combinations of CT, MRI, T1, T2-weighted MRI, PET, and SPECT. First, in color image transformation, the PET and SPECT images into YUV space. Then fuse the Y component with the MR-T2 image. Finally, convert the middle fusion back to RGB using the U and V components to produce the fused image. The experiments with the proposed algorithm and the comparisons of MIF results are demonstrated in Figs. 7, 8, 9, 10, 11, 12, 13 and 14.

Figure 7 shows the experimental results on “data 1”, which contains the source images (CT and MRI) and the fused image using various methods. Figure 7a is a CT image. Figure 7b is an MRI image, and Fig. 7c shows the result with color distoration. Figure 7d loses some valuable information from the source image Fig. 7a. Figure 7e shows the blurred image and loses some details. Figure 7h shows the brain structure is clear than in Fig. 7g and f and preserves the valuable information from the CT and MRI images without color distortions.

Figure 8 shows the experimental results on “data 2” for the source images (CT and MRI) and the fused image using various methods. Figure 8a is a CT image, Fig. 8b is an MRI image, and Fig. 8c d show results that contain some artifacts, color distortions, and loss of some meaningful information from the source images. Figure 8e shows a blurred image. In Fig. 8f, some brain structures are not clearly defined from the CT image Fig. 8a. Figure 8h: The brain structure is clearer than in Fig. 8f and g and preserves the valuable information from the CT and MRI images without distortions.

Figure 9 shows the fusion results for “data 3”. Figure 9a is an MR-T1 image, Fig. 9b is an MR-T2 image, and Fig. 9c d show results that contain color distortions. Figure 9g shows a clearer image than Fig. 9e and f, but also loses some details. Figure 9h: The brain structure is clearer than in Fig. 9g, which preserves the valuable information from the MR-T1 and MR-T2.

Figure 10 shows the fusion results for “data 4”. Figure 10a is an MR-T1 image, Fig. 9b is an MR-T2 image, and Fig. 10c d show results with color distortions. Figure 10e shows a better result than Fig. 10c and d but loses valuable information from Fig. 10a. Figure 10h: The brain structure is better than in Fig. 10g and f, preserving the valuable information from the MR-T1 and MR-T2 without any artifacts.

Figure 11 shows the fusion results for “data 5”, which involves MRI and PET images Fig. 11a is an MRI image, and Fig. 11b is a PET image. The fusion results from the NSCT, NSCT-SR, and NSCT-PCLP methods perform well in extracting details from MRI images but still have color distortion as well as the brain edges cannot be detected successfully in Fig. 11c, d, and e. The NSST-PAPCNN, the NSCT-PCNN, and the proposed method can preserve color information and achieve higher quality than other methods; see Fig. 11f, g and h. Figure 11h shows that the proposed method performs better than NSST-PAPCNN and NSCT-PCNN; see Fig. 11f-g on extraction details in some regions.

Figure 12 shows the fusion results on “data 6” for MRI and PET images. Figure 12a is an MRI image, and Fig. 12b is a PET image. The fusion results from the NSCT and NSCT-SR contains color distortion results in Fig. 12c, d. The NSCT-PCLP methods perform well in extraction details from MRI images as well as the brain edges cannot detect successfully in Fig. 12c, d, and 12e. The NSST-PAPCNN, the NSCT-PCNN, and the proposed method can preserve color information and achieve higher quality than other methods see Fig. 12f, g, and 12h. Figure 12h shows that the proposed method performs better than NSCT-PCNN as in Fig. 12g on extraction details in some regions without any distortions.

Figure 13 shows the fusion results on “data 7” for MRI and SPECT images. Figure 13a is an MRI image, and Fig. 13b is a SPECT image. The fusion results from the NSCT and NSCT-SR preserve the detailed MRI information with the color fidelity problem in Fig. 13c and d. Figure 13e is better than Fig. 13d in color fidelity issue but loses some details from the MRI image. The NSST-PAPCNN can preserve functional information from the SPECT image, but some edge and structure information cannot be detected accurately (see Fig. 13f). Figure 13g is better than Fig. 13f but contains some artifacts. In Fig. 13h, the proposed method is able to preserve color and structure information from the source images and achieve higher-quality images than other methods.

Figure 14 shows the fusion results on “data 8” for MRT2 and SPECT images. Figure 14b is an MRI image, and Fig. 14b is a SPECT image. The fusion results from the NSCT and NSCT-SR can preserve the detailed MRI information with the color fidelity problem in Fig. 14c and d. Figure 14e is better than Fig. 14d in the color fidelity issue but loses some details from the MRI image. The NSST-PAPCNN can preserve functional information from the SPECT image, but some edge and structure information cannot be detected accurately (see Fig. 14f). Figure 14f shows a better than Fig. 14g. In Fig. 14h, the proposed method can preserve color and structure information from the source images and achieve higher quality images than other methods.

Assessment metrics for the proposed method and each compared method are shown in Tables 2, 3, 4, 5, 6, 7, 8, and 9. Table 2 shows the comparative assessments of the proposed algorithm and the compared methods on “data 1” for the CT and MRI images. Our proposed method has the best values in EN 5.3803, Q\(^{AB/F}\) 0.5819, Q\(_{ncie}\) 0.8063, and AG 10.1030. On the other hand, the NSST-PAPCNN method has the best value in MI 2.8512, and the NSCT is faster than other compared methods. The bolded values are the best in every column.

Table 3 shows the assessment of the proposed algorithm and the compared methods on the “data 2” for the CT and MRI. Our proposed method has values of EN 5.0058, MI 3.0729, Q\(^{AB/F}\) 0.6656, Q\(_{ncie}\) .8072, AG 7.2829, and time 4.3944 sec. The proposed algorithm metrics are bolded values because these values are the best in every column. The proposed algorithm is better than other compared methods in all metrics values.

Table 4 shows the assessments of the proposed algorithm and the compared methods for the MR-T1 and MR-T2 images in Fig. 9. Our method has EN 5.7306, MI 3.1123, Q\(^{AB/F}\) 0.5871, Q\(_{ncie}\) .8078, AG 10.8864, and time 1.8272 sec. The bolded values are the best in every column. The proposed algorithm is better than other compared methods in EN, AG, and time metrics, but the NSCT-PCNN method is the best in MI, Q\(^{AB/F}\), and Q\(_{ncie}\) metrics.

Table 5 shows the assessments in Fig. 10. Our method has EN 5.6301, MI 3.1816, Q\(^{AB/F}\) 0.6230, Q\(_{ncie}\) 0.8077, AG 10.3080, and time 2.0821 sec. The proposed algorithm is better than other compared methods in MI, Q\(_{ncie}\), and AG metrics; otherwise, the NSCT-PCNN method is the best MI value, the NSST-PAPCNN method is the best in Q\(^{AB/F}\) value, and the NSCT method has a low computational time. Table 6 shows the assessments of the proposed algorithm and the compared methods on MRI and PET images shown in Fig. 11. The proposed algorithm is better than other compared methods in EN, MI, Q\(^{AB/F}\), and Q\(_{ncie}\). On the other hand, the NSCT-SR method has a high AG value, and the NSCT method has a low computational time.

Table 7 shows the result metrics of Fig. 12 for the MRI and PET images. Our proposed method has EN 5.3461, MI 3.3457, Q\(^{AB/F}\) 0.7425, Q\(_{ncie}\) 0.8092, AG 11.1658, and time 4.1552 sec. The proposed algorithm is better than other compared methods in EN, MI, Q\(^{AB/F}\), Q\(_{ncie}\), and time values. Table 8 shows the result metrics of Fig. 13 involving combinations of MRI and SPECT images. Our proposed algorithm has higher values than other compared methods in EN, MI, Q\(^{AB/F}\), and Q\(_{ncie}\), but the NSCT-PCNN method has the best AG value. The time in the NSCT is better than the proposed algorithm time.

Table 9 shows the comparative assessments of the proposed algorithm and the compared methods on MRI and SPECT images shown in Fig. 14. Our proposed algorithm has higher values than other compared methods in EN, MI, Q\(^{AB/F}\), Q\(_{ncie}\), AG, and time. The results show that the proposed algorithm performs better than other compared methods while retaining more information from the source images.

In this paper, five objective metrics (EN, MI, Q\(^{AB/F}\), Q\(_{ncie}\), and AG) have evaluated the fusion performance for the NSCT, NSCT-SR, NSCT-PCLP, NSST-PAPCNN, and the proposed algorithm using CT/MRI, MR-T1/MR-T2, MRI/PET, and MRI/SPECT images. These metrics have been represented in Figs. 15, 16, , , 19, and 20. Figure 15 shows the graphical representation of the entropy metric between 8 groups of medical images over various fusion methods: NSCT, NSCT-SR, NSCT-PCLP, NSST-PAPCNN, NSCT-PCNN, and the proposed method. The proposed method has the highest entropy value compared with other methods.

Figure 16 shows the graphical representation of the mutual information metric between 8 groups of medical images over various fusion methods: NSCT, NSCT-SR, NSCT-PCLP, NNST-PAPCNN, NSCT-PCNN, and the proposed method. The proposed method has a high mutual information value, except Tables 2 and 4 are not efficient. Figure  shows the graphical representation of the Q\(^{AB/F}\) metric between 8 groups of medical images over various fusion methods: NSCT, NSCT-SR, NSCT-PCLP, NSST-PAPCNN, NSCT-PCNN, and the proposed method. The proposed method has a high Q\(^{AB/F}\) value, except Tables 4 and 5 are not efficient.

Figure  shows the graphical representation of the Q\(_{ncie}\) metric between 8 groups of medical images over various fusion methods: NSCT, NSCT-SR, NSCT-PCLP, NNST-PAPCNN, NSCT-PCNN, and the proposed method. The proposed method has a high Q\(_{ncie}\) value, except that Table 4 is not efficient. Figure 19 shows the graphical representation of the average gradient metric between 8 groups of medical images over various fusion methods: NSCT, NSCT-SR, NSCT-PCLP, NNSTPAPCNN, NSCT-PCNN, and the proposed method. The proposed method has a high average gradient value, except Tables 6 and 8 are not efficient. Figure 20 shows the graphical representation of the time metric between 8 groups of medical images over various fusion methods: NSCT, NSCT-SR, NSCT-PCLP, NNST-PAPCNN, NSCT-PCNN, and the proposed method. The proposed method consumes more time (see Tables 2, 5, 6, and 8).

Table 2 Quality assessment of different MMIF methods of the “data 1” for the CT and MR images
Full size table
Table 3 Quality assessment of different MMIF methods for the “data 2” of the CT and MR images
Full size table
Table 4 Quality assessment of different MIFF methods for the “data 3” of the MR-T1 and MR-T2 images
Full size table
Table 5 Quality assessment of different MIFF methods for the “data 4” of the MR-T1 and MR-T2 images
Full size table
Table 6 Quality assessment of different MMIF methods for the “data 5” of the MRI and PET images
Full size table
Table 7 Quality assessment of different MMIF methods for the “data 6” of the MRI and PET images
Full size table
Table 8 Quality assessment of different MMIF methods for the “data 7” of the MRI and SPECT images
Full size table
Table 9 Quality assessment of different MMIF methods for the “data 8” of the MRI and SPECT images
Full size table

5 Discussion

To summarize and discuss the experimental results, the fused medical images based on the PA-PCNN method in the NSCT domain. The proposed MMIF method presented results visually and using quality assessments. We performed the experiments on eight groups of medical images as described in Section 4.1. The visual results of the proposed method are compared with the NSCT, NSCT-SR, NSCT-PCLP, NSST-PAPCNN, and NSCT-PCNN methods. Figures 7, 8, 9, 10, 11, 12, 13, 14 demonstrate the results of the proposed method and five MIF methods. It is shown that the NSCT and NSCT-SR methods have poor performance with color distortions and artifacts. The NSCT-PCLP is better than the NSCT and NSCT-SR methods, except it has blurred images. The NSST-PAPCNN preserves the source image information without detecting edges and time-consuming. The NSCT-PCNN is better than the NSST-PAPCNN method, except it contains some artifacts. On the other hand, the proposed method achieves high contrast and clear fusion results by preserving salient features for bone structure and soft tissues of brain images. Next, we summarize the quality assessments of the fused image using the five fusion metrics, such as EN, MI, Q\(^{AB/F}\), Q\(_{ncie}\), and AG. Tables 2, 3, 4, 5, 6, 7, 8, and 9 show the comparative assessment results of the proposed method over the five compared methods. The average values for EN, MI, Q\(^{AB/F}\), Q\(_{ncie}\), and AG with the proposed approach are 5.2144, 3.1282, 0.6600, 0.8071, and 8.9874, respectively.

Fig. 15
figure 15

Entropy assessment of different fusion methods compared to proposed method

Full size image
Fig. 16
figure 16

Mutual Information assessment of different fusion methods compared to proposed method

Full size image
Fig. 17
figure 17

Q\(^{AB/F}\) assessment of different fusion methods compared to proposed method

Full size image
Fig. 18
figure 18

Q\(_{ncie}\) assessment of different fusion methods compared to proposed method

Full size image
Fig. 19
figure 19

Average Gradient assessment of different fusion methods compared to proposed method

Full size image
Fig. 20
figure 20

Time assessment of different fusion methods compared to proposed method

Full size image

The proposed method retains significant and salient features such as high contrast, sharp boundaries, and smooth edges without noise. It also helps doctors diagnose the disease accurately from one image with limited energy and time. The main problem with the proposed method is that it is not effective for fusing MR-T1 and MR-T2 images as compared to other fusion methods, so we need to work on it in the future.

6 Conclusion

In this paper, we introduce a novel multimodal MIF algorithm. The proposed algorithm depends on the PA-PCNN and the NSCT methods. This algorithm consists of the following steps: decomposition, fusion rule, and reconstruction. First, the NSCT decomposes two input images into low and high-frequency subbands. Then, the PA-PCNN fused the low and high-frequency subbands. Finally, the inverse of the NSCT produces the final image. The experiments were conducted on various groups of grayscale and colored medical source images from the Whole Brain Atlas database. The proposed algorithms performance was evaluated using five fusion metrics: entropy, mutual information, \(Q^{AB/F}\), \(Q_{ncie}\), and average gradient. The results achieve high performance compared with other methods in objective and visual quality while preserving more information from these images.