Introduction

Magnetic spin textures on the micro- and nano-scale are of continued importance for both a fundamental understanding of their physical behavior and for potential applications in novel computing paradigms1,2,29,30. SITIE, however, is only accurate in the small defocus limit and assumes that all contrast is magnetic in origin. This makes it very susceptible to noise and to the influence of non-magnetic contrast such as diffraction contrast. In addition, real-world samples can often contain non-magnetic regions, including focused ion beam damage, ice that accumulates during cryogenic experiments, or surface contaminants. These all add contrast that will be incorrectly reconstructed as magnetic domains. Despite these drawbacks, SITIE is frequently used for map** the integrated magnetic induction of thin films and 2D materials31,32,33. In almost all cases it is used qualitatively, as techniques for dealing with noise, including Tikhonov filtering or image pre-processing, can make the resulting reconstructions non-quantitative34.

In this work, we present the development and demonstration of Single Image Phase Reconstruction via Automatic Differentiation (SIPRAD): a method for reconstructing the magnetic phase shift, ϕm, from a single defocused LTEM image. SIPRAD is shown to be quantitatively accurate and robust to high levels of noise across a wide defocus range. It is also able to isolate ϕm in samples with amplitude and ϕe variations that are present because of artifacts or sample heterogeneity. These characteristics make SIPRAD ideally suited for performing phase reconstructions from in situ experiments.

Results

Single image phase retrieval

Figure 1 depicts a schematic of the SIPRAD algorithm that uses AD to reconstruct ϕm. In previous work, AD was applied directly to an image formation model in order to reconstruct the total phase shift ϕ from a TFS of images28. With only a single, often noisy, image, applying this same technique leads the algorithm either to over-fit to noise or diverge. To avoid this, a key feature of our algorithm is that it uses up to two deep-image-priors (DIPs) in place of other regularization techniques. By doing so we are able to perform accurate phase reconstructions from a single noisy image.

Fig. 1: Overview of the SIPRAD magnetic phase retrieval algorithm.
figure 1

A single defocused LTEM image and microscope parameters are given as input, and a deep image prior (DIP) is trained to output ϕm. If the amplitude reconstruction branch is activated, a second DIP will be trained to output an amplitude map which will also be scaled according to material parameters and used as the electrostatic phase shift ϕe.

Full size image

As described in Ulyanov et al., a DIP is a generative convolutional neural network (CNN) that is trained to create a desired image35. The DIP relies upon the CNN architecture to generate images with lower patch-wise entropy, and DIPs have been successfully applied for tasks including image de-noising, segmentation, and tomographic reconstructions35,36,37,38. We find that using a DIP both stabilizes the algorithm and provides superior noise robustness compared to other techniques such as total-variance regularization, with the additional benefit of eliminating user-optimized parameters. Therefore, a key distinction in our algorithm from previous AD phase retrieval approaches is that, rather than attempting to learn the phase shift directly, a DIP is trained to generate the reconstructed phase shift.

We will begin by considering a case where the amplitude and ϕe are both uniform across a sample, such as for a uniformly thick flake of vdW material or amorphous film. The branch of the algorithm that reconstructs variations in the electron wave amplitude, which will be discussed further, is therefore turned off. In all cases, the algorithm begins with an untrained DIP that is given an approximate phase input and which iteratively learns to output an improved ϕm. In many previous applications of DIPs, random noise is used as the input35,36,37,38. We have found that using an approximate phase shift, obtained with the SITIE method, as an input for the DIP, can allow the algorithm to converge more quickly. The output phase shift of the DIP, ϕm, is used as the input of an image-formation forward model with microscope parameters set by the user to match experimental values. The mean-squared error (MSE) loss is calculated between this resulting simulated LTEM image and the input experimental image. Gradients are calculated by back-propagating the loss through the algorithm and are then used to update the weights of the DIP, iteratively improving the accuracy of its output phase.

The image formation forward model begins by writing the electron exit wavefunction at the sample as \(\psi \left({{{{\bf{r}}}}}_{\perp }\right)=a\left({{{{\bf{r}}}}}_{\perp }\right){e}^{i\phi \left({{{{\bf{r}}}}}_{\perp }\right)}\), where \(a\left({{{{\bf{r}}}}}_{\perp }\right)\) is the amplitude function of the electron wave that depends on the sample shape function and composition, ϕ the total phase shift imparted by the sample, and r⊥ is a radial vector perpendicular to the electron beam direction, which is taken to be z. This wavefunction is then convolved with the microscope transfer function, which consists of an aperture function, phase transfer function, and dam** envelope. This linear image formation model has been shown to work well for LTEM, as the deflection angle due to the Lorentz force is very small compared to that of the diffracted beams28,29,33. Additional details regarding the forward model are given in the “Methods” section.

Our goal is to reconstruct ϕm and, for samples with uniform amplitude and uniform ϕe, a single DIP is sufficient because ϕ ~ ϕm except for a uniform offset. The SIPRAD method also provides a way to simultaneously reconstruct ϕm and separate it from the non-uniform amplitude function and ϕe that arise from sample heterogeneity and contamination. For the examples in this work, we assume that the origin of a non-uniform electron wave amplitude are particles on the sample surface which also contribute to an electrostatic phase shift. This is a situation that frequently arises when performing cryogenic LTEM. For these cases, the optional amplitude reconstruction branch can be enabled. A second DIP is then trained to output an amplitude map that corresponds to surface contaminants, which is both input directly into the forward model and also scaled to generate the ϕe induced by the contaminants. This method works because a non-uniform amplitude is required to accurately reproduce the very low image intensity that occurs at the location of surface contaminants, and from this amplitude map, we can also approximate ϕe, which is added to ϕm to give the total phase shift that is input to the forward model. The amplitude DIP thus learns to generate the image contrast from the contaminants while the phase DIP learns to output the phase shift corresponding to the magnetic contrast. We have chosen to focus on the case of surface contaminants that generate both a non-uniform amplitude and a contribution to ϕe, but we believe this method could be adapted and extended to account for sources that generate only a non-uniform ϕe, such as variations in local electrostatic potential in multiferroics, or only a non-uniform amplitude, for example bend contours that generate strong amplitude contrast but do not contribute to the Aharonov-Bohm phase shift.

Demonstration of SIPRAD with simulated data

We first compare the phase retrieval accuracy of SIPRAD and SITIE for stripe domains in a simulated, uniformly thick CGT sample. Figure 2a shows the in-plane magnetization component that has been simulated using the Mumax3 micromagnetics package39. The calculated electron phase shift (without the uniform electrostatic offset) is shown in Fig. 2b and the integrated in-plane magnetic induction map, B⊥, from the boxed region in b is shown in Fig. 2c. Figure 2e shows a simulated LTEM image for the phase shift shown in b, with Δz = − 1 mm, and the corresponding phase reconstructions and integrated induction maps are shown in Fig. 2f–i using the SIPRAD (f, g) and SITIE (h, i) techniques respectively. The accuracy of each reconstruction is quantified as the correlation between the reconstruction and the ground truth and is printed in brackets. For the integrated in-plane magnetic induction, the accuracy of the x and y components is calculated individually and then averaged.

Fig. 2: Demonstration of SIPRAD and SITIE reconstruction approaches for simulated data.
figure 2

a In-plane component of the simulated magnetization for stripe domains in CGT. b Ground-truth phase shift corresponding to a. c Ground truth in-plane integrated magnetic induction map corresponding to the red boxed region in b. d Comparison between line profiles of the y component of the integrated induction across the lines shown in c, g and i. Dashed lines show domain wall centers. e Simulated LTEM image for Δz = − 500 μm. f Phase image reconstructed from e using SIPRAD. g Integrated in-plane magnetic induction map reconstructed using SIPRAD (of red boxed region). h, i SITIE-reconstructed phase and in-plane integrated magnetic induction map using e as an input image. j LTEM image of same area as shown in e but with 50% Gaussian noise added. Phase reconstructions of the noised image carried out using (k) SIPRAD and (m) SITIE, with corresponding in-plane integrated magnetic induction maps ((l) and (n)). For all images, the number in bracket denotes accuracy compared to the ground truth.

Full size image

The SIPRAD approach reconstructs the phase with slightly higher accuracy than the SITIE approach, but both techniques perform well. The integrated induction map, however, is much more accurate for SIPRAD than for SITIE, and this is confirmed in Fig. 2d, which shows a line profile of the y component of the in-plane integrated magnetic induction across two domain walls. The SIPRAD profile matches well to the ground truth both in terms of domain wall location and width, although the true peak intensity is not reached. The SITIE reconstruction, by contrast, shows a domain wall that is both wider than the true wall and centered in the wrong location. The inaccurate placement of domain walls when using the SITIE method is due to an assumption that ignores the unequal intensities of converging and diverging domain walls, and is a known issue with the technique29.

We next demonstrate the application of the SIPRAD method to a more experimentally-relevant, noisy input image. Figure 2j shows the same LTEM image with the addition of 50% Gaussian noise, and the corresponding phase reconstructions and integrated magnetic induction maps are shown in Fig. 2k–n. The accuracies of both the phase and the integrated induction remain high for the SIPRAD method, showing only a slight decrease with the addition of noise. The phase accuracy for the SITIE method decreases sharply, and the accuracy of the in-plane integrated magnetic induction is low for both the noise-free and noisy images. The accuracy of the SITIE-reconstructed integrated induction does not significantly decrease because the additional noise manifests in longer-range variations in the phase shift. These are not reflected in the in-plane magnetic induction, which is obtained by taking a gradient of the phase shift.

For this simple example, SIPRAD outperforms SITIE with respect to the accuracy of both the phase shift and the integrated magnetic induction maps for all cases. The SIPRAD technique accurately determines the location and width of the domain walls, which can be important when trying to use domain information to calculate magnetic material parameters such as the micromagnetic exchange stiffness40,41,42.

Increased robustness against noise

To better compare the accuracy of the SIPRAD and SITIE techniques, we compute the accuracy of both the reconstructed phase and the in-plane integrated magnetic induction for a wide range of defocus and noise values. There are several noise sources that contribute to LTEM images, including Poisson, Gaussian, and salt & pepper noise. We have found that SIPRAD performs similarly well when given input with equal amounts of either pure Gaussian noise or a mix of noise types, while SITIE performs slightly better for pure Gaussian noise (see Supplementary Fig. 1). Gaussian noise was therefore used in the following accuracy measurements as it is easier to quantify. The level of Gaussian noise refers to the standard deviation of the noise distribution relative to the mean image intensity.

Figure 3 shows the accuracy of the SIPRAD and SITIE techniques applied to an input image simulated from the true phase in Fig. 2, for defocus values ranging from −0.1 mm to −8 mm, and noise values ranging from 0% to 300%. The SIPRAD phase reconstruction is very accurate across a large portion of the parameter space (yellow region), while at high defocus values the accuracy of the in-plane integrated magnetic induction reduces somewhat. This is caused by the extreme blurring that occurs in images with very high defocus. The SITIE reconstructions, by contrast, are only accurate within a narrow region of low defocus and low noise.

Fig. 3: Comparison of accuracy between SIPRAD and SITIE.
figure 3

a, b Phase accuracy for the SIPRAD and SITIE approaches, plotted on a color scale as a function of % noise and defocus of the input image. c, d Accuracy of the in-plane integrated magnetic induction displayed in the same way as for a and b.

Full size image

For both reconstruction methods, noise is more impactful at low defocus values. This is because the intensity of the magnetic contrast increases with increased defocus. At Δz = − 100 μm, even small amounts of noise can completely obscure the magnetic contrast in the image, which is why real-world LTEM imaging is normally performed at relatively high defocus values, outside the small defocus limit within which the SITIE approach is accurate. The SIPRAD technique, by contrast, shows an increase in accuracy up to, and beyond, a defocus of -1 mm, meaning that it is most accurate for real-world imaging conditions. Likewise, the SIPRAD technique maintains a high accuracy in both the reconstructed phase images and the in-plane integrated magnetic induction maps up to noise values approaching 100%. Both of these trends are also reflected when measuring accuracy using the structural similarity index measure (SSIM), as shown in Supplementary Fig. 2.

The SIPRAD method is accurate when reconstructing the phase shift from images taken at high defocus, and it can therefore be applied to samples that induce only a small magnetic phase shift. Due to the high noise tolerance, shorter acquisition times can be used which makes SIPRAD viable for imaging highly beam-sensitive samples and for imaging with increased time resolution.

Phase reconstruction with amplitude variation

The most important development enabled by our SIPRAD approach is not the increased accuracy or robustness to noise, but the ability to perform magnetic phase reconstructions on samples across which the amplitude and electrostatic phase shift of the exit wave are varying. All of the previously-introduced phase retrieval techniques, including the TIE approach and off-axis holography, reconstruct the total electron phase shift ϕ of the sample. If the electrostatic phase shift, ϕe, is uniform across the sample, then the gradients of ϕ and ϕm are equal and B⊥ can be obtained. However, if the mean inner potential or the sample thickness is not uniform, then the amplitude of the electron wave and ϕe are not constant and a single phase reconstruction cannot be used to calculate B⊥. There are methods for separating ϕe and ϕm, such as performing a second phase reconstruction with the sample inverted in the microscope43, but this is not often applicable during in situ experiments as this requires the sample to be fully removed from the microscope44.

The distinction between ϕ and ϕm is especially important when considering SITIE reconstructions. A major assumption for the SITIE technique is that all contrast in the defocused image arises from magnetic contributions in the sample. This requires samples that do not display diffraction or amplitude contrast in the images, and that are perfectly clean and of uniform thickness. These conditions are frequently not met. It is especially difficult to ensure these conditions when imaging single-crystal samples that generate strong diffraction contrast, or when performing cryogenic in situ experiments that can easily lead to sample surface contamination. Contamination that collects on the sample is particularly problematic, as it can create variations in both the amplitude and the electrostatic phase shift. While diffraction contrast can frequently be avoided by adjusting sample tilt and imaging conditions, contrast from contamination cannot be removed once the sample is in the microscope. Especially troubling is that surface contamination can lead to phase reconstructions and to integrated magnetic induction maps with features that appear similar to viable magnetic features, but which are actually non-magnetic in origin.

Performing the amplitude reconstruction poses an additional computational problem, namely that an amplitude map can be created that generates contrast that matches magnetic contrast. The reconstruction algorithm is therefore less stable when performing amplitude and ϕe reconstructions and will often diverge or generate non-physical amplitude maps. We remedy this by adding the additional constraint that the amplitude should be dual-valued, i.e. uniform except in finite regions that correspond to the surface contamination (which is visible in the bright-field, in-focus image), and this sufficiently constrains the model such that reconstructions can be performed in many cases. The general workflow of the algorithm does not change when performing amplitude reconstructions, but the additional branch shown in Fig. 1 is included. The input into the amplitude DIP is a thresholded version of the input image. This is created with a manually-chosen threshold value that approximately isolates the contaminants. The output of the amplitude DIP is then used directly as the amplitude in the forward model. This output amplitude is scaled according to user-input material parameters to convert it to ϕe (for cases where both amplitude and ϕe are locally modified), which is added to the value of ϕm from the phase DIP to give the total phase shift input to the forward model. When backpropagating, both DIPs are optimized with every iteration. Other training methods, including alternating between optimizing the phase and amplitude DIPs, made the algorithm less accurate and stable.

Figure 4 shows an application of amplitude learning to a simulated sample of CGT containing bubble domains and with added surface contamination. Figure 4a and b show the true magnetic phase shift, ϕm, and the corresponding in-plane integrated magnetic induction map. Two regions of surface contamination are then added that contribute to a non-uniform electrostatic phase shift, ϕe, and the corresponding amplitude map is shown in Fig. 4c. An LTEM image, simulated with Δz = − 500 μm and 30% added noise, is shown in Fig. 4d; the surface contamination is clearly visible. Figure 4e–g show the SIPRAD-reconstructed map of ϕm, the in-plane integrated magnetic induction map, and the amplitude map, respectively. The amplitude matches the ground truth well, and the phase map is accurate except in regions obscured by the surface particles. There are some artifacts in the magnetic induction map near the edge of the surface particles, but there is otherwise a very good match to the ground truth.

Fig. 4: Application of phase reconstruction to samples with surface contamination.
figure 4

a True magnetic phase shift and b integrated magnetic induction map showing simulated bubble domains in CGT. c True amplitude map including two pieces of surface contamination that impart both amplitude variation and electrostatic phase shift. d Resulting LTEM image with 30% noise added. e SIPRAD-reconstructed magnetic phase shift and f corresponding integrated magnetic induction map. g SIPRAD-reconstructed amplitude map. SITIE-reconstructed maps of h the total phase shift, and i the in-plane magnetic induction for the same input image. j, k SITIE-reconstructed maps of the total phase shift and of the magnetic induction with a manually-optimized Tikhonov filter applied.

Full size image

By contrast, the SITIE approach does not produce an accurate reconstruction of either ϕm or B⊥ from the LTEM image shown in d. Figure 4h and i show the SITIE-reconstructed phase shift (h) and integrated magnetic induction map (i) from the same input image (d). The scale of the phase shift is very inaccurate due to the additional contrast from the amplitude and ϕe introduced by the surface particles. The particles also dominate the integrated magnetic induction maps, and the actual magnetic domains are not visible. One can improve the quality of the SITIE reconstruction by applying a Tikhonov filter, as shown in Fig. 4j for a manually-optimized Tikhonov frequency of qc = 0.003 nm−1. The Tikhonov filter makes the magnetic components of the phase shift more visible, but the image is still both quantitatively and qualitatively inaccurate. The magnetic information in the in-plane integrated induction map shown in Fig. 4k is also improved, but the false contrast from the surface particles still dominates, and in this case could be easily mistaken for a magnetic bubble with opposite chirality to that of the other magnetic bubbles.

Application to experimental data

We now apply the SIPRAD reconstruction technique to an experimental LTEM image of an exfoliated CGT flake, shown in Fig. 5a. The sample was field cooled through the Curie temperature to 23 K in a 500 Oe field to stabilize magnetic bubble domains, and was then imaged using a defocus of Δz = − 700 μm. The experimental image shows clear bubble domains along with additional contrast due to surface contamination. The image is very noisy, and the SIPRAD algorithm will often diverge when trying simultaneously to reconstruct both the amplitude and phase for this image. The contrast from the contaminants can still be taken into account, however, by using a fixed, non-uniform amplitude map and corresponding electrostatic phase shift, ϕe. We first approximate the amplitude map by manually thresholding the input image to create the binary image shown in Fig. 5b, which is used to obtain the amplitude function and scaled to create ϕe. These components of the wave function contribute to the forward model but remain fixed during the reconstruction process that optimizes the magnetic phase shift, ϕm. The training process is thus stable while still taking into account local contributions to the amplitude and to ϕe, which allows the reconstruction of ϕm isolated from the total phase shift, ϕ. Figure 5c and d show the SIPRAD-reconstructed maps of ϕm and of the in-plane integrated magnetic induction, respectively. The reconstructed map of ϕm matches well to the simulated phase map of bubble domains shown in Fig. 4a. The reconstruction shows inaccuracies around the areas of surface contamination, where the magnetic information is lost. Errors are also observed near the edge of the image due to periodic boundary conditions enforced by the forward model. Performing the same reconstruction with a uniform amplitude map leads to a significantly worse reconstruction with visible artifacts from the surface particles (Supplementary Fig. 3).

Fig. 5: Application of SIPRAD to experimental data.
figure 5

a Experimental LTEM image of bubble domains in CGT recorded with Δz = − 700 μm and at 23 K. b Amplitude map obtained from thresholding the input image to separate out the surface particles visible in a. SIPRAD-reconstructed maps of c the magnetic phase shift, and d the in-plane integrated magnetic induction. e, f SITIE-reconstructed maps of the total phase shift and induction map from a. g, h TIE-reconstructed maps of the total phase shift and the integrated magnetic induction using a TFS with Δz = ± 700 μm.

Full size image

Figure 5e and f show maps of the SITIE-reconstructed total phase shift and of the in-plane integrated magnetic induction from the same experimental image (shown in a) with an optimized value of the Tikhonov filter frequency, qc. The phase map is dominated by the contribution from the surface particles, which lead to artifacts in the induction map, such as varying intensities for different bubbles and an apparent bubble with opposite chirality to the others. We also performed a TIE reconstruction of the same region using a TFS of three images taken at defocus values of Δz = ± 700 μm and Δz = 0 μm. Note that the underfocus image from the TFS is what was used for the SIPRAD and SITIE reconstructions. The full TFS is shown in Supplementary Fig. 4 and surface contaminants are clearly visible in the in-focus image. An optimal value of qc was determined manually as for the SITIE reconstruction. The TIE-reconstructed phase shift map is significantly improved compared to the map obtained for the SITIE reconstruction, primarily because the amplitude contrast from the surface particles is present in all images in the TFS and is thus accounted for, but the contribution from ϕe is still evident.

Even though the SIPRAD approach uses only one image, rather than a TFS of three images, the SIPRAD-reconstructed maps of the phase shift and integrated magnetic induction both appear qualitatively to better represent the true domain structure than those produced by the TIE reconstruction. This is further supported by comparing the domain wall profiles in the reconstructed magnetic induction maps to the true profile of a simulated bubble, which is shown in Supplementary Fig. 5. The SIPRAD reconstruction with a fixed, dual-valued amplitude map, therefore, appears to enable the magnetic phase shift to be quantitatively and qualitatively determined from the experimental data shown in Fig. 5.

We next show the application of SIPRAD to experimental in situ LTEM cooling data. This is an example of a situation in which acquiring a TFS of images is not feasible. Figure 6a–e show a series of LTEM images of a CGT flake recorded during field-cooling of the sample from above the Curie temperature to 24 K. As the temperature is reduced, the sample expands due to magnetostrictive effects; this leads to the formation of bend contours that can obscure the magnetic contrast, as can be seen in Fig. 6e45.

Fig. 6: Application of SIPRAD to an in situ field-cooling experiment.
figure 6

a–e Experimental LTEM images of CGT during field cooling in a 500 Oe field, recorded with Δz = − 700 μm. Temperatures are shown above each image. f–j Magnetic phase reconstructions performed with the SIPRAD method of the images shown in a–e. Red boxes highlight an area where a bubble has disappeared between 39 K and 35 K. k–o Integrated magnetic induction maps generated from the magnetic phase reconstructions shown in f–j.

Full size image

Figure 6f–j shows the SIPRAD-reconstructed phase shift maps. The corresponding integrated induction maps are shown in Fig. 6k–o. The phase and induction maps allow easy visualization of how the magnetic bubbles rearrange when cooling, and show that some bubbles are driven out of the sample, as highlighted by the red boxes in Fig. 6g and h. Between each temperature step, the magnetic bubble lattice shifts and distorts due to both the disappearance of bubbles and Brownian motion46,47. The SIPRAD method enables quantitative analysis of the bubble-lattice time evolution, because phase reconstruction allows accurate tracking of bubbles between sequential images and gives further insight into local changes of the magnetic domain structure.

Discussion

In this work, we have demonstrated that we can reconstruct the magnetic phase shift, and thus the integrated magnetic induction, from a single input LTEM image, using a DIP-enabled automatic differentiation method. We applied the SIPRAD approach to reconstructing images of Bloch type magnetic bubbles and stripe domains in CGT, though the technique would work equally well applied to images of other spin textures. The SIPRAD algorithm is shown to be significantly more accurate than existing techniques over a wide range of input conditions, including moderate to large defocus values and noisy inputs that best represent experimental data. The use of a large defocus is often required when performing experimental LTEM, as the increased phase sensitivity at large defocus allows magnetic contrast to be more easily distinguished from noise and other contrast contributions. Other phase reconstruction methods, such as those based on solving the TIE, are only accurate in the small defocus limit and lose both spatial and phase resolution when experimentally-relevant defocus values are used. The SIPRAD method, by contrast, is based on a forward model that retains high accuracy of the reconstructed phase to defocus values as large as−6 mm, and accuracy of the in-plane integrated magnetic induction to defocus values as large as −3 mm. The robustness of the approach against high noise levels is primarily due to the implementation of a DIP. DIPs have been shown to be an effective way of regularizing noisy input data without relying on manually-chosen parameters, and this work further reinforces these findings. The downside of using a DIP in AD-based phase reconstruction is that computationally it is much slower than inference by a conventional machine-learning model, though with the benefit of not needing to train such a model in the first place. Reconstructing each image requires optimizing a new CNN, but we find that the ~ 5000 iterations required to converge the model for a 512 x 512 image takes only 35 seconds on an NVIDIA A100 GPU. Although this is not fast enough for real-time image processing, it is more than sufficient for integration into other data processing pipelines. We are able to achieve this reconstruction time by inputting an estimated phase shift calculated with the SITIE method, which reduces the total number of iterations required. By demonstrating the high accuracy at both large defocus and low signal-to-noise levels, we have shown that SIPRAD is an excellent technique for reconstructing LTEM images in these experimental conditions, such as would occur when imaging weakly magnetic or beam-sensitive samples.

Although the increased accuracy and robustness towards noise is promising, the most important aspect of the SIPRAD approach is that it allows ϕm to be reconstructed even in the presence of image contrast arising from variations in amplitude and ϕe. Real-world samples inevitably contain heterogeneities that can make magnetic phase reconstruction challenging. This is especially true when imaging spin textures across a large field of view. We have demonstrated that the SIPRAD approach enables simultaneous reconstruction of both the amplitude and ϕm, but we find that to optimize both components from a single image can be difficult and will often lead the algorithm to diverge at high noise levels. Even in these cases, however, the magnetic phase can still be effectively isolated and reconstructed after creating a thresholded binary amplitude map (or otherwise determining the amplitude distribution), as demonstrated in Fig. 5. This allows for reconstructing phase maps and in-plane integrated magnetic induction maps from images for which the TIE method and a single TFS do not work. This also provides an example of the adaptability of the SIPRAD method. Known information about the sample, such as topographical data for patterned nanostructures, could be additionally included into the forward model and used to constrain the reconstruction and improve the isolation of ϕm.

The fact that SIPRAD can accommodate large defocus values, high noise levels, and heterogeneous samples or those with surface particulates demonstrates that it is most useful when given noisy images recorded in conditions that do not allow for the sample to be inverted or even a TFS to be recorded. This makes the technique especially appealing for application to in situ experiments. When studying the time evolution of magnetic domains it is frequently impossible to collect a TFS at each point of interest–normally a movie of a series of defocused images will be the only data available. SIPRAD is a viable technique able to perform both quantitatively- and qualitatively-accurate reconstructions of ϕm and of the integrated magnetic induction from single movie frames. In addition, in these scenarios it is often paramount to study the evolution of one particular region over time, and SIPRAD is able to isolate ϕm even for changes in the imaging conditions, or additional surface features, that develop during the experiment. We, therefore, hope that SIPRAD can be broadly applied to future LTEM in situ experiments and studies of difficult-to-observe magnetic spin textures.

Methods

Single image phase reconstruction via automatic differentiation

SIPRAD is implemented in PyTorch using the Adam optimizer. The DIPs are convolutional autoencoders as depicted in Supplementary Fig. 6. The input for the phase DIP is a SITIE phase reconstruction of the input image, and the input for the amplitude DIP is a thresholded version of the input image with the threshold value manually set to isolate features such as surface contamination. Similar to other implementations, we augment our input image by adding small amounts of noise in each iteration35. Each DIP is pre-trained for 200 iterations to produce its input image before the full algorithm is run with the forward model. The final reconstructed ϕm is smoothed by a Gaussian filter with σ = 2 pixels.

When quantifying the accuracy for Fig. 3, 10 randomly-noised images (with the same noise level) were created for each defocus value and used for the phase reconstructions. The average accuracy of the 10 reconstructions of the phase maps and the integrated magnetic induction maps is displayed for both SIPRAD and SITIE.

When training with amplitude reconstruction, the dual-valued constraint is implemented every 100 iterations of the reconstruction as well as after the final iteration. The high and low values are determined from the mode and minimum of all pixels, and the threshold value is chosen as 3/4 of the mode. Early stop** is sometimes necessary, especially when performing amplitude reconstruction, to prevent over-fitting. The amplitude map that we are reconstructing is based on the thickness and mean inner potential of the material, as described in the forward model section.

Experimental dataset

Experimental cryo-LTEM imaging of CGT was performed on a JEOL JEM-2100F TEM instrument using a Gatan double-tilt liquid helium holder. TEM samples were created by dry-exfoliating flakes from a bulk crystal, which were then placed on a silicon nitride TEM membrane. The thickness of the flake was measured with atomic force microscopy to be 150 nm. The sample was field-cooled in the microscope in a 500 Oe out-of-plane magnetic field to 23 K in order to nucleate and stabilize magnetic bubbles.

Phase retrieval using the transport of intensity equation

The transport of intensity equation can be written as

$$\nabla \cdot \left({I}_{{{{\rm{0}}}}}\nabla \phi \right)=-\frac{2\pi }{\lambda }\frac{\partial I}{\partial z},$$
(1)

where I0 is the in-focus image, λ the electron wavelength, ∂I/∂z the through-focal image intensity derivative, with z being the electron propagation direction. Under the single-image TIE approximation, the equation becomes

$${\nabla }^{2}\phi =\frac{2\pi }{\lambda \Delta z}\left(1-\frac{{I}_{\Delta z}}{{I}_{{{{\rm{0}}}}}}\right),$$
(2)

where Δz is the defocus and the in-focus image I0 is approximated as a uniform image with the mean intensity of the defocused image IΔz29.

The TIE and SITIE methods were implemented for both the simulated and experimental data using the open-source PyLorentz software33,48.

LTEM image formation model

Here we show how the LTEM images are calculated. We begin with the Aharonov-Bohm equation19, which describes how electrons that pass through the sample are subject to a phase shift induced by the electrostatic potential (V) and magnetic vector potential (A),

$$\begin{array}{l}\phi\,=\,{\phi }_{{{{\rm{e}}}}}+{\phi }_{{{{\rm{m}}}}},\\\quad=\,\frac{\pi }{\lambda E}\int\,Vd{{{\bf{l}}}}-\frac{\pi }{{\varphi }_{{{{\rm{0}}}}}}\int\,{{{\bf{A}}}}\cdot d{{{\bf{l}}}},\end{array}$$
(3)

where ϕ is the total phase shift, ϕe and ϕm the electrostatic and magnetic components of the phase shift respectively, λ is the electron wavelength, E the relativistic electron energy, l the electron propagation direction, and φ0 = h/2e the magnetic flux quantum. The in-plane component of the integrated magnetic induction can be obtained from ϕm,

$$\left({B}_{x},{B}_{y}\right)=\frac{{\varphi }_{{{{\rm{0}}}}}}{\pi }\left(-\frac{\partial {\phi }_{{{{\rm{m}}}}}}{\partial y},\frac{\partial {\phi }_{{{{\rm{m}}}}}}{\partial x}\right).$$
(4)

The total phase shift is used to write the electron wave function at the exit surface of the sample as

$$\psi ({{{{\bf{r}}}}}_{\perp })=a({{{{\bf{r}}}}}_{\perp }){{{{\rm{e}}}}}^{{{{\rm{i}}}}\phi ({{{{\bf{r}}}}}_{\perp })},$$
(5)

where a(r⊥) is an amplitude function and ϕ is the total electron phase shift. We do not include detailed electron scattering when calculating the amplitude function, but only the absorption of the electrons, which is given by

$$a({{{{\bf{r}}}}}_{\perp })={{{{\rm{e}}}}}^{-{t}_{\perp }/{\xi }_{0}},$$
(6)

where t⊥ is the sample thickness shape function, and ξ0 is the absorption coefficient for the sample material.

In order to calculate the resulting image intensity, we propagate this wave function to the exit plane by convolving it with the transfer function of the microscope in the back focal plane and determining the intensity of the wave function,

$$I({{{{\bf{r}}}}}_{\perp })={\left\vert \psi ({{{{\bf{r}}}}}_{\perp })* {{{\mathcal{T}}}}({{{{\bf{r}}}}}_{\perp })\right\vert }^{2},$$
(7)

where \(*\) is a convolution operation, and \({{{\mathcal{T}}}}({{{{\bf{r}}}}}_{\perp })\) is the microscope transfer function. This operation is written here in real space but is computed in Fourier space. The LTEM transfer function is composed of three parts49:

$${{{\mathcal{T}}}}({{{{\bf{k}}}}}_{\perp })=A({{{{\bf{k}}}}}_{\perp }){{{{\rm{e}}}}}^{-{{{\rm{i}}}}\chi ({{{{\bf{k}}}}}_{\perp })}{{{{\rm{e}}}}}^{-g({{{{\bf{k}}}}}_{\perp })},$$
(8)

where A(k⊥) is the objective aperture function, \({{{{\rm{e}}}}}^{-{{{\rm{i}}}}\chi ({{{{\bf{k}}}}}_{\perp })}\) the phase transfer function, \({{{{\rm{e}}}}}^{-g({{{{\bf{k}}}}}_{\perp })}\) the dam** envelope, and k⊥ the reciprocal space wave vector perpendicular to the beam direction. The aperture is a binary function (1 inside and 0 outside) centered in reciprocal space for the Fresnel imaging mode. We can define the phase transfer function as

$$\chi ({{{{\bf{k}}}}}_{\perp })=\pi \lambda \left[\Delta z+{C}_{{{{\rm{a}}}}}\cos \left(2{\phi }_{a}\right)\right]{\left\vert k\right\vert }^{2}+\frac{\pi }{2}{C}_{{{{\rm{s}}}}}{\lambda }^{3}{\left\vert k\right\vert }^{4},$$
(9)

where Δz is the defocus, Ca and ϕa are the magnitude and orientation of the two-fold astigmatism, and Cs is the spherical aberration coefficient.

The dam** envelope can be written as

$$g({{{{\bf{k}}}}}_{\perp })=\frac{{\pi }^{2}{\theta }_{{{{\rm{c}}}}}^{2}}{{\lambda }^{2}\,u}{\left({C}_{{{{\rm{s}}}}}{\lambda }^{3}{\left\vert k\right\vert }^{3}-\Delta z\lambda \left\vert k\right\vert \right)}^{2}+\frac{{\left(\pi \lambda \Delta \right)}^{2}}{2u}{\left\vert k\right\vert }^{4},$$
(10)

where \(u=1+2{\left(\pi {\theta }_{{{{\rm{c}}}}}\Delta \right)}^{2}{\left\vert k\right\vert }^{2}\), θc is the beam divergence angle, and Δ is the defocus spread50. LTEM image simulations are performed using the PyLorentz software33,48.

Simulated datasets

Micromagnetic simulations of CGT were performed with Mumax339. The following parameters were used: cell size \(\left(4\,{{{\rm{nm}}}}\times 4\,{{{\rm{nm}}}}\times 10\,{{{\rm{nm}}}}\right)\), grid size \(\left(512\times 512\times 10\right)\), Ms = 2 × 105 Am−1, Aex = 1.2 × 10−12 Jm−1, Ku = 2 × 104 Jm−3, Bext = 0.08 T along the z direction. For simulations of magnetic bubbles, a random starting magnetization of Bloch type domains and bubbles was placed and then relaxed. The PyLorentz software was used to calculating the total electron phase shift of the simulated samples33,48. Gaussian noise was added to the images after they are created. The level of Gaussian noise refers to the standard deviation of the distribution divided by the mean intensity of the image.