Introduction

Self-assembled quantum dots (QDs) have attracted great interest due to their applications in various optoelectronic devices1. The so-called Stranski-Krastanow (SK) mode in molecular beam epitaxy (MBE) is widely used for growing these high-quality QDs2. For specific applications, such as QD lasers, high QD densities are required; while low-density QDs are necessary for other applications, such as single photon sources1. However, the outcomes of any QD growth process are a complex function of a large number of variables including the substrate temperature, III/V ratio, and growth rate, etc. Building a comprehensive analytical model that describes complex physical processes occurring during growth is an intractable problem3. The optimization of material growth largely depends on the skills and experience of MBE researchers. Time-consuming trial-and-error testing is inevitably required to establish optimal process parameters for the intended material specification. It is needed to build a real-time connection between in situ characterization and material growth status.

Machine learning (ML) is revolutionary due to its exceptional capability for pattern recognition and its potential in approximating the empirical functions of complex systems. It enables researchers to extract valuable insights and identify hidden patterns from large datasets, leading to a better understanding of complex phenomena required to build predictive models, generate new hypotheses, and determine optimal growth conditions for MBE4,5,6,7,8. It offers an alternative approach where the growth outcomes for an arbitrary set of parameters can be predicted via a trained neural network, which has been applied to extract film thickness and growth rate information9,10. Moreover, by enabling the direct adjustment of parameters during material growth, ML-based in situ control can detect and correct any deviation from expected values in a timely manner11,12,13. Meng et al. have utilized a feedback control to adjust cell temperatures after an amount of film being deposited, thereby capable of regulating AlxGa1-xAs compositions in situ13. However, these remain posteriori ML-based approaches, since they require the completion of the growth10. Therefore, once the sample characterization results deviate from expectation, it becomes challenging and time-consuming to identify reasons11.

To fully exploit the effectiveness of real-time connection based on ML, an in situ characterization with continuous and transient working mode is highly desirable to unveil the material status. Reflection high-energy electron diffraction (RHEED) has been widely used to capture a wealth of growth information in situ14. However, it still faces the challenges of extracting information from noisy and overlap** images. Traditionally, identifying RHEED patterns during material growth was mainly depended on the experience of growers. Progress has been made in the automatic classification of RHEED patterns through ML that incorporates principal component analysis (PCA) and clustering algorithms9,14,15,16,17. It surpasses human analysis when a single static RHEED image is gathered with the substrate held at a fixed angle16,18. However, this may result in an incomplete utilization of the temporal information available in RHEED videos taken with the substrate rotating continuously16,19,20. Although few researchers have highlighted the importance of analyzing RHEED videos, their ML models were still based on PCA and encoder-decoder architecture with a single image input16,21. To date, it is still hard to build a real-time connection between in situ characterization and material growth status, which is a prospective way to realize the precise control on material specifications.

In this work, we proposed a real-time feedback control based on ML to connect in situ RHEED videos and QD density, automatically and intelligently achieving a low-cost, efficient, reliable, and reproduceable growth of QDs with required density. By investigating the temporal evolution of RHEED information during the growth of InAs QDs on GaAs substrates, the real-time status of the sample was better reflected. We applied two structurally identical 3D ResNet 50 models in ML deployment, which takes three-dimensional variables as inputs to determine the QD formation and classify density22,23. We showed that ML can simulate and predict the post-growth material specifications, and based on which, growth parameters are continuously monitored and adjusted. As a result, the QD densities were successfully tuned from 1.5 × 1010 cm−2 down to 3.8 × 108 cm−2, and up to 1.4 × 1011 cm−2, respectively. The methodology can also be adapted to other QD systems, such as the droplet QDs, the local droplet etching, and manhole infilling QDs with distinct RHEED features24,25,26,27. The effectiveness of our approach by taking full advantages of in situ characterization marks a significant achievement of establishing a precise growth control scheme, with the capabilities and potentials of being extended to large-scale material growth, reducing the impact on material growth due to instability and uncertainty of MBE operation, shortening the parameter optimization cycle, and improving the final yield of material growth.

Results

Sample structure and data labeling

Our method relied on an existing database of growth parameters with corresponding QD characteristics, which helped us decide the growth parameters needed to achieve a target density for QDs. The sample structure is shown in Fig. 1a, with density results of 30 QD samples summarized in Table 1, which are correlated with the RHEED videos. Each QD growth was repeated on average 4 times, leading to the generation of 120 RHEED videos. The samples grown without QDs have flat surfaces determined by an atomic force microscope (AFM) as shown in Fig. 1b, corresponding to streaky RHEED patterns in Fig. 1f, which was artificially labeled with “zero” referring to the density of QDs on the surface27. With the increase of QD density, it was found that the RHEED gradually converted from streaky to spotty patterns. The QDs density of ~2.9 × 109 cm−2 from Fig. 1c corresponds to the RHEED pattern consisted of streaks and spots in Fig. 1g27. As shown in Fig. 1d, h, with the further increase of QDs density reaching 1.9 × 1010 cm−2, the arrangement of RHEED pattern showed typical spot features28,29,30. Upon conducting a more thorough analysis of the RHEED patterns, we identified the patterns transition from having both streaks and spots to only spots, corresponding to a density of 1 × 1010 cm−2 in AFM images. So, we assigned the “low” label to densities below 1 × 1010 cm−2, the “middle” label to densities ranging from 1 × 1010 cm−2 to 4 × 1010 cm−2. As the density exceeds 4 × 1010 cm−2, the spot features become more rounded and show higher brightness, we labeled densities exceeding 4 × 1010 cm−2 as “high” (see Supplementary Information for RHEED characteristics of QDs with different labels, S1)31,32,33. Fig. 1e, i show the typical AFM and RHEED images after the QD formation, respectively, with the QD density of ~1.0 × 1011 cm−2. It is worth noting that we keep the labels to the RHEED videos the same before and after the QD formation. The RHEED information before the QD formation can be used to determine the resulting QD density grown under the same conditions, enabling us to adjust the material growth parameters before the QD formation.

Fig. 1: Quantum dot (QD) sample structure and data labeling.
figure 1

a The schematic of the sample structure. The thick GaAs buffer layer is grown on an n-type GaAs substrate at 600 °C. Subsequently, the substrate is cooled to 450–500 °C for the growth of InAs QDs. b–e The 2 μm × 2 μm atomic force microscope (AFM) images of QDs with varied labels. “zero”: there are no QDs on the surface. “low”: the InAs QDs sample with density less than 1 × 1010 cm−2. “middle”: the InAs QDs sample with density range from 1 × 1010 cm−2 to 4 × 1010 cm−2. “high”: the InAs QDs sample with density higher than 4 × 1010 cm−2. f–i Corresponding reflection high-energy electron diffraction (RHEED) images with varied labels.

Full size image
Table 1 QD densities and labels assigned for the samples in the dataset, the density of QDs was determined by counting the number of QDs in the AFM image
Full size table

Program framework and video processing

The design framework of our program is illustrated in Fig. 2a. An appropriate scheme was designed to pre-process the RHEED video data. Then, a ML model was selected based on the pre-processing results. Furthermore, we also determined the way to adjust parameters according to output results of the model.

Fig. 2: Control program framework and video processing principle.
figure 2

a The framework of the program. An electron beam generated by an electron gun is continuously applied to the rotating substrate, creating a diffraction image frame by frame on the fluorescent screen. Multiple images are then converted to a new matrix, and the reshaped matrix is then transferred to the model for generation of results to guide the adjustment of material growth parameters. b A typical reflection high-energy electron diffraction (RHEED) image taken from the camera and the crop** area. The area marked by the blue square is an effective selection area that the software must handle and subsequently provide as input to the model. c Continuous sampling method for RHEED images. The software processes images with data at position T during each iteration. T is comprised of several sub-images taken sequentially from time t, including t-1, t-2, and so forth. Similarly, the data positions processed by the software in the previous iteration are T-1, T-2, and so on. d Processing method for sampled images. The data at a specific position T is initially acquired. Subsequently, each data point within T, denoted as t, t-1, t-2, and so forth, is transformed into a 2D matrix containing only black and white channels. Finally, these individual 2D matrices are stitched into a 3D matrix.

Full size image

RHEED videos taken during growth were first deconstructed into multiple temporal images and utilized as input for our model, which is a technique for breaking down video processing, offering a more versatile and efficient approach to analyze and process information in dynamic scenes34. The original image collected has uncompressed 4 channels of 8-bit depth color with a resolution of 1920 × 1200. The software cropped a square area from the image, as shown in Fig. 2b, immediately compressed to 300 × 300 by zero-order sampling and converted it to a single 4 × 300 × 300 matrix (see Supplementary Information for the crop** area of RHEED images, S2). To efficiently utilize the temporal information within RHEED data, we used the latest RHEED image as a starting point, acquired an additional several consecutive images before, and bundled as the original RHEED dataset for the model, as shown in Fig. 2c35. Subsequently, each image was converted to a single-channel 8-bit grayscale image based on the luminance information16,21, and stitched into an N × 300 × 300 3D matrix through shift registers, as shown in Fig. 2d. Additionally, we modified the convolutional filtering order. Traditional Convolutional Neural Networks (CNNs) tend to uniformly sample color channels, which is effective for images with rich color information, but not suitable for our data36,37. To address this issue, we designed an image processing approach that uses longitudinal size information instead of color channel information, which can improve the correlation between channels, reduce unnecessary calculations, and improve data processing efficiency38,39,40. Therefore, the size of the preprocessed data was N × 300 × 1 × 300, representing the image number, image width, image color channel, and image length. We then compared the impact of choosing different numbers of images on model training accuracy and speed, ultimately determined that selecting 8 images was the appropriate choice (see Supplementary Information for the number of images selected and speed management, S3 and S4). Finally, we conducted preprocessing for all data before model training to reduce the time. We have also applied data augmentation techniques to the training data, including adjusting image curves, crop** images, and scaling images, which effectively increase data diversity and improve the model’s generalization ability. This process resulted in the acquisition of ~360,000 NumPy arrays with each sample ~704 KB in size.

Model construction and evaluation

The ResNet 50 model has emerged as our preferred choice for several reasons, including its capability to address gradient vanishing issues and automatically learn methods for extracting features from raw data without manual feature engineering or dimensionality reduction, rapid convergence, and suitability for deep training. When processing video frame data, 3D convolution proves more effective in extracting spatial and temporal dimension information from videos than the 2D convolution used in the standard ResNet model. The difference between our network and the original ResNet lies in the dimensionality of the convolutional kernel and batch normalization operations23. Our 3D ResNet 50 model employs 3D convolutions and 3D batch normalization. This spatiotemporal-feature-learning method enables the 3D ResNet 50 model to better capture the dynamic information in videos compared to 2D CNN, resulting in an enhanced ability to identify and differentiate the categories of videos53. A C-type thermocouple measured the substrate temperature, and the growth rates were calibrated through the RHEED oscillations of extra layers grown on GaAs substrates. The substrate was outgassed at 350 °C in the buffer chamber, then heated to 620 °C in the growth chamber for deoxidation. The substrate was then cooled to 600 °C for the growth of the GaAs buffer layer. The sample structure is shown in Fig. 1a. The BEP of In was 1.4–1.5 × 10-8 Torr, while the BEP of As was 0.95–1.0 × 10-7 Torr. The QDs were formed in the SK growth mode after depositing 1.7 ML of InAs at a temperature of 450–500 °C and a rate of 0.01 ML/s. The initial growth showed a relatively flat planar structure, pertaining to the wetting layer growth54,55,56,57,58,59. The formation of InAs QDs was verified by the streaky-spotty RHEED pattern transition60.

Material characterization

RHEED in the MBE growth chamber enabled us to analyze and monitor the surface of the epilayer during the growth. RHEED patterns were recorded at an electron energy of 12 kV (RHEED 12, from STAIB Instruments). A darkroom equipped with a camera was placed outside the chamber to continuously collect RHEED videos with the substrate rotating at 20 revolutions per minute. The exposure time was 100 ms, with a sampling rate of 8 frames per second. To achieve a clear correlation of RHEED with QD density, we also characterized the surface morphology of the InAs QDs using AFM (Dimension Icon, from Burker).

Hardware wiring scheme

Our model is deployed on a Windows 10 computer system with an AMD R9 7950X CPU, 64GB of memory, an NVIDIA 3090 graphics card, and a 2TB solid-state drive. The program primarily obtains information from the temperature controllers, shutter controllers, and the camera (Fig. S11). We use a USB 3.0 interface to connect the camera to the computer. Temperature controllers are connected in series using the Modbus protocol to simplify the wiring. To reduce the delay in substrate temperature control, our program solely looks up information by address from the temperature controllers connected to the substrate.

Model construction environment

The model development occurred in the Jupyter Notebook environment on the Ubuntu system, utilizing Python version 3.9. We installed PyTorch based on CUDA 11.8 in this environment to leverage graphics card operations. The original video data was rapidly converted into the NumPy arrays format using code with random data augmentation within the Ubuntu system. Subsequently, the processed data was stored on our computer’s hard disk.

Program interface and deployment environment

The program was developed using LabVIEW, the built-in NI VISA, NI VISION, and Python libraries for data acquisition and processing (see Supplementary Information for program interface and deployment environment, S12). The program also employs ONNX for model deployment and TensorRT for inference acceleration and allows users to set targets for the desired QD density. Once the substrate and In cell temperatures stabilize, the program can be initiated. Firstly, the program checks the shutter status and controls the substrate and cell temperatures. Once the In shutter is open, the model starts to analyze RHEED data in real-time. The model outputs are displayed as numerical values at the top of the interface and are converted to corresponding label characters on the right side. The growth status can be displayed in the “Reminder Information”. Before and after the QDs formation, the “Reminder Information” will show up as “Stage 1” and “Stage 2”, respectively. It will display “Finished” when the model results meet the targets several times after the QD formation.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.