Introduction

Recent experimental1,2,3,4,5 and theoretical6,7,8 works have demonstrated that topological surface states (TSS) support massless spin-polarised Dirac fermions due to a strong spin-orbit coupling effect. They are robust and immune to backscattering by non-magnetic defects because of protection by time-reversal symmetry9. Unlike the Dirac fermions in graphene with pseudospin texture, the real helical spin polarization TSS exhibit has been widely studied by surface-sensitive experiments such as angle-resolved photoemission spectroscopy (ARPES), scanning tunneling microscopy (STM) and scanning tunneling spectroscopy (STS) as summarized in recent review articles10,11. TSS are attractive not only in fundamental condensed matter physics for realising novel entities such as dyons12, imaging magnetic monopoles13 and Majorana fermions12,14, but also in practical applications in spintronics and error-tolerant quantum computing. Motivated by such perspective, special attention has been focused on thin films hosting TSS suitable for device applications. Therefore, three-dimensional (3D) topological insulator (TI) thin films approaching 2D limit have been explored. Theoretical calculations predict that an energy gap can be opened at the Dirac point by inter-surface coupling15. Some of these films may exhibit quantum spin Hall effect as that observed in HgTe quantum wells16,17. Experimental studies on Bi2Se318,19 and Bi2Te320 confirm the existence of a thickness-dependent bandgap. However, the detail of inter-surface coupling effect, especially the intermediate state between weak and strong coupling of surface states (SS) from two surfaces, still needs to be interrogated. From topological nontrivial to trivial phases, interesting transformation must take place progressively as the overlap of electron wave functions on two surfaces increases with reducing film thickness.

Here we investigate the inter-surface coupling issue in antimony (Sb), which has a rhombohedral crystal structure and can be considered as a stacking of (111) bilayers (BL, 1 BL = 3.75 Å). There are several reasons to study Sb(111) films. Firstly, as the “parent” of the first-generation 3D TI Bi1−xSbx21, although bulk Sb is semimetal due to its negative indirect bandgap, its band order is inverted at the L point of the Brillouin zone. Sb(111) has been confirmed to possess TSS, leading to the absence of 180° backscattering and exotic transmission through atomic steps22. Next, the strongly distorted TSS Dirac cone on Sb(111) yields various scattering channels in QPI patterns, from which TSS dispersion and spin information can be extracted. Measuring the patterns at different film thickness helps us understand how the spin texture evolves as inter-surface coupling varies. Thirdly, a thin film with a large surface-to-volume ratio can make surface effect more observable. Lastly, compared with well-studied compound TIs such as Bi2Se3, Sb thin films provide a single-element simple system for demonstrating topological properties without much influence of residual bulk carriers from self-do** states2,4,23 or spatial fluctuations of charge and potential24. Therefore, by using Fourier-transform scanning tunneling spectroscopy (FT-STS), we measure the quasi-particle interference (QPI) patterns of TSS on thick and thin Sb(111) films. Combining with density functional theory (DFT) calculations, we aim to identify the scattering features in the patterns generated by the intra- and inter-surface couplings of TSS and to examine the dependence of the couplings on film thickness and wavevector k.

Results

We grew Sb films on Si(111)-√3 × √3:Bi-β surface in an ultrahigh vacuum (UHV) chamber25,26,27,28. Shown in Figure 1a is a STM image of an Sb(111) film of thickness ~30 BL. Steps here are all 3.75 Å corresponding to 1 BL of Sb(111). The atomic resolution image in the inset exhibits the hexagonal lattice on the (111) surface. In Fig. 1b, a typical differential tunneling conductance dI/dVb spectrum, where Vb is sample bias voltage and I the tunneling current, is displayed to reveal the local density of states (LDOS) of electrons in certain energy range. Unlike Bi2Se3 and Bi2Te3, Sb(111) has a strongly distorted TSS Dirac cone, with its lower part bowing up rapidly near point of the first surface Brillouin zone (SBZ) before bending down to merge with bulk valence bands, as shown schematically in Fig. 1b inset. Several extraordinary features of Sb(111) show up starting from the Dirac point ED at E ≈ −240 meV (with the Fermi energy EF at E = 0), which have also been confirmed by previous ARPES measurements1,29. The specific dispersion of TSS on Sb(111) allows us to determine ED being at the appearance E of QPI patterns22,Fig. 2c–d.